Two TopK Sparse Autoencoders (SAEs) trained on identical data but with different initializations exhibit significant differences in learned features, sharing only about 53% of their features. This variability highlights that larger SAEs are more prone to learning distinct feature sets compared to smaller ones, emphasizing the importance for content creators to consider model size and initialization when aiming for consistent feature extraction.
Read the full article at Blog on EleutherAI Blog
This is a brief trending article summary.
Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.
Views30
Ali NematiBlogUI.author_articles
