AI & Machine Learning

SAEs trained on the same data don't learn the same features

Ali NematiAli NematiDec 12, 2024
29 sec read81 views0 listens

Two TopK Sparse Autoencoders (SAEs) trained on identical data but with different initializations exhibit significant differences in learned features, sharing only about 53% of their features. This variability highlights that larger SAEs are more prone to learning distinct feature sets compared to smaller ones, emphasizing the importance for content creators to consider model size and initialization when aiming for consistent feature extraction.

Read the full article at Blog on EleutherAI Blog

This is a brief trending article summary.


Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.

Views81
Ali Nemati
Ali NematiBlogUI.author_articles
BlogUI.view_profile