On Implications of Scaling Laws on Feature Superposition
<|reference_start|>On Implications of Scaling Laws on Feature Superposition: Using results from scaling laws, this theoretical note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer is a complete theory of feature representation. 2. Features are universal, meaning two models trained on the same data and achieving equal performance will learn identical features.<|reference_end|>
@article{katta2024on,
    title={On Implications of Scaling Laws on Feature Superposition},
    author={Pavan Katta},
    journal={arXiv preprint arXiv:2407.01459},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01459},
    primaryClass={cs.LG cs.AI}
}