One-layer transformers fail to solve the induction heads task
<|reference_start|>One-layer transformers fail to solve the induction heads task: A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.<|reference_end|>
@article{sanford2024one-layer,
    title={One-layer transformers fail to solve the induction heads task},
    author={Clayton Sanford and Daniel Hsu and Matus Telgarsky},
    journal={arXiv preprint arXiv:2408.14332},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14332},
    primaryClass={cs.LG stat.ML}
}