Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
<|reference_start|>Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks: Large language models can solve tasks that were not present in the training set. This capability is believed to be due to in-context learning and skill composition. In this work, we study the emergence of in-context learning and skill composition in a collection of modular arithmetic tasks. Specifically, we consider a finite collection of linear modular functions $z = a \, x + b \, y \;\mathrm{mod}\; p$ labeled by the vector $(a, b) \in \mathbb{Z}_p^2$. We use some of these tasks for pre-training and the rest for out-of-distribution testing. We empirically show that a GPT-style transformer exhibits a transition from in-distribution to out-of-distribution generalization as the number of pre-training tasks increases. We find that the smallest model capable of out-of-distribution generalization requires two transformer blocks, while for deeper models, the out-of-distribution generalization phase is \emph{transient}, necessitating early stopping. Finally, we perform an interpretability study of the pre-trained models, revealing the highly structured representations in both phases; and discuss the learnt algorithm.<|reference_end|>
@article{he2024learning,
    title={Learning to grok: Emergence of in-context learning and skill composition
  in modular arithmetic tasks},
    author={Tianyu He, Darshil Doshi, Aritra Das, Andrey Gromov},
    journal={arXiv preprint arXiv:2406.02550},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02550},
    primaryClass={cs.LG cond-mat.dis-nn hep-th stat.ML}
}