The Developmental Landscape of In-Context Learning
<|reference_start|>The Developmental Landscape of In-Context Learning: We show that in-context learning emerges in transformers in discrete developmental stages, when they are trained on either language modeling or linear regression tasks. We introduce two methods for detecting the milestones that separate these stages, by probing the geometry of the population loss in both parameter space and function space. We study the stages revealed by these new methods using a range of behavioral and structural metrics to establish their validity.<|reference_end|>
@article{hoogland2024the,
    title={The Developmental Landscape of In-Context Learning},
    author={Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll,
  Susan Wei, Daniel Murfet},
    journal={arXiv preprint arXiv:2402.02364},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02364},
    primaryClass={cs.LG cs.AI cs.CL}
}