A Data Generation Perspective to the Mechanism of In-Context Learning
<|reference_start|>A Data Generation Perspective to the Mechanism of In-Context Learning: In-Context Learning (ICL) empowers Large Language Models (LLMs) with the capacity to learn in context, achieving downstream generalization without gradient updates but with a few in-context examples. Despite the encouraging empirical success, the underlying mechanism of ICL remains unclear, and existing research offers various viewpoints of understanding. These studies propose intuition-driven and ad-hoc technical solutions for interpreting ICL, illustrating an ambiguous road map. In this paper, we leverage a data generation perspective to reinterpret recent efforts and demonstrate the potential broader usage of popular technical solutions, approaching a systematic angle. For a conceptual definition, we rigorously adopt the terms of skill learning and skill recognition. The difference between them is skill learning can learn new data generation functions from in-context data. We also provide a comprehensive study on the merits and weaknesses of different solutions, and highlight the uniformity among them given the perspective of data generation, establishing a technical foundation for future research to incorporate the strengths of different lines of research.<|reference_end|>
@article{mao2024a,
    title={A Data Generation Perspective to the Mechanism of In-Context Learning},
    author={Haitao Mao, Guangliang Liu, Yao Ma, Rongrong Wang, Kristen Johnson,
  Jiliang Tang},
    journal={arXiv preprint arXiv:2402.02212},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02212},
    primaryClass={cs.CL}
}