title: Understanding In-Context Learning via Supportive Pretraining Data
summary: In-context learning (ICL) improves language models' performance on a variety
  of NLP tasks by simply demonstrating a handful of examples at inference time.
  It is not well understood why ICL ability emerges, as the model has never been
  specifically trained on such demonstrations. Unlike prior work that explores
  implicit mechanisms behind ICL, we study ICL via investigating the pretraining
  data. Specifically, we first adapt an iterative, gradient-based approach to
  find a small subset of pretraining data that supports ICL. We observe that a
  continued pretraining on this small subset significantly improves the model's
  ICL ability, by up to 18%. We then compare the supportive subset constrastively
  with random subsets of pretraining data and discover: (1) The supportive
  pretraining data to ICL do not have a higher domain relevance to downstream
  tasks. (2) The supportive pretraining data have a higher mass of rarely
  occurring, long-tail tokens. (3) The supportive pretraining data are
  challenging examples where the information gain from long-range context is
  below average, indicating learning to incorporate difficult long-range context
  encourages ICL. Our work takes a first step towards understanding ICL via
  analyzing instance-level pretraining data. Our insights have a potential to
  enhance the ICL ability of language models by actively guiding the construction
  of pretraining data in the future.
paper_authors._id: 649b9396b38579156347eca6
paper_authors.name: Xiaochuang Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-29T20:55:52.123Z
paper_authors._id: 635ea3969f24f6db0a1e2d0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
paper_authors.fullname: Xiaochuang Han
paper_authors.type: user
paper_authors.user: xhan77
paper_authors._id: 649b9396b38579156347eca7
paper_authors.name: Daniel Simig
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-08-24T05:37:15.784Z
paper_authors._id: 64e6ea7a3b9c1e300efb79ca
paper_authors.avatarUrl: /avatars/393898e6437d82dd143e137d6f9d1647.svg
paper_authors.fullname: Daniel Simig
paper_authors.type: user
paper_authors.user: danielsimig
paper_authors._id: 649b9396b38579156347eca8
paper_authors.name: Todor Mihaylov
paper_authors._id: 649b9396b38579156347eca9
paper_authors.name: Yulia Tsvetkov
paper_authors._id: 649b9396b38579156347ecaa
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 649b9396b38579156347ecab
paper_authors.name: Tianlu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:50.503Z
paper_authors._id: 63f6a0e6b4c9a104f4b95d86
paper_authors.avatarUrl: /avatars/660c3e44bea976f3e0560bf95466b03c.svg
paper_authors.fullname: Tianlu Wang
paper_authors.type: user
paper_authors.user: Tianlu
paper_title: Understanding In-Context Learning via Supportive Pretraining Data
paper_summary: In-context learning (ICL) improves language models' performance on a variety
  of NLP tasks by simply demonstrating a handful of examples at inference time.
  It is not well understood why ICL ability emerges, as the model has never been
  specifically trained on such demonstrations. Unlike prior work that explores
  implicit mechanisms behind ICL, we study ICL via investigating the pretraining
  data. Specifically, we first adapt an iterative, gradient-based approach to
  find a small subset of pretraining data that supports ICL. We observe that a
  continued pretraining on this small subset significantly improves the model's
  ICL ability, by up to 18%. We then compare the supportive subset constrastively
  with random subsets of pretraining data and discover: (1) The supportive
  pretraining data to ICL do not have a higher domain relevance to downstream
  tasks. (2) The supportive pretraining data have a higher mass of rarely
  occurring, long-tail tokens. (3) The supportive pretraining data are
  challenging examples where the information gain from long-range context is
  below average, indicating learning to incorporate difficult long-range context
  encourages ICL. Our work takes a first step towards understanding ICL via
  analyzing instance-level pretraining data. Our insights have a potential to
  enhance the ICL ability of language models by actively guiding the construction
  of pretraining data in the future.
paper_ai_keywords: in-context learning (ICL) gradient-based approach long-tail tokens long-range context