title: In-Context Learning Creates Task Vectors
summary: In-context learning (ICL) in Large Language Models (LLMs) has emerged as a
  powerful new learning paradigm. However, its underlying mechanism is still not
  well understood. In particular, it is challenging to map it to the "standard"
  machine learning framework, where one uses a training set S to find a
  best-fitting function f(x) in some hypothesis class. Here we make progress on
  this problem by showing that the functions learned by ICL often have a very
  simple structure: they correspond to the transformer LLM whose only inputs are
  the query x and a single "task vector" calculated from the training set.
  Thus, ICL can be seen as compressing S into a single task vector
  theta(S) and then using this task vector to modulate the
  transformer to produce the output. We support the above claim via comprehensive
  experiments across a range of models and tasks.
paper_authors._id: 65388a1d1b3eaa722ddc7fc9
paper_authors.name: Roee Hendel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T07:33:25.828Z
paper_authors._id: 6383773c834d3558a393dcba
paper_authors.avatarUrl: /avatars/dd0e9c017dbe13438a1007d59380c41d.svg
paper_authors.fullname: Roee Hendel
paper_authors.type: user
paper_authors.user: roeehendel
paper_authors._id: 65388a1d1b3eaa722ddc7fca
paper_authors.name: Mor Geva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T07:33:29.300Z
paper_authors._id: 610b729f9da682cd54ad9adf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
paper_authors.fullname: Mor Geva
paper_authors.type: user
paper_authors.user: mega
paper_authors._id: 65388a1d1b3eaa722ddc7fcb
paper_authors.name: Amir Globerson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:36:46.074Z
paper_authors._id: 62cd956bc589e4a9e23ea347
paper_authors.avatarUrl: /avatars/dbdccd9258c9d18173d312a8bf14dd6e.svg
paper_authors.fullname: Amir Globerson
paper_authors.type: user
paper_authors.user: amirgloberson
paper_title: In-Context Learning Creates Task Vectors
paper_summary: In-context learning (ICL) in Large Language Models (LLMs) has emerged as a
  powerful new learning paradigm. However, its underlying mechanism is still not
  well understood. In particular, it is challenging to map it to the "standard"
  machine learning framework, where one uses a training set S to find a
  best-fitting function f(x) in some hypothesis class. Here we make progress on
  this problem by showing that the functions learned by ICL often have a very
  simple structure: they correspond to the transformer LLM whose only inputs are
  the query x and a single "task vector" calculated from the training set.
  Thus, ICL can be seen as compressing S into a single task vector
  theta(S) and then using this task vector to modulate the
  transformer to produce the output. We support the above claim via comprehensive
  experiments across a range of models and tasks.
paper_ai_summary: In-Context Learning in Large Language Models can be understood as compressing a training set into a task vector that modulates a transformer for output generation.
paper_ai_keywords: in-context learning large language models transformer task vector output generation