title: A Closer Look at In-Context Learning under Distribution Shifts
summary: In-context learning, a capability that enables a model to learn from input
  examples on the fly without necessitating weight updates, is a defining
  characteristic of large language models. In this work, we follow the setting
  proposed in (Garg et al., 2022) to better understand the generality and
  limitations of in-context learning from the lens of the simple yet fundamental
  task of linear regression. The key question we aim to address is: Are
  transformers more adept than some natural and simpler architectures at
  performing in-context learning under varying distribution shifts? To compare
  transformers, we propose to use a simple architecture based on set-based
  Multi-Layer Perceptrons (MLPs). We find that both transformers and set-based
  MLPs exhibit in-context learning under in-distribution evaluations, but
  transformers more closely emulate the performance of ordinary least squares
  (OLS). Transformers also display better resilience to mild distribution shifts,
  where set-based MLPs falter. However, under severe distribution shifts, both
  models' in-context learning abilities diminish.
paper_authors._id: 647430c771f07ae738d5cdda
paper_authors.name: Kartik Ahuja
paper_authors._id: 647430c771f07ae738d5cddb
paper_authors.name: David Lopez-Paz
paper_title: A Closer Look at In-Context Learning under Distribution Shifts
paper_summary: In-context learning, a capability that enables a model to learn from input
  examples on the fly without necessitating weight updates, is a defining
  characteristic of large language models. In this work, we follow the setting
  proposed in (Garg et al., 2022) to better understand the generality and
  limitations of in-context learning from the lens of the simple yet fundamental
  task of linear regression. The key question we aim to address is: Are
  transformers more adept than some natural and simpler architectures at
  performing in-context learning under varying distribution shifts? To compare
  transformers, we propose to use a simple architecture based on set-based
  Multi-Layer Perceptrons (MLPs). We find that both transformers and set-based
  MLPs exhibit in-context learning under in-distribution evaluations, but
  transformers more closely emulate the performance of ordinary least squares
  (OLS). Transformers also display better resilience to mild distribution shifts,
  where set-based MLPs falter. However, under severe distribution shifts, both
  models' in-context learning abilities diminish.
paper_ai_keywords: in-context learning large language models Generality limitations linear regression transformers set-based Multi-Layer Perceptrons (MLPs) ordinary evaluations ordinary least squares (OLS) resilience distribution shifts