title: Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
    Tasks in Superposition
summary: Large Language Models (LLMs) have demonstrated remarkable in-context learning
  (ICL) capabilities. In this study, we explore a surprising phenomenon related
  to ICL: LLMs can perform multiple, computationally distinct ICL tasks
  simultaneously, during a single inference call, a capability we term "task
  superposition". We provide empirical evidence of this phenomenon across various
  LLM families and scales and show that this phenomenon emerges even if we train
  the model to in-context learn one task at a time. We offer theoretical
  explanations that this capability is well within the expressive power of
  transformers. We also explore how LLMs internally compose task vectors during
  superposition. Furthermore, we show that larger models can solve more ICL tasks
  in parallel, and better calibrate their output distribution. Our findings offer
  insights into the latent capabilities of LLMs, further substantiate the
  perspective of "LLMs as superposition of simulators", and raise questions about
  the mechanisms enabling simultaneous task execution.
paper_authors._id: 67093c6d10d26dfd98500536
paper_authors.name: Zheyang Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:27:51.579Z
paper_authors._id: 67084dbb652deb3c09d8f633
paper_authors.avatarUrl: /avatars/6489cc414ec6b7be3523e335ba14888f.svg
paper_authors.fullname: Zheyang Xiong
paper_authors.type: user
paper_authors.user: zheyang-hf
paper_authors._id: 67093c6d10d26dfd98500537
paper_authors.name: Ziyang Cai
paper_authors._id: 67093c6d10d26dfd98500538
paper_authors.name: John Cooper
paper_authors._id: 67093c6d10d26dfd98500539
paper_authors.name: Albert Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:43:29.776Z
paper_authors._id: 650263c89a612aa33a018383
paper_authors.avatarUrl: /avatars/a1415a4696da98ffebc1ac798cbab052.svg
paper_authors.fullname: Albert Ge
paper_authors.type: user
paper_authors.user: albertge
paper_authors._id: 67093c6d10d26dfd9850053a
paper_authors.name: Vasilis Papageorgiou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:16:05.070Z
paper_authors._id: 631bd99e9b7736c65a62287c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662769518341-noauth.jpeg
paper_authors.fullname: Vasilis Papageorgiou
paper_authors.type: user
paper_authors.user: vpapageorgiou
paper_authors._id: 67093c6d10d26dfd9850053b
paper_authors.name: Zack Sifakis
paper_authors._id: 67093c6d10d26dfd9850053c
paper_authors.name: Angeliki Giannou
paper_authors._id: 67093c6d10d26dfd9850053d
paper_authors.name: Ziqian Lin
paper_authors._id: 67093c6d10d26dfd9850053e
paper_authors.name: Liu Yang
paper_authors._id: 67093c6d10d26dfd9850053f
paper_authors.name: Saurabh Agarwal
paper_authors._id: 67093c6d10d26dfd98500540
paper_authors.name: Grigorios G Chrysos
paper_authors._id: 67093c6d10d26dfd98500541
paper_authors.name: Samet Oymak
paper_authors._id: 67093c6d10d26dfd98500542
paper_authors.name: Kangwook Lee
paper_authors._id: 67093c6d10d26dfd98500543
paper_authors.name: Dimitris Papailiopoulos
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-11T14:55:42.107Z
paper_authors._id: 64428cf07feb866811b0be6a
paper_authors.avatarUrl: /avatars/f30d339a02ce070c7e77315b59e20359.svg
paper_authors.fullname: Dimitris Papailiopoulos
paper_authors.type: user
paper_authors.user: anadim
paper_title: Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
    Tasks in Superposition
paper_summary: Large Language Models (LLMs) have demonstrated remarkable in-context learning
  (ICL) capabilities. In this study, we explore a surprising phenomenon related
  to ICL: LLMs can perform multiple, computationally distinct ICL tasks
  simultaneously, during a single inference call, a capability we term "task
  superposition". We provide empirical evidence of this phenomenon across various
  LLM families and scales and show that this phenomenon emerges even if we train
  the model to in-context learn one task at a time. We offer theoretical
  explanations that this capability is well within the expressive power of
  transformers. We also explore how LLMs internally compose task vectors during
  superposition. Furthermore, we show that larger models can solve more ICL tasks
  in parallel, and better calibrate their output distribution. Our findings offer
  insights into the latent capabilities of LLMs, further substantiate the
  perspective of "LLMs as superposition of simulators", and raise questions about
  the mechanisms enabling simultaneous task execution.
paper_ai_summary: LLMs can perform multiple distinct in-context learning tasks simultaneously during a single inference, a phenomenon termed "task superposition," which scales with model size and improves output calibration.
paper_ai_keywords: Large Language Models in-context learning task superposition transformers task vectors output distribution calibration