title: Many-Shot In-Context Learning in Multimodal Foundation Models
summary: Large language models are well-known to be effective at few-shot in-context
  learning (ICL). Recent advancements in multimodal foundation models have
  enabled unprecedentedly long context windows, presenting an opportunity to
  explore their capability to perform ICL with many more demonstrating examples.
  In this work, we evaluate the performance of multimodal foundation models
  scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
  across 10 datasets spanning multiple domains (natural imagery, medical imagery,
  remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
  fine-grained classification). We observe that many-shot ICL, including up to
  almost 2,000 multimodal demonstrating examples, leads to substantial
  improvements compared to few-shot (<100 examples) ICL across all of the
  datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
  up to the maximum number of tested examples on many datasets. Given the high
  inference costs associated with the long prompts required for many-shot ICL, we
  also explore the impact of batching multiple queries in a single API call. We
  show that batching up to 50 queries can lead to performance improvements under
  zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
  multiple datasets, while drastically reducing per-query cost and latency.
  Finally, we measure ICL data efficiency of the models, or the rate at which the
  models learn from more demonstrating examples. We find that while GPT-4o and
  Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
  Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
  datasets. Our results suggest that many-shot ICL could enable users to
  efficiently adapt multimodal foundation models to new applications and domains.
  Our codebase is publicly available at
  https://github.com/stanfordmlgroup/ManyICL .
paper_authors._id: 6646ba2032fbd804f34d68aa
paper_authors.name: Yixing Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-20T07:38:33.700Z
paper_authors._id: 64bbcbdde10455688fde3b51
paper_authors.avatarUrl: /avatars/d5be06d419a1a6f3e53706c8e44cd7bc.svg
paper_authors.fullname: Yixing Jiang
paper_authors.type: user
paper_authors.user: jyx-su
paper_authors._id: 6646ba2032fbd804f34d68ab
paper_authors.name: Jeremy Irvin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:31:08.730Z
paper_authors._id: 632cb34a4a4991e71155d4a6
paper_authors.avatarUrl: /avatars/d87701cf30f425e018a7f89cb2d7cc5c.svg
paper_authors.fullname: Jeremy Irvin
paper_authors.type: user
paper_authors.user: jirvin16
paper_authors._id: 6646ba2032fbd804f34d68ac
paper_authors.name: Ji Hun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:31:14.499Z
paper_authors._id: 638003b79f1f158b014b4550
paper_authors.avatarUrl: /avatars/9c268b96e22b6d01b2fbf805e6990d56.svg
paper_authors.fullname: Ji Hun Wang
paper_authors.type: user
paper_authors.user: JiHunWang
paper_authors._id: 6646ba2032fbd804f34d68ad
paper_authors.name: Muhammad Ahmed Chaudhry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:31:20.896Z
paper_authors._id: 65ca76cd8e0d94e487338d97
paper_authors.avatarUrl: /avatars/6853fa7ba90a8e727cff940eb5272a2c.svg
paper_authors.fullname: Muhammad Ahmed Chaudhry
paper_authors.type: user
paper_authors.user: MAhmedCh
paper_authors._id: 6646ba2032fbd804f34d68ae
paper_authors.name: Jonathan H. Chen
paper_authors._id: 6646ba2032fbd804f34d68af
paper_authors.name: Andrew Y. Ng
paper_title: Many-Shot In-Context Learning in Multimodal Foundation Models
paper_summary: Large language models are well-known to be effective at few-shot in-context
  learning (ICL). Recent advancements in multimodal foundation models have
  enabled unprecedentedly long context windows, presenting an opportunity to
  explore their capability to perform ICL with many more demonstrating examples.
  In this work, we evaluate the performance of multimodal foundation models
  scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
  across 10 datasets spanning multiple domains (natural imagery, medical imagery,
  remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
  fine-grained classification). We observe that many-shot ICL, including up to
  almost 2,000 multimodal demonstrating examples, leads to substantial
  improvements compared to few-shot (<100 examples) ICL across all of the
  datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
  up to the maximum number of tested examples on many datasets. Given the high
  inference costs associated with the long prompts required for many-shot ICL, we
  also explore the impact of batching multiple queries in a single API call. We
  show that batching up to 50 queries can lead to performance improvements under
  zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
  multiple datasets, while drastically reducing per-query cost and latency.
  Finally, we measure ICL data efficiency of the models, or the rate at which the
  models learn from more demonstrating examples. We find that while GPT-4o and
  Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
  Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
  datasets. Our results suggest that many-shot ICL could enable users to
  efficiently adapt multimodal foundation models to new applications and domains.
  Our codebase is publicly available at
  https://github.com/stanfordmlgroup/ManyICL .
paper_ai_summary: Multimodal foundation models exhibit significant performance improvements with many-shot in-context learning compared to few-shot, with Gemini 1.5 Pro showing higher data efficiency and better batch processing capabilities across various domains.
paper_ai_keywords: few-shot in-context learning GPT-4o Gemini 1.5 Pro multimodal foundation models in-context learning data efficiency batch processing