title: Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
summary: The recent success of interleaved Large Multimodal Models (LMMs) in few-shot
  learning suggests that in-context learning (ICL) with many examples can be
  promising for learning new tasks. However, this many-shot multimodal ICL
  setting has one crucial problem: it is fundamentally limited by the model's
  context length set at pretraining. The problem is especially prominent in the
  multimodal domain, which processes both text and images, requiring additional
  tokens. This motivates the need for a multimodal method to compress many shots
  into fewer tokens without finetuning. In this work, we enable LMMs to perform
  multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors
  (MTV)--compact implicit representations of in-context examples compressed in
  the model's attention heads. Specifically, we first demonstrate the existence
  of such MTV in LMMs and then leverage these extracted MTV to enable many-shot
  in-context learning for various vision-and-language tasks. Our experiments
  suggest that MTV can scale in performance with the number of compressed shots
  and generalize to similar out-of-domain tasks without additional context length
  for inference.
paper_authors._id: 667dd9b3e677941031e4f9fc
paper_authors.name: Brandon Huang
paper_authors._id: 667dd9b3e677941031e4f9fd
paper_authors.name: Chancharik Mitra
paper_authors._id: 667dd9b3e677941031e4f9fe
paper_authors.name: Assaf Arbelle
paper_authors._id: 667dd9b3e677941031e4f9ff
paper_authors.name: Leonid Karlinsky
paper_authors._id: 667dd9b3e677941031e4fa00
paper_authors.name: Trevor Darrell
paper_authors._id: 667dd9b3e677941031e4fa01
paper_authors.name: Roei Herzig
paper_title: Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
paper_summary: The recent success of interleaved Large Multimodal Models (LMMs) in few-shot
  learning suggests that in-context learning (ICL) with many examples can be
  promising for learning new tasks. However, this many-shot multimodal ICL
  setting has one crucial problem: it is fundamentally limited by the model's
  context length set at pretraining. The problem is especially prominent in the
  multimodal domain, which processes both text and images, requiring additional
  tokens. This motivates the need for a multimodal method to compress many shots
  into fewer tokens without finetuning. In this work, we enable LMMs to perform
  multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors
  (MTV)--compact implicit representations of in-context examples compressed in
  the model's attention heads. Specifically, we first demonstrate the existence
  of such MTV in LMMs and then leverage these extracted MTV to enable many-shot
  in-context learning for various vision-and-language tasks. Our experiments
  suggest that MTV can scale in performance with the number of compressed shots
  and generalize to similar out-of-domain tasks without additional context length
  for inference.
paper_ai_summary: Multimodal Task Vectors enable large multimodal models to perform many-shot in-context learning by compressing examples into compact representations, overcoming context length limitations.
paper_ai_keywords: Large Multimodal Models (LMMs) few-shot learning in-context learning (ICL) many-shot multimodal ICL context length Multimodal Task Vectors (MTV) vision-and-language tasks attention heads