Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
The recent success of interleaved Large Multimodal Models (LMMs) in few-shot
learning suggests that in-context learning (ICL) with many examples can be
promising for learning new tasks. However, this many-shot multimodal ICL
setting has one crucial problem: it is fundamentally limited by the model's
context length set at pretraining. The problem is especially prominent in the
multimodal domain, which processes both text and images, requiring additional
tokens. This motivates the need for a multimodal method to compress many shots
into fewer tokens without finetuning. In this work, we enable LMMs to perform
multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors
(MTV)--compact implicit representations of in-context examples compressed in
the model's attention heads. Specifically, we first demonstrate the existence
of such MTV in LMMs and then leverage these extracted MTV to enable many-shot
in-context learning for various vision-and-language tasks. Our experiments
suggest that MTV can scale in performance with the number of compressed shots
and generalize to similar out-of-domain tasks without additional context length
for inference.
667dd9b3e677941031e4f9fc
Brandon Huang
667dd9b3e677941031e4f9fd
Chancharik Mitra
667dd9b3e677941031e4f9fe
Assaf Arbelle
667dd9b3e677941031e4f9ff
Leonid Karlinsky
667dd9b3e677941031e4fa00
Trevor Darrell
667dd9b3e677941031e4fa01
Roei Herzig
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
The recent success of interleaved Large Multimodal Models (LMMs) in few-shot
learning suggests that in-context learning (ICL) with many examples can be
promising for learning new tasks. However, this many-shot multimodal ICL
setting has one crucial problem: it is fundamentally limited by the model's
context length set at pretraining. The problem is especially prominent in the
multimodal domain, which processes both text and images, requiring additional
tokens. This motivates the need for a multimodal method to compress many shots
into fewer tokens without finetuning. In this work, we enable LMMs to perform
multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors
(MTV)--compact implicit representations of in-context examples compressed in
the model's attention heads. Specifically, we first demonstrate the existence
of such MTV in LMMs and then leverage these extracted MTV to enable many-shot
in-context learning for various vision-and-language tasks. Our experiments
suggest that MTV can scale in performance with the number of compressed shots
and generalize to similar out-of-domain tasks without additional context length
for inference.
Multimodal Task Vectors enable large multimodal models to perform many-shot in-context learning by compressing examples into compact representations, overcoming context length limitations.
Large Multimodal Models (LMMs) few-shot learning in-context learning (ICL) many-shot multimodal ICL context length Multimodal Task Vectors (MTV) vision-and-language tasks attention heads