What Makes Multimodal In-Context Learning Work?
<|reference_start|>What Makes Multimodal In-Context Learning Work?: Large Language Models have demonstrated remarkable performance across various tasks, exhibiting the capacity to swiftly acquire new skills, such as through In-Context Learning (ICL) with minimal demonstration examples. In this work, we present a comprehensive framework for investigating Multimodal ICL (M-ICL) in the context of Large Multimodal Models. We consider the best open-source multimodal models (e.g., IDEFICS, OpenFlamingo) and a wide range of multimodal tasks. Our study unveils several noteworthy findings: (1) M-ICL primarily relies on text-driven mechanisms, showing little to no influence from the image modality. (2) When used with advanced-ICL strategy (like RICES), M-ICL is not better than a simple strategy based on majority voting over context examples. Moreover, we identify several biases and limitations of M-ICL that warrant consideration prior to deployment. Code available at https://gitlab.com/folbaeni/multimodal-icl<|reference_end|>
@article{baldassini2024what,
    title={What Makes Multimodal In-Context Learning Work?},
    author={Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure
  Soulier, Benjamin Piwowarski},
    journal={arXiv preprint arXiv:2404.15736},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.15736},
    primaryClass={cs.CV cs.AI}
}