MCIF: Multimodal Crosslingual Instruction-Following Benchmark from
  Scientific Talks
Recent advances in large language models have catalyzed the development of
multimodal LLMs (MLLMs) that integrate text, speech, and vision within unified
frameworks. As MLLMs evolve from narrow, monolingual, task-specific systems to
general-purpose instruction-following models, a key frontier lies in evaluating
their multilingual and multimodal capabilities over both long and short
contexts. However, existing benchmarks fall short in evaluating these
dimensions jointly: they are often limited to English, mostly focus on one
single modality at a time, rely on short-form contexts, or lack human
annotations -- hindering comprehensive assessment of model performance across
languages, modalities, and task complexity. To address these gaps, we introduce
MCIF (Multimodal Crosslingual Instruction Following), the first multilingual
human-annotated benchmark based on scientific talks that is designed to
evaluate instruction-following in crosslingual, multimodal settings over both
short- and long-form inputs. MCIF spans three core modalities -- speech,
vision, and text -- and four diverse languages (English, German, Italian, and
Chinese), enabling a comprehensive evaluation of MLLMs' abilities to interpret
instructions across languages and combine them with multimodal contextual
information. MCIF is released under a CC-BY 4.0 license to encourage open
research and progress in MLLMs development.
6890e6d6f01a094725f833d9
Sara Papi
6890e6d6f01a094725f833da
Maike Züfle
6890e6d6f01a094725f833db
Marco Gaido
6890e6d6f01a094725f833dc
Beatrice Savoldi
6890e6d6f01a094725f833dd
Danni Liu
6890e6d6f01a094725f833de
Ioannis Douros
6890e6d6f01a094725f833df
Luisa Bentivogli
6890e6d6f01a094725f833e0
Jan Niehues
MCIF: Multimodal Crosslingual Instruction-Following Benchmark from
  Scientific Talks
Recent advances in large language models have catalyzed the development of
multimodal LLMs (MLLMs) that integrate text, speech, and vision within unified
frameworks. As MLLMs evolve from narrow, monolingual, task-specific systems to
general-purpose instruction-following models, a key frontier lies in evaluating
their multilingual and multimodal capabilities over both long and short
contexts. However, existing benchmarks fall short in evaluating these
dimensions jointly: they are often limited to English, mostly focus on one
single modality at a time, rely on short-form contexts, or lack human
annotations -- hindering comprehensive assessment of model performance across
languages, modalities, and task complexity. To address these gaps, we introduce
MCIF (Multimodal Crosslingual Instruction Following), the first multilingual
human-annotated benchmark based on scientific talks that is designed to
evaluate instruction-following in crosslingual, multimodal settings over both
short- and long-form inputs. MCIF spans three core modalities -- speech,
vision, and text -- and four diverse languages (English, German, Italian, and
Chinese), enabling a comprehensive evaluation of MLLMs' abilities to interpret
instructions across languages and combine them with multimodal contextual
information. MCIF is released under a CC-BY 4.0 license to encourage open
research and progress in MLLMs development.
MCIF is a multilingual, human-annotated benchmark for evaluating instruction-following in crosslingual, multimodal settings using scientific talks.
multimodal LLMs MLLMs instruction-following multilingual multimodal scientific talks speech vision text human-annotated benchmark CC-BY 4.0 license