Aya Vision: Advancing the Frontier of Multilingual Multimodality
Building multimodal language models is fundamentally challenging: it requires
aligning vision and language modalities, curating high-quality instruction
data, and avoiding the degradation of existing text-only capabilities once
vision is introduced. These difficulties are further magnified in the
multilingual setting, where the need for multimodal data in different languages
exacerbates existing data scarcity, machine translation often distorts meaning,
and catastrophic forgetting is more pronounced. To address the aforementioned
challenges, we introduce novel techniques spanning both data and modeling.
First, we develop a synthetic annotation framework that curates high-quality,
diverse multilingual multimodal instruction data, enabling Aya Vision models to
produce natural, human-preferred responses to multimodal inputs across many
languages. Complementing this, we propose a cross-modal model merging technique
that mitigates catastrophic forgetting, effectively preserving text-only
capabilities while simultaneously enhancing multimodal generative performance.
Aya-Vision-8B achieves best-in-class performance compared to strong multimodal
models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger
Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which
outperforms models more than twice its size, such as Molmo-72B and
LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the
multi-modal frontier, and provides insights into techniques that effectively
bend the need for compute while delivering extremely high performance.
68249d57f08a44ee1f1c7139
Saurabh Dash
68249d57f08a44ee1f1c713a
Yiyang Nan
claimed_verified
2025-05-15T10:32:16.613Z
66ad9237091a147c4f154dbb
/avatars/c80258e877bafc08452f241a6ecfec04.svg
Yiyang Nan
user
olivernan
68249d57f08a44ee1f1c713b
John Dang
68249d57f08a44ee1f1c713c
Arash Ahmadian
68249d57f08a44ee1f1c713d
Shivalika Singh
68249d57f08a44ee1f1c713e
Madeline Smith
68249d57f08a44ee1f1c713f
Bharat Venkitesh
68249d57f08a44ee1f1c7140
Vlad Shmyhlo
68249d57f08a44ee1f1c7141
Viraat Aryabumi
68249d57f08a44ee1f1c7142
Walter Beller-Morales
68249d57f08a44ee1f1c7143
Jeremy Pekmez
68249d57f08a44ee1f1c7144
Jason Ozuzu
68249d57f08a44ee1f1c7145
Pierre Richemond
68249d57f08a44ee1f1c7146
Acyr Locatelli
68249d57f08a44ee1f1c7147
Nick Frosst
68249d57f08a44ee1f1c7148
Phil Blunsom
68249d57f08a44ee1f1c7149
Aidan Gomez
68249d57f08a44ee1f1c714a
Ivan Zhang
68249d57f08a44ee1f1c714b
Marzieh Fadaee
68249d57f08a44ee1f1c714c
Manoj Govindassamy
68249d57f08a44ee1f1c714d
Sudip Roy
68249d57f08a44ee1f1c714e
Matthias Gallé
68249d57f08a44ee1f1c714f
Beyza Ermis
68249d57f08a44ee1f1c7150
Ahmet Üstün
68249d57f08a44ee1f1c7151
Sara Hooker
Aya Vision: Advancing the Frontier of Multilingual Multimodality
Building multimodal language models is fundamentally challenging: it requires
aligning vision and language modalities, curating high-quality instruction
data, and avoiding the degradation of existing text-only capabilities once
vision is introduced. These difficulties are further magnified in the
multilingual setting, where the need for multimodal data in different languages
exacerbates existing data scarcity, machine translation often distorts meaning,
and catastrophic forgetting is more pronounced. To address the aforementioned
challenges, we introduce novel techniques spanning both data and modeling.
First, we develop a synthetic annotation framework that curates high-quality,
diverse multilingual multimodal instruction data, enabling Aya Vision models to
produce natural, human-preferred responses to multimodal inputs across many
languages. Complementing this, we propose a cross-modal model merging technique
that mitigates catastrophic forgetting, effectively preserving text-only
capabilities while simultaneously enhancing multimodal generative performance.
Aya-Vision-8B achieves best-in-class performance compared to strong multimodal
models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger
Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which
outperforms models more than twice its size, such as Molmo-72B and
LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the
multi-modal frontier, and provides insights into techniques that effectively
bend the need for compute while delivering extremely high performance.
Multi-modal language models are enhanced through synthetic data creation and cross-modal merging techniques, achieving superior performance in multilingual settings.
synthetic annotation framework cross-modal model merging technique catastrophic forgetting multilingual multimodal instruction data multimodal generative performance