title: Aya Vision: Advancing the Frontier of Multilingual Multimodality
summary: Building multimodal language models is fundamentally challenging: it requires
  aligning vision and language modalities, curating high-quality instruction
  data, and avoiding the degradation of existing text-only capabilities once
  vision is introduced. These difficulties are further magnified in the
  multilingual setting, where the need for multimodal data in different languages
  exacerbates existing data scarcity, machine translation often distorts meaning,
  and catastrophic forgetting is more pronounced. To address the aforementioned
  challenges, we introduce novel techniques spanning both data and modeling.
  First, we develop a synthetic annotation framework that curates high-quality,
  diverse multilingual multimodal instruction data, enabling Aya Vision models to
  produce natural, human-preferred responses to multimodal inputs across many
  languages. Complementing this, we propose a cross-modal model merging technique
  that mitigates catastrophic forgetting, effectively preserving text-only
  capabilities while simultaneously enhancing multimodal generative performance.
  Aya-Vision-8B achieves best-in-class performance compared to strong multimodal
  models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger
  Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which
  outperforms models more than twice its size, such as Molmo-72B and
  LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the
  multi-modal frontier, and provides insights into techniques that effectively
  bend the need for compute while delivering extremely high performance.
paper_authors._id: 68249d57f08a44ee1f1c7139
paper_authors.name: Saurabh Dash
paper_authors._id: 68249d57f08a44ee1f1c713a
paper_authors.name: Yiyang Nan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:32:16.613Z
paper_authors._id: 66ad9237091a147c4f154dbb
paper_authors.avatarUrl: /avatars/c80258e877bafc08452f241a6ecfec04.svg
paper_authors.fullname: Yiyang Nan
paper_authors.type: user
paper_authors.user: olivernan
paper_authors._id: 68249d57f08a44ee1f1c713b
paper_authors.name: John Dang
paper_authors._id: 68249d57f08a44ee1f1c713c
paper_authors.name: Arash Ahmadian
paper_authors._id: 68249d57f08a44ee1f1c713d
paper_authors.name: Shivalika Singh
paper_authors._id: 68249d57f08a44ee1f1c713e
paper_authors.name: Madeline Smith
paper_authors._id: 68249d57f08a44ee1f1c713f
paper_authors.name: Bharat Venkitesh
paper_authors._id: 68249d57f08a44ee1f1c7140
paper_authors.name: Vlad Shmyhlo
paper_authors._id: 68249d57f08a44ee1f1c7141
paper_authors.name: Viraat Aryabumi
paper_authors._id: 68249d57f08a44ee1f1c7142
paper_authors.name: Walter Beller-Morales
paper_authors._id: 68249d57f08a44ee1f1c7143
paper_authors.name: Jeremy Pekmez
paper_authors._id: 68249d57f08a44ee1f1c7144
paper_authors.name: Jason Ozuzu
paper_authors._id: 68249d57f08a44ee1f1c7145
paper_authors.name: Pierre Richemond
paper_authors._id: 68249d57f08a44ee1f1c7146
paper_authors.name: Acyr Locatelli
paper_authors._id: 68249d57f08a44ee1f1c7147
paper_authors.name: Nick Frosst
paper_authors._id: 68249d57f08a44ee1f1c7148
paper_authors.name: Phil Blunsom
paper_authors._id: 68249d57f08a44ee1f1c7149
paper_authors.name: Aidan Gomez
paper_authors._id: 68249d57f08a44ee1f1c714a
paper_authors.name: Ivan Zhang
paper_authors._id: 68249d57f08a44ee1f1c714b
paper_authors.name: Marzieh Fadaee
paper_authors._id: 68249d57f08a44ee1f1c714c
paper_authors.name: Manoj Govindassamy
paper_authors._id: 68249d57f08a44ee1f1c714d
paper_authors.name: Sudip Roy
paper_authors._id: 68249d57f08a44ee1f1c714e
paper_authors.name: Matthias Gallé
paper_authors._id: 68249d57f08a44ee1f1c714f
paper_authors.name: Beyza Ermis
paper_authors._id: 68249d57f08a44ee1f1c7150
paper_authors.name: Ahmet Üstün
paper_authors._id: 68249d57f08a44ee1f1c7151
paper_authors.name: Sara Hooker
paper_title: Aya Vision: Advancing the Frontier of Multilingual Multimodality
paper_summary: Building multimodal language models is fundamentally challenging: it requires
  aligning vision and language modalities, curating high-quality instruction
  data, and avoiding the degradation of existing text-only capabilities once
  vision is introduced. These difficulties are further magnified in the
  multilingual setting, where the need for multimodal data in different languages
  exacerbates existing data scarcity, machine translation often distorts meaning,
  and catastrophic forgetting is more pronounced. To address the aforementioned
  challenges, we introduce novel techniques spanning both data and modeling.
  First, we develop a synthetic annotation framework that curates high-quality,
  diverse multilingual multimodal instruction data, enabling Aya Vision models to
  produce natural, human-preferred responses to multimodal inputs across many
  languages. Complementing this, we propose a cross-modal model merging technique
  that mitigates catastrophic forgetting, effectively preserving text-only
  capabilities while simultaneously enhancing multimodal generative performance.
  Aya-Vision-8B achieves best-in-class performance compared to strong multimodal
  models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger
  Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which
  outperforms models more than twice its size, such as Molmo-72B and
  LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the
  multi-modal frontier, and provides insights into techniques that effectively
  bend the need for compute while delivering extremely high performance.
paper_ai_summary: Multi-modal language models are enhanced through synthetic data creation and cross-modal merging techniques, achieving superior performance in multilingual settings.
paper_ai_keywords: synthetic annotation framework cross-modal model merging technique catastrophic forgetting multilingual multimodal instruction data multimodal generative performance