title: Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
summary: Despite recent advances in multimodal large language models (MLLMs), their
  development has predominantly focused on English- and western-centric datasets
  and tasks, leaving most of the world's languages and diverse cultural contexts
  underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
  trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
  PangeaIns features: 1) high-quality English instructions, 2) carefully
  machine-translated instructions, and 3) culturally relevant multimodal tasks to
  ensure cross-cultural coverage. To rigorously assess models' capabilities, we
  introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
  covering 47 languages. Results show that Pangea significantly outperforms
  existing open-source models in multilingual settings and diverse cultural
  contexts. Ablation studies further reveal the importance of English data
  proportions, language popularity, and the number of multimodal training samples
  on overall performance. We fully open-source our data, code, and trained
  checkpoints, to facilitate the development of inclusive and robust multilingual
  MLLMs, promoting equity and accessibility across a broader linguistic and
  cultural spectrum.
paper_authors._id: 671718ed63c851e167b775c5
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:57:23.127Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 671718ed63c851e167b775c6
paper_authors.name: Yueqi Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:17:30.353Z
paper_authors._id: 63e7bf7be02ee67e8e53f78d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e7bf7be02ee67e8e53f78d/z9j1PWrurEyIA2JXzpz0i.png
paper_authors.fullname: Yueqi Song
paper_authors.type: user
paper_authors.user: yueqis
paper_authors._id: 671718ed63c851e167b775c7
paper_authors.name: Akari Asai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:17:24.248Z
paper_authors._id: 6266e0cb7a1f5a1562c4e86e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266e0cb7a1f5a1562c4e86e/kkJZPssa76mdRPnKH-q3e.jpeg
paper_authors.fullname: Akari Asai
paper_authors.type: user
paper_authors.user: akariasai
paper_authors._id: 671718ed63c851e167b775c8
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:57:21.588Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 671718ed63c851e167b775c9
paper_authors.name: Jean de Dieu Nyandwi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T09:05:59.579Z
paper_authors._id: 62989cd743e990340bead77c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654168730944-noauth.png
paper_authors.fullname: Jean de Dieu Nyandwi
paper_authors.type: user
paper_authors.user: Nyandwi
paper_authors._id: 671718ed63c851e167b775ca
paper_authors.name: Simran Khanuja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:17:15.486Z
paper_authors._id: 63a4d079658851481f7e4394
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4d079658851481f7e4394/yptj_hJiaqHvEaqxLs0jT.jpeg
paper_authors.fullname: Simran Khanuja
paper_authors.type: user
paper_authors.user: skhanuja
paper_authors._id: 671718ed63c851e167b775cb
paper_authors.name: Anjali Kantharuban
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:16:49.370Z
paper_authors._id: 634e6d7ea51d5df8c2da699d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666084174110-noauth.jpeg
paper_authors.fullname: Anjali Kantharuban
paper_authors.type: user
paper_authors.user: AnjaliRuban
paper_authors._id: 671718ed63c851e167b775cc
paper_authors.name: Lintang Sutawika
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:17:08.786Z
paper_authors._id: 5f666a6b5e78cc6b0ed319fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635412047803-5f666a6b5e78cc6b0ed319fc.jpeg
paper_authors.fullname: Lintang Sutawika
paper_authors.type: user
paper_authors.user: lintang
paper_authors._id: 671718ed63c851e167b775cd
paper_authors.name: Sathyanarayanan Ramamoorthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:17:02.898Z
paper_authors._id: 643b87db0e5495afdeff8de5
paper_authors.avatarUrl: /avatars/5c0bca81a52ab6b7beb7a52d82e53a10.svg
paper_authors.fullname: Sathyanarayanan Ramamoorthy
paper_authors.type: user
paper_authors.user: sathyaram
paper_authors._id: 671718ed63c851e167b775ce
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:16:57.085Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
paper_summary: Despite recent advances in multimodal large language models (MLLMs), their
  development has predominantly focused on English- and western-centric datasets
  and tasks, leaving most of the world's languages and diverse cultural contexts
  underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
  trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
  PangeaIns features: 1) high-quality English instructions, 2) carefully
  machine-translated instructions, and 3) culturally relevant multimodal tasks to
  ensure cross-cultural coverage. To rigorously assess models' capabilities, we
  introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
  covering 47 languages. Results show that Pangea significantly outperforms
  existing open-source models in multilingual settings and diverse cultural
  contexts. Ablation studies further reveal the importance of English data
  proportions, language popularity, and the number of multimodal training samples
  on overall performance. We fully open-source our data, code, and trained
  checkpoints, to facilitate the development of inclusive and robust multilingual
  MLLMs, promoting equity and accessibility across a broader linguistic and
  cultural spectrum.
paper_ai_summary: Pangea, a multilingual multimodal LLM, is trained on a diverse 6M instruction dataset across 39 languages and evaluated using PangeaBench, demonstrating superior performance in multilingual and cross-cultural settings.
paper_ai_keywords: multimodal large language models (MLLMs) Pangea PangeaIns PangeaBench multilingual multimodal evaluation suite ablation studies