title: Centurio: On Drivers of Multilingual Ability of Large Vision-Language
    Model
summary: Most Large Vision-Language Models (LVLMs) to date are trained predominantly
  on English data, which makes them struggle to understand non-English input and
  fail to generate output in the desired target language. Existing efforts
  mitigate these issues by adding multilingual training data, but do so in a
  largely ad-hoc manner, lacking insight into how different training mixes tip
  the scale for different groups of languages. In this work, we present a
  comprehensive investigation into the training strategies for massively
  multilingual LVLMs. First, we conduct a series of multi-stage experiments
  spanning 13 downstream vision-language tasks and 43 languages, systematically
  examining: (1) the number of training languages that can be included without
  degrading English performance and (2) optimal language distributions of
  pre-training as well as (3) instruction-tuning data. Further, we (4)
  investigate how to improve multilingual text-in-image understanding, and
  introduce a new benchmark for the task. Surprisingly, our analysis reveals that
  one can (i) include as many as 100 training languages simultaneously (ii) with
  as little as 25-50\% of non-English data, to greatly improve multilingual
  performance while retaining strong English performance. We further find that
  (iii) including non-English OCR data in pre-training and instruction-tuning is
  paramount for improving multilingual text-in-image understanding. Finally, we
  put all our findings together and train Centurio, a 100-language LVLM, offering
  state-of-the-art performance in an evaluation covering 14 tasks and 56
  languages.
paper_authors._id: 6780ce57c759e70936ba6da5
paper_authors.name: Gregor Geigle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T13:35:02.224Z
paper_authors._id: 613f291fe5a812eff913808e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/613f291fe5a812eff913808e/aQUoRv6tllC2j-jg6meUt.jpeg
paper_authors.fullname: Gregor Geigle
paper_authors.type: user
paper_authors.user: Gregor
paper_authors._id: 6780ce57c759e70936ba6da6
paper_authors.name: Florian Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:35.865Z
paper_authors._id: 62dfd54798815401141c47fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
paper_authors.fullname: Flo Schneider
paper_authors.type: user
paper_authors.user: floschne
paper_authors._id: 6780ce57c759e70936ba6da7
paper_authors.name: Carolin Holtermann
paper_authors._id: 6780ce57c759e70936ba6da8
paper_authors.name: Chris Biemann
paper_authors._id: 6780ce57c759e70936ba6da9
paper_authors.name: Radu Timofte
paper_authors._id: 6780ce57c759e70936ba6daa
paper_authors.name: Anne Lauscher
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:33.626Z
paper_authors._id: 626c02e7703f3b27dd590896
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
paper_authors.fullname: Anne Lauscher
paper_authors.type: user
paper_authors.user: anlausch
paper_authors._id: 6780ce57c759e70936ba6dab
paper_authors.name: Goran Glavaš
paper_title: Centurio: On Drivers of Multilingual Ability of Large Vision-Language
    Model
paper_summary: Most Large Vision-Language Models (LVLMs) to date are trained predominantly
  on English data, which makes them struggle to understand non-English input and
  fail to generate output in the desired target language. Existing efforts
  mitigate these issues by adding multilingual training data, but do so in a
  largely ad-hoc manner, lacking insight into how different training mixes tip
  the scale for different groups of languages. In this work, we present a
  comprehensive investigation into the training strategies for massively
  multilingual LVLMs. First, we conduct a series of multi-stage experiments
  spanning 13 downstream vision-language tasks and 43 languages, systematically
  examining: (1) the number of training languages that can be included without
  degrading English performance and (2) optimal language distributions of
  pre-training as well as (3) instruction-tuning data. Further, we (4)
  investigate how to improve multilingual text-in-image understanding, and
  introduce a new benchmark for the task. Surprisingly, our analysis reveals that
  one can (i) include as many as 100 training languages simultaneously (ii) with
  as little as 25-50\% of non-English data, to greatly improve multilingual
  performance while retaining strong English performance. We further find that
  (iii) including non-English OCR data in pre-training and instruction-tuning is
  paramount for improving multilingual text-in-image understanding. Finally, we
  put all our findings together and train Centurio, a 100-language LVLM, offering
  state-of-the-art performance in an evaluation covering 14 tasks and 56
  languages.
paper_ai_summary: A comprehensive investigation into training strategies for multilingual vision-language models reveals optimal ways to include multiple languages without degrading English performance and introduces a new benchmark for text-in-image understanding.
paper_ai_keywords: Large Vision-Language Models LVLMs multilingual training downstream vision-language tasks pre-training instruction-tuning multilingual text-in-image understanding benchmark