title: Centurio: On Drivers of Multilingual Ability of Large Vision-Language
    Model
summary: Most Large Vision-Language Models (LVLMs) to date are trained predominantly
  on English data, which makes them struggle to understand non-English input and
  fail to generate output in the desired target language. Existing efforts
  mitigate these issues by adding multilingual training data, but do so in a
  largely ad-hoc manner, lacking insight into how different training mixes tip
  the scale for different groups of languages. In this work, we present a
  comprehensive investigation into the training strategies for massively
  multilingual LVLMs. First, we conduct a series of multi-stage experiments
  spanning 13 downstream vision-language tasks and 43 languages, systematically
  examining: (1) the number of training languages that can be included without
  degrading English performance and (2) optimal language distributions of
  pre-training as well as (3) instruction-tuning data. Further, we (4)
  investigate how to improve multilingual text-in-image understanding, and
  introduce a new benchmark for the task. Surprisingly, our analysis reveals that
  one can (i) include as many as 100 training languages simultaneously (ii) with
  as little as 25-50\% of non-English data, to greatly improve multilingual
  performance while retaining strong English performance. We further find that
  (iii) including non-English OCR data in pre-training and instruction-tuning is
  paramount for improving multilingual text-in-image understanding. Finally, we
  put all our findings together and train Centurio, a 100-language LVLM, offering
  state-of-the-art performance in an evaluation covering 14 tasks and 56
  languages.
paper_authors._id: 6780ce57c759e70936ba6da5
paper_authors.name: Gregor Geigle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T13:35:02.224Z
paper_authors._id: 613f291fe5a812eff913808e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/613f291fe5a812eff913808e/aQUoRv6tllC2j-jg6meUt.jpeg
paper_authors.fullname: Gregor Geigle
paper_authors.type: user
paper_authors.user: Gregor
paper_authors._id: 6780ce57c759e70936ba6da6
paper_authors.name: Florian Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:35.865Z
paper_authors._id: 62dfd54798815401141c47fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
paper_authors.fullname: Flo Schneider
paper_authors.type: user
paper_authors.user: floschne
paper_authors._id: 6780ce57c759e70936ba6da7
paper_authors.name: Carolin Holtermann
paper_authors._id: 6780ce57c759e70936ba6da8
paper_authors.name: Chris Biemann
paper_authors._id: 6780ce57c759e70936ba6da9
paper_authors.name: Radu Timofte
paper_authors._id: 6780ce57c759e70936ba6daa
paper_authors.name: Anne Lauscher
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:57:33.626Z
paper_authors._id: 626c02e7703f3b27dd590896
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
paper_authors.fullname: Anne Lauscher
paper_authors.type: user
paper_authors.user: anlausch
paper_authors._id: 6780ce57c759e70936ba6dab
paper_authors.name: Goran Glavaš
paper_title: Centurio: On Drivers of Multilingual Ability of Large Vision-Language
    Model
paper_summary: Most Large Vision-Language Models (LVLMs) to date are trained predominantly
  on English data, which makes them struggle to understand non-English input and
  fail to generate output in the desired target language. Existing efforts
  mitigate these issues by adding multilingual training data, but do so in a
  largely ad-hoc manner, lacking insight into how different training mixes tip
  the scale for different groups of languages. In this work, we present a
  comprehensive investigation into the training strategies for massively
  multilingual LVLMs. First, we conduct a series of multi-stage experiments
  spanning 13 downstream vision-language tasks and 43 languages, systematically
  examining: (1) the number of training languages that can be included without
  degrading English performance and (2) optimal language distributions of
  pre-training as well as (3) instruction-tuning data. Further, we (4)
  investigate how to improve multilingual text-in-image understanding, and
  introduce a new benchmark for the task. Surprisingly, our analysis reveals that
  one can (i) include as many as 100 training languages simultaneously (ii) with
  as little as 25-50\% of non-English data, to greatly improve multilingual
  performance while retaining strong English performance. We further find that
  (iii) including non-English OCR data in pre-training and instruction-tuning is
  paramount for improving multilingual text-in-image understanding. Finally, we
  put all our findings together and train Centurio, a 100-language LVLM, offering
  state-of-the-art performance in an evaluation covering 14 tasks and 56
  languages.
paper_ai_keywords: Large Vision-Language Models (LVLMs) multilingual training data downstream vision-language tasks English performance multilingual text-in-image understanding non-English OCR data pre-training instruction-tuning adiabetic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adiabatic adi