title: PALO: A Polyglot Large Multimodal Model for 5B People
summary: In pursuit of more inclusive Vision-Language Models (VLMs), this study
  introduces a Large Multilingual Multimodal Model called Palo.
  Palo offers visual reasoning capabilities in 10 major languages,
  including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
  Urdu, and Japanese, that span a total of sim5B people (65\% of the world
  population). Our approach involves a semi-automated translation approach to
  adapt the multimodal instruction dataset from English to the target languages
  using a fine-tuned Large Language Model, thereby ensuring high linguistic
  fidelity while allowing scalability due to minimal manual effort. The
  incorporation of diverse instruction sets helps us boost overall performance
  across multiple languages especially those that are underrepresented like
  Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
  scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
  where we observe substantial improvements compared to strong baselines. We also
  propose the first multilingual multimodal benchmark for the forthcoming
  approaches to evaluate their vision-language reasoning capabilities across
  languages. Code: https://github.com/mbzuai-oryx/PALO.
paper_authors._id: 65d813fa0be3fc13a9de8fab
paper_authors.name: Muhammad Maaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-26T09:25:48.560Z
paper_authors._id: 64807585856901b0edb8d68b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
paper_authors.fullname: Muhammad Maaz
paper_authors.type: user
paper_authors.user: mmaaz60
paper_authors._id: 65d813fa0be3fc13a9de8fac
paper_authors.name: Hanoona Rasheed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:48:18.019Z
paper_authors._id: 64636b2551fa6e6306046293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
paper_authors.fullname: Hanoona Rasheed
paper_authors.type: user
paper_authors.user: Hanoona
paper_authors._id: 65d813fa0be3fc13a9de8fad
paper_authors.name: Abdelrahman Shaker
paper_authors._id: 65d813fa0be3fc13a9de8fae
paper_authors.name: Salman Khan
paper_authors._id: 65d813fa0be3fc13a9de8faf
paper_authors.name: Hisham Cholakal
paper_authors._id: 65d813fa0be3fc13a9de8fb0
paper_authors.name: Rao M. Anwer
paper_authors._id: 65d813fa0be3fc13a9de8fb1
paper_authors.name: Tim Baldwin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-23T13:02:29.589Z
paper_authors._id: 65d895d8d435dce78392065f
paper_authors.avatarUrl: /avatars/68c7185d3dd1adf78927b91849978d62.svg
paper_authors.fullname: Tim Baldwin
paper_authors.type: user
paper_authors.user: tbaldwin
paper_authors._id: 65d813fa0be3fc13a9de8fb2
paper_authors.name: Michael Felsberg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:49:10.186Z
paper_authors._id: 63d38a1a9e1b81818c816905
paper_authors.avatarUrl: /avatars/a2160bb9fbb8a8eec129b8eab559294b.svg
paper_authors.fullname: Michael Felsberg
paper_authors.type: user
paper_authors.user: CVL-LiU
paper_authors._id: 65d813fa0be3fc13a9de8fb3
paper_authors.name: Fahad S. Khan
paper_title: PALO: A Polyglot Large Multimodal Model for 5B People
paper_summary: In pursuit of more inclusive Vision-Language Models (VLMs), this study
  introduces a Large Multilingual Multimodal Model called Palo.
  Palo offers visual reasoning capabilities in 10 major languages,
  including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
  Urdu, and Japanese, that span a total of sim5B people (65\% of the world
  population). Our approach involves a semi-automated translation approach to
  adapt the multimodal instruction dataset from English to the target languages
  using a fine-tuned Large Language Model, thereby ensuring high linguistic
  fidelity while allowing scalability due to minimal manual effort. The
  incorporation of diverse instruction sets helps us boost overall performance
  across multiple languages especially those that are underrepresented like
  Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
  scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
  where we observe substantial improvements compared to strong baselines. We also
  propose the first multilingual multimodal benchmark for the forthcoming
  approaches to evaluate their vision-language reasoning capabilities across
  languages. Code: https://github.com/mbzuai-oryx/PALO.
paper_ai_summary: A multilingual multimodal model called Palo enhances visual reasoning across 10 languages using a semi-automated translation approach and achieves substantial performance improvements across multiple scales.
paper_ai_keywords: Large Multilingual Multimodal Model visual reasoning Palo semi-automated translation fine-tuned Large Language Model multilingual multimodal benchmark