title: PaLI-X: On Scaling up a Multilingual Vision and Language Model
summary: We present the training recipe and results of scaling up PaLI-X, a
  multilingual vision and language model, both in terms of size of the components
  and the breadth of its training task mixture. Our model achieves new levels of
  performance on a wide-range of varied and complex tasks, including multiple
  image-based captioning and question-answering tasks, image-based document
  understanding and few-shot (in-context) learning, as well as object detection,
  video question answering, and video captioning. PaLI-X advances the
  state-of-the-art on most vision-and-language benchmarks considered (25+ of
  them). Finally, we observe emerging capabilities, such as complex counting and
  multilingual object detection, tasks that are not explicitly in the training
  mix.
paper_authors._id: 6476c48d9b260a1739bb981b
paper_authors.name: Xi Chen
paper_authors._id: 6476c48d9b260a1739bb981c
paper_authors.name: Josip Djolonga
paper_authors._id: 6476c48d9b260a1739bb981d
paper_authors.name: Piotr Padlewski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:01:15.940Z
paper_authors._id: 64770ddafe51edaf1fc0322d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770ddafe51edaf1fc0322d/jxN1_ALBmz7hn-IG8q75P.jpeg
paper_authors.fullname: Piotr Padlewski
paper_authors.type: user
paper_authors.user: prazek
paper_authors._id: 6476c48d9b260a1739bb981e
paper_authors.name: Basil Mustafa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:03:43.926Z
paper_authors._id: 63cfcad6e23b90128c66685c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
paper_authors.fullname: Basil Mustafa
paper_authors.type: user
paper_authors.user: BasilMustafa
paper_authors._id: 6476c48d9b260a1739bb981f
paper_authors.name: Soravit Changpinyo
paper_authors._id: 6476c48d9b260a1739bb9820
paper_authors.name: Jialin Wu
paper_authors._id: 6476c48d9b260a1739bb9821
paper_authors.name: Carlos Riquelme Ruiz
paper_authors._id: 6476c48d9b260a1739bb9822
paper_authors.name: Sebastian Goodman
paper_authors._id: 6476c48d9b260a1739bb9823
paper_authors.name: Xiao Wang
paper_authors._id: 6476c48d9b260a1739bb9824
paper_authors.name: Yi Tay
paper_authors._id: 6476c48d9b260a1739bb9825
paper_authors.name: Siamak Shakeri
paper_authors._id: 6476c48d9b260a1739bb9826
paper_authors.name: Mostafa Dehghani
paper_authors._id: 6476c48d9b260a1739bb9827
paper_authors.name: Daniel Salz
paper_authors._id: 6476c48d9b260a1739bb9828
paper_authors.name: Mario Lucic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:05:49.667Z
paper_authors._id: 643e8cc492159b909ffe8dd2
paper_authors.avatarUrl: /avatars/fc0b506916912e60971376997b3941ba.svg
paper_authors.fullname: Mario Lucic
paper_authors.type: user
paper_authors.user: lucic
paper_authors._id: 6476c48d9b260a1739bb9829
paper_authors.name: Michael Tschannen
paper_authors.statusLastChangedAt: 2023-05-31T08:06:07.746Z
paper_authors._id: 6476c48d9b260a1739bb982a
paper_authors.name: Arsha Nagrani
paper_authors._id: 6476c48d9b260a1739bb982b
paper_authors.name: Hexiang Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:06:47.365Z
paper_authors._id: 643441ccd55dea2d0ec2c309
paper_authors.avatarUrl: /avatars/82e99d445e2b513ad7270fa852adbcbb.svg
paper_authors.fullname: Hexiang Hu
paper_authors.type: user
paper_authors.user: hexianghu
paper_authors._id: 6476c48d9b260a1739bb982c
paper_authors.name: Mandar Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:07:07.413Z
paper_authors._id: 640f5e6cc025ddf618988565
paper_authors.avatarUrl: /avatars/d3983ed6f7b909d6b5c6b7f68408fefd.svg
paper_authors.fullname: Mandar Joshi
paper_authors.type: user
paper_authors.user: mandarjoshi
paper_authors._id: 6476c48d9b260a1739bb982d
paper_authors.name: Bo Pang
paper_authors._id: 6476c48d9b260a1739bb982e
paper_authors.name: Ceslee Montgomery
paper_authors._id: 6476c48d9b260a1739bb982f
paper_authors.name: Paulina Pietrzyk
paper_authors._id: 6476c48d9b260a1739bb9830
paper_authors.name: Marvin Ritter
paper_authors._id: 6476c48d9b260a1739bb9831
paper_authors.name: AJ Piergiovanni
paper_authors._id: 6476c48d9b260a1739bb9832
paper_authors.name: Matthias Minderer
paper_authors._id: 6476c48d9b260a1739bb9833
paper_authors.name: Filip Pavetic
paper_authors._id: 6476c48d9b260a1739bb9834
paper_authors.name: Austin Waters
paper_authors._id: 6476c48d9b260a1739bb9835
paper_authors.name: Gang Li
paper_authors._id: 6476c48d9b260a1739bb9836
paper_authors.name: Ibrahim Alabdulmohsin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:08:24.966Z
paper_authors._id: 630545da20668afe24860235
paper_authors.avatarUrl: /avatars/5d82be2e7412bff1af15cc5eafa60b7d.svg
paper_authors.fullname: Ibrahim Alabdulmohsin
paper_authors.type: user
paper_authors.user: ibomohsin
paper_authors._id: 6476c48d9b260a1739bb9837
paper_authors.name: Lucas Beyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:08:49.062Z
paper_authors._id: 642d334ff65714b4585f2de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
paper_authors.fullname: Lucas Beyer
paper_authors.type: user
paper_authors.user: giffmana
paper_authors._id: 6476c48d9b260a1739bb9838
paper_authors.name: Julien Amelot
paper_authors._id: 6476c48d9b260a1739bb9839
paper_authors.name: Kenton Lee
paper_authors._id: 6476c48d9b260a1739bb983a
paper_authors.name: Andreas Peter Steiner
paper_authors._id: 6476c48d9b260a1739bb983b
paper_authors.name: Yang Li
paper_authors._id: 6476c48d9b260a1739bb983c
paper_authors.name: Daniel Keysers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T10:05:34.589Z
paper_authors._id: 662fc73ccee11621a1067db1
paper_authors.avatarUrl: /avatars/4fbfbc0621f631d8e95a2a642fa0cd27.svg
paper_authors.fullname: Daniel Keysers
paper_authors.type: user
paper_authors.user: dkeysers
paper_authors._id: 6476c48d9b260a1739bb983d
paper_authors.name: Anurag Arnab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T08:10:14.969Z
paper_authors._id: 63d7f0466ac3104e50c5311e
paper_authors.avatarUrl: /avatars/2421bfb7747664d1acefbac1c00c3475.svg
paper_authors.fullname: Anurag Arnab
paper_authors.type: user
paper_authors.user: anurag-arnab
paper_authors._id: 6476c48d9b260a1739bb983e
paper_authors.name: Yuanzhong Xu
paper_authors._id: 6476c48d9b260a1739bb983f
paper_authors.name: Keran Rong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:28:05.635Z
paper_authors._id: 648a9083abcf427a9a498679
paper_authors.avatarUrl: /avatars/e32d413ce0a48d83f95e29c11a8a8ae8.svg
paper_authors.fullname: Keran 
paper_authors.type: user
paper_authors.user: Keeera
paper_authors._id: 6476c48d9b260a1739bb9840
paper_authors.name: Alexander Kolesnikov
paper_authors._id: 6476c48d9b260a1739bb9841
paper_authors.name: Mojtaba Seyedhosseini
paper_authors._id: 6476c48d9b260a1739bb9842
paper_authors.name: Anelia Angelova
paper_authors._id: 6476c48d9b260a1739bb9843
paper_authors.name: Xiaohua Zhai
paper_authors._id: 6476c48d9b260a1739bb9844
paper_authors.name: Neil Houlsby
paper_authors._id: 6476c48d9b260a1739bb9845
paper_authors.name: Radu Soricut
paper_title: PaLI-X: On Scaling up a Multilingual Vision and Language Model
paper_summary: We present the training recipe and results of scaling up PaLI-X, a
  multilingual vision and language model, both in terms of size of the components
  and the breadth of its training task mixture. Our model achieves new levels of
  performance on a wide-range of varied and complex tasks, including multiple
  image-based captioning and question-answering tasks, image-based document
  understanding and few-shot (in-context) learning, as well as object detection,
  video question answering, and video captioning. PaLI-X advances the
  state-of-the-art on most vision-and-language benchmarks considered (25+ of
  them). Finally, we observe emerging capabilities, such as complex counting and
  multilingual object detection, tasks that are not explicitly in the training
  mix.
paper_ai_summary: PaLI-X, a multilingual vision and language model, achieves state-of-the-art performance on various tasks by scaling its components and task diversity.
paper_ai_keywords: multilingual vision and language model image-based captioning image-based question-answering image-based document understanding few-shot learning object detection video question answering video captioning vision-and-language benchmarks complex counting multilingual object detection