title: PaLI-3 Vision Language Models: Smaller, Faster, Stronger
summary: This paper presents PaLI-3, a smaller, faster, and stronger vision language
  model (VLM) that compares favorably to similar models that are 10x larger. As
  part of arriving at this strong performance, we compare Vision Transformer
  (ViT) models pretrained using classification objectives to contrastively
  (SigLIP) pretrained ones. We find that, while slightly underperforming on
  standard image classification benchmarks, SigLIP-based PaLI shows superior
  performance across various multimodal benchmarks, especially on localization
  and visually-situated text understanding. We scale the SigLIP image encoder up
  to 2 billion parameters, and achieves a new state-of-the-art on multilingual
  cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
  research on fundamental pieces of complex VLMs, and could fuel a new generation
  of scaled-up models.
paper_authors._id: 652caebdff2202020ecbd639
paper_authors.name: Xi Chen
paper_authors._id: 652caebdff2202020ecbd63a
paper_authors.name: Xiao Wang
paper_authors._id: 652caebdff2202020ecbd63b
paper_authors.name: Lucas Beyer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-09T14:25:56.488Z
paper_authors._id: 642d334ff65714b4585f2de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
paper_authors.fullname: Lucas Beyer
paper_authors.type: user
paper_authors.user: giffmana
paper_authors._id: 652caebdff2202020ecbd63c
paper_authors.name: Alexander Kolesnikov
paper_authors._id: 652caebdff2202020ecbd63d
paper_authors.name: Jialin Wu
paper_authors._id: 652caebdff2202020ecbd63e
paper_authors.name: Paul Voigtlaender
paper_authors._id: 652caebdff2202020ecbd63f
paper_authors.name: Basil Mustafa
paper_authors._id: 652caebdff2202020ecbd640
paper_authors.name: Sebastian Goodman
paper_authors._id: 652caebdff2202020ecbd641
paper_authors.name: Ibrahim Alabdulmohsin
paper_authors._id: 652caebdff2202020ecbd642
paper_authors.name: Piotr Padlewski
paper_authors._id: 652caebdff2202020ecbd643
paper_authors.name: Daniel Salz
paper_authors._id: 652caebdff2202020ecbd644
paper_authors.name: Xi Xiong
paper_authors._id: 652caebdff2202020ecbd645
paper_authors.name: Daniel Vlasic
paper_authors._id: 652caebdff2202020ecbd646
paper_authors.name: Filip Pavetic
paper_authors._id: 652caebdff2202020ecbd647
paper_authors.name: Keran Rong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:28:22.875Z
paper_authors._id: 648a9083abcf427a9a498679
paper_authors.avatarUrl: /avatars/e32d413ce0a48d83f95e29c11a8a8ae8.svg
paper_authors.fullname: Keran 
paper_authors.type: user
paper_authors.user: Keeera
paper_authors._id: 652caebdff2202020ecbd648
paper_authors.name: Tianli Yu
paper_authors._id: 652caebdff2202020ecbd649
paper_authors.name: Daniel Keysers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T10:05:29.144Z
paper_authors._id: 662fc73ccee11621a1067db1
paper_authors.avatarUrl: /avatars/4fbfbc0621f631d8e95a2a642fa0cd27.svg
paper_authors.fullname: Daniel Keysers
paper_authors.type: user
paper_authors.user: dkeysers
paper_authors._id: 652caebdff2202020ecbd64a
paper_authors.name: Xiaohua Zhai
paper_authors._id: 652caebdff2202020ecbd64b
paper_authors.name: Radu Soricut
paper_title: PaLI-3 Vision Language Models: Smaller, Faster, Stronger
paper_summary: This paper presents PaLI-3, a smaller, faster, and stronger vision language
  model (VLM) that compares favorably to similar models that are 10x larger. As
  part of arriving at this strong performance, we compare Vision Transformer
  (ViT) models pretrained using classification objectives to contrastively
  (SigLIP) pretrained ones. We find that, while slightly underperforming on
  standard image classification benchmarks, SigLIP-based PaLI shows superior
  performance across various multimodal benchmarks, especially on localization
  and visually-situated text understanding. We scale the SigLIP image encoder up
  to 2 billion parameters, and achieves a new state-of-the-art on multilingual
  cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
  research on fundamental pieces of complex VLMs, and could fuel a new generation
  of scaled-up models.
paper_ai_summary: PaLI-3, a smaller vision language model, outperforms larger models on multimodal tasks, particularly localization and text understanding, using SigLIP pretraining and achieves state-of-the-art results in multilingual cross-modal retrieval.
paper_ai_keywords: Vision Transformer SigLIP image encoder multilingual cross-modal retrieval localization visually-situated text understanding