PaLI-3 Vision Language Models: Smaller, Faster, Stronger
This paper presents PaLI-3, a smaller, faster, and stronger vision language
model (VLM) that compares favorably to similar models that are 10x larger. As
part of arriving at this strong performance, we compare Vision Transformer
(ViT) models pretrained using classification objectives to contrastively
(SigLIP) pretrained ones. We find that, while slightly underperforming on
standard image classification benchmarks, SigLIP-based PaLI shows superior
performance across various multimodal benchmarks, especially on localization
and visually-situated text understanding. We scale the SigLIP image encoder up
to 2 billion parameters, and achieves a new state-of-the-art on multilingual
cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
research on fundamental pieces of complex VLMs, and could fuel a new generation
of scaled-up models.
652caebdff2202020ecbd639
Xi Chen
652caebdff2202020ecbd63a
Xiao Wang
652caebdff2202020ecbd63b
Lucas Beyer
claimed_verified
2024-01-09T14:25:56.488Z
642d334ff65714b4585f2de4
https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
Lucas Beyer
user
giffmana
652caebdff2202020ecbd63c
Alexander Kolesnikov
652caebdff2202020ecbd63d
Jialin Wu
652caebdff2202020ecbd63e
Paul Voigtlaender
652caebdff2202020ecbd63f
Basil Mustafa
652caebdff2202020ecbd640
Sebastian Goodman
652caebdff2202020ecbd641
Ibrahim Alabdulmohsin
652caebdff2202020ecbd642
Piotr Padlewski
652caebdff2202020ecbd643
Daniel Salz
652caebdff2202020ecbd644
Xi Xiong
652caebdff2202020ecbd645
Daniel Vlasic
652caebdff2202020ecbd646
Filip Pavetic
652caebdff2202020ecbd647
Keran Rong
claimed_verified
2025-02-13T08:28:22.875Z
648a9083abcf427a9a498679
/avatars/e32d413ce0a48d83f95e29c11a8a8ae8.svg
Keran 
user
Keeera
652caebdff2202020ecbd648
Tianli Yu
652caebdff2202020ecbd649
Daniel Keysers
claimed_verified
2024-07-11T10:05:29.144Z
662fc73ccee11621a1067db1
/avatars/4fbfbc0621f631d8e95a2a642fa0cd27.svg
Daniel Keysers
user
dkeysers
652caebdff2202020ecbd64a
Xiaohua Zhai
652caebdff2202020ecbd64b
Radu Soricut
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
This paper presents PaLI-3, a smaller, faster, and stronger vision language
model (VLM) that compares favorably to similar models that are 10x larger. As
part of arriving at this strong performance, we compare Vision Transformer
(ViT) models pretrained using classification objectives to contrastively
(SigLIP) pretrained ones. We find that, while slightly underperforming on
standard image classification benchmarks, SigLIP-based PaLI shows superior
performance across various multimodal benchmarks, especially on localization
and visually-situated text understanding. We scale the SigLIP image encoder up
to 2 billion parameters, and achieves a new state-of-the-art on multilingual
cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
research on fundamental pieces of complex VLMs, and could fuel a new generation
of scaled-up models.
PaLI-3, a smaller vision language model, outperforms larger models on multimodal tasks, particularly localization and text understanding, using SigLIP pretraining and achieves state-of-the-art results in multilingual cross-modal retrieval.
Vision Transformer SigLIP image encoder multilingual cross-modal retrieval localization visually-situated text understanding