Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
67e375d3cc93cc8c42da7699
Baifeng Shi
claimed_verified
2025-03-26T20:44:24.349Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
67e375d3cc93cc8c42da769a
Boyi Li
67e375d3cc93cc8c42da769b
Han Cai
67e375d3cc93cc8c42da769c
Yao Lu
67e375d3cc93cc8c42da769d
Sifei Liu
67e375d3cc93cc8c42da769e
Marco Pavone
67e375d3cc93cc8c42da769f
Jan Kautz
67e375d3cc93cc8c42da76a0
Song Han
67e375d3cc93cc8c42da76a1
Trevor Darrell
67e375d3cc93cc8c42da76a2
Pavlo Molchanov
claimed_verified
2025-04-20T15:04:07.484Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
67e375d3cc93cc8c42da76a3
Hongxu Yin
claimed_verified
2025-03-27T09:04:59.461Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
PS3 enables high-resolution vision pre-training with reduced computational cost, improving multi-modal LLM performance and efficiency.
CLIP-style vision pre-training PS3 contrastive learning local regions detailed captions high-resolution representation learning saliency text prompt multi-modal LLM VILA-HD AnyRes S^2 high-resolution visual perception MLLM NVILA Qwen2-VL 4KPro image QA token pruning