title: Scaling Vision Pre-Training to 4K Resolution
summary: High-resolution perception of visual details is crucial for daily tasks.
  Current vision pre-training, however, is still limited to low resolutions
  (e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
  We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
  with a near-constant cost. Instead of contrastive learning on global image
  representation, PS3 is pre-trained by selectively processing local regions and
  contrasting them with local detailed captions, enabling high-resolution
  representation learning with greatly reduced computational overhead. The
  pre-trained PS3 is able to both encode the global image at low resolution and
  selectively process local high-resolution regions based on their saliency or
  relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
  resulting model, named VILA-HD, significantly improves high-resolution visual
  perception compared to baselines without high-resolution vision pre-training
  such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
  appealing scaling properties of VILA-HD, including scaling up resolution for
  free and scaling up test-time compute for better performance. Compared to state
  of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
  across multiple benchmarks and achieves better efficiency than latest token
  pruning approaches. Finally, we find current benchmarks do not require
  4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
  of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
  including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
  speedup over Qwen2-VL.
paper_authors._id: 67e375d3cc93cc8c42da7699
paper_authors.name: Baifeng Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:24.349Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 67e375d3cc93cc8c42da769a
paper_authors.name: Boyi Li
paper_authors._id: 67e375d3cc93cc8c42da769b
paper_authors.name: Han Cai
paper_authors._id: 67e375d3cc93cc8c42da769c
paper_authors.name: Yao Lu
paper_authors._id: 67e375d3cc93cc8c42da769d
paper_authors.name: Sifei Liu
paper_authors._id: 67e375d3cc93cc8c42da769e
paper_authors.name: Marco Pavone
paper_authors._id: 67e375d3cc93cc8c42da769f
paper_authors.name: Jan Kautz
paper_authors._id: 67e375d3cc93cc8c42da76a0
paper_authors.name: Song Han
paper_authors._id: 67e375d3cc93cc8c42da76a1
paper_authors.name: Trevor Darrell
paper_authors._id: 67e375d3cc93cc8c42da76a2
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:04:07.484Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 67e375d3cc93cc8c42da76a3
paper_authors.name: Hongxu Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:04:59.461Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_title: Scaling Vision Pre-Training to 4K Resolution
paper_summary: High-resolution perception of visual details is crucial for daily tasks.
  Current vision pre-training, however, is still limited to low resolutions
  (e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
  We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
  with a near-constant cost. Instead of contrastive learning on global image
  representation, PS3 is pre-trained by selectively processing local regions and
  contrasting them with local detailed captions, enabling high-resolution
  representation learning with greatly reduced computational overhead. The
  pre-trained PS3 is able to both encode the global image at low resolution and
  selectively process local high-resolution regions based on their saliency or
  relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
  resulting model, named VILA-HD, significantly improves high-resolution visual
  perception compared to baselines without high-resolution vision pre-training
  such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
  appealing scaling properties of VILA-HD, including scaling up resolution for
  free and scaling up test-time compute for better performance. Compared to state
  of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
  across multiple benchmarks and achieves better efficiency than latest token
  pruning approaches. Finally, we find current benchmarks do not require
  4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
  of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
  including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
  speedup over Qwen2-VL.
paper_ai_summary: PS3 enables high-resolution vision pre-training with reduced computational cost, improving multi-modal LLM performance and efficiency.
paper_ai_keywords: CLIP-style vision pre-training PS3 contrastive learning local regions detailed captions high-resolution representation learning saliency text prompt multi-modal LLM VILA-HD AnyRes S^2 high-resolution visual perception MLLM NVILA Qwen2-VL 4KPro image QA token pruning