title: NVILA: Efficient Frontier Visual Language Models
summary: Visual language models (VLMs) have made significant advances in accuracy in
  recent years. However, their efficiency has received much less attention. This
  paper introduces NVILA, a family of open VLMs designed to optimize both
  efficiency and accuracy. Building on top of VILA, we improve its model
  architecture by first scaling up the spatial and temporal resolutions, and then
  compressing visual tokens. This "scale-then-compress" approach enables NVILA to
  efficiently process high-resolution images and long videos. We also conduct a
  systematic investigation to enhance the efficiency of NVILA throughout its
  entire lifecycle, from training and fine-tuning to deployment. NVILA matches or
  surpasses the accuracy of many leading open and proprietary VLMs across a wide
  range of image and video benchmarks. At the same time, it reduces training
  costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by
  1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and
  models available to facilitate reproducibility.
paper_authors._id: 67530600886878c8868445a9
paper_authors.name: Zhijian Liu
paper_authors._id: 67530600886878c8868445aa
paper_authors.name: Ligeng Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:37:12.108Z
paper_authors._id: 62b4b5beb25cb80fcf278354
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
paper_authors.fullname: Ligeng Zhu
paper_authors.type: user
paper_authors.user: Ligeng-Zhu
paper_authors._id: 67530600886878c8868445ab
paper_authors.name: Baifeng Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:06:59.994Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 67530600886878c8868445ac
paper_authors.name: Zhuoyang Zhang
paper_authors._id: 67530600886878c8868445ad
paper_authors.name: Yuming Lou
paper_authors._id: 67530600886878c8868445ae
paper_authors.name: Shang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T10:00:24.530Z
paper_authors._id: 641d8bacd526196afc12766d
paper_authors.avatarUrl: /avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
paper_authors.fullname: Shang Yang
paper_authors.type: user
paper_authors.user: Shangy
paper_authors._id: 67530600886878c8868445af
paper_authors.name: Haocheng Xi
paper_authors._id: 67530600886878c8868445b0
paper_authors.name: Shiyi Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:06:56.294Z
paper_authors._id: 64ebbae6895a36ab28de811a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ebbae6895a36ab28de811a/gBiaQP4paS4L13eu-yRm7.jpeg
paper_authors.fullname: Shiyi Cao
paper_authors.type: user
paper_authors.user: eva98
paper_authors._id: 67530600886878c8868445b1
paper_authors.name: Yuxian Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:06:58.017Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 67530600886878c8868445b2
paper_authors.name: Dacheng Li
paper_authors._id: 67530600886878c8868445b3
paper_authors.name: Xiuyu Li
paper_authors._id: 67530600886878c8868445b4
paper_authors.name: Yunhao Fang
paper_authors._id: 67530600886878c8868445b5
paper_authors.name: Yukang Chen
paper_authors._id: 67530600886878c8868445b6
paper_authors.name: Cheng-Yu Hsieh
paper_authors._id: 67530600886878c8868445b7
paper_authors.name: De-An Huang
paper_authors._id: 67530600886878c8868445b8
paper_authors.name: An-Chieh Cheng
paper_authors._id: 67530600886878c8868445b9
paper_authors.name: Vishwesh Nath
paper_authors._id: 67530600886878c8868445ba
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:06:54.373Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 67530600886878c8868445bb
paper_authors.name: Sifei Liu
paper_authors._id: 67530600886878c8868445bc
paper_authors.name: Ranjay Krishna
paper_authors._id: 67530600886878c8868445bd
paper_authors.name: Daguang Xu
paper_authors._id: 67530600886878c8868445be
paper_authors.name: Xiaolong Wang
paper_authors._id: 67530600886878c8868445bf
paper_authors.name: Pavlo Molchanov
paper_authors._id: 67530600886878c8868445c0
paper_authors.name: Jan Kautz
paper_authors._id: 67530600886878c8868445c1
paper_authors.name: Hongxu Yin
paper_authors._id: 67530600886878c8868445c2
paper_authors.name: Song Han
paper_authors._id: 67530600886878c8868445c3
paper_authors.name: Yao Lu
paper_title: NVILA: Efficient Frontier Visual Language Models
paper_summary: Visual language models (VLMs) have made significant advances in accuracy in
  recent years. However, their efficiency has received much less attention. This
  paper introduces NVILA, a family of open VLMs designed to optimize both
  efficiency and accuracy. Building on top of VILA, we improve its model
  architecture by first scaling up the spatial and temporal resolutions, and then
  compressing visual tokens. This "scale-then-compress" approach enables NVILA to
  efficiently process high-resolution images and long videos. We also conduct a
  systematic investigation to enhance the efficiency of NVILA throughout its
  entire lifecycle, from training and fine-tuning to deployment. NVILA matches or
  surpasses the accuracy of many leading open and proprietary VLMs across a wide
  range of image and video benchmarks. At the same time, it reduces training
  costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by
  1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and
  models available to facilitate reproducibility.
paper_ai_keywords: Visual language models (VLMs) VILA NVILA model architecture spatial resolutions temporal resolutions visual tokens scale-then-compress training fine-tuning deployment pre-filling latency decoding latency