title: LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
summary: Visual encoding constitutes the basis of large multimodal models (LMMs) in
  understanding the visual world. Conventional LMMs process images in fixed sizes
  and limited resolutions, while recent explorations in this direction are
  limited in adaptivity, efficiency, and even correctness. In this work, we first
  take GPT-4V and LLaVA-1.5 as representative examples and expose systematic
  flaws rooted in their visual encoding strategy. To address the challenges, we
  present LLaVA-UHD, a large multimodal model that can efficiently perceive
  images in any aspect ratio and high resolution. LLaVA-UHD includes three key
  components: (1) An image modularization strategy that divides native-resolution
  images into smaller variable-sized slices for efficient and extensible
  encoding, (2) a compression module that further condenses image tokens from
  visual encoders, and (3) a spatial schema to organize slice tokens for LLMs.
  Comprehensive experiments show that LLaVA-UHD outperforms established LMMs
  trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our
  model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088)
  resolution images using only 94% inference computation, and achieves 6.4
  accuracy improvement on TextVQA. Moreover, the model can be efficiently trained
  in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of
  LLaVA-1.5). We make the data and code publicly available at
  https://github.com/thunlp/LLaVA-UHD.
paper_authors._id: 65f918b8ee1763d31ade7c29
paper_authors.name: Ruyi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:35:09.694Z
paper_authors._id: 656db11a271c5c4e334e9c4f
paper_authors.avatarUrl: /avatars/274dbf4c7f982397498c70b91bd6712f.svg
paper_authors.fullname: Xu Ruyi
paper_authors.type: user
paper_authors.user: xrorrim
paper_authors._id: 65f918b8ee1763d31ade7c2a
paper_authors.name: Yuan Yao
paper_authors._id: 65f918b8ee1763d31ade7c2b
paper_authors.name: Zonghao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:35:37.443Z
paper_authors._id: 6491af36c1741666238f3bff
paper_authors.avatarUrl: /avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
paper_authors.fullname: Zonghao Guo
paper_authors.type: user
paper_authors.user: guozonghao96
paper_authors._id: 65f918b8ee1763d31ade7c2c
paper_authors.name: Junbo Cui
paper_authors._id: 65f918b8ee1763d31ade7c2d
paper_authors.name: Zanlin Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:35:52.289Z
paper_authors._id: 63987ffb2ceb55aabe0852f3
paper_authors.avatarUrl: /avatars/343b796ff6b8906203904e8c620d7eb5.svg
paper_authors.fullname: Zanlin Ni
paper_authors.type: user
paper_authors.user: nzl-thu
paper_authors._id: 65f918b8ee1763d31ade7c2e
paper_authors.name: Chunjiang Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:35:59.676Z
paper_authors._id: 645e054ff7a55f0d780a8ff7
paper_authors.avatarUrl: /avatars/9614510443bee3bd5d6266efd1c39fc1.svg
paper_authors.fullname: Chunjiang Ge
paper_authors.type: user
paper_authors.user: HelloJiang
paper_authors._id: 65f918b8ee1763d31ade7c2f
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:36:06.392Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_authors._id: 65f918b8ee1763d31ade7c30
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:37:19.924Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 65f918b8ee1763d31ade7c31
paper_authors.name: Maosong Sun
paper_authors._id: 65f918b8ee1763d31ade7c32
paper_authors.name: Gao Huang
paper_title: LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
paper_summary: Visual encoding constitutes the basis of large multimodal models (LMMs) in
  understanding the visual world. Conventional LMMs process images in fixed sizes
  and limited resolutions, while recent explorations in this direction are
  limited in adaptivity, efficiency, and even correctness. In this work, we first
  take GPT-4V and LLaVA-1.5 as representative examples and expose systematic
  flaws rooted in their visual encoding strategy. To address the challenges, we
  present LLaVA-UHD, a large multimodal model that can efficiently perceive
  images in any aspect ratio and high resolution. LLaVA-UHD includes three key
  components: (1) An image modularization strategy that divides native-resolution
  images into smaller variable-sized slices for efficient and extensible
  encoding, (2) a compression module that further condenses image tokens from
  visual encoders, and (3) a spatial schema to organize slice tokens for LLMs.
  Comprehensive experiments show that LLaVA-UHD outperforms established LMMs
  trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our
  model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088)
  resolution images using only 94% inference computation, and achieves 6.4
  accuracy improvement on TextVQA. Moreover, the model can be efficiently trained
  in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of
  LLaVA-1.5). We make the data and code publicly available at
  https://github.com/thunlp/LLaVA-UHD.
paper_ai_summary: LLaVA-UHD, a large multimodal model, efficiently processes images of any aspect ratio and high resolution by modularizing and compressing image tokens, outperforming established models on various benchmarks.
paper_ai_keywords: GPT-4V LLaVA-1.5 LLaVA-UHD image modularization compression module spatial schema native-resolution images variable-sized slices image tokens visual encoders TextVQA LLMs academic settings A100 GPUs