title: LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
    Hierarchical Window Transformer
summary: In multimodal large language models (MLLMs), vision transformers (ViTs) are
  widely employed for visual encoding. However, their performance in solving
  universal MLLM tasks is not satisfactory. We attribute it to a lack of
  information from diverse visual levels, impeding alignment with the various
  semantic granularity required for language generation. To address this issue,
  we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
  transformer that enables capturing diverse visual granularity by constructing
  and integrating a high-resolution feature pyramid. As a vision-language
  projector, Hiwin transformer comprises two primary modules: (i) an inverse
  feature pyramid, constructed by a ViT-derived feature up-sampling process
  utilizing high-frequency details from an image pyramid, and (ii) hierarchical
  window attention, focusing on a set of key sampling features within cross-scale
  windows to condense multi-level feature maps. Extensive experiments demonstrate
  that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
  benchmarks. Notably, our design brings an average boost of 3.7% across 14
  benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
  make all the data, model checkpoint, and code publicly available to facilitate
  future research.
paper_authors._id: 67640aa2eb7128270fa0404a
paper_authors.name: Yipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:30.587Z
paper_authors._id: 666e91b1623133f1ce35acc5
paper_authors.avatarUrl: /avatars/cc78520e6cfb83817c1d0c1ac867ebdd.svg
paper_authors.fullname: YipengZhang
paper_authors.type: user
paper_authors.user: YipengZhang
paper_authors._id: 67640aa2eb7128270fa0404b
paper_authors.name: Yifan Liu
paper_authors._id: 67640aa2eb7128270fa0404c
paper_authors.name: Zonghao Guo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-19T11:59:32.252Z
paper_authors._id: 6491af36c1741666238f3bff
paper_authors.avatarUrl: /avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
paper_authors.fullname: Zonghao Guo
paper_authors.type: user
paper_authors.user: guozonghao96
paper_authors._id: 67640aa2eb7128270fa0404d
paper_authors.name: Yidan Zhang
paper_authors._id: 67640aa2eb7128270fa0404e
paper_authors.name: Xuesong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:44:05.061Z
paper_authors._id: 64b708fa53d91a364aa32bbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b708fa53d91a364aa32bbf/f8gnDFShkI3VBhadko8qr.jpeg
paper_authors.fullname: Xuesong Yang
paper_authors.type: user
paper_authors.user: magicyoung8
paper_authors._id: 67640aa2eb7128270fa0404f
paper_authors.name: Chi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:21:37.240Z
paper_authors._id: 642086ed290342c5df85662d
paper_authors.avatarUrl: /avatars/915a4d7b89455ae97b8544c79286ddf8.svg
paper_authors.fullname: Chi Chen
paper_authors.type: user
paper_authors.user: carboncoo
paper_authors._id: 67640aa2eb7128270fa04050
paper_authors.name: Jun Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:44:46.935Z
paper_authors._id: 645b3bbb4093d5c140deff2d
paper_authors.avatarUrl: /avatars/70a7c860f42af143c1f925d9bd2328d1.svg
paper_authors.fullname: sj
paper_authors.type: user
paper_authors.user: songjun
paper_authors._id: 67640aa2eb7128270fa04051
paper_authors.name: Bo Zheng
paper_authors._id: 67640aa2eb7128270fa04052
paper_authors.name: Yuan Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:44:53.591Z
paper_authors._id: 664618d83a7681520570b58a
paper_authors.avatarUrl: /avatars/c03e25ee9bbf3504cdf2959acf81e0fe.svg
paper_authors.fullname: Yuan Yao
paper_authors.type: user
paper_authors.user: yuanyao
paper_authors._id: 67640aa2eb7128270fa04053
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:43:24.984Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67640aa2eb7128270fa04054
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:43:10.504Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_authors._id: 67640aa2eb7128270fa04055
paper_authors.name: Maosong Sun
paper_title: LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
    Hierarchical Window Transformer
paper_summary: In multimodal large language models (MLLMs), vision transformers (ViTs) are
  widely employed for visual encoding. However, their performance in solving
  universal MLLM tasks is not satisfactory. We attribute it to a lack of
  information from diverse visual levels, impeding alignment with the various
  semantic granularity required for language generation. To address this issue,
  we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
  transformer that enables capturing diverse visual granularity by constructing
  and integrating a high-resolution feature pyramid. As a vision-language
  projector, Hiwin transformer comprises two primary modules: (i) an inverse
  feature pyramid, constructed by a ViT-derived feature up-sampling process
  utilizing high-frequency details from an image pyramid, and (ii) hierarchical
  window attention, focusing on a set of key sampling features within cross-scale
  windows to condense multi-level feature maps. Extensive experiments demonstrate
  that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
  benchmarks. Notably, our design brings an average boost of 3.7% across 14
  benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
  make all the data, model checkpoint, and code publicly available to facilitate
  future research.
paper_ai_keywords: multimodal large language models (MLLMs) vision transformers (ViTs) visual encoding semantic granularity LLaVA-UHD v2 Hierarchical window transformer high-resolution feature pyramid vision-language projector Hiwin transformer inverse feature pyramid ViT-derived feature up-sampling image pyramid hierarchical window attention cross-scale windows feature maps DocVQA