title: ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal
    Models
summary: High-resolution Large Multimodal Models (LMMs) encounter the challenges of
  excessive visual tokens and quadratic visual complexity. Current
  high-resolution LMMs address the quadratic complexity while still generating
  excessive visual tokens. However, the redundancy in visual tokens is the key
  problem as it leads to more substantial compute. To mitigate this issue, we
  propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the
  visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses
  high-resolution images into information-rich visual features, effectively
  preventing the generation of excessive visual tokens. To enhance the
  capabilities of ConvLLaVA, we propose two critical optimizations. Since the
  low-resolution pretrained ConvNeXt underperforms when directly applied on high
  resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original
  compression ratio is inadequate for much higher resolution inputs, we train a
  successive stage to further compress the visual tokens, thereby reducing
  redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536
  resolution generating only 576 visual tokens, capable of handling images of
  arbitrary aspect ratios. Experimental results demonstrate that our method
  achieves competitive performance with state-of-the-art models on mainstream
  benchmarks. The ConvLLaVA model series are publicly available at
  https://github.com/alibaba/conv-llava.
paper_authors._id: 66540321d200a2ecdf784ffe
paper_authors.name: Chunjiang Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T07:54:25.329Z
paper_authors._id: 66249e5c5e23c3941aa6cb09
paper_authors.avatarUrl: /avatars/d3f68c2eba04f2698402cc532779a549.svg
paper_authors.fullname: Chunjiang Ge
paper_authors.type: user
paper_authors.user: ChunjiangGe
paper_authors._id: 66540321d200a2ecdf784fff
paper_authors.name: Sijie Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T07:54:42.620Z
paper_authors._id: 639003391c5a623727252735
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639003391c5a623727252735/fp_wZEhGzxGlyr64XF-x9.jpeg
paper_authors.fullname: Sijie Cheng
paper_authors.type: user
paper_authors.user: SijieCheng
paper_authors._id: 66540321d200a2ecdf785000
paper_authors.name: Ziming Wang
paper_authors._id: 66540321d200a2ecdf785001
paper_authors.name: Jiale Yuan
paper_authors._id: 66540321d200a2ecdf785002
paper_authors.name: Yuan Gao
paper_authors._id: 66540321d200a2ecdf785003
paper_authors.name: Jun Song
paper_authors._id: 66540321d200a2ecdf785004
paper_authors.name: Shiji Song
paper_authors._id: 66540321d200a2ecdf785005
paper_authors.name: Gao Huang
paper_authors._id: 66540321d200a2ecdf785006
paper_authors.name: Bo Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T07:57:05.815Z
paper_authors._id: 62c695ad5aae1c624ca992e2
paper_authors.avatarUrl: /avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
paper_authors.fullname: Bo Zheng
paper_authors.type: user
paper_authors.user: bzheng
paper_title: ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal
    Models
paper_summary: High-resolution Large Multimodal Models (LMMs) encounter the challenges of
  excessive visual tokens and quadratic visual complexity. Current
  high-resolution LMMs address the quadratic complexity while still generating
  excessive visual tokens. However, the redundancy in visual tokens is the key
  problem as it leads to more substantial compute. To mitigate this issue, we
  propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the
  visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses
  high-resolution images into information-rich visual features, effectively
  preventing the generation of excessive visual tokens. To enhance the
  capabilities of ConvLLaVA, we propose two critical optimizations. Since the
  low-resolution pretrained ConvNeXt underperforms when directly applied on high
  resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original
  compression ratio is inadequate for much higher resolution inputs, we train a
  successive stage to further compress the visual tokens, thereby reducing
  redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536
  resolution generating only 576 visual tokens, capable of handling images of
  arbitrary aspect ratios. Experimental results demonstrate that our method
  achieves competitive performance with state-of-the-art models on mainstream
  benchmarks. The ConvLLaVA model series are publicly available at
  https://github.com/alibaba/conv-llava.
paper_ai_keywords: Large multimodal Models (LMMs) Visual tokens Quadratic visual complexity ConvLLaVA ConvNeXt Visual encoder Vision Transformer (ViT) Information-develop visual features High-resolution images Low-resolution pretrained ConvNeXt Compression ratio Arbitrary aspect ratios State-of-the-art models Mainstream benchmarks