ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal
  Models
High-resolution Large Multimodal Models (LMMs) encounter the challenges of
excessive visual tokens and quadratic visual complexity. Current
high-resolution LMMs address the quadratic complexity while still generating
excessive visual tokens. However, the redundancy in visual tokens is the key
problem as it leads to more substantial compute. To mitigate this issue, we
propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the
visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses
high-resolution images into information-rich visual features, effectively
preventing the generation of excessive visual tokens. To enhance the
capabilities of ConvLLaVA, we propose two critical optimizations. Since the
low-resolution pretrained ConvNeXt underperforms when directly applied on high
resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original
compression ratio is inadequate for much higher resolution inputs, we train a
successive stage to further compress the visual tokens, thereby reducing
redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536
resolution generating only 576 visual tokens, capable of handling images of
arbitrary aspect ratios. Experimental results demonstrate that our method
achieves competitive performance with state-of-the-art models on mainstream
benchmarks. The ConvLLaVA model series are publicly available at
https://github.com/alibaba/conv-llava.
66540321d200a2ecdf784ffe
Chunjiang Ge
admin_assigned
2024-05-27T07:54:25.329Z
66249e5c5e23c3941aa6cb09
/avatars/d3f68c2eba04f2698402cc532779a549.svg
Chunjiang Ge
user
ChunjiangGe
66540321d200a2ecdf784fff
Sijie Cheng
admin_assigned
2024-05-27T07:54:42.620Z
639003391c5a623727252735
https://cdn-avatars.huggingface.co/v1/production/uploads/639003391c5a623727252735/fp_wZEhGzxGlyr64XF-x9.jpeg
Sijie Cheng
user
SijieCheng
66540321d200a2ecdf785000
Ziming Wang
66540321d200a2ecdf785001
Jiale Yuan
66540321d200a2ecdf785002
Yuan Gao
66540321d200a2ecdf785003
Jun Song
66540321d200a2ecdf785004
Shiji Song
66540321d200a2ecdf785005
Gao Huang
66540321d200a2ecdf785006
Bo Zheng
admin_assigned
2024-05-27T07:57:05.815Z
62c695ad5aae1c624ca992e2
/avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
Bo Zheng
user
bzheng
ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal
  Models
High-resolution Large Multimodal Models (LMMs) encounter the challenges of
excessive visual tokens and quadratic visual complexity. Current
high-resolution LMMs address the quadratic complexity while still generating
excessive visual tokens. However, the redundancy in visual tokens is the key
problem as it leads to more substantial compute. To mitigate this issue, we
propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the
visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses
high-resolution images into information-rich visual features, effectively
preventing the generation of excessive visual tokens. To enhance the
capabilities of ConvLLaVA, we propose two critical optimizations. Since the
low-resolution pretrained ConvNeXt underperforms when directly applied on high
resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original
compression ratio is inadequate for much higher resolution inputs, we train a
successive stage to further compress the visual tokens, thereby reducing
redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536
resolution generating only 576 visual tokens, capable of handling images of
arbitrary aspect ratios. Experimental results demonstrate that our method
achieves competitive performance with state-of-the-art models on mainstream
benchmarks. The ConvLLaVA model series are publicly available at
https://github.com/alibaba/conv-llava.
ConvLLaVA addresses excessive visual tokens and quadratic complexity in high-resolution multimodal models by using ConvNeXt as a hierarchical backbone, optimizing pretrained ConvNeXt for high resolution, and adding a successive stage for further compression, achieving competitive performance.
High-resolution Large Multimodal Models LMMs quadratic complexity visual tokens ConvLLaVA ConvNeXt hierarchical backbone Vision Transformer ViT image compression visual features aspect ratios