LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture
<|reference_start|>LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture: Expanding the long-context capabilities of Multi-modal Large Language Models~(MLLMs) is crucial for video understanding, high-resolution image understanding, and multi-modal agents. This involves a series of systematic optimizations, including model architecture, data construction and training strategy, particularly addressing challenges such as \textit{degraded performance with more images} and \textit{high computational costs}. In this paper, we adapt the model architecture to a hybrid of Mamba and Transformer blocks, approach data construction with both temporal and spatial dependencies among multiple images and employ a progressive training strategy. The released model \textbf{LongLLaVA}~(\textbf{Long}-Context \textbf{L}arge \textbf{L}anguage \textbf{a}nd \textbf{V}ision \textbf{A}ssistant) is the first hybrid MLLM, which achieved a better balance between efficiency and effectiveness. LongLLaVA not only achieves competitive results across various benchmarks, but also maintains high throughput and low memory consumption. Especially, it could process nearly a thousand images on a single A100 80GB GPU, showing promising application prospects for a wide range of tasks.<|reference_end|>
@article{wang2024longllava:,
    title={LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a
  Hybrid Architecture},
    author={Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, Benyou Wang},
    journal={arXiv preprint arXiv:2409.02889},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.02889},
    primaryClass={cs.CL cs.AI cs.CV cs.MM}
}