title: LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
    Hybrid Architecture
summary: Expanding the long-context capabilities of Multi-modal Large Language
  Models~(MLLMs) is crucial for video understanding, high-resolution image
  understanding, and multi-modal agents. This involves a series of systematic
  optimizations, including model architecture, data construction and training
  strategy, particularly addressing challenges such as degraded
  performance with more images and high computational costs. In this
  paper, we adapt the model architecture to a hybrid of Mamba and Transformer
  blocks, approach data construction with both temporal and spatial dependencies
  among multiple images and employ a progressive training strategy. The released
  model LongLLaVA~(Long-Context Large
  Language and Vision Assistant) is the first
  hybrid MLLM, which achieved a better balance between efficiency and
  effectiveness. LongLLaVA not only achieves competitive results across various
  benchmarks, but also maintains high throughput and low memory consumption.
  Especially, it could process nearly a thousand images on a single A100 80GB
  GPU, showing promising application prospects for a wide range of tasks.
paper_authors._id: 66d917fe8506f9b6cfd098df
paper_authors.name: Xidong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T15:31:08.187Z
paper_authors._id: 640ed3e9f2d7c41a1e9a9fde
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
paper_authors.fullname: Xidong Wang
paper_authors.type: user
paper_authors.user: Xidong
paper_authors._id: 66d917fe8506f9b6cfd098e0
paper_authors.name: Dingjie Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:23:57.982Z
paper_authors._id: 619f01b8cc04eadf54fa5d5d
paper_authors.avatarUrl: /avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
paper_authors.fullname: Song Dingjie
paper_authors.type: user
paper_authors.user: songdj
paper_authors._id: 66d917fe8506f9b6cfd098e1
paper_authors.name: Shunian Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T15:31:17.133Z
paper_authors._id: 623be9e1d1eb227788764959
paper_authors.avatarUrl: /avatars/b6521b795a59754dbb40123fd4f63b8c.svg
paper_authors.fullname: Shunian Chen
paper_authors.type: user
paper_authors.user: Shunian
paper_authors._id: 66d917fe8506f9b6cfd098e2
paper_authors.name: Chen Zhang
paper_authors._id: 66d917fe8506f9b6cfd098e3
paper_authors.name: Benyou Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:35.614Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_title: LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
    Hybrid Architecture
paper_summary: Expanding the long-context capabilities of Multi-modal Large Language
  Models~(MLLMs) is crucial for video understanding, high-resolution image
  understanding, and multi-modal agents. This involves a series of systematic
  optimizations, including model architecture, data construction and training
  strategy, particularly addressing challenges such as degraded
  performance with more images and high computational costs. In this
  paper, we adapt the model architecture to a hybrid of Mamba and Transformer
  blocks, approach data construction with both temporal and spatial dependencies
  among multiple images and employ a progressive training strategy. The released
  model LongLLaVA~(Long-Context Large
  Language and Vision Assistant) is the first
  hybrid MLLM, which achieved a better balance between efficiency and
  effectiveness. LongLLaVA not only achieves competitive results across various
  benchmarks, but also maintains high throughput and low memory consumption.
  Especially, it could process nearly a thousand images on a single A100 80GB
  GPU, showing promising application prospects for a wide range of tasks.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) video understanding high-resolution image understanding multi-modal agents model architecture data construction training strategy degraded performance high computational costs Mamba blocks Transformer blocks temporal dependencies spatial dependencies progressive training strategy LongLLaVA long-context large language vision assistant hybrid MLLM competitive results high throughput low memory consumption