title: Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient
    Inference
summary: In recent years, the application of multimodal large language models (MLLM)
  in various fields has achieved remarkable success. However, as the foundation
  model for many downstream tasks, current MLLMs are composed of the well-known
  Transformer network, which has a less efficient quadratic computation
  complexity. To improve the efficiency of such basic models, we propose Cobra, a
  linear computational complexity MLLM. Specifically, Cobra integrates the
  efficient Mamba language model into the visual modality. Moreover, we explore
  and study various modal fusion schemes to create an effective multi-modal
  Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely
  competitive performance with current computationally efficient state-of-the-art
  methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster
  speed due to Cobra's linear sequential modeling. (2) Interestingly, the results
  of closed-set challenging prediction benchmarks show that Cobra performs well
  in overcoming visual illusions and spatial relationship judgments. (3) Notably,
  Cobra even achieves comparable performance to LLaVA with about 43% of the
  number of parameters. We will make all codes of Cobra open-source and hope that
  the proposed method can facilitate future research on complexity problems in
  MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.
paper_authors._id: 65fd0a404d36be78e694bd0d
paper_authors.name: Han Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T13:24:59.524Z
paper_authors._id: 646dbbc8075bbcc48ddcecbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646dbbc8075bbcc48ddcecbf/V52Em-78O5F3QxRbRwG5O.jpeg
paper_authors.fullname: Han Zhao
paper_authors.type: user
paper_authors.user: han1997
paper_authors._id: 65fd0a404d36be78e694bd0e
paper_authors.name: Min Zhang
paper_authors._id: 65fd0a404d36be78e694bd0f
paper_authors.name: Wei Zhao
paper_authors._id: 65fd0a404d36be78e694bd10
paper_authors.name: Pengxiang Ding
paper_authors._id: 65fd0a404d36be78e694bd11
paper_authors.name: Siteng Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T13:25:01.255Z
paper_authors._id: 65fd82762bf2cd20ddaa193f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
paper_authors.fullname: Siteng Huang
paper_authors.type: user
paper_authors.user: huangsiteng
paper_authors._id: 65fd0a404d36be78e694bd12
paper_authors.name: Donglin Wang
paper_title: Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient
    Inference
paper_summary: In recent years, the application of multimodal large language models (MLLM)
  in various fields has achieved remarkable success. However, as the foundation
  model for many downstream tasks, current MLLMs are composed of the well-known
  Transformer network, which has a less efficient quadratic computation
  complexity. To improve the efficiency of such basic models, we propose Cobra, a
  linear computational complexity MLLM. Specifically, Cobra integrates the
  efficient Mamba language model into the visual modality. Moreover, we explore
  and study various modal fusion schemes to create an effective multi-modal
  Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely
  competitive performance with current computationally efficient state-of-the-art
  methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster
  speed due to Cobra's linear sequential modeling. (2) Interestingly, the results
  of closed-set challenging prediction benchmarks show that Cobra performs well
  in overcoming visual illusions and spatial relationship judgments. (3) Notably,
  Cobra even achieves comparable performance to LLaVA with about 43% of the
  number of parameters. We will make all codes of Cobra open-source and hope that
  the proposed method can facilitate future research on complexity problems in
  MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.
paper_ai_keywords: multimodal large language models (MLLM) Transformer network Mamba language model modal fusion schemes linear computational complexity LLaVA-Phi TinyLLaVA MobileVLM v2 closed-set challenging prediction benchmarks visual illusions spatial relationship judgments linear sequential modeling hash tables