title: EVLM: An Efficient Vision-Language Model for Visual Understanding
summary: In the field of multi-modal language models, the majority of methods are
  built on an architecture similar to LLaVA. These models use a single-layer ViT
  feature as a visual prompt, directly feeding it into the language models
  alongside textual tokens. However, when dealing with long sequences of visual
  signals or inputs such as videos, the self-attention mechanism of language
  models can lead to significant computational overhead. Additionally, using
  single-layer ViT features makes it challenging for large language models to
  perceive visual signals fully. This paper proposes an efficient multi-modal
  language model to minimize computational costs while enabling the model to
  perceive visual signals as comprehensively as possible. Our method primarily
  includes: (1) employing cross-attention to image-text interaction similar to
  Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of
  Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
  competitive scores on public multi-modal benchmarks and performs well in tasks
  such as image captioning and video captioning.
paper_authors._id: 669dc8559a4bf63e08f55291
paper_authors.name: Kaibing Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:12:34.593Z
paper_authors._id: 66951c0ffdd2b31d42aa4053
paper_authors.avatarUrl: /avatars/b293625bb703df53d0436a6a0ab6261a.svg
paper_authors.fullname: Kaibing
paper_authors.type: user
paper_authors.user: Kaibing1109
paper_authors._id: 669dc8559a4bf63e08f55292
paper_authors.name: Dong Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:39:48.081Z
paper_authors._id: 64337e4382ca403c44df29bf
paper_authors.avatarUrl: /avatars/9a0a2a9928079adfc192da24173b51eb.svg
paper_authors.fullname: shen
paper_authors.type: user
paper_authors.user: sean29
paper_authors._id: 669dc8559a4bf63e08f55293
paper_authors.name: Hanwen Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:23:50.241Z
paper_authors._id: 64cb1d70a81988d0733a9cef
paper_authors.avatarUrl: /avatars/367db3734a9a0f68ae01ef620c32161d.svg
paper_authors.fullname: zhonghanwen
paper_authors.type: user
paper_authors.user: zhonghanwen
paper_authors._id: 669dc8559a4bf63e08f55294
paper_authors.name: Huasong Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:24:04.188Z
paper_authors._id: 650bf48dbffe400573631244
paper_authors.avatarUrl: /avatars/635ee35127eba5fcdef8ad3091232e04.svg
paper_authors.fullname: huasong
paper_authors.type: user
paper_authors.user: zhonghuasong
paper_authors._id: 669dc8559a4bf63e08f55295
paper_authors.name: Kui Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:31.631Z
paper_authors._id: 6464c4e134037a50516b4db2
paper_authors.avatarUrl: /avatars/09c7b0a7294a27d927baa9633e44f60a.svg
paper_authors.fullname: xiakui
paper_authors.type: user
paper_authors.user: xiakui
paper_authors._id: 669dc8559a4bf63e08f55296
paper_authors.name: Di Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:24:25.767Z
paper_authors._id: 638194c94761ddfa0091a981
paper_authors.avatarUrl: /avatars/ca9974ac8f3e4bdd2806f65e64b383f8.svg
paper_authors.fullname: Di Xu
paper_authors.type: user
paper_authors.user: dixu
paper_authors._id: 669dc8559a4bf63e08f55297
paper_authors.name: Wei Yuan
paper_authors._id: 669dc8559a4bf63e08f55298
paper_authors.name: Yifei Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T10:20:09.832Z
paper_authors._id: 669e0f5d004e418e71a626c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-YWo6xdFV1uk0CN2AU13r.jpeg
paper_authors.fullname: Yifei Hu
paper_authors.type: user
paper_authors.user: yifei-adam
paper_authors._id: 669dc8559a4bf63e08f55299
paper_authors.name: Bin Wen
paper_authors._id: 669dc8559a4bf63e08f5529a
paper_authors.name: Tianke Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:48:01.244Z
paper_authors._id: 64c7537571947b03ff2bd375
paper_authors.avatarUrl: /avatars/f48d57e72d08caea42b03aa02f0c92ad.svg
paper_authors.fullname: ktz
paper_authors.type: user
paper_authors.user: Victoryztk23
paper_authors._id: 669dc8559a4bf63e08f5529b
paper_authors.name: Changyi Liu
paper_authors._id: 669dc8559a4bf63e08f5529c
paper_authors.name: Dewen Fan
paper_authors._id: 669dc8559a4bf63e08f5529d
paper_authors.name: Huihui Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:26:24.599Z
paper_authors._id: 64096da29e9f790c905b75f0
paper_authors.avatarUrl: /avatars/a5e0bca66c5e698830ad640fe77e2242.svg
paper_authors.fullname: huxiaohui
paper_authors.type: user
paper_authors.user: xiaohuihui
paper_authors._id: 669dc8559a4bf63e08f5529e
paper_authors.name: Jiahong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:26:33.007Z
paper_authors._id: 64abac22ac49ad6717cc3866
paper_authors.avatarUrl: /avatars/b075a3898862b52f03e513e49f6c40e7.svg
paper_authors.fullname: jiahong wu
paper_authors.type: user
paper_authors.user: dogewu
paper_authors._id: 669dc8559a4bf63e08f5529f
paper_authors.name: Fan Yang
paper_authors._id: 669dc8559a4bf63e08f552a0
paper_authors.name: Size Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:27:29.116Z
paper_authors._id: 6218739e2036397c13e7c087
paper_authors.avatarUrl: /avatars/e692a07353f604cd62715281a0e98ef5.svg
paper_authors.fullname: size.li
paper_authors.type: user
paper_authors.user: size
paper_authors._id: 669dc8559a4bf63e08f552a1
paper_authors.name: Di Zhang
paper_title: EVLM: An Efficient Vision-Language Model for Visual Understanding
paper_summary: In the field of multi-modal language models, the majority of methods are
  built on an architecture similar to LLaVA. These models use a single-layer ViT
  feature as a visual prompt, directly feeding it into the language models
  alongside textual tokens. However, when dealing with long sequences of visual
  signals or inputs such as videos, the self-attention mechanism of language
  models can lead to significant computational overhead. Additionally, using
  single-layer ViT features makes it challenging for large language models to
  perceive visual signals fully. This paper proposes an efficient multi-modal
  language model to minimize computational costs while enabling the model to
  perceive visual signals as comprehensively as possible. Our method primarily
  includes: (1) employing cross-attention to image-text interaction similar to
  Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of
  Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
  competitive scores on public multi-modal benchmarks and performs well in tasks
  such as image captioning and video captioning.
paper_ai_keywords: multi-modal language models LLaVA ViT feature self-attention mechanism visual signals videos computational overhead cross-attention image-text interaction Flamingo hierarchical ViT features Mixture of Experts (MoE) multi-modal benchmarks image captioning video captioning