EVLM: An Efficient Vision-Language Model for Visual Understanding
In the field of multi-modal language models, the majority of methods are
built on an architecture similar to LLaVA. These models use a single-layer ViT
feature as a visual prompt, directly feeding it into the language models
alongside textual tokens. However, when dealing with long sequences of visual
signals or inputs such as videos, the self-attention mechanism of language
models can lead to significant computational overhead. Additionally, using
single-layer ViT features makes it challenging for large language models to
perceive visual signals fully. This paper proposes an efficient multi-modal
language model to minimize computational costs while enabling the model to
perceive visual signals as comprehensively as possible. Our method primarily
includes: (1) employing cross-attention to image-text interaction similar to
Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of
Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
competitive scores on public multi-modal benchmarks and performs well in tasks
such as image captioning and video captioning.
669dc8559a4bf63e08f55291
Kaibing Chen
claimed_verified
2024-08-15T10:12:34.593Z
66951c0ffdd2b31d42aa4053
/avatars/b293625bb703df53d0436a6a0ab6261a.svg
Kaibing
user
Kaibing1109
669dc8559a4bf63e08f55292
Dong Shen
claimed_verified
2024-08-13T07:39:48.081Z
64337e4382ca403c44df29bf
/avatars/9a0a2a9928079adfc192da24173b51eb.svg
shen
user
sean29
669dc8559a4bf63e08f55293
Hanwen Zhong
admin_assigned
2024-07-22T07:23:50.241Z
64cb1d70a81988d0733a9cef
/avatars/367db3734a9a0f68ae01ef620c32161d.svg
zhonghanwen
user
zhonghanwen
669dc8559a4bf63e08f55294
Huasong Zhong
admin_assigned
2024-07-22T07:24:04.188Z
650bf48dbffe400573631244
/avatars/635ee35127eba5fcdef8ad3091232e04.svg
huasong
user
zhonghuasong
669dc8559a4bf63e08f55295
Kui Xia
claimed_verified
2024-07-22T08:43:31.631Z
6464c4e134037a50516b4db2
/avatars/09c7b0a7294a27d927baa9633e44f60a.svg
xiakui
user
xiakui
669dc8559a4bf63e08f55296
Di Xu
admin_assigned
2024-07-22T07:24:25.767Z
638194c94761ddfa0091a981
/avatars/ca9974ac8f3e4bdd2806f65e64b383f8.svg
Di Xu
user
dixu
669dc8559a4bf63e08f55297
Wei Yuan
669dc8559a4bf63e08f55298
Yifei Hu
claimed_verified
2024-09-24T10:20:09.832Z
669e0f5d004e418e71a626c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-YWo6xdFV1uk0CN2AU13r.jpeg
Yifei Hu
user
yifei-adam
669dc8559a4bf63e08f55299
Bin Wen
669dc8559a4bf63e08f5529a
Tianke Zhang
claimed_verified
2024-07-23T07:48:01.244Z
64c7537571947b03ff2bd375
/avatars/f48d57e72d08caea42b03aa02f0c92ad.svg
ktz
user
Victoryztk23
669dc8559a4bf63e08f5529b
Changyi Liu
669dc8559a4bf63e08f5529c
Dewen Fan
669dc8559a4bf63e08f5529d
Huihui Xiao
admin_assigned
2024-07-22T07:26:24.599Z
64096da29e9f790c905b75f0
/avatars/a5e0bca66c5e698830ad640fe77e2242.svg
huxiaohui
user
xiaohuihui
669dc8559a4bf63e08f5529e
Jiahong Wu
admin_assigned
2024-07-22T07:26:33.007Z
64abac22ac49ad6717cc3866
/avatars/b075a3898862b52f03e513e49f6c40e7.svg
jiahong wu
user
dogewu
669dc8559a4bf63e08f5529f
Fan Yang
669dc8559a4bf63e08f552a0
Size Li
admin_assigned
2024-07-22T07:27:29.116Z
6218739e2036397c13e7c087
/avatars/e692a07353f604cd62715281a0e98ef5.svg
size.li
user
size
669dc8559a4bf63e08f552a1
Di Zhang
EVLM: An Efficient Vision-Language Model for Visual Understanding
In the field of multi-modal language models, the majority of methods are
built on an architecture similar to LLaVA. These models use a single-layer ViT
feature as a visual prompt, directly feeding it into the language models
alongside textual tokens. However, when dealing with long sequences of visual
signals or inputs such as videos, the self-attention mechanism of language
models can lead to significant computational overhead. Additionally, using
single-layer ViT features makes it challenging for large language models to
perceive visual signals fully. This paper proposes an efficient multi-modal
language model to minimize computational costs while enabling the model to
perceive visual signals as comprehensively as possible. Our method primarily
includes: (1) employing cross-attention to image-text interaction similar to
Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of
Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
competitive scores on public multi-modal benchmarks and performs well in tasks
such as image captioning and video captioning.
A multi-modal language model using cross-attention, hierarchical ViT features, and Mixture of Experts mechanism achieves competitive performance in image and video captioning tasks with reduced computational costs.
cross-attention hierarchical ViT features Mixture of Experts (MoE) image captioning video captioning