Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of
  Encoders
The ability to accurately interpret complex visual information is a crucial
topic of multimodal large language models (MLLMs). Recent work indicates that
enhanced visual perception significantly reduces hallucinations and improves
performance on resolution-sensitive tasks, such as optical character
recognition and document analysis. A number of recent MLLMs achieve this goal
using a mixture of vision encoders. Despite their success, there is a lack of
systematic comparisons and detailed ablation studies addressing critical
aspects, such as expert selection and the integration of multiple vision
experts. This study provides an extensive exploration of the design space for
MLLMs using a mixture of vision encoders and resolutions. Our findings reveal
several underlying principles common to various existing strategies, leading to
a streamlined yet effective design approach. We discover that simply
concatenating visual tokens from a set of complementary vision encoders is as
effective as more complex mixing architectures or strategies. We additionally
introduce Pre-Alignment to bridge the gap between vision-focused encoders and
language tokens, enhancing model coherence. The resulting family of MLLMs,
Eagle, surpasses other leading open-source models on major MLLM benchmarks.
Models and code: https://github.com/NVlabs/Eagle
66cfdffc469a44ba82b1827a
Min Shi
66cfdffc469a44ba82b1827b
Fuxiao Liu
claimed_verified
2024-08-29T07:23:12.985Z
64618b9496259bec21d44704
https://cdn-avatars.huggingface.co/v1/production/uploads/64618b9496259bec21d44704/FCzN0p7Elh8MeUTuM-hya.jpeg
Fuxiao Liu
user
Fuxiao
66cfdffc469a44ba82b1827c
Shihao Wang
66cfdffc469a44ba82b1827d
Shijia Liao
66cfdffc469a44ba82b1827e
Subhashree Radhakrishnan
admin_assigned
2024-08-29T07:35:35.932Z
63b609f50d5913eee489be7a
/avatars/2d1a6cdd11a76d5c03a97bbf8c52b7e5.svg
Subhashree Radhakrishnan
user
subhashreer
66cfdffc469a44ba82b1827f
De-An Huang
admin_assigned
2024-08-29T07:35:42.520Z
641d1c5ec3983aa94915c162
/avatars/127985b837ecf61e43c835deee578b5e.svg
De-An Huang
user
deahuang
66cfdffc469a44ba82b18280
Hongxu Yin
admin_assigned
2024-08-29T07:35:49.314Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
66cfdffc469a44ba82b18281
Karan Sapra
66cfdffc469a44ba82b18282
Yaser Yacoob
admin_assigned
2024-08-29T07:36:09.204Z
65ca73ea76ba4c0ddc519286
/avatars/04d8191b850c6e4c81a8dd9d31d7227e.svg
Yaser Yacoob
user
yaseryacoob
66cfdffc469a44ba82b18283
Humphrey Shi
admin_assigned
2024-08-29T07:36:15.370Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
66cfdffc469a44ba82b18284
Bryan Catanzaro
admin_assigned
2024-08-29T07:36:23.906Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66cfdffc469a44ba82b18285
Andrew Tao
66cfdffc469a44ba82b18286
Jan Kautz
66cfdffc469a44ba82b18287
Zhiding Yu
claimed_verified
2024-08-29T07:23:05.194Z
66c8037c737ba92ae3fe0322
https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
Zhiding Yu
user
Zhiding
66cfdffc469a44ba82b18288
Guilin Liu
admin_assigned
2024-08-29T07:36:58.396Z
6656eb16e50d7c40881a14f0
/avatars/c6822a51c8d5918debf6ee1d25fe1825.svg
GuilinLiu
user
GuilinLiu
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of
  Encoders
The ability to accurately interpret complex visual information is a crucial
topic of multimodal large language models (MLLMs). Recent work indicates that
enhanced visual perception significantly reduces hallucinations and improves
performance on resolution-sensitive tasks, such as optical character
recognition and document analysis. A number of recent MLLMs achieve this goal
using a mixture of vision encoders. Despite their success, there is a lack of
systematic comparisons and detailed ablation studies addressing critical
aspects, such as expert selection and the integration of multiple vision
experts. This study provides an extensive exploration of the design space for
MLLMs using a mixture of vision encoders and resolutions. Our findings reveal
several underlying principles common to various existing strategies, leading to
a streamlined yet effective design approach. We discover that simply
concatenating visual tokens from a set of complementary vision encoders is as
effective as more complex mixing architectures or strategies. We additionally
introduce Pre-Alignment to bridge the gap between vision-focused encoders and
language tokens, enhancing model coherence. The resulting family of MLLMs,
Eagle, surpasses other leading open-source models on major MLLM benchmarks.
Models and code: https://github.com/NVlabs/Eagle
Mixture of vision encoders and resolutions in multimodal large language models improves performance through concatenation of visual tokens and a Pre-Alignment mechanism, leading to superior results on benchmarks.
multimodal large language models MLLMs vision encoders optical character recognition document analysis visual perception hallucinations resolution-sensitive tasks vision tokens mixing architectures Pre-Alignment