title: Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of
    Encoders
summary: The ability to accurately interpret complex visual information is a crucial
  topic of multimodal large language models (MLLMs). Recent work indicates that
  enhanced visual perception significantly reduces hallucinations and improves
  performance on resolution-sensitive tasks, such as optical character
  recognition and document analysis. A number of recent MLLMs achieve this goal
  using a mixture of vision encoders. Despite their success, there is a lack of
  systematic comparisons and detailed ablation studies addressing critical
  aspects, such as expert selection and the integration of multiple vision
  experts. This study provides an extensive exploration of the design space for
  MLLMs using a mixture of vision encoders and resolutions. Our findings reveal
  several underlying principles common to various existing strategies, leading to
  a streamlined yet effective design approach. We discover that simply
  concatenating visual tokens from a set of complementary vision encoders is as
  effective as more complex mixing architectures or strategies. We additionally
  introduce Pre-Alignment to bridge the gap between vision-focused encoders and
  language tokens, enhancing model coherence. The resulting family of MLLMs,
  Eagle, surpasses other leading open-source models on major MLLM benchmarks.
  Models and code: https://github.com/NVlabs/Eagle
paper_authors._id: 66cfdffc469a44ba82b1827a
paper_authors.name: Min Shi
paper_authors._id: 66cfdffc469a44ba82b1827b
paper_authors.name: Fuxiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:12.985Z
paper_authors._id: 64618b9496259bec21d44704
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64618b9496259bec21d44704/FCzN0p7Elh8MeUTuM-hya.jpeg
paper_authors.fullname: Fuxiao Liu
paper_authors.type: user
paper_authors.user: Fuxiao
paper_authors._id: 66cfdffc469a44ba82b1827c
paper_authors.name: Shihao Wang
paper_authors._id: 66cfdffc469a44ba82b1827d
paper_authors.name: Shijia Liao
paper_authors._id: 66cfdffc469a44ba82b1827e
paper_authors.name: Subhashree Radhakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:35:35.932Z
paper_authors._id: 63b609f50d5913eee489be7a
paper_authors.avatarUrl: /avatars/2d1a6cdd11a76d5c03a97bbf8c52b7e5.svg
paper_authors.fullname: Subhashree Radhakrishnan
paper_authors.type: user
paper_authors.user: subhashreer
paper_authors._id: 66cfdffc469a44ba82b1827f
paper_authors.name: De-An Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:35:42.520Z
paper_authors._id: 641d1c5ec3983aa94915c162
paper_authors.avatarUrl: /avatars/127985b837ecf61e43c835deee578b5e.svg
paper_authors.fullname: De-An Huang
paper_authors.type: user
paper_authors.user: deahuang
paper_authors._id: 66cfdffc469a44ba82b18280
paper_authors.name: Hongxu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:35:49.314Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 66cfdffc469a44ba82b18281
paper_authors.name: Karan Sapra
paper_authors._id: 66cfdffc469a44ba82b18282
paper_authors.name: Yaser Yacoob
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:36:09.204Z
paper_authors._id: 65ca73ea76ba4c0ddc519286
paper_authors.avatarUrl: /avatars/04d8191b850c6e4c81a8dd9d31d7227e.svg
paper_authors.fullname: Yaser Yacoob
paper_authors.type: user
paper_authors.user: yaseryacoob
paper_authors._id: 66cfdffc469a44ba82b18283
paper_authors.name: Humphrey Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:36:15.370Z
paper_authors._id: 61e1188afc27c0f5e3641eb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
paper_authors.fullname: Humphrey Shi
paper_authors.type: user
paper_authors.user: Humphrey
paper_authors._id: 66cfdffc469a44ba82b18284
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:36:23.906Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66cfdffc469a44ba82b18285
paper_authors.name: Andrew Tao
paper_authors._id: 66cfdffc469a44ba82b18286
paper_authors.name: Jan Kautz
paper_authors._id: 66cfdffc469a44ba82b18287
paper_authors.name: Zhiding Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:05.194Z
paper_authors._id: 66c8037c737ba92ae3fe0322
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
paper_authors.fullname: Zhiding Yu
paper_authors.type: user
paper_authors.user: Zhiding
paper_authors._id: 66cfdffc469a44ba82b18288
paper_authors.name: Guilin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:36:58.396Z
paper_authors._id: 6656eb16e50d7c40881a14f0
paper_authors.avatarUrl: /avatars/c6822a51c8d5918debf6ee1d25fe1825.svg
paper_authors.fullname: GuilinLiu
paper_authors.type: user
paper_authors.user: GuilinLiu
paper_title: Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of
    Encoders
paper_summary: The ability to accurately interpret complex visual information is a crucial
  topic of multimodal large language models (MLLMs). Recent work indicates that
  enhanced visual perception significantly reduces hallucinations and improves
  performance on resolution-sensitive tasks, such as optical character
  recognition and document analysis. A number of recent MLLMs achieve this goal
  using a mixture of vision encoders. Despite their success, there is a lack of
  systematic comparisons and detailed ablation studies addressing critical
  aspects, such as expert selection and the integration of multiple vision
  experts. This study provides an extensive exploration of the design space for
  MLLMs using a mixture of vision encoders and resolutions. Our findings reveal
  several underlying principles common to various existing strategies, leading to
  a streamlined yet effective design approach. We discover that simply
  concatenating visual tokens from a set of complementary vision encoders is as
  effective as more complex mixing architectures or strategies. We additionally
  introduce Pre-Alignment to bridge the gap between vision-focused encoders and
  language tokens, enhancing model coherence. The resulting family of MLLMs,
  Eagle, surpasses other leading open-source models on major MLLM benchmarks.
  Models and code: https://github.com/NVlabs/Eagle
paper_ai_summary: Mixture of vision encoders and resolutions in multimodal large language models improves performance through concatenation of visual tokens and a Pre-Alignment mechanism, leading to superior results on benchmarks.
paper_ai_keywords: multimodal large language models MLLMs vision encoders optical character recognition document analysis visual perception hallucinations resolution-sensitive tasks vision tokens mixing architectures Pre-Alignment