title: MouSi: Poly-Visual-Expert Vision-Language Models
summary: Current large vision-language models (VLMs) often encounter challenges such
  as insufficient capabilities of a single visual component and excessively long
  visual tokens. These issues can limit the model's effectiveness in accurately
  interpreting complex visual information and over-lengthy contextual
  information. Addressing these challenges is crucial for enhancing the
  performance and applicability of VLMs. This paper proposes the use of ensemble
  experts technique to synergizes the capabilities of individual visual encoders,
  including those skilled in image-text matching, OCR, image segmentation, etc.
  This technique introduces a fusion network to unify the processing of outputs
  from different visual experts, while bridging the gap between image encoders
  and pre-trained LLMs. In addition, we explore different positional encoding
  schemes to alleviate the waste of positional encoding caused by lengthy image
  feature sequences, effectively addressing the issue of position overflow and
  length limitations. For instance, in our implementation, this technique
  significantly reduces the positional occupancy in models like SAM, from a
  substantial 4096 to a more efficient and manageable 64 or even down to 1.
  Experimental results demonstrate that VLMs with multiple experts exhibit
  consistently superior performance over isolated visual encoders and mark a
  significant performance boost as more experts are integrated. We have
  open-sourced the training code used in this report. All of these resources can
  be found on our project website.
paper_authors._id: 65b9bb0ba1880eed44a350e1
paper_authors.name: Xiaoran Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:20:52.671Z
paper_authors._id: 6424fa5f988b8b19edba8ed4
paper_authors.avatarUrl: /avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
paper_authors.fullname: Xiaoran Fan
paper_authors.type: user
paper_authors.user: cnxup
paper_authors._id: 65b9bb0ba1880eed44a350e2
paper_authors.name: Tao Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:20:17.759Z
paper_authors._id: 647f5f845e1bc4753745ffaa
paper_authors.avatarUrl: /avatars/86a09927413702103e04f4f4a209efe9.svg
paper_authors.fullname: Tao Ji
paper_authors.type: user
paper_authors.user: TaoJi
paper_authors._id: 65b9bb0ba1880eed44a350e3
paper_authors.name: Changhao Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:21:12.863Z
paper_authors._id: 646875cd2a180a1378356d2f
paper_authors.avatarUrl: /avatars/6eb7024030e184de1257cd66e5fd7ed4.svg
paper_authors.fullname: Changhao Jiang
paper_authors.type: user
paper_authors.user: jiangchanghao
paper_authors._id: 65b9bb0ba1880eed44a350e4
paper_authors.name: Shuo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-31T17:47:42.178Z
paper_authors._id: 64c2a8ee5d95081bd2c1c2ba
paper_authors.avatarUrl: /avatars/547b7d0b12b7926a11f3d28a3d2c6010.svg
paper_authors.fullname: Shuo Li
paper_authors.type: user
paper_authors.user: lishuo123
paper_authors._id: 65b9bb0ba1880eed44a350e5
paper_authors.name: Senjie Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:22:10.361Z
paper_authors._id: 63bbbe1fb8c61b8aa4922213
paper_authors.avatarUrl: /avatars/8199d6181794b8feee8fe9df73267af2.svg
paper_authors.fullname: jinsenjie
paper_authors.type: user
paper_authors.user: yizhidalaohu
paper_authors._id: 65b9bb0ba1880eed44a350e6
paper_authors.name: Sirui Song
paper_authors._id: 65b9bb0ba1880eed44a350e7
paper_authors.name: Junke Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:22:27.019Z
paper_authors._id: 642e1e7147833318f0eb3755
paper_authors.avatarUrl: /avatars/7ef4f5a099d3eb0fab99b589f33903fa.svg
paper_authors.fullname: Junke Wang
paper_authors.type: user
paper_authors.user: Daniel0724
paper_authors._id: 65b9bb0ba1880eed44a350e8
paper_authors.name: Boyang Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:22:41.728Z
paper_authors._id: 6407d786a7bc7c3865ae01c5
paper_authors.avatarUrl: /avatars/96d8c2591b8aa8f6fa03e7e091500cc4.svg
paper_authors.fullname: hby
paper_authors.type: user
paper_authors.user: hongboyang
paper_authors._id: 65b9bb0ba1880eed44a350e9
paper_authors.name: Lu Chen
paper_authors._id: 65b9bb0ba1880eed44a350ea
paper_authors.name: Guodong Zheng
paper_authors._id: 65b9bb0ba1880eed44a350eb
paper_authors.name: Ming Zhang
paper_authors._id: 65b9bb0ba1880eed44a350ec
paper_authors.name: Caishuang Huang
paper_authors._id: 65b9bb0ba1880eed44a350ed
paper_authors.name: Rui Zheng
paper_authors._id: 65b9bb0ba1880eed44a350ee
paper_authors.name: Zhiheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:23:32.730Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 65b9bb0ba1880eed44a350ef
paper_authors.name: Yuhao Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:23:40.294Z
paper_authors._id: 646dbfcaf813cfe153ef2964
paper_authors.avatarUrl: /avatars/8e73f91b35ac0d11671fd7399a43e723.svg
paper_authors.fullname: Yuhao Zhou
paper_authors.type: user
paper_authors.user: YuhaoZhou
paper_authors._id: 65b9bb0ba1880eed44a350f0
paper_authors.name: Shihan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:23:48.647Z
paper_authors._id: 64ae0b6ccf90fe27556cf56e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
paper_authors.fullname: Shihan Dou
paper_authors.type: user
paper_authors.user: Ablustrund
paper_authors._id: 65b9bb0ba1880eed44a350f1
paper_authors.name: Junjie Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:42:05.160Z
paper_authors._id: 66384be673c2c55f2ded89fa
paper_authors.avatarUrl: /avatars/1d8721074f0f51fab405f81474f2035f.svg
paper_authors.fullname: Junjie Ye
paper_authors.type: user
paper_authors.user: Junjie-Ye
paper_authors._id: 65b9bb0ba1880eed44a350f2
paper_authors.name: Hang Yan
paper_authors._id: 65b9bb0ba1880eed44a350f3
paper_authors.name: Tao Gui
paper_authors._id: 65b9bb0ba1880eed44a350f4
paper_authors.name: Qi Zhang
paper_authors._id: 65b9bb0ba1880eed44a350f5
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:25:03.795Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_authors._id: 65b9bb0ba1880eed44a350f6
paper_authors.name: Xuanjing Huang
paper_authors._id: 65b9bb0ba1880eed44a350f7
paper_authors.name: Zuxuan Wu
paper_authors._id: 65b9bb0ba1880eed44a350f8
paper_authors.name: Yu-Gang Jiang
paper_title: MouSi: Poly-Visual-Expert Vision-Language Models
paper_summary: Current large vision-language models (VLMs) often encounter challenges such
  as insufficient capabilities of a single visual component and excessively long
  visual tokens. These issues can limit the model's effectiveness in accurately
  interpreting complex visual information and over-lengthy contextual
  information. Addressing these challenges is crucial for enhancing the
  performance and applicability of VLMs. This paper proposes the use of ensemble
  experts technique to synergizes the capabilities of individual visual encoders,
  including those skilled in image-text matching, OCR, image segmentation, etc.
  This technique introduces a fusion network to unify the processing of outputs
  from different visual experts, while bridging the gap between image encoders
  and pre-trained LLMs. In addition, we explore different positional encoding
  schemes to alleviate the waste of positional encoding caused by lengthy image
  feature sequences, effectively addressing the issue of position overflow and
  length limitations. For instance, in our implementation, this technique
  significantly reduces the positional occupancy in models like SAM, from a
  substantial 4096 to a more efficient and manageable 64 or even down to 1.
  Experimental results demonstrate that VLMs with multiple experts exhibit
  consistently superior performance over isolated visual encoders and mark a
  significant performance boost as more experts are integrated. We have
  open-sourced the training code used in this report. All of these resources can
  be found on our project website.
paper_ai_summary: The ensemble experts technique with a fusion network improves VLM performance by combining visual encoders and optimizing positional encoding for lengthy image feature sequences.
paper_ai_keywords: ensemble experts visual encoders image-text matching OCR image segmentation fusion network positional encoding SAM positional occupancy