MouSi: Poly-Visual-Expert Vision-Language Models
Current large vision-language models (VLMs) often encounter challenges such
as insufficient capabilities of a single visual component and excessively long
visual tokens. These issues can limit the model's effectiveness in accurately
interpreting complex visual information and over-lengthy contextual
information. Addressing these challenges is crucial for enhancing the
performance and applicability of VLMs. This paper proposes the use of ensemble
experts technique to synergizes the capabilities of individual visual encoders,
including those skilled in image-text matching, OCR, image segmentation, etc.
This technique introduces a fusion network to unify the processing of outputs
from different visual experts, while bridging the gap between image encoders
and pre-trained LLMs. In addition, we explore different positional encoding
schemes to alleviate the waste of positional encoding caused by lengthy image
feature sequences, effectively addressing the issue of position overflow and
length limitations. For instance, in our implementation, this technique
significantly reduces the positional occupancy in models like SAM, from a
substantial 4096 to a more efficient and manageable 64 or even down to 1.
Experimental results demonstrate that VLMs with multiple experts exhibit
consistently superior performance over isolated visual encoders and mark a
significant performance boost as more experts are integrated. We have
open-sourced the training code used in this report. All of these resources can
be found on our project website.
65b9bb0ba1880eed44a350e1
Xiaoran Fan
admin_assigned
2024-01-31T10:20:52.671Z
6424fa5f988b8b19edba8ed4
/avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
Xiaoran Fan
user
cnxup
65b9bb0ba1880eed44a350e2
Tao Ji
claimed_verified
2025-03-02T20:20:17.759Z
647f5f845e1bc4753745ffaa
/avatars/86a09927413702103e04f4f4a209efe9.svg
Tao Ji
user
TaoJi
65b9bb0ba1880eed44a350e3
Changhao Jiang
admin_assigned
2024-01-31T10:21:12.863Z
646875cd2a180a1378356d2f
/avatars/6eb7024030e184de1257cd66e5fd7ed4.svg
Changhao Jiang
user
jiangchanghao
65b9bb0ba1880eed44a350e4
Shuo Li
claimed_verified
2024-01-31T17:47:42.178Z
64c2a8ee5d95081bd2c1c2ba
/avatars/547b7d0b12b7926a11f3d28a3d2c6010.svg
Shuo Li
user
lishuo123
65b9bb0ba1880eed44a350e5
Senjie Jin
admin_assigned
2024-01-31T10:22:10.361Z
63bbbe1fb8c61b8aa4922213
/avatars/8199d6181794b8feee8fe9df73267af2.svg
jinsenjie
user
yizhidalaohu
65b9bb0ba1880eed44a350e6
Sirui Song
65b9bb0ba1880eed44a350e7
Junke Wang
admin_assigned
2024-01-31T10:22:27.019Z
642e1e7147833318f0eb3755
/avatars/7ef4f5a099d3eb0fab99b589f33903fa.svg
Junke Wang
user
Daniel0724
65b9bb0ba1880eed44a350e8
Boyang Hong
admin_assigned
2024-01-31T10:22:41.728Z
6407d786a7bc7c3865ae01c5
/avatars/96d8c2591b8aa8f6fa03e7e091500cc4.svg
hby
user
hongboyang
65b9bb0ba1880eed44a350e9
Lu Chen
65b9bb0ba1880eed44a350ea
Guodong Zheng
65b9bb0ba1880eed44a350eb
Ming Zhang
65b9bb0ba1880eed44a350ec
Caishuang Huang
65b9bb0ba1880eed44a350ed
Rui Zheng
65b9bb0ba1880eed44a350ee
Zhiheng Xi
admin_assigned
2024-01-31T10:23:32.730Z
653a6e5cae155b92bae77b74
https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
Zhiheng Xi
user
WooooDyy
65b9bb0ba1880eed44a350ef
Yuhao Zhou
admin_assigned
2024-01-31T10:23:40.294Z
646dbfcaf813cfe153ef2964
/avatars/8e73f91b35ac0d11671fd7399a43e723.svg
Yuhao Zhou
user
YuhaoZhou
65b9bb0ba1880eed44a350f0
Shihan Dou
admin_assigned
2024-01-31T10:23:48.647Z
64ae0b6ccf90fe27556cf56e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
Shihan Dou
user
Ablustrund
65b9bb0ba1880eed44a350f1
Junjie Ye
claimed_verified
2024-12-30T19:42:05.160Z
66384be673c2c55f2ded89fa
/avatars/1d8721074f0f51fab405f81474f2035f.svg
Junjie Ye
user
Junjie-Ye
65b9bb0ba1880eed44a350f2
Hang Yan
65b9bb0ba1880eed44a350f3
Tao Gui
65b9bb0ba1880eed44a350f4
Qi Zhang
65b9bb0ba1880eed44a350f5
Xipeng Qiu
admin_assigned
2024-01-31T10:25:03.795Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
65b9bb0ba1880eed44a350f6
Xuanjing Huang
65b9bb0ba1880eed44a350f7
Zuxuan Wu
65b9bb0ba1880eed44a350f8
Yu-Gang Jiang
MouSi: Poly-Visual-Expert Vision-Language Models
Current large vision-language models (VLMs) often encounter challenges such
as insufficient capabilities of a single visual component and excessively long
visual tokens. These issues can limit the model's effectiveness in accurately
interpreting complex visual information and over-lengthy contextual
information. Addressing these challenges is crucial for enhancing the
performance and applicability of VLMs. This paper proposes the use of ensemble
experts technique to synergizes the capabilities of individual visual encoders,
including those skilled in image-text matching, OCR, image segmentation, etc.
This technique introduces a fusion network to unify the processing of outputs
from different visual experts, while bridging the gap between image encoders
and pre-trained LLMs. In addition, we explore different positional encoding
schemes to alleviate the waste of positional encoding caused by lengthy image
feature sequences, effectively addressing the issue of position overflow and
length limitations. For instance, in our implementation, this technique
significantly reduces the positional occupancy in models like SAM, from a
substantial 4096 to a more efficient and manageable 64 or even down to 1.
Experimental results demonstrate that VLMs with multiple experts exhibit
consistently superior performance over isolated visual encoders and mark a
significant performance boost as more experts are integrated. We have
open-sourced the training code used in this report. All of these resources can
be found on our project website.
The ensemble experts technique with a fusion network improves VLM performance by combining visual encoders and optimizing positional encoding for lengthy image feature sequences.
ensemble experts visual encoders image-text matching OCR image segmentation fusion network positional encoding SAM positional occupancy