Chimera: Improving Generalist Model with Domain-Specific Experts
Recent advancements in Large Multi-modal Models (LMMs) underscore the
importance of scaling by increasing image-text paired data, achieving
impressive performance on general tasks. Despite their effectiveness in broad
applications, generalist models are primarily trained on web-scale datasets
dominated by natural images, resulting in the sacrifice of specialized
capabilities for domain-specific tasks that require extensive domain prior
knowledge. Moreover, directly integrating expert models tailored for specific
domains is challenging due to the representational gap and imbalanced
optimization between the generalist model and experts. To address these
challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline
designed to boost the ability of existing LMMs with domain-specific experts.
Specifically, we design a progressive training strategy to integrate features
from expert models into the input of a generalist LMM. To address the
imbalanced optimization caused by the well-aligned general visual encoder, we
introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism.
This results in a versatile model that excels across the chart, table, math,
and document domains, achieving state-of-the-art performance on multi-modal
reasoning and visual content extraction tasks, both of which are challenging
tasks for assessing existing LMMs.
675958c2528f1296b6789dc8
Tianshuo Peng
admin_assigned
2024-12-11T13:51:28.371Z
6538dd471ad9b3ba7c2df861
https://cdn-avatars.huggingface.co/v1/production/uploads/6538dd471ad9b3ba7c2df861/MbEa7KHAK6u7PRb7WiPUC.jpeg
Tianshuo Peng
user
Potentialts
675958c2528f1296b6789dc9
Mingsheng Li
admin_assigned
2024-12-11T13:51:34.844Z
674d8c97d086436faf52c890
/avatars/c4bde24b0796f725a9f4e68f72fe6a44.svg
mingsheng li
user
mingsheng1
675958c2528f1296b6789dca
Hongbin Zhou
675958c2528f1296b6789dcb
Renqiu Xia
admin_assigned
2024-12-11T13:51:47.602Z
65b7ae76768464877cdb2e39
https://cdn-avatars.huggingface.co/v1/production/uploads/65b7ae76768464877cdb2e39/uAWPo4tpkqbZoDeEkc7y0.jpeg
Renqiu Xia
user
renqiux0302
675958c2528f1296b6789dcc
Renrui Zhang
675958c2528f1296b6789dcd
Lei Bai
675958c2528f1296b6789dce
Song Mao
675958c2528f1296b6789dcf
Bin Wang
admin_assigned
2024-12-11T13:52:09.628Z
5e49e8cf37cb5b49818287ae
https://cdn-avatars.huggingface.co/v1/production/uploads/5e49e8cf37cb5b49818287ae/IV9b5Z70NhgmBNfAlc_co.jpeg
Bin Wang
user
binwang
675958c2528f1296b6789dd0
Conghui He
admin_assigned
2024-12-11T13:52:15.773Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
675958c2528f1296b6789dd1
Aojun Zhou
claimed_verified
2024-12-11T13:39:14.372Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
675958c2528f1296b6789dd2
Botian Shi
admin_assigned
2024-12-11T13:52:21.114Z
643df87f7cd64d872cb9fabd
/avatars/c53bfabcee08de448dde973915e8b31d.svg
Botian Shi
user
friskit
675958c2528f1296b6789dd3
Tao Chen
admin_assigned
2024-12-11T13:52:28.097Z
64fc225152992431a6013ebd
/avatars/c3320cfb88b4fd21884bc401f71caa6f.svg
Chen Tao
user
taochen
675958c2528f1296b6789dd4
Bo Zhang
claimed_verified
2024-12-18T09:53:45.594Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
675958c2528f1296b6789dd5
Xiangyu Yue
admin_assigned
2024-12-11T13:51:21.617Z
666a8f24e2990b0cb16b7bf9
/avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
Xiangyu Yue
user
xyyue
Chimera: Improving Generalist Model with Domain-Specific Experts
Recent advancements in Large Multi-modal Models (LMMs) underscore the
importance of scaling by increasing image-text paired data, achieving
impressive performance on general tasks. Despite their effectiveness in broad
applications, generalist models are primarily trained on web-scale datasets
dominated by natural images, resulting in the sacrifice of specialized
capabilities for domain-specific tasks that require extensive domain prior
knowledge. Moreover, directly integrating expert models tailored for specific
domains is challenging due to the representational gap and imbalanced
optimization between the generalist model and experts. To address these
challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline
designed to boost the ability of existing LMMs with domain-specific experts.
Specifically, we design a progressive training strategy to integrate features
from expert models into the input of a generalist LMM. To address the
imbalanced optimization caused by the well-aligned general visual encoder, we
introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism.
This results in a versatile model that excels across the chart, table, math,
and document domains, achieving state-of-the-art performance on multi-modal
reasoning and visual content extraction tasks, both of which are challenging
tasks for assessing existing LMMs.
Chimera, a scalable multi-modal pipeline, integrates domain-specific experts into LMMs using progressive training and GSCM to achieve state-of-the-art performance in multi-modal reasoning and visual content extraction.
Large Multi-modal Models (LMMs) domain-specific experts progressive training strategy Generalist-Specialist Collaboration Masking (GSCM) multi-modal reasoning visual content extraction