title: Chimera: Improving Generalist Model with Domain-Specific Experts
summary: Recent advancements in Large Multi-modal Models (LMMs) underscore the
  importance of scaling by increasing image-text paired data, achieving
  impressive performance on general tasks. Despite their effectiveness in broad
  applications, generalist models are primarily trained on web-scale datasets
  dominated by natural images, resulting in the sacrifice of specialized
  capabilities for domain-specific tasks that require extensive domain prior
  knowledge. Moreover, directly integrating expert models tailored for specific
  domains is challenging due to the representational gap and imbalanced
  optimization between the generalist model and experts. To address these
  challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline
  designed to boost the ability of existing LMMs with domain-specific experts.
  Specifically, we design a progressive training strategy to integrate features
  from expert models into the input of a generalist LMM. To address the
  imbalanced optimization caused by the well-aligned general visual encoder, we
  introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism.
  This results in a versatile model that excels across the chart, table, math,
  and document domains, achieving state-of-the-art performance on multi-modal
  reasoning and visual content extraction tasks, both of which are challenging
  tasks for assessing existing LMMs.
paper_authors._id: 675958c2528f1296b6789dc8
paper_authors.name: Tianshuo Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:51:28.371Z
paper_authors._id: 6538dd471ad9b3ba7c2df861
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6538dd471ad9b3ba7c2df861/MbEa7KHAK6u7PRb7WiPUC.jpeg
paper_authors.fullname: Tianshuo Peng
paper_authors.type: user
paper_authors.user: Potentialts
paper_authors._id: 675958c2528f1296b6789dc9
paper_authors.name: Mingsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:51:34.844Z
paper_authors._id: 674d8c97d086436faf52c890
paper_authors.avatarUrl: /avatars/c4bde24b0796f725a9f4e68f72fe6a44.svg
paper_authors.fullname: mingsheng li
paper_authors.type: user
paper_authors.user: mingsheng1
paper_authors._id: 675958c2528f1296b6789dca
paper_authors.name: Hongbin Zhou
paper_authors._id: 675958c2528f1296b6789dcb
paper_authors.name: Renqiu Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:51:47.602Z
paper_authors._id: 65b7ae76768464877cdb2e39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b7ae76768464877cdb2e39/uAWPo4tpkqbZoDeEkc7y0.jpeg
paper_authors.fullname: Renqiu Xia
paper_authors.type: user
paper_authors.user: renqiux0302
paper_authors._id: 675958c2528f1296b6789dcc
paper_authors.name: Renrui Zhang
paper_authors._id: 675958c2528f1296b6789dcd
paper_authors.name: Lei Bai
paper_authors._id: 675958c2528f1296b6789dce
paper_authors.name: Song Mao
paper_authors._id: 675958c2528f1296b6789dcf
paper_authors.name: Bin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:52:09.628Z
paper_authors._id: 5e49e8cf37cb5b49818287ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e49e8cf37cb5b49818287ae/IV9b5Z70NhgmBNfAlc_co.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: binwang
paper_authors._id: 675958c2528f1296b6789dd0
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:52:15.773Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 675958c2528f1296b6789dd1
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T13:39:14.372Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 675958c2528f1296b6789dd2
paper_authors.name: Botian Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:52:21.114Z
paper_authors._id: 643df87f7cd64d872cb9fabd
paper_authors.avatarUrl: /avatars/c53bfabcee08de448dde973915e8b31d.svg
paper_authors.fullname: Botian Shi
paper_authors.type: user
paper_authors.user: friskit
paper_authors._id: 675958c2528f1296b6789dd3
paper_authors.name: Tao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:52:28.097Z
paper_authors._id: 64fc225152992431a6013ebd
paper_authors.avatarUrl: /avatars/c3320cfb88b4fd21884bc401f71caa6f.svg
paper_authors.fullname: Chen Tao
paper_authors.type: user
paper_authors.user: taochen
paper_authors._id: 675958c2528f1296b6789dd4
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:53:45.594Z
paper_authors._id: 643dfd235aafbdca3a5792c0
paper_authors.avatarUrl: /avatars/ce8553cf5936012c692e08054ee27937.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: BoZhang
paper_authors._id: 675958c2528f1296b6789dd5
paper_authors.name: Xiangyu Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:51:21.617Z
paper_authors._id: 666a8f24e2990b0cb16b7bf9
paper_authors.avatarUrl: /avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
paper_authors.fullname: Xiangyu Yue
paper_authors.type: user
paper_authors.user: xyyue
paper_title: Chimera: Improving Generalist Model with Domain-Specific Experts
paper_summary: Recent advancements in Large Multi-modal Models (LMMs) underscore the
  importance of scaling by increasing image-text paired data, achieving
  impressive performance on general tasks. Despite their effectiveness in broad
  applications, generalist models are primarily trained on web-scale datasets
  dominated by natural images, resulting in the sacrifice of specialized
  capabilities for domain-specific tasks that require extensive domain prior
  knowledge. Moreover, directly integrating expert models tailored for specific
  domains is challenging due to the representational gap and imbalanced
  optimization between the generalist model and experts. To address these
  challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline
  designed to boost the ability of existing LMMs with domain-specific experts.
  Specifically, we design a progressive training strategy to integrate features
  from expert models into the input of a generalist LMM. To address the
  imbalanced optimization caused by the well-aligned general visual encoder, we
  introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism.
  This results in a versatile model that excels across the chart, table, math,
  and document domains, achieving state-of-the-art performance on multi-modal
  reasoning and visual content extraction tasks, both of which are challenging
  tasks for assessing existing LMMs.
paper_ai_summary: Chimera, a scalable multi-modal pipeline, integrates domain-specific experts into LMMs using progressive training and GSCM to achieve state-of-the-art performance in multi-modal reasoning and visual content extraction.
paper_ai_keywords: Large Multi-modal Models (LMMs) domain-specific experts progressive training strategy Generalist-Specialist Collaboration Masking (GSCM) multi-modal reasoning visual content extraction