title: On Domain-Specific Post-Training for Multimodal Large Language Models
summary: Recent years have witnessed the rapid development of general multimodal large
  language models (MLLMs). However, adapting general MLLMs to specific domains,
  such as scientific fields and industrial applications, remains less explored.
  This paper systematically investigates domain adaptation of MLLMs through
  post-training, focusing on data synthesis, training pipelines, and task
  evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
  instruction synthesizer that effectively generates diverse visual instruction
  tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
  those generated by manual rules, GPT-4, and GPT-4V in enhancing the
  domain-specific performance of MLLMs. (2) Training Pipeline: While the
  two-stage training--initially on image-caption pairs followed by visual
  instruction tasks--is commonly adopted for developing general MLLMs, we apply a
  single-stage training pipeline to enhance task diversity for domain-specific
  post-training. (3) Task Evaluation: We conduct experiments in two domains,
  biomedicine and food, by post-training MLLMs of different sources and scales
  (e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
  performance on various domain-specific tasks. To support further research in
  MLLM domain adaptation, we will open-source our implementations.
paper_authors._id: 674d32f63965f56df21c340c
paper_authors.name: Daixuan Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:49:33.196Z
paper_authors._id: 649e6761f9134a06ed1e0cea
paper_authors.avatarUrl: /avatars/00b5dcb744c54a4aa18fe08efd70d6ff.svg
paper_authors.fullname: Daixuan Cheng
paper_authors.type: user
paper_authors.user: daixuancheng
paper_authors._id: 674d32f63965f56df21c340d
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:04:44.158Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 674d32f63965f56df21c340e
paper_authors.name: Ziyu Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:05:38.250Z
paper_authors._id: 64d86d66d7e30889c6a2e955
paper_authors.avatarUrl: /avatars/222fcbe4af3bd897f260d019a54cfb6d.svg
paper_authors.fullname: ziyu zhu
paper_authors.type: user
paper_authors.user: edward2021
paper_authors._id: 674d32f63965f56df21c340f
paper_authors.name: Xintong Zhang
paper_authors._id: 674d32f63965f56df21c3410
paper_authors.name: Wayne Xin Zhao
paper_authors._id: 674d32f63965f56df21c3411
paper_authors.name: Zhongzhi Luan
paper_authors._id: 674d32f63965f56df21c3412
paper_authors.name: Bo Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T08:56:41.196Z
paper_authors._id: 665e2bd0fbaa279db369b488
paper_authors.avatarUrl: /avatars/8cbc18d17d2813617976dbe709235799.svg
paper_authors.fullname: Bo Dai
paper_authors.type: user
paper_authors.user: doubling
paper_authors._id: 674d32f63965f56df21c3413
paper_authors.name: Zhenliang Zhang
paper_title: On Domain-Specific Post-Training for Multimodal Large Language Models
paper_summary: Recent years have witnessed the rapid development of general multimodal large
  language models (MLLMs). However, adapting general MLLMs to specific domains,
  such as scientific fields and industrial applications, remains less explored.
  This paper systematically investigates domain adaptation of MLLMs through
  post-training, focusing on data synthesis, training pipelines, and task
  evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
  instruction synthesizer that effectively generates diverse visual instruction
  tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
  those generated by manual rules, GPT-4, and GPT-4V in enhancing the
  domain-specific performance of MLLMs. (2) Training Pipeline: While the
  two-stage training--initially on image-caption pairs followed by visual
  instruction tasks--is commonly adopted for developing general MLLMs, we apply a
  single-stage training pipeline to enhance task diversity for domain-specific
  post-training. (3) Task Evaluation: We conduct experiments in two domains,
  biomedicine and food, by post-training MLLMs of different sources and scales
  (e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
  performance on various domain-specific tasks. To support further research in
  MLLM domain adaptation, we will open-source our implementations.
paper_ai_keywords: multimodal large language models (MLLMs) domain adaptation post-training data synthesis visual instruction synthesizer image-caption pairs visual instruction tasks GPT-4 GPT-4V single-stage training pipeline task diversity domain-specific performance biomedicine food task evaluation Qwen2-VL-2B LLaVA-v1.6-8B Llama-3.2-11B