On Domain-Specific Post-Training for Multimodal Large Language Models
Recent years have witnessed the rapid development of general multimodal large
language models (MLLMs). However, adapting general MLLMs to specific domains,
such as scientific fields and industrial applications, remains less explored.
This paper systematically investigates domain adaptation of MLLMs through
post-training, focusing on data synthesis, training pipelines, and task
evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
instruction synthesizer that effectively generates diverse visual instruction
tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
those generated by manual rules, GPT-4, and GPT-4V in enhancing the
domain-specific performance of MLLMs. (2) Training Pipeline: While the
two-stage training--initially on image-caption pairs followed by visual
instruction tasks--is commonly adopted for developing general MLLMs, we apply a
single-stage training pipeline to enhance task diversity for domain-specific
post-training. (3) Task Evaluation: We conduct experiments in two domains,
biomedicine and food, by post-training MLLMs of different sources and scales
(e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
performance on various domain-specific tasks. To support further research in
MLLM domain adaptation, we will open-source our implementations.
674d32f63965f56df21c340c
Daixuan Cheng
claimed_verified
2024-12-02T08:49:33.196Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
674d32f63965f56df21c340d
Shaohan Huang
admin_assigned
2024-12-02T09:04:44.158Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
674d32f63965f56df21c340e
Ziyu Zhu
admin_assigned
2024-12-02T09:05:38.250Z
64d86d66d7e30889c6a2e955
/avatars/222fcbe4af3bd897f260d019a54cfb6d.svg
ziyu zhu
user
edward2021
674d32f63965f56df21c340f
Xintong Zhang
674d32f63965f56df21c3410
Wayne Xin Zhao
674d32f63965f56df21c3411
Zhongzhi Luan
674d32f63965f56df21c3412
Bo Dai
admin_assigned
2024-12-02T08:56:41.196Z
665e2bd0fbaa279db369b488
/avatars/8cbc18d17d2813617976dbe709235799.svg
Bo Dai
user
doubling
674d32f63965f56df21c3413
Zhenliang Zhang
On Domain-Specific Post-Training for Multimodal Large Language Models
Recent years have witnessed the rapid development of general multimodal large
language models (MLLMs). However, adapting general MLLMs to specific domains,
such as scientific fields and industrial applications, remains less explored.
This paper systematically investigates domain adaptation of MLLMs through
post-training, focusing on data synthesis, training pipelines, and task
evaluation. (1) Data Synthesis: Using open-source models, we develop a visual
instruction synthesizer that effectively generates diverse visual instruction
tasks from domain-specific image-caption pairs. Our synthetic tasks surpass
those generated by manual rules, GPT-4, and GPT-4V in enhancing the
domain-specific performance of MLLMs. (2) Training Pipeline: While the
two-stage training--initially on image-caption pairs followed by visual
instruction tasks--is commonly adopted for developing general MLLMs, we apply a
single-stage training pipeline to enhance task diversity for domain-specific
post-training. (3) Task Evaluation: We conduct experiments in two domains,
biomedicine and food, by post-training MLLMs of different sources and scales
(e.g., Qwen2-VL-2B, LLaVA-v1.6-8B, Llama-3.2-11B), and then evaluating MLLM
performance on various domain-specific tasks. To support further research in
MLLM domain adaptation, we will open-source our implementations.
The paper explores the domain adaptation of multimodal large language models through post-training, utilizing a visual instruction synthesizer and single-stage training pipeline to improve performance on specific tasks in domains like biomedicine and food.
multimodal large language models domain adaptation post-training visual instruction synthesizer single-stage training pipeline image-caption pairs domain-specific tasks biomedicine food