From Generalist to Specialist: Adapting Vision Language Models via
  Task-Specific Visual Instruction Tuning
Large vision language models (VLMs) combine large language models with vision
encoders, demonstrating promise across various tasks. However, they often
underperform in task-specific applications due to domain gaps between
pre-training and fine-tuning. We introduce VITask, a novel framework that
enhances task-specific adaptability of VLMs by integrating task-specific models
(TSMs). VITask employs three key strategies: exemplar prompting (EP), response
distribution alignment (RDA), and contrastive response tuning (CRT) to improve
the task-specific performance of VLMs by adjusting their response
distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to
adapt without TSMs during inference by learning from exemplar-prompted models.
CRT further optimizes the ranking of correct image-response pairs, thereby
reducing the risk of generating undesired responses. Experiments on 12 medical
diagnosis datasets across 9 imaging modalities show that VITask outperforms
both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to
integrate complementary features from both models effectively. Additionally,
VITask offers practical advantages such as flexible TSM integration and
robustness to incomplete instructions, making it a versatile and efficient
solution for task-specific VLM tuning. Our code are available at
https://github.com/baiyang4/VITask.
670a14c002d531812e0467a3
Yang Bai
claimed_verified
2024-10-13T20:15:39.915Z
64802423c7f87934d081bad3
/avatars/32b07a84662cbf73a0adbb325088440b.svg
Yang bai
user
yangbai123
670a14c002d531812e0467a4
Yang Zhou
claimed_verified
2024-10-30T09:55:43.894Z
670cc0b60c2a160841c8a343
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/21wJ_Eg4jBvTfaGR4Y7CY.png
ZHOU Yang
user
youngzhou12
670a14c002d531812e0467a5
Jun Zhou
670a14c002d531812e0467a6
Rick Siow Mong Goh
670a14c002d531812e0467a7
Daniel Shu Wei Ting
670a14c002d531812e0467a8
Yong Liu
From Generalist to Specialist: Adapting Vision Language Models via
  Task-Specific Visual Instruction Tuning
Large vision language models (VLMs) combine large language models with vision
encoders, demonstrating promise across various tasks. However, they often
underperform in task-specific applications due to domain gaps between
pre-training and fine-tuning. We introduce VITask, a novel framework that
enhances task-specific adaptability of VLMs by integrating task-specific models
(TSMs). VITask employs three key strategies: exemplar prompting (EP), response
distribution alignment (RDA), and contrastive response tuning (CRT) to improve
the task-specific performance of VLMs by adjusting their response
distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to
adapt without TSMs during inference by learning from exemplar-prompted models.
CRT further optimizes the ranking of correct image-response pairs, thereby
reducing the risk of generating undesired responses. Experiments on 12 medical
diagnosis datasets across 9 imaging modalities show that VITask outperforms
both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to
integrate complementary features from both models effectively. Additionally,
VITask offers practical advantages such as flexible TSM integration and
robustness to incomplete instructions, making it a versatile and efficient
solution for task-specific VLM tuning. Our code are available at
https://github.com/baiyang4/VITask.
VITask enhances vision language models with task-specific models using exemplar prompting, response distribution alignment, and contrastive response tuning to improve task-specific performance across medical diagnosis datasets.
VLMs vision encoders task-specific models exemplar prompting response distribution alignment contrastive response tuning medical diagnosis imaging modalities