title: From Generalist to Specialist: Adapting Vision Language Models via
    Task-Specific Visual Instruction Tuning
summary: Large vision language models (VLMs) combine large language models with vision
  encoders, demonstrating promise across various tasks. However, they often
  underperform in task-specific applications due to domain gaps between
  pre-training and fine-tuning. We introduce VITask, a novel framework that
  enhances task-specific adaptability of VLMs by integrating task-specific models
  (TSMs). VITask employs three key strategies: exemplar prompting (EP), response
  distribution alignment (RDA), and contrastive response tuning (CRT) to improve
  the task-specific performance of VLMs by adjusting their response
  distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to
  adapt without TSMs during inference by learning from exemplar-prompted models.
  CRT further optimizes the ranking of correct image-response pairs, thereby
  reducing the risk of generating undesired responses. Experiments on 12 medical
  diagnosis datasets across 9 imaging modalities show that VITask outperforms
  both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to
  integrate complementary features from both models effectively. Additionally,
  VITask offers practical advantages such as flexible TSM integration and
  robustness to incomplete instructions, making it a versatile and efficient
  solution for task-specific VLM tuning. Our code are available at
  https://github.com/baiyang4/VITask.
paper_authors._id: 670a14c002d531812e0467a3
paper_authors.name: Yang Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:15:39.915Z
paper_authors._id: 64802423c7f87934d081bad3
paper_authors.avatarUrl: /avatars/32b07a84662cbf73a0adbb325088440b.svg
paper_authors.fullname: Yang bai
paper_authors.type: user
paper_authors.user: yangbai123
paper_authors._id: 670a14c002d531812e0467a4
paper_authors.name: Yang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:43.894Z
paper_authors._id: 670cc0b60c2a160841c8a343
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/21wJ_Eg4jBvTfaGR4Y7CY.png
paper_authors.fullname: ZHOU Yang
paper_authors.type: user
paper_authors.user: youngzhou12
paper_authors._id: 670a14c002d531812e0467a5
paper_authors.name: Jun Zhou
paper_authors._id: 670a14c002d531812e0467a6
paper_authors.name: Rick Siow Mong Goh
paper_authors._id: 670a14c002d531812e0467a7
paper_authors.name: Daniel Shu Wei Ting
paper_authors._id: 670a14c002d531812e0467a8
paper_authors.name: Yong Liu
paper_title: From Generalist to Specialist: Adapting Vision Language Models via
    Task-Specific Visual Instruction Tuning
paper_summary: Large vision language models (VLMs) combine large language models with vision
  encoders, demonstrating promise across various tasks. However, they often
  underperform in task-specific applications due to domain gaps between
  pre-training and fine-tuning. We introduce VITask, a novel framework that
  enhances task-specific adaptability of VLMs by integrating task-specific models
  (TSMs). VITask employs three key strategies: exemplar prompting (EP), response
  distribution alignment (RDA), and contrastive response tuning (CRT) to improve
  the task-specific performance of VLMs by adjusting their response
  distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to
  adapt without TSMs during inference by learning from exemplar-prompted models.
  CRT further optimizes the ranking of correct image-response pairs, thereby
  reducing the risk of generating undesired responses. Experiments on 12 medical
  diagnosis datasets across 9 imaging modalities show that VITask outperforms
  both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to
  integrate complementary features from both models effectively. Additionally,
  VITask offers practical advantages such as flexible TSM integration and
  robustness to incomplete instructions, making it a versatile and efficient
  solution for task-specific VLM tuning. Our code are available at
  https://github.com/baiyang4/VITask.
paper_ai_keywords: VLMs (Vision Language Models) vision encoders task-specific models (TSMs) exemplar prompting (EP) response distribution alignment (RDA) contrastive response tuning (CRT) image-response pairs