title: Task Preference Optimization: Improving Multimodal Large Language Models
    with Vision Task Alignment
summary: Current multimodal large language models (MLLMs) struggle with fine-grained
  or precise understanding of visuals though they give comprehensive perception
  and reasoning in a spectrum of vision applications. Recent studies either
  develop tool-using or unify specific visual tasks into the autoregressive
  framework, often at the expense of overall multimodal performance. To address
  this issue and enhance MLLMs with visual tasks in a scalable fashion, we
  propose Task Preference Optimization (TPO), a novel method that utilizes
  differentiable task preferences derived from typical fine-grained visual tasks.
  TPO introduces learnable task tokens that establish connections between
  multiple task-specific heads and the MLLM. By leveraging rich visual labels
  during training, TPO significantly enhances the MLLM's multimodal capabilities
  and task-specific performance. Through multi-task co-training within TPO, we
  observe synergistic benefits that elevate individual task performance beyond
  what is achievable through single-task training methodologies. Our
  instantiation of this approach with VideoChat and LLaVA demonstrates an overall
  14.6% improvement in multimodal performance compared to baseline models.
  Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across
  various tasks, performing comparably to state-of-the-art supervised models. The
  code will be released at https://github.com/OpenGVLab/TPO
paper_authors._id: 67722817633a6043c33212aa
paper_authors.name: Ziang Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:44:33.925Z
paper_authors._id: 65499e5f2a292b3e2e5715a3
paper_authors.avatarUrl: /avatars/087b3e36dfb66e044265b856bab31657.svg
paper_authors.fullname: ziang yan
paper_authors.type: user
paper_authors.user: Aurorana
paper_authors._id: 67722817633a6043c33212ab
paper_authors.name: Zhilin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:44:41.063Z
paper_authors._id: 64412af71a80f6d83cb03318
paper_authors.avatarUrl: /avatars/113ad7064e7230eff3ae86ad24c88818.svg
paper_authors.fullname: zhilin li
paper_authors.type: user
paper_authors.user: white213
paper_authors._id: 67722817633a6043c33212ac
paper_authors.name: Yinan He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:44:47.500Z
paper_authors._id: 65b9d9961fe588f824fde191
paper_authors.avatarUrl: /avatars/a9245958cc998a4b4b870bf2490fdaee.svg
paper_authors.fullname: Yinan He
paper_authors.type: user
paper_authors.user: yinanhe
paper_authors._id: 67722817633a6043c33212ad
paper_authors.name: Chenting Wang
paper_authors._id: 67722817633a6043c33212ae
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:45:06.653Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 67722817633a6043c33212af
paper_authors.name: Xinhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:45:14.329Z
paper_authors._id: 672f8a28c53c174f39b08ac1
paper_authors.avatarUrl: /avatars/9d865f757667de14381d7c4d7ba7e4c4.svg
paper_authors.fullname: XINHAO LI
paper_authors.type: user
paper_authors.user: xinhaoli
paper_authors._id: 67722817633a6043c33212b0
paper_authors.name: Xiangyu Zeng
paper_authors._id: 67722817633a6043c33212b1
paper_authors.name: Zilei Wang
paper_authors._id: 67722817633a6043c33212b2
paper_authors.name: Yali Wang
paper_authors._id: 67722817633a6043c33212b3
paper_authors.name: Yu Qiao
paper_authors._id: 67722817633a6043c33212b4
paper_authors.name: Limin Wang
paper_authors._id: 67722817633a6043c33212b5
paper_authors.name: Yi Wang
paper_title: Task Preference Optimization: Improving Multimodal Large Language Models
    with Vision Task Alignment
paper_summary: Current multimodal large language models (MLLMs) struggle with fine-grained
  or precise understanding of visuals though they give comprehensive perception
  and reasoning in a spectrum of vision applications. Recent studies either
  develop tool-using or unify specific visual tasks into the autoregressive
  framework, often at the expense of overall multimodal performance. To address
  this issue and enhance MLLMs with visual tasks in a scalable fashion, we
  propose Task Preference Optimization (TPO), a novel method that utilizes
  differentiable task preferences derived from typical fine-grained visual tasks.
  TPO introduces learnable task tokens that establish connections between
  multiple task-specific heads and the MLLM. By leveraging rich visual labels
  during training, TPO significantly enhances the MLLM's multimodal capabilities
  and task-specific performance. Through multi-task co-training within TPO, we
  observe synergistic benefits that elevate individual task performance beyond
  what is achievable through single-task training methodologies. Our
  instantiation of this approach with VideoChat and LLaVA demonstrates an overall
  14.6% improvement in multimodal performance compared to baseline models.
  Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across
  various tasks, performing comparably to state-of-the-art supervised models. The
  code will be released at https://github.com/OpenGVLab/TPO
paper_ai_summary: Task Preference Optimization (TPO) enhances multimodal large language models with fine-grained visual tasks by using learnable task tokens and multi-task co-training, leading to significant performance improvements and robust zero-shot capabilities.
paper_ai_keywords: multimodal large language models Task Preference Optimization TPO learnable task tokens multi-task co-training fine-grained visual tasks multimodal capabilities zero-shot capabilities