title: mDPO: Conditional Preference Optimization for Multimodal Large Language
    Models
summary: Direct preference optimization (DPO) has shown to be an effective method for
  large language model (LLM) alignment. Recent works have attempted to apply DPO
  to multimodal scenarios but have found it challenging to achieve consistent
  improvement. Through a comparative experiment, we identify the unconditional
  preference problem in multimodal preference optimization, where the model
  overlooks the image condition. To address this problem, we propose mDPO, a
  multimodal DPO objective that prevents the over-prioritization of language-only
  preferences by also optimizing image preference. Moreover, we introduce a
  reward anchor that forces the reward to be positive for chosen responses,
  thereby avoiding the decrease in their likelihood -- an intrinsic problem of
  relative preference optimization. Experiments on two multimodal LLMs of
  different sizes and three widely used benchmarks demonstrate that mDPO
  effectively addresses the unconditional preference problem in multimodal
  preference optimization and significantly improves model performance,
  particularly in reducing hallucination.
paper_authors._id: 6670efee86ff335801c6754d
paper_authors.name: Fei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:18:01.857Z
paper_authors._id: 652d9db6442fb6963b778295
paper_authors.avatarUrl: /avatars/318334e5ddae8fde0cbd06a1b9583ed8.svg
paper_authors.fullname: Fei Wang
paper_authors.type: user
paper_authors.user: fwnlp
paper_authors._id: 6670efee86ff335801c6754e
paper_authors.name: Wenxuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:30:18.982Z
paper_authors._id: 619414c7979eb76ec3cc5bab
paper_authors.avatarUrl: /avatars/c1a7055db6150b1bba92e7b13266b489.svg
paper_authors.fullname: Wenxuan Zhou
paper_authors.type: user
paper_authors.user: wzhouad
paper_authors._id: 6670efee86ff335801c6754f
paper_authors.name: James Y. Huang
paper_authors._id: 6670efee86ff335801c67550
paper_authors.name: Nan Xu
paper_authors._id: 6670efee86ff335801c67551
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:27:31.381Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 6670efee86ff335801c67552
paper_authors.name: Hoifung Poon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:31:32.388Z
paper_authors._id: 664d07456083f276c4feb1a4
paper_authors.avatarUrl: /avatars/1bfa6d8f82e9223b47630cefd79d7d0e.svg
paper_authors.fullname: Hoifung Poon
paper_authors.type: user
paper_authors.user: hoifung
paper_authors._id: 6670efee86ff335801c67553
paper_authors.name: Muhao Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-18T03:43:23.648Z
paper_authors._id: 666be8ef81f01fbd60e84f01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
paper_authors.fullname: Muhao Chen
paper_authors.type: user
paper_authors.user: Muhao
paper_title: mDPO: Conditional Preference Optimization for Multimodal Large Language
    Models
paper_summary: Direct preference optimization (DPO) has shown to be an effective method for
  large language model (LLM) alignment. Recent works have attempted to apply DPO
  to multimodal scenarios but have found it challenging to achieve consistent
  improvement. Through a comparative experiment, we identify the unconditional
  preference problem in multimodal preference optimization, where the model
  overlooks the image condition. To address this problem, we propose mDPO, a
  multimodal DPO objective that prevents the over-prioritization of language-only
  preferences by also optimizing image preference. Moreover, we introduce a
  reward anchor that forces the reward to be positive for chosen responses,
  thereby avoiding the decrease in their likelihood -- an intrinsic problem of
  relative preference optimization. Experiments on two multimodal LLMs of
  different sizes and three widely used benchmarks demonstrate that mDPO
  effectively addresses the unconditional preference problem in multimodal
  preference optimization and significantly improves model performance,
  particularly in reducing hallucination.
paper_ai_keywords: Direct preference optimization (DPO) large language model (LLM) multimodal scenarios unconditional preference problem preference optimization model architecture mDP (multimodal DPO) image condition image preference reward anchor relative preference optimization multimodal LLMs benchmarks hallucination