title: Enhancing the Reasoning Ability of Multimodal Large Language Models via
    Mixed Preference Optimization
summary: Existing open-source multimodal large language models (MLLMs) generally
  follow a training process involving pre-training and supervised fine-tuning.
  However, these models suffer from distribution shifts, which limit their
  multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance.
  To address this, we introduce a preference optimization (PO) process to enhance
  the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data
  side, we design an automated preference data construction pipeline to create
  MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and
  (2) on the model side, we explore integrating PO with MLLMs, developing a
  simple yet effective method, termed Mixed Preference Optimization (MPO), which
  boosts multimodal CoT performance. Our approach demonstrates improved
  performance across multiple benchmarks, particularly in multimodal reasoning
  tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on
  MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance
  comparable to the 10x larger InternVL2-76B. We hope this study could inspire
  further advancements in MLLMs. Code, data, and model shall be publicly
  released.
paper_authors._id: 673acbc08127cd120caeb52d
paper_authors.name: Weiyun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:03:42.369Z
paper_authors._id: 619507e7b74b6c591f794340
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
paper_authors.fullname: Weiyun Wang
paper_authors.type: user
paper_authors.user: Weiyun1025
paper_authors._id: 673acbc08127cd120caeb52e
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:05.699Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 673acbc08127cd120caeb52f
paper_authors.name: Wenhai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:03:58.775Z
paper_authors._id: 64d1c560c0c627dfa71bdbe0
paper_authors.avatarUrl: /avatars/f42794fe25bffcd870a1bcee69b95298.svg
paper_authors.fullname: wenhai.wang
paper_authors.type: user
paper_authors.user: wangwhcore
paper_authors._id: 673acbc08127cd120caeb530
paper_authors.name: Yue Cao
paper_authors._id: 673acbc08127cd120caeb531
paper_authors.name: Yangzhou Liu
paper_authors._id: 673acbc08127cd120caeb532
paper_authors.name: Zhangwei Gao
paper_authors._id: 673acbc08127cd120caeb533
paper_authors.name: Jinguo Zhu
paper_authors._id: 673acbc08127cd120caeb534
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:04:52.411Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 673acbc08127cd120caeb535
paper_authors.name: Lewei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:04:57.809Z
paper_authors._id: 65ead3ea908526a39082e641
paper_authors.avatarUrl: /avatars/dcf870695fd56b06ca03d82f831e9019.svg
paper_authors.fullname: Lewei Lu
paper_authors.type: user
paper_authors.user: luotto
paper_authors._id: 673acbc08127cd120caeb536
paper_authors.name: Yu Qiao
paper_authors._id: 673acbc08127cd120caeb537
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:05:03.791Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_title: Enhancing the Reasoning Ability of Multimodal Large Language Models via
    Mixed Preference Optimization
paper_summary: Existing open-source multimodal large language models (MLLMs) generally
  follow a training process involving pre-training and supervised fine-tuning.
  However, these models suffer from distribution shifts, which limit their
  multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance.
  To address this, we introduce a preference optimization (PO) process to enhance
  the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data
  side, we design an automated preference data construction pipeline to create
  MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and
  (2) on the model side, we explore integrating PO with MLLMs, developing a
  simple yet effective method, termed Mixed Preference Optimization (MPO), which
  boosts multimodal CoT performance. Our approach demonstrates improved
  performance across multiple benchmarks, particularly in multimodal reasoning
  tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on
  MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance
  comparable to the 10x larger InternVL2-76B. We hope this study could inspire
  further advancements in MLLMs. Code, data, and model shall be publicly
  released.
paper_ai_summary: A preference optimization process enhances multimodal reasoning, particularly Chain-of-Thought performance, in large language models through automated data construction and mixed preference optimization, demonstrating significant improvements on benchmarks.
paper_ai_keywords: multimodal large language models MLLMs distribution shifts Chain-of-Thought preference optimization PO MMPR Mixed Preference Optimization MPO multimodal CoT benchmarks MathVista InternVL2