title: MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
    Vision-Language Models
summary: Visual preference alignment involves training Large Vision-Language Models
  (LVLMs) to predict human preferences between visual inputs. This is typically
  achieved by using labeled datasets of chosen/rejected pairs and employing
  optimization algorithms like direct preference optimization (DPO). Existing
  visual alignment methods, primarily designed for single-image scenarios,
  struggle to effectively handle the complexity of multi-image tasks due to the
  scarcity of diverse training data and the high cost of annotating
  chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
  Optimization (MIA-DPO), a visual preference alignment approach that effectively
  handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
  multi-image training data by extending single-image data with unrelated images
  arranged in grid collages or pic-in-pic formats, significantly reducing the
  costs associated with multi-image data annotations. Our observation reveals
  that attention values of LVLMs vary considerably across different images. We
  use attention values to identify and filter out rejected responses the model
  may have mistakenly focused on. Our attention-aware selection for constructing
  the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
  data, and (iii) external models or APIs. MIA-DPO is compatible with various
  architectures and outperforms existing methods on five multi-image benchmarks,
  achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
  recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
  ability to understand single images.
paper_authors._id: 6719b2470b8f923fdda13623
paper_authors.name: Ziyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:13:22.757Z
paper_authors._id: 66fe1334ff3ee1f7569fab6d
paper_authors.avatarUrl: /avatars/6868b1a545028a9b8bbded52490dc093.svg
paper_authors.fullname: ziyuliu
paper_authors.type: user
paper_authors.user: ziyuliu
paper_authors._id: 6719b2470b8f923fdda13624
paper_authors.name: Yuhang Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:13:12.075Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 6719b2470b8f923fdda13625
paper_authors.name: Xiaoyi Dong
paper_authors._id: 6719b2470b8f923fdda13626
paper_authors.name: Pan Zhang
paper_authors._id: 6719b2470b8f923fdda13627
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:13:03.746Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 6719b2470b8f923fdda13628
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:24.930Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 6719b2470b8f923fdda13629
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:00:44.324Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 6719b2470b8f923fdda1362a
paper_authors.name: Yuanjun Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:12:24.611Z
paper_authors._id: 66209605f44be495f123b055
paper_authors.avatarUrl: /avatars/b72081a0383a709899f9ba80b4211690.svg
paper_authors.fullname: Yuanjun Xiong
paper_authors.type: user
paper_authors.user: TheYJ
paper_authors._id: 6719b2470b8f923fdda1362b
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:12:18.459Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 6719b2470b8f923fdda1362c
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:21:39.835Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
    Vision-Language Models
paper_summary: Visual preference alignment involves training Large Vision-Language Models
  (LVLMs) to predict human preferences between visual inputs. This is typically
  achieved by using labeled datasets of chosen/rejected pairs and employing
  optimization algorithms like direct preference optimization (DPO). Existing
  visual alignment methods, primarily designed for single-image scenarios,
  struggle to effectively handle the complexity of multi-image tasks due to the
  scarcity of diverse training data and the high cost of annotating
  chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
  Optimization (MIA-DPO), a visual preference alignment approach that effectively
  handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
  multi-image training data by extending single-image data with unrelated images
  arranged in grid collages or pic-in-pic formats, significantly reducing the
  costs associated with multi-image data annotations. Our observation reveals
  that attention values of LVLMs vary considerably across different images. We
  use attention values to identify and filter out rejected responses the model
  may have mistakenly focused on. Our attention-aware selection for constructing
  the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
  data, and (iii) external models or APIs. MIA-DPO is compatible with various
  architectures and outperforms existing methods on five multi-image benchmarks,
  achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
  recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
  ability to understand single images.
paper_ai_summary: MIA-DPO is a method for visual preference alignment that improves handling of multi-image inputs by extending single-image data and using attention values to refine chosen/rejected pairs without extra annotations or data.
paper_ai_keywords: Large Vision-Language Models LVLMs direct preference optimization DPO Multi-Image Augmented Direct Preference Optimization MIA-DPO attention values LLaVA-v1.5 InternLM-XC2.5