MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
  Vision-Language Models
Visual preference alignment involves training Large Vision-Language Models
(LVLMs) to predict human preferences between visual inputs. This is typically
achieved by using labeled datasets of chosen/rejected pairs and employing
optimization algorithms like direct preference optimization (DPO). Existing
visual alignment methods, primarily designed for single-image scenarios,
struggle to effectively handle the complexity of multi-image tasks due to the
scarcity of diverse training data and the high cost of annotating
chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
Optimization (MIA-DPO), a visual preference alignment approach that effectively
handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
multi-image training data by extending single-image data with unrelated images
arranged in grid collages or pic-in-pic formats, significantly reducing the
costs associated with multi-image data annotations. Our observation reveals
that attention values of LVLMs vary considerably across different images. We
use attention values to identify and filter out rejected responses the model
may have mistakenly focused on. Our attention-aware selection for constructing
the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
data, and (iii) external models or APIs. MIA-DPO is compatible with various
architectures and outperforms existing methods on five multi-image benchmarks,
achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
ability to understand single images.
6719b2470b8f923fdda13623
Ziyu Liu
admin_assigned
2024-10-24T08:13:22.757Z
66fe1334ff3ee1f7569fab6d
/avatars/6868b1a545028a9b8bbded52490dc093.svg
ziyuliu
user
ziyuliu
6719b2470b8f923fdda13624
Yuhang Zang
admin_assigned
2024-10-24T08:13:12.075Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
6719b2470b8f923fdda13625
Xiaoyi Dong
6719b2470b8f923fdda13626
Pan Zhang
6719b2470b8f923fdda13627
Yuhang Cao
admin_assigned
2024-10-24T08:13:03.746Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
6719b2470b8f923fdda13628
Haodong Duan
claimed_verified
2024-10-24T07:15:24.930Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
6719b2470b8f923fdda13629
Conghui He
admin_assigned
2024-10-24T08:00:44.324Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
6719b2470b8f923fdda1362a
Yuanjun Xiong
admin_assigned
2024-10-24T08:12:24.611Z
66209605f44be495f123b055
/avatars/b72081a0383a709899f9ba80b4211690.svg
Yuanjun Xiong
user
TheYJ
6719b2470b8f923fdda1362b
Dahua Lin
admin_assigned
2024-10-24T08:12:18.459Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
6719b2470b8f923fdda1362c
Jiaqi Wang
claimed_verified
2024-12-11T09:21:39.835Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
  Vision-Language Models
Visual preference alignment involves training Large Vision-Language Models
(LVLMs) to predict human preferences between visual inputs. This is typically
achieved by using labeled datasets of chosen/rejected pairs and employing
optimization algorithms like direct preference optimization (DPO). Existing
visual alignment methods, primarily designed for single-image scenarios,
struggle to effectively handle the complexity of multi-image tasks due to the
scarcity of diverse training data and the high cost of annotating
chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
Optimization (MIA-DPO), a visual preference alignment approach that effectively
handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
multi-image training data by extending single-image data with unrelated images
arranged in grid collages or pic-in-pic formats, significantly reducing the
costs associated with multi-image data annotations. Our observation reveals
that attention values of LVLMs vary considerably across different images. We
use attention values to identify and filter out rejected responses the model
may have mistakenly focused on. Our attention-aware selection for constructing
the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
data, and (iii) external models or APIs. MIA-DPO is compatible with various
architectures and outperforms existing methods on five multi-image benchmarks,
achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
ability to understand single images.
MIA-DPO is a method for visual preference alignment that improves handling of multi-image inputs by extending single-image data and using attention values to refine chosen/rejected pairs without extra annotations or data.
Large Vision-Language Models LVLMs direct preference optimization DPO Multi-Image Augmented Direct Preference Optimization MIA-DPO attention values LLaVA-v1.5 InternLM-XC2.5