title: Aligning Multimodal LLM with Human Preference: A Survey
summary: Large language models (LLMs) can handle a wide variety of general tasks with
  simple prompts, without the need for task-specific training. Multimodal Large
  Language Models (MLLMs), built upon LLMs, have demonstrated impressive
  potential in tackling complex tasks involving visual, auditory, and textual
  data. However, critical issues related to truthfulness, safety, o1-like
  reasoning, and alignment with human preference remain insufficiently addressed.
  This gap has spurred the emergence of various alignment algorithms, each
  targeting different application scenarios and optimization goals. Recent
  studies have shown that alignment algorithms are a powerful approach to
  resolving the aforementioned challenges. In this paper, we aim to provide a
  comprehensive and systematic review of alignment algorithms for MLLMs.
  Specifically, we explore four key aspects: (1) the application scenarios
  covered by alignment algorithms, including general image understanding,
  multi-image, video, and audio, and extended multimodal applications; (2) the
  core factors in constructing alignment datasets, including data sources, model
  responses, and preference annotations; (3) the benchmarks used to evaluate
  alignment algorithms; and (4) a discussion of potential future directions for
  the development of alignment algorithms. This work seeks to help researchers
  organize current advancements in the field and inspire better alignment
  methods. The project page of this paper is available at
  https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Alignment.
paper_authors._id: 67da436711b6db6920802e9e
paper_authors.name: Tao Yu
paper_authors._id: 67da436711b6db6920802e9f
paper_authors.name: Yi-Fan Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-20T03:10:01.854Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 67da436711b6db6920802ea0
paper_authors.name: Chaoyou Fu
paper_authors._id: 67da436711b6db6920802ea1
paper_authors.name: Junkang Wu
paper_authors._id: 67da436711b6db6920802ea2
paper_authors.name: Jinda Lu
paper_authors._id: 67da436711b6db6920802ea3
paper_authors.name: Kun Wang
paper_authors._id: 67da436711b6db6920802ea4
paper_authors.name: Xingyu Lu
paper_authors._id: 67da436711b6db6920802ea5
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:38:55.153Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 67da436711b6db6920802ea6
paper_authors.name: Guibin Zhang
paper_authors._id: 67da436711b6db6920802ea7
paper_authors.name: Dingjie Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:38:06.473Z
paper_authors._id: 619f01b8cc04eadf54fa5d5d
paper_authors.avatarUrl: /avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
paper_authors.fullname: Song Dingjie
paper_authors.type: user
paper_authors.user: songdj
paper_authors._id: 67da436711b6db6920802ea8
paper_authors.name: Yibo Yan
paper_authors._id: 67da436711b6db6920802ea9
paper_authors.name: Tianlong Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:37:52.755Z
paper_authors._id: 66b65be6308cd3b74061c71a
paper_authors.avatarUrl: /avatars/e8e9120a25044fde0a9483b4553ab8fa.svg
paper_authors.fullname: Tianlong Xu
paper_authors.type: user
paper_authors.user: xtl090994
paper_authors._id: 67da436711b6db6920802eaa
paper_authors.name: Qingsong Wen
paper_authors._id: 67da436711b6db6920802eab
paper_authors.name: Zhang Zhang
paper_authors._id: 67da436711b6db6920802eac
paper_authors.name: Yan Huang
paper_authors._id: 67da436711b6db6920802ead
paper_authors.name: Liang Wang
paper_authors._id: 67da436711b6db6920802eae
paper_authors.name: Tieniu Tan
paper_title: Aligning Multimodal LLM with Human Preference: A Survey
paper_summary: Large language models (LLMs) can handle a wide variety of general tasks with
  simple prompts, without the need for task-specific training. Multimodal Large
  Language Models (MLLMs), built upon LLMs, have demonstrated impressive
  potential in tackling complex tasks involving visual, auditory, and textual
  data. However, critical issues related to truthfulness, safety, o1-like
  reasoning, and alignment with human preference remain insufficiently addressed.
  This gap has spurred the emergence of various alignment algorithms, each
  targeting different application scenarios and optimization goals. Recent
  studies have shown that alignment algorithms are a powerful approach to
  resolving the aforementioned challenges. In this paper, we aim to provide a
  comprehensive and systematic review of alignment algorithms for MLLMs.
  Specifically, we explore four key aspects: (1) the application scenarios
  covered by alignment algorithms, including general image understanding,
  multi-image, video, and audio, and extended multimodal applications; (2) the
  core factors in constructing alignment datasets, including data sources, model
  responses, and preference annotations; (3) the benchmarks used to evaluate
  alignment algorithms; and (4) a discussion of potential future directions for
  the development of alignment algorithms. This work seeks to help researchers
  organize current advancements in the field and inspire better alignment
  methods. The project page of this paper is available at
  https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Alignment.
paper_ai_summary: A comprehensive review of alignment algorithms for multimodal large language models, covering their application scenarios, dataset construction, evaluation benchmarks, and future directions.
paper_ai_keywords: Large language models (LLMs) Multimodal Large Language Models (MLLMs) alignment algorithms truthfulness safety reasoning human preference image understanding multi-image video audio data sources model responses preference annotations benchmarks future directions