title: MM-IFEngine: Towards Multimodal Instruction Following
summary: The Instruction Following (IF) ability measures how well Multi-modal Large
  Language Models (MLLMs) understand exactly what users are telling them and
  whether they are doing it right. Existing multimodal instruction following
  training data is scarce, the benchmarks are simple with atomic instructions,
  and the evaluation strategies are imprecise for tasks demanding exact output
  constraints. To address this, we present MM-IFEngine, an effective pipeline to
  generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields
  large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which
  is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for
  Direct Preference Optimization (DPO). We further introduce MM-IFEval, a
  challenging and diverse multi-modal instruction-following benchmark that
  includes (1) both compose-level constraints for output responses and
  perception-level constraints tied to the input images, and (2) a comprehensive
  evaluation pipeline incorporating both rule-based assessment and judge model.
  We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on
  MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF
  benchmarks, such as MM-IFEval (+10.2%), MIA (+7.6%), and IFEval
  (+12.3%). The full data and evaluation code will be released on
  https://github.com/SYuan03/MM-IFEngine.
paper_authors._id: 67f8914516d43f88b3177ec1
paper_authors.name: Shengyuan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:25:40.694Z
paper_authors._id: 646cd947da8e99940b6e55cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
paper_authors.fullname: Shengyuan Ding
paper_authors.type: user
paper_authors.user: ChrisDing1105
paper_authors._id: 67f8914516d43f88b3177ec2
paper_authors.name: Shenxi Wu
paper_authors._id: 67f8914516d43f88b3177ec3
paper_authors.name: Xiangyu Zhao
paper_authors._id: 67f8914516d43f88b3177ec4
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:25:05.800Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 67f8914516d43f88b3177ec5
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:26:11.929Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67f8914516d43f88b3177ec6
paper_authors.name: Xiaoyi Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:26:18.423Z
paper_authors._id: 67c0849ee08c178ef8d4e05c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mQ6VdnjZnRhb0H_waPclo.png
paper_authors.fullname: Xiaoyi Dong
paper_authors.type: user
paper_authors.user: sweetFruit
paper_authors._id: 67f8914516d43f88b3177ec7
paper_authors.name: Pan Zhang
paper_authors._id: 67f8914516d43f88b3177ec8
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:26:51.425Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 67f8914516d43f88b3177ec9
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:26:59.045Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 67f8914516d43f88b3177eca
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:27:20.727Z
paper_authors._id: 64638c4d51fa6e63060521b5
paper_authors.avatarUrl: /avatars/c863ace5b1dc788a341bcf4ddbdfaec1.svg
paper_authors.fullname: JIaqi
paper_authors.type: user
paper_authors.user: Jiaqiwang
paper_title: MM-IFEngine: Towards Multimodal Instruction Following
paper_summary: The Instruction Following (IF) ability measures how well Multi-modal Large
  Language Models (MLLMs) understand exactly what users are telling them and
  whether they are doing it right. Existing multimodal instruction following
  training data is scarce, the benchmarks are simple with atomic instructions,
  and the evaluation strategies are imprecise for tasks demanding exact output
  constraints. To address this, we present MM-IFEngine, an effective pipeline to
  generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields
  large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which
  is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for
  Direct Preference Optimization (DPO). We further introduce MM-IFEval, a
  challenging and diverse multi-modal instruction-following benchmark that
  includes (1) both compose-level constraints for output responses and
  perception-level constraints tied to the input images, and (2) a comprehensive
  evaluation pipeline incorporating both rule-based assessment and judge model.
  We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on
  MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF
  benchmarks, such as MM-IFEval (+10.2%), MIA (+7.6%), and IFEval
  (+12.3%). The full data and evaluation code will be released on
  https://github.com/SYuan03/MM-IFEngine.
paper_ai_summary: MM-IFEngine generates high-quality image-instruction pairs for training Multi-modal Large Language Models, leading to improved performance in instruction-following tasks.
paper_ai_keywords: Multi-modal Large Language Models MM-IFEngine MM-IFInstruct-23k MM-IFDPO-23k Supervised Fine-Tuning Direct Preference Optimization MM-IFEval instruction-following benchmark compose-level constraints perception-level constraints rule-based assessment judge model