MM-IFEngine: Towards Multimodal Instruction Following
The Instruction Following (IF) ability measures how well Multi-modal Large
Language Models (MLLMs) understand exactly what users are telling them and
whether they are doing it right. Existing multimodal instruction following
training data is scarce, the benchmarks are simple with atomic instructions,
and the evaluation strategies are imprecise for tasks demanding exact output
constraints. To address this, we present MM-IFEngine, an effective pipeline to
generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields
large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which
is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for
Direct Preference Optimization (DPO). We further introduce MM-IFEval, a
challenging and diverse multi-modal instruction-following benchmark that
includes (1) both compose-level constraints for output responses and
perception-level constraints tied to the input images, and (2) a comprehensive
evaluation pipeline incorporating both rule-based assessment and judge model.
We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on
MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF
benchmarks, such as MM-IFEval (+10.2%), MIA (+7.6%), and IFEval
(+12.3%). The full data and evaluation code will be released on
https://github.com/SYuan03/MM-IFEngine.
67f8914516d43f88b3177ec1
Shengyuan Ding
admin_assigned
2025-04-11T08:25:40.694Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
user
ChrisDing1105
67f8914516d43f88b3177ec2
Shenxi Wu
67f8914516d43f88b3177ec3
Xiangyu Zhao
67f8914516d43f88b3177ec4
Yuhang Zang
claimed_verified
2025-04-11T07:25:05.800Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
67f8914516d43f88b3177ec5
Haodong Duan
admin_assigned
2025-04-11T08:26:11.929Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67f8914516d43f88b3177ec6
Xiaoyi Dong
admin_assigned
2025-04-11T08:26:18.423Z
67c0849ee08c178ef8d4e05c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mQ6VdnjZnRhb0H_waPclo.png
Xiaoyi Dong
user
sweetFruit
67f8914516d43f88b3177ec7
Pan Zhang
67f8914516d43f88b3177ec8
Yuhang Cao
admin_assigned
2025-04-11T08:26:51.425Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67f8914516d43f88b3177ec9
Dahua Lin
admin_assigned
2025-04-11T08:26:59.045Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67f8914516d43f88b3177eca
Jiaqi Wang
admin_assigned
2025-04-11T08:27:20.727Z
64638c4d51fa6e63060521b5
/avatars/c863ace5b1dc788a341bcf4ddbdfaec1.svg
JIaqi
user
Jiaqiwang
MM-IFEngine: Towards Multimodal Instruction Following
The Instruction Following (IF) ability measures how well Multi-modal Large
Language Models (MLLMs) understand exactly what users are telling them and
whether they are doing it right. Existing multimodal instruction following
training data is scarce, the benchmarks are simple with atomic instructions,
and the evaluation strategies are imprecise for tasks demanding exact output
constraints. To address this, we present MM-IFEngine, an effective pipeline to
generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields
large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which
is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for
Direct Preference Optimization (DPO). We further introduce MM-IFEval, a
challenging and diverse multi-modal instruction-following benchmark that
includes (1) both compose-level constraints for output responses and
perception-level constraints tied to the input images, and (2) a comprehensive
evaluation pipeline incorporating both rule-based assessment and judge model.
We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on
MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF
benchmarks, such as MM-IFEval (+10.2%), MIA (+7.6%), and IFEval
(+12.3%). The full data and evaluation code will be released on
https://github.com/SYuan03/MM-IFEngine.
MM-IFEngine generates high-quality image-instruction pairs for training Multi-modal Large Language Models, leading to improved performance in instruction-following tasks.
Multi-modal Large Language Models MM-IFEngine MM-IFInstruct-23k MM-IFDPO-23k Supervised Fine-Tuning Direct Preference Optimization MM-IFEval instruction-following benchmark compose-level constraints perception-level constraints rule-based assessment judge model