GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing
Editing images using natural language instructions has become a natural and
expressive way to modify visual content; yet, evaluating the performance of
such models remains challenging. Existing evaluation approaches often rely on
image-text similarity metrics like CLIP, which lack precision. In this work, we
introduce a new benchmark designed to evaluate text-guided image editing models
in a more grounded manner, along two critical dimensions: (i) functional
correctness, assessed via automatically generated multiple-choice questions
that verify whether the intended change was successfully applied; and (ii)
image content preservation, which ensures that non-targeted regions of the
image remain visually consistent using an object-aware masking technique and
preservation scoring. The benchmark includes over 1000 high-quality editing
examples across 20 diverse content categories, each annotated with detailed
editing instructions, evaluation questions, and spatial object masks. We
conduct a large-scale study comparing GPT-Image-1, the latest flagship in the
text-guided image editing space, against several state-of-the-art editing
models, and validate our automatic metrics against human ratings. Results show
that GPT-Image-1 leads in instruction-following accuracy, but often
over-modifies irrelevant image regions, highlighting a key trade-off in the
current model behavior. GIE-Bench provides a scalable, reproducible framework
for advancing more accurate evaluation of text-guided image editing.
682b70a7ebc1db7ecb14ee63
Yusu Qian
claimed_verified
2025-06-12T12:44:03.646Z
65d5d6e590119106db895670
https://cdn-avatars.huggingface.co/v1/production/uploads/65d5d6e590119106db895670/SphTxh2E0MtT8T_bH0_zb.png
Yusu Qian
user
YusuQian
682b70a7ebc1db7ecb14ee64
Jiasen Lu
682b70a7ebc1db7ecb14ee65
Tsu-Jui Fu
682b70a7ebc1db7ecb14ee66
Xinze Wang
682b70a7ebc1db7ecb14ee67
Chen Chen
682b70a7ebc1db7ecb14ee68
Yinfei Yang
682b70a7ebc1db7ecb14ee69
Wenze Hu
682b70a7ebc1db7ecb14ee6a
Zhe Gan
GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing
Editing images using natural language instructions has become a natural and
expressive way to modify visual content; yet, evaluating the performance of
such models remains challenging. Existing evaluation approaches often rely on
image-text similarity metrics like CLIP, which lack precision. In this work, we
introduce a new benchmark designed to evaluate text-guided image editing models
in a more grounded manner, along two critical dimensions: (i) functional
correctness, assessed via automatically generated multiple-choice questions
that verify whether the intended change was successfully applied; and (ii)
image content preservation, which ensures that non-targeted regions of the
image remain visually consistent using an object-aware masking technique and
preservation scoring. The benchmark includes over 1000 high-quality editing
examples across 20 diverse content categories, each annotated with detailed
editing instructions, evaluation questions, and spatial object masks. We
conduct a large-scale study comparing GPT-Image-1, the latest flagship in the
text-guided image editing space, against several state-of-the-art editing
models, and validate our automatic metrics against human ratings. Results show
that GPT-Image-1 leads in instruction-following accuracy, but often
over-modifies irrelevant image regions, highlighting a key trade-off in the
current model behavior. GIE-Bench provides a scalable, reproducible framework
for advancing more accurate evaluation of text-guided image editing.
A new benchmark evaluates text-guided image editing models for functional correctness and image content preservation using automatic metrics and human ratings.
Multiple-choice questions object-aware masking preservation scoring GPT-Image-1 instruction-following accuracy