title: GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing
summary: Editing images using natural language instructions has become a natural and
  expressive way to modify visual content; yet, evaluating the performance of
  such models remains challenging. Existing evaluation approaches often rely on
  image-text similarity metrics like CLIP, which lack precision. In this work, we
  introduce a new benchmark designed to evaluate text-guided image editing models
  in a more grounded manner, along two critical dimensions: (i) functional
  correctness, assessed via automatically generated multiple-choice questions
  that verify whether the intended change was successfully applied; and (ii)
  image content preservation, which ensures that non-targeted regions of the
  image remain visually consistent using an object-aware masking technique and
  preservation scoring. The benchmark includes over 1000 high-quality editing
  examples across 20 diverse content categories, each annotated with detailed
  editing instructions, evaluation questions, and spatial object masks. We
  conduct a large-scale study comparing GPT-Image-1, the latest flagship in the
  text-guided image editing space, against several state-of-the-art editing
  models, and validate our automatic metrics against human ratings. Results show
  that GPT-Image-1 leads in instruction-following accuracy, but often
  over-modifies irrelevant image regions, highlighting a key trade-off in the
  current model behavior. GIE-Bench provides a scalable, reproducible framework
  for advancing more accurate evaluation of text-guided image editing.
paper_authors._id: 682b70a7ebc1db7ecb14ee63
paper_authors.name: Yusu Qian
paper_authors._id: 682b70a7ebc1db7ecb14ee64
paper_authors.name: Jiasen Lu
paper_authors._id: 682b70a7ebc1db7ecb14ee65
paper_authors.name: Tsu-Jui Fu
paper_authors._id: 682b70a7ebc1db7ecb14ee66
paper_authors.name: Xinze Wang
paper_authors._id: 682b70a7ebc1db7ecb14ee67
paper_authors.name: Chen Chen
paper_authors._id: 682b70a7ebc1db7ecb14ee68
paper_authors.name: Yinfei Yang
paper_authors._id: 682b70a7ebc1db7ecb14ee69
paper_authors.name: Wenze Hu
paper_authors._id: 682b70a7ebc1db7ecb14ee6a
paper_authors.name: Zhe Gan
paper_title: GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing
paper_summary: Editing images using natural language instructions has become a natural and
  expressive way to modify visual content; yet, evaluating the performance of
  such models remains challenging. Existing evaluation approaches often rely on
  image-text similarity metrics like CLIP, which lack precision. In this work, we
  introduce a new benchmark designed to evaluate text-guided image editing models
  in a more grounded manner, along two critical dimensions: (i) functional
  correctness, assessed via automatically generated multiple-choice questions
  that verify whether the intended change was successfully applied; and (ii)
  image content preservation, which ensures that non-targeted regions of the
  image remain visually consistent using an object-aware masking technique and
  preservation scoring. The benchmark includes over 1000 high-quality editing
  examples across 20 diverse content categories, each annotated with detailed
  editing instructions, evaluation questions, and spatial object masks. We
  conduct a large-scale study comparing GPT-Image-1, the latest flagship in the
  text-guided image editing space, against several state-of-the-art editing
  models, and validate our automatic metrics against human ratings. Results show
  that GPT-Image-1 leads in instruction-following accuracy, but often
  over-modifies irrelevant image regions, highlighting a key trade-off in the
  current model behavior. GIE-Bench provides a scalable, reproducible framework
  for advancing more accurate evaluation of text-guided image editing.
paper_ai_summary: A new benchmark evaluates text-guided image editing models for functional correctness and image content preservation using automatic metrics and human ratings.
paper_ai_keywords: Multiple-choice questions object-aware masking preservation scoring GPT-Image-1 instruction-following accuracy