title: Complex-Edit: CoT-Like Instruction Generation for
    Complexity-Controllable Image Editing Benchmark
summary: We introduce Complex-Edit, a comprehensive benchmark designed to
  systematically evaluate instruction-based image editing models across
  instructions of varying complexity. To develop this benchmark, we harness
  GPT-4o to automatically collect a diverse set of editing instructions at scale.
  Our approach follows a well-structured ``Chain-of-Edit'' pipeline: we first
  generate individual atomic editing tasks independently and then integrate them
  to form cohesive, complex instructions. Additionally, we introduce a suite of
  metrics to assess various aspects of editing performance, along with a
  VLM-based auto-evaluation pipeline that supports large-scale assessments. Our
  benchmark yields several notable insights: 1) Open-source models significantly
  underperform relative to proprietary, closed-source models, with the
  performance gap widening as instruction complexity increases; 2) Increased
  instructional complexity primarily impairs the models' ability to retain key
  elements from the input images and to preserve the overall aesthetic quality;
  3) Decomposing a complex instruction into a sequence of atomic steps, executed
  in a step-by-step manner, substantially degrades performance across multiple
  metrics; 4) A straightforward Best-of-N selection strategy improves results for
  both direct editing and the step-by-step sequential approach; and 5) We observe
  a ``curse of synthetic data'': when synthetic data is involved in model
  training, the edited images from such models tend to appear increasingly
  synthetic as the complexity of the editing instructions rises -- a phenomenon
  that intriguingly also manifests in the latest GPT-4o outputs.
paper_authors._id: 6802d28a5c4268810bdd148e
paper_authors.name: Siwei Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:13:17.157Z
paper_authors._id: 6433fd1095b8ab049385d2b3
paper_authors.avatarUrl: /avatars/718117cd4631b7c297add7e559032b2f.svg
paper_authors.fullname: Siwei Yang
paper_authors.type: user
paper_authors.user: LAOS-Y
paper_authors._id: 6802d28a5c4268810bdd148f
paper_authors.name: Mude Hui
paper_authors._id: 6802d28a5c4268810bdd1490
paper_authors.name: Bingchen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:15.504Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 6802d28a5c4268810bdd1491
paper_authors.name: Yuyin Zhou
paper_authors._id: 6802d28a5c4268810bdd1492
paper_authors.name: Nataniel Ruiz
paper_authors._id: 6802d28a5c4268810bdd1493
paper_authors.name: Cihang Xie
paper_title: Complex-Edit: CoT-Like Instruction Generation for
    Complexity-Controllable Image Editing Benchmark
paper_summary: We introduce Complex-Edit, a comprehensive benchmark designed to
  systematically evaluate instruction-based image editing models across
  instructions of varying complexity. To develop this benchmark, we harness
  GPT-4o to automatically collect a diverse set of editing instructions at scale.
  Our approach follows a well-structured ``Chain-of-Edit'' pipeline: we first
  generate individual atomic editing tasks independently and then integrate them
  to form cohesive, complex instructions. Additionally, we introduce a suite of
  metrics to assess various aspects of editing performance, along with a
  VLM-based auto-evaluation pipeline that supports large-scale assessments. Our
  benchmark yields several notable insights: 1) Open-source models significantly
  underperform relative to proprietary, closed-source models, with the
  performance gap widening as instruction complexity increases; 2) Increased
  instructional complexity primarily impairs the models' ability to retain key
  elements from the input images and to preserve the overall aesthetic quality;
  3) Decomposing a complex instruction into a sequence of atomic steps, executed
  in a step-by-step manner, substantially degrades performance across multiple
  metrics; 4) A straightforward Best-of-N selection strategy improves results for
  both direct editing and the step-by-step sequential approach; and 5) We observe
  a ``curse of synthetic data'': when synthetic data is involved in model
  training, the edited images from such models tend to appear increasingly
  synthetic as the complexity of the editing instructions rises -- a phenomenon
  that intriguingly also manifests in the latest GPT-4o outputs.
paper_ai_summary: A benchmark using GPT-4o evaluates instruction-based image editing models across varying complexities, revealing performance gaps, retention issues, and synthetic data challenges.
paper_ai_keywords: GPT-4o Chain-of-Edit VLM-based auto-evaluation Best-of-N selection strategy