title: Image Editing As Programs with Diffusion Models
summary: While diffusion models have achieved remarkable success in text-to-image
  generation, they encounter significant challenges with instruction-driven image
  editing. Our research highlights a key challenge: these models particularly
  struggle with structurally inconsistent edits that involve substantial layout
  changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a
  unified image editing framework built upon the Diffusion Transformer (DiT)
  architecture. At its core, IEAP approaches instructional editing through a
  reductionist lens, decomposing complex editing instructions into sequences of
  atomic operations. Each operation is implemented via a lightweight adapter
  sharing the same DiT backbone and is specialized for a specific type of edit.
  Programmed by a vision-language model (VLM)-based agent, these operations
  collaboratively support arbitrary and structurally inconsistent
  transformations. By modularizing and sequencing edits in this way, IEAP
  generalizes robustly across a wide range of editing tasks, from simple
  adjustments to substantial structural changes. Extensive experiments
  demonstrate that IEAP significantly outperforms state-of-the-art methods on
  standard benchmarks across various editing scenarios. In these evaluations, our
  framework delivers superior accuracy and semantic fidelity, particularly for
  complex, multi-step instructions. Codes are available at
  https://github.com/YujiaHu1109/IEAP.
paper_authors._id: 6840fb71d4e16ff5f95108aa
paper_authors.name: Yujia Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:42:05.838Z
paper_authors._id: 67da3534e6387c9c9bc954b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1IiNF_DgWQcvMYIlW8ikC.png
paper_authors.fullname: HU YUJIA
paper_authors.type: user
paper_authors.user: Cicici1109
paper_authors._id: 6840fb71d4e16ff5f95108ab
paper_authors.name: Songhua Liu
paper_authors._id: 6840fb71d4e16ff5f95108ac
paper_authors.name: Zhenxiong Tan
paper_authors._id: 6840fb71d4e16ff5f95108ad
paper_authors.name: Xingyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:27:11.256Z
paper_authors._id: 634cfebc350bcee9bed20a4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
paper_authors.fullname: Xingyi Yang
paper_authors.type: user
paper_authors.user: adamdad
paper_authors._id: 6840fb71d4e16ff5f95108ae
paper_authors.name: Xinchao Wang
paper_title: Image Editing As Programs with Diffusion Models
paper_summary: While diffusion models have achieved remarkable success in text-to-image
  generation, they encounter significant challenges with instruction-driven image
  editing. Our research highlights a key challenge: these models particularly
  struggle with structurally inconsistent edits that involve substantial layout
  changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a
  unified image editing framework built upon the Diffusion Transformer (DiT)
  architecture. At its core, IEAP approaches instructional editing through a
  reductionist lens, decomposing complex editing instructions into sequences of
  atomic operations. Each operation is implemented via a lightweight adapter
  sharing the same DiT backbone and is specialized for a specific type of edit.
  Programmed by a vision-language model (VLM)-based agent, these operations
  collaboratively support arbitrary and structurally inconsistent
  transformations. By modularizing and sequencing edits in this way, IEAP
  generalizes robustly across a wide range of editing tasks, from simple
  adjustments to substantial structural changes. Extensive experiments
  demonstrate that IEAP significantly outperforms state-of-the-art methods on
  standard benchmarks across various editing scenarios. In these evaluations, our
  framework delivers superior accuracy and semantic fidelity, particularly for
  complex, multi-step instructions. Codes are available at
  https://github.com/YujiaHu1109/IEAP.
paper_ai_summary: A unified image editing framework, IEAP, built on Diffusion Transformer (DiT) decomposes complex editing instructions into operations performed by vision-language models for robust editing across various tasks.
paper_ai_keywords: diffusion models text-to-image generation instruction-driven image editing structurally inconsistent edits Image Editing As Programs (IEAP) Diffusion Transformer (DiT) atomic operations lightweight adapter vision-language model (VLM) modularizing edits