title: Learning Action and Reasoning-Centric Image Editing from Videos and
    Simulations
summary: An image editing model should be able to perform diverse edits, ranging from
  object replacement, changing attributes or style, to performing actions or
  movement, which require many forms of reasoning. Current general
  instruction-guided editing models have significant shortcomings with action and
  reasoning-centric edits. Object, attribute or stylistic changes can be learned
  from visually static datasets. On the other hand, high-quality data for action
  and reasoning-centric edits is scarce and has to come from entirely different
  sources that cover e.g. physical dynamics, temporality and spatial reasoning.
  To this end, we meticulously curate the AURORA Dataset
  (Action-Reasoning-Object-Attribute), a collection of high-quality training
  data, human-annotated and curated from videos and simulation engines. We focus
  on a key aspect of quality training data: triplets (source image, prompt,
  target image) contain a single meaningful visual change described by the
  prompt, i.e., truly minimal changes between source and target images. To
  demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
  a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
  Our model significantly outperforms previous editing models as judged by human
  raters. For automatic evaluations, we find important flaws in previous metrics
  and caution their use for semantically hard editing tasks. Instead, we propose
  a new automatic metric that focuses on discriminative understanding. We hope
  that our efforts : (1) curating a quality training dataset and an evaluation
  benchmark, (2) developing critical evaluations, and (3) releasing a
  state-of-the-art model, will fuel further progress on general image editing.
paper_authors._id: 668d7a65963fdbcecc61f8fd
paper_authors.name: Benno Krojer
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-09T20:17:59.541Z
paper_authors._id: 6270c58780d5f35f8dbe42be
paper_authors.avatarUrl: /avatars/d4d6e5eadfe9b1f47bce1c66728b24fc.svg
paper_authors.fullname: Benno Krojer
paper_authors.type: user
paper_authors.user: BennoKrojer
paper_authors._id: 668d7a65963fdbcecc61f8fe
paper_authors.name: Dheeraj Vattikonda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:08:30.712Z
paper_authors._id: 65fb7b1c876d4642e8340e95
paper_authors.avatarUrl: /avatars/c258c4528315243b98c1e60ba398e98b.svg
paper_authors.fullname: Dheeraj Vattikonda
paper_authors.type: user
paper_authors.user: Dheeraj46329
paper_authors._id: 668d7a65963fdbcecc61f8ff
paper_authors.name: Luis Lara
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:08:53.780Z
paper_authors._id: 63e972f1ccae1fe5c6211759
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e972f1ccae1fe5c6211759/AfKPgMdAraUtvbtJpoHFY.jpeg
paper_authors.fullname: Luis Lara
paper_authors.type: user
paper_authors.user: ludolara
paper_authors._id: 668d7a65963fdbcecc61f900
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:08:38.230Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_authors._id: 668d7a65963fdbcecc61f901
paper_authors.name: Eva Portelance
paper_authors._id: 668d7a65963fdbcecc61f902
paper_authors.name: Christopher Pal
paper_authors._id: 668d7a65963fdbcecc61f903
paper_authors.name: Siva Reddy
paper_title: Learning Action and Reasoning-Centric Image Editing from Videos and
    Simulations
paper_summary: An image editing model should be able to perform diverse edits, ranging from
  object replacement, changing attributes or style, to performing actions or
  movement, which require many forms of reasoning. Current general
  instruction-guided editing models have significant shortcomings with action and
  reasoning-centric edits. Object, attribute or stylistic changes can be learned
  from visually static datasets. On the other hand, high-quality data for action
  and reasoning-centric edits is scarce and has to come from entirely different
  sources that cover e.g. physical dynamics, temporality and spatial reasoning.
  To this end, we meticulously curate the AURORA Dataset
  (Action-Reasoning-Object-Attribute), a collection of high-quality training
  data, human-annotated and curated from videos and simulation engines. We focus
  on a key aspect of quality training data: triplets (source image, prompt,
  target image) contain a single meaningful visual change described by the
  prompt, i.e., truly minimal changes between source and target images. To
  demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
  a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
  Our model significantly outperforms previous editing models as judged by human
  raters. For automatic evaluations, we find important flaws in previous metrics
  and caution their use for semantically hard editing tasks. Instead, we propose
  a new automatic metric that focuses on discriminative understanding. We hope
  that our efforts : (1) curating a quality training dataset and an evaluation
  benchmark, (2) developing critical evaluations, and (3) releasing a
  state-of-the-art model, will fuel further progress on general image editing.
paper_ai_summary: AURORA Dataset and benchmark improve general image editing by focusing on action, reasoning, and minimal visual changes, outperforming previous models with better automatic metrics.
paper_ai_keywords: AURORA Dataset image editing object replacement attribute changes style changes action edits reasoning edits physical dynamics temporality spatial reasoning AURORA-Bench triplets source image prompt target image human-annotated expert-curated automatic evaluation discriminative understanding