title: Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
summary: We propose a generative model that, given a coarsely edited image,
  synthesizes a photorealistic output that follows the prescribed layout. Our
  method transfers fine details from the original image and preserves the
  identity of its parts. Yet, it adapts it to the lighting and context defined by
  the new layout. Our key insight is that videos are a powerful source of
  supervision for this task: objects and camera motions provide many observations
  of how the world changes with viewpoint, lighting, and physical interactions.
  We construct an image dataset in which each sample is a pair of source and
  target frames extracted from the same video at randomly chosen time intervals.
  We warp the source frame toward the target using two motion models that mimic
  the expected test-time user edits. We supervise our model to translate the
  warped image into the ground truth, starting from a pretrained diffusion model.
  Our model design explicitly enables fine detail transfer from the source frame
  to the generated image, while closely following the user-specified layout. We
  show that by using simple segmentations and coarse 2D manipulations, we can
  synthesize a photorealistic edit faithful to the user's input while addressing
  second-order effects like harmonizing the lighting and physical interactions
  between edited objects.
paper_authors._id: 65fba1cfbeb503364b7ccc83
paper_authors.name: Hadi Alzayer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:46:11.494Z
paper_authors._id: 648d554fba166fc822582b14
paper_authors.avatarUrl: /avatars/93072cde421c9d7d25614f967a9aa45a.svg
paper_authors.fullname: Hadi Alzayer
paper_authors.type: user
paper_authors.user: HadiZayer
paper_authors._id: 65fba1cfbeb503364b7ccc84
paper_authors.name: Zhihao Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:46:24.364Z
paper_authors._id: 64df0d5a1961da512204db2c
paper_authors.avatarUrl: /avatars/e08625382ba8244b72d2e94a11ff325a.svg
paper_authors.fullname: xiazhihao
paper_authors.type: user
paper_authors.user: xiazhihao
paper_authors._id: 65fba1cfbeb503364b7ccc85
paper_authors.name: Xuaner Zhang
paper_authors._id: 65fba1cfbeb503364b7ccc86
paper_authors.name: Eli Shechtman
paper_authors._id: 65fba1cfbeb503364b7ccc87
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:46:44.108Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_authors._id: 65fba1cfbeb503364b7ccc88
paper_authors.name: Michael Gharbi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T11:46:49.904Z
paper_authors._id: 658161ad208369a9403ca9a1
paper_authors.avatarUrl: /avatars/e2324ab2a712100681b0b8127305212b.svg
paper_authors.fullname: Michael Gharbi
paper_authors.type: user
paper_authors.user: mgharbi
paper_title: Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
paper_summary: We propose a generative model that, given a coarsely edited image,
  synthesizes a photorealistic output that follows the prescribed layout. Our
  method transfers fine details from the original image and preserves the
  identity of its parts. Yet, it adapts it to the lighting and context defined by
  the new layout. Our key insight is that videos are a powerful source of
  supervision for this task: objects and camera motions provide many observations
  of how the world changes with viewpoint, lighting, and physical interactions.
  We construct an image dataset in which each sample is a pair of source and
  target frames extracted from the same video at randomly chosen time intervals.
  We warp the source frame toward the target using two motion models that mimic
  the expected test-time user edits. We supervise our model to translate the
  warped image into the ground truth, starting from a pretrained diffusion model.
  Our model design explicitly enables fine detail transfer from the source frame
  to the generated image, while closely following the user-specified layout. We
  show that by using simple segmentations and coarse 2D manipulations, we can
  synthesize a photorealistic edit faithful to the user's input while addressing
  second-order effects like harmonizing the lighting and physical interactions
  between edited objects.
paper_ai_summary: A generative model utilizes video-based supervision to synthesize photorealistic edits from coarse layouts, transferring details and harmonizing lighting and interactions.
paper_ai_keywords: generative model photorealistic output prescribed layout fine details identity preservation lighting context motion models image dataset source frames target frames pretrained diffusion model fine detail transfer user-specified layout segmentation 2D manipulations second-order effects harmonizing lighting physical interactions