title: Zero-shot Image Editing with Reference Imitation
summary: Image editing serves as a practical yet challenging task considering the
  diverse demands from users, where one of the hardest parts is to precisely
  describe how the edited image should look like. In this work, we present a new
  form of editing, termed imitative editing, to help users exercise their
  creativity more conveniently. Concretely, to edit an image region of interest,
  users are free to directly draw inspiration from some in-the-wild references
  (e.g., some relative pictures come across online), without having to cope with
  the fit between the reference and the source. Such a design requires the system
  to automatically figure out what to expect from the reference to perform the
  editing. For this purpose, we propose a generative training framework, dubbed
  MimicBrush, which randomly selects two frames from a video clip, masks some
  regions of one frame, and learns to recover the masked regions using the
  information from the other frame. That way, our model, developed from a
  diffusion prior, is able to capture the semantic correspondence between
  separate images in a self-supervised manner. We experimentally show the
  effectiveness of our method under various test cases as well as its superiority
  over existing alternatives. We also construct a benchmark to facilitate further
  research.
paper_authors._id: 6669102952faf47d18f0e07e
paper_authors.name: Xi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:29.878Z
paper_authors._id: 644a1b6401e18bf93a6f45c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
paper_authors.fullname: xichen
paper_authors.type: user
paper_authors.user: xichenhku
paper_authors._id: 6669102952faf47d18f0e07f
paper_authors.name: Yutong Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:47:03.236Z
paper_authors._id: 64a54e468cfaa458bd6844bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
paper_authors.fullname: Yutong Feng
paper_authors.type: user
paper_authors.user: fengyutong
paper_authors._id: 6669102952faf47d18f0e080
paper_authors.name: Mengting Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:47:46.418Z
paper_authors._id: 641073eb928400b416444af5
paper_authors.avatarUrl: /avatars/0eed14a8929682f182e474f3693fc442.svg
paper_authors.fullname: Chenmengting
paper_authors.type: user
paper_authors.user: MengTingChen
paper_authors._id: 6669102952faf47d18f0e081
paper_authors.name: Yiyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:47:53.032Z
paper_authors._id: 65664c8fcc78541029e4f4ac
paper_authors.avatarUrl: /avatars/7182aa41dd4c7f74f7b566506543d33a.svg
paper_authors.fullname: Yiyang Wang
paper_authors.type: user
paper_authors.user: dwyy
paper_authors._id: 6669102952faf47d18f0e082
paper_authors.name: Shilong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:47:59.760Z
paper_authors._id: 6424ffce46d202ad3d918a67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
paper_authors.fullname: Shilong Zhang
paper_authors.type: user
paper_authors.user: shilongz
paper_authors._id: 6669102952faf47d18f0e083
paper_authors.name: Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:48:29.016Z
paper_authors._id: 636de85cc4a7a729c164d2b5
paper_authors.avatarUrl: /avatars/3e281e547e1697e1c06805e7e63f3918.svg
paper_authors.fullname: Yu Liu
paper_authors.type: user
paper_authors.user: YuLiu
paper_authors._id: 6669102952faf47d18f0e084
paper_authors.name: Yujun Shen
paper_authors._id: 6669102952faf47d18f0e085
paper_authors.name: Hengshuang Zhao
paper_title: Zero-shot Image Editing with Reference Imitation
paper_summary: Image editing serves as a practical yet challenging task considering the
  diverse demands from users, where one of the hardest parts is to precisely
  describe how the edited image should look like. In this work, we present a new
  form of editing, termed imitative editing, to help users exercise their
  creativity more conveniently. Concretely, to edit an image region of interest,
  users are free to directly draw inspiration from some in-the-wild references
  (e.g., some relative pictures come across online), without having to cope with
  the fit between the reference and the source. Such a design requires the system
  to automatically figure out what to expect from the reference to perform the
  editing. For this purpose, we propose a generative training framework, dubbed
  MimicBrush, which randomly selects two frames from a video clip, masks some
  regions of one frame, and learns to recover the masked regions using the
  information from the other frame. That way, our model, developed from a
  diffusion prior, is able to capture the semantic correspondence between
  separate images in a self-supervised manner. We experimentally show the
  effectiveness of our method under various test cases as well as its superiority
  over existing alternatives. We also construct a benchmark to facilitate further
  research.
paper_ai_summary: MimicBrush, a generative training framework developed from a diffusion prior, enables imitative image editing by leveraging video frames and masked regions to capture semantic correspondence and improve editing quality.
paper_ai_keywords: generative training framework MimicBrush diffusion prior video frames masked regions semantic correspondence imitative editing