title: FramePainter: Endowing Interactive Image Editing with Video Diffusion
    Priors
summary: Interactive image editing allows users to modify images through visual
  interaction operations such as drawing, clicking, and dragging. Existing
  methods construct such supervision signals from videos, as they capture how
  objects change with various physical interactions. However, these models are
  usually built upon text-to-image diffusion models, so necessitate (i) massive
  training samples and (ii) an additional reference encoder to learn real-world
  dynamics and visual consistency. In this paper, we reformulate this task as an
  image-to-video generation problem, so that inherit powerful video diffusion
  priors to reduce training costs and ensure temporal consistency. Specifically,
  we introduce FramePainter as an efficient instantiation of this formulation.
  Initialized with Stable Video Diffusion, it only uses a lightweight sparse
  control encoder to inject editing signals. Considering the limitations of
  temporal attention in handling large motion between two frames, we further
  propose matching attention to enlarge the receptive field while encouraging
  dense correspondence between edited and source image tokens. We highlight the
  effectiveness and efficiency of FramePainter across various of editing signals:
  it domainantly outperforms previous state-of-the-art methods with far less
  training data, achieving highly seamless and coherent editing of images, \eg,
  automatically adjust the reflection of the cup. Moreover, FramePainter also
  exhibits exceptional generalization in scenarios not present in real-world
  videos, \eg, transform the clownfish into shark-like shape. Our code will be
  available at https://github.com/YBYBZhang/FramePainter.
paper_authors._id: 67872d3659aa2a284c52ad5f
paper_authors.name: Yabo Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:48:10.703Z
paper_authors._id: 62cd752ba9be5c19555c2b4c
paper_authors.avatarUrl: /avatars/ed72684f0f2139516ccde24cd467cea6.svg
paper_authors.fullname: YaboZhang
paper_authors.type: user
paper_authors.user: Yabo
paper_authors._id: 67872d3659aa2a284c52ad60
paper_authors.name: Xinpeng Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:48:17.107Z
paper_authors._id: 66a7044a315d9b5c32d2a399
paper_authors.avatarUrl: /avatars/e2da8af4a4f758aa5de7bf34b085f764.svg
paper_authors.fullname: zhou
paper_authors.type: user
paper_authors.user: xinpengzhou
paper_authors._id: 67872d3659aa2a284c52ad61
paper_authors.name: Yihan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:48:26.078Z
paper_authors._id: 65beb318358734fd090f742e
paper_authors.avatarUrl: /avatars/962420a21b2aa8689dd0e1d4531fbf35.svg
paper_authors.fullname: Yihan Zeng
paper_authors.type: user
paper_authors.user: vikyzeng2
paper_authors._id: 67872d3659aa2a284c52ad62
paper_authors.name: Hang Xu
paper_authors._id: 67872d3659aa2a284c52ad63
paper_authors.name: Hui Li
paper_authors._id: 67872d3659aa2a284c52ad64
paper_authors.name: Wangmeng Zuo
paper_title: FramePainter: Endowing Interactive Image Editing with Video Diffusion
    Priors
paper_summary: Interactive image editing allows users to modify images through visual
  interaction operations such as drawing, clicking, and dragging. Existing
  methods construct such supervision signals from videos, as they capture how
  objects change with various physical interactions. However, these models are
  usually built upon text-to-image diffusion models, so necessitate (i) massive
  training samples and (ii) an additional reference encoder to learn real-world
  dynamics and visual consistency. In this paper, we reformulate this task as an
  image-to-video generation problem, so that inherit powerful video diffusion
  priors to reduce training costs and ensure temporal consistency. Specifically,
  we introduce FramePainter as an efficient instantiation of this formulation.
  Initialized with Stable Video Diffusion, it only uses a lightweight sparse
  control encoder to inject editing signals. Considering the limitations of
  temporal attention in handling large motion between two frames, we further
  propose matching attention to enlarge the receptive field while encouraging
  dense correspondence between edited and source image tokens. We highlight the
  effectiveness and efficiency of FramePainter across various of editing signals:
  it domainantly outperforms previous state-of-the-art methods with far less
  training data, achieving highly seamless and coherent editing of images, \eg,
  automatically adjust the reflection of the cup. Moreover, FramePainter also
  exhibits exceptional generalization in scenarios not present in real-world
  videos, \eg, transform the clownfish into shark-like shape. Our code will be
  available at https://github.com/YBYBZhang/FramePainter.
paper_ai_summary: FramePainter, an image-to-video generation model, efficiently handles interactive image editing with sparse control and matching attention for seamless and coherent edits.
paper_ai_keywords: image-to-video generation video diffusion priors FramePainter Stable Video Diffusion sparse control encoder matching attention temporal attention dense correspondence image editing real-world dynamics visual consistency