title: NaRCan: Natural Refined Canonical Image with Integration of Diffusion
    Prior for Video Editing
summary: We propose a video editing framework, NaRCan, which integrates a hybrid
  deformation field and diffusion prior to generate high-quality natural
  canonical images to represent the input video. Our approach utilizes homography
  to model global motion and employs multi-layer perceptrons (MLPs) to capture
  local residual deformations, enhancing the model's ability to handle complex
  video dynamics. By introducing a diffusion prior from the early stages of
  training, our model ensures that the generated images retain a high-quality
  natural appearance, making the produced canonical images suitable for various
  downstream tasks in video editing, a capability not achieved by current
  canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
  fine-tuning and introduce a noise and diffusion prior update scheduling
  technique that accelerates the training process by 14 times. Extensive
  experimental results show that our method outperforms existing approaches in
  various video editing tasks and produces coherent and high-quality edited video
  sequences. See our project page for video results at
  https://koi953215.github.io/NaRCan_page/.
paper_authors._id: 6669defe69a08ea4aa8d66da
paper_authors.name: Ting-Hsuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:26.100Z
paper_authors._id: 65d78eb72a17b13e8fa32aa6
paper_authors.avatarUrl: /avatars/1c7d598d67dc0e233d65dac6d9260e1c.svg
paper_authors.fullname: Ting-Hsuan Chen
paper_authors.type: user
paper_authors.user: Koi953215
paper_authors._id: 6669defe69a08ea4aa8d66db
paper_authors.name: Jiewen Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T12:01:46.784Z
paper_authors._id: 6630a67d6b3957f5983658b1
paper_authors.avatarUrl: /avatars/4cc0c264e0d8cc0ebc346bb9b1abe8de.svg
paper_authors.fullname: Chan Jiewen
paper_authors.type: user
paper_authors.user: JiewenChan
paper_authors._id: 6669defe69a08ea4aa8d66dc
paper_authors.name: Hau-Shiang Shiu
paper_authors._id: 6669defe69a08ea4aa8d66dd
paper_authors.name: Shih-Han Yen
paper_authors._id: 6669defe69a08ea4aa8d66de
paper_authors.name: Chang-Han Yeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T11:09:17.840Z
paper_authors._id: 65db54a31ed8dc6d82837094
paper_authors.avatarUrl: /avatars/668ad7105f6e3c71bbe3ff5c75e2860e.svg
paper_authors.fullname: Yeh, Chang-Han
paper_authors.type: user
paper_authors.user: jimmycv07
paper_authors._id: 6669defe69a08ea4aa8d66df
paper_authors.name: Yu-Lun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:23.917Z
paper_authors._id: 6459d5da3b6fafd9664807ab
paper_authors.avatarUrl: /avatars/57430d1bbde3a2fe5586e5fbcafb0e74.svg
paper_authors.fullname: Yu-Lun Liu
paper_authors.type: user
paper_authors.user: yulunliu
paper_title: NaRCan: Natural Refined Canonical Image with Integration of Diffusion
    Prior for Video Editing
paper_summary: We propose a video editing framework, NaRCan, which integrates a hybrid
  deformation field and diffusion prior to generate high-quality natural
  canonical images to represent the input video. Our approach utilizes homography
  to model global motion and employs multi-layer perceptrons (MLPs) to capture
  local residual deformations, enhancing the model's ability to handle complex
  video dynamics. By introducing a diffusion prior from the early stages of
  training, our model ensures that the generated images retain a high-quality
  natural appearance, making the produced canonical images suitable for various
  downstream tasks in video editing, a capability not achieved by current
  canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
  fine-tuning and introduce a noise and diffusion prior update scheduling
  technique that accelerates the training process by 14 times. Extensive
  experimental results show that our method outperforms existing approaches in
  various video editing tasks and produces coherent and high-quality edited video
  sequences. See our project page for video results at
  https://koi953215.github.io/NaRCan_page/.
paper_ai_summary: A video editing framework combining deformation fields and diffusion priors produces high-quality canonical images for diverse video editing tasks with accelerated training using low-rank adaptation.
paper_ai_keywords: deformation field diffusion prior homography multi-layer perceptrons low-rank adaptation noise and diffusion prior update scheduling