NaRCan: Natural Refined Canonical Image with Integration of Diffusion
  Prior for Video Editing
We propose a video editing framework, NaRCan, which integrates a hybrid
deformation field and diffusion prior to generate high-quality natural
canonical images to represent the input video. Our approach utilizes homography
to model global motion and employs multi-layer perceptrons (MLPs) to capture
local residual deformations, enhancing the model's ability to handle complex
video dynamics. By introducing a diffusion prior from the early stages of
training, our model ensures that the generated images retain a high-quality
natural appearance, making the produced canonical images suitable for various
downstream tasks in video editing, a capability not achieved by current
canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
fine-tuning and introduce a noise and diffusion prior update scheduling
technique that accelerates the training process by 14 times. Extensive
experimental results show that our method outperforms existing approaches in
various video editing tasks and produces coherent and high-quality edited video
sequences. See our project page for video results at
https://koi953215.github.io/NaRCan_page/.
6669defe69a08ea4aa8d66da
Ting-Hsuan Chen
claimed_verified
2024-06-13T09:31:26.100Z
65d78eb72a17b13e8fa32aa6
/avatars/1c7d598d67dc0e233d65dac6d9260e1c.svg
Ting-Hsuan Chen
user
Koi953215
6669defe69a08ea4aa8d66db
Jiewen Chan
admin_assigned
2024-06-13T12:01:46.784Z
6630a67d6b3957f5983658b1
/avatars/4cc0c264e0d8cc0ebc346bb9b1abe8de.svg
Chan Jiewen
user
JiewenChan
6669defe69a08ea4aa8d66dc
Hau-Shiang Shiu
6669defe69a08ea4aa8d66dd
Shih-Han Yen
6669defe69a08ea4aa8d66de
Chang-Han Yeh
claimed_verified
2024-06-14T11:09:17.840Z
65db54a31ed8dc6d82837094
/avatars/668ad7105f6e3c71bbe3ff5c75e2860e.svg
Yeh, Chang-Han
user
jimmycv07
6669defe69a08ea4aa8d66df
Yu-Lun Liu
claimed_verified
2024-06-13T09:31:23.917Z
6459d5da3b6fafd9664807ab
/avatars/57430d1bbde3a2fe5586e5fbcafb0e74.svg
Yu-Lun Liu
user
yulunliu
NaRCan: Natural Refined Canonical Image with Integration of Diffusion
  Prior for Video Editing
We propose a video editing framework, NaRCan, which integrates a hybrid
deformation field and diffusion prior to generate high-quality natural
canonical images to represent the input video. Our approach utilizes homography
to model global motion and employs multi-layer perceptrons (MLPs) to capture
local residual deformations, enhancing the model's ability to handle complex
video dynamics. By introducing a diffusion prior from the early stages of
training, our model ensures that the generated images retain a high-quality
natural appearance, making the produced canonical images suitable for various
downstream tasks in video editing, a capability not achieved by current
canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
fine-tuning and introduce a noise and diffusion prior update scheduling
technique that accelerates the training process by 14 times. Extensive
experimental results show that our method outperforms existing approaches in
various video editing tasks and produces coherent and high-quality edited video
sequences. See our project page for video results at
https://koi953215.github.io/NaRCan_page/.
A video editing framework combining deformation fields and diffusion priors produces high-quality canonical images for diverse video editing tasks with accelerated training using low-rank adaptation.
deformation field diffusion prior homography multi-layer perceptrons low-rank adaptation noise and diffusion prior update scheduling