Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
In this paper, we introduce Fairy, a minimalist yet robust adaptation of
image-editing diffusion models, enhancing them for video editing applications.
Our approach centers on the concept of anchor-based cross-frame attention, a
mechanism that implicitly propagates diffusion features across frames, ensuring
superior temporal coherence and high-fidelity synthesis. Fairy not only
addresses limitations of previous models, including memory and processing
speed. It also improves temporal consistency through a unique data augmentation
strategy. This strategy renders the model equivariant to affine transformations
in both source and target images. Remarkably efficient, Fairy generates
120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds,
outpacing prior works by at least 44x. A comprehensive user study, involving
1000 generated samples, confirms that our approach delivers superior quality,
decisively outperforming established methods.
6584f1bfebe2be20806e3e51
Bichen Wu
admin_assigned
2023-12-22T08:25:31.748Z
6363749520c2cf37bccd9e84
/avatars/e9de7911e67090d883f15a7a58f18e4e.svg
Bichen Wu
user
bichenwu
6584f1bfebe2be20806e3e52
Ching-Yao Chuang
6584f1bfebe2be20806e3e53
Xiaoyan Wang
admin_assigned
2023-12-22T08:28:22.573Z
63a50790e3623e66842f5fc8
/avatars/4cc919b7b9fd393595599b4897a75e05.svg
Xiaoyan Wang
user
Horizonian
6584f1bfebe2be20806e3e54
Yichen Jia
6584f1bfebe2be20806e3e55
Kapil Krishnakumar
6584f1bfebe2be20806e3e56
Tong Xiao
6584f1bfebe2be20806e3e57
Feng Liang
claimed_verified
2024-01-02T09:27:37.952Z
63631be58f43a912fc7595d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
Jeff Liang
user
JeffLiang
6584f1bfebe2be20806e3e58
Licheng Yu
admin_assigned
2023-12-22T08:26:55.357Z
6372b1f9bd81fae2b3a712e8
/avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
Licheng Yu
user
lichengyu
6584f1bfebe2be20806e3e59
Peter Vajda
admin_assigned
2023-12-22T11:02:07.614Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
In this paper, we introduce Fairy, a minimalist yet robust adaptation of
image-editing diffusion models, enhancing them for video editing applications.
Our approach centers on the concept of anchor-based cross-frame attention, a
mechanism that implicitly propagates diffusion features across frames, ensuring
superior temporal coherence and high-fidelity synthesis. Fairy not only
addresses limitations of previous models, including memory and processing
speed. It also improves temporal consistency through a unique data augmentation
strategy. This strategy renders the model equivariant to affine transformations
in both source and target images. Remarkably efficient, Fairy generates
120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds,
outpacing prior works by at least 44x. A comprehensive user study, involving
1000 generated samples, confirms that our approach delivers superior quality,
decisively outperforming established methods.
Fairy, an adaptation of image-editing diffusion models for video editing, uses anchor-based cross-frame attention to enhance temporal coherence and efficiency, outperforming existing methods.
diffusion models anchor-based cross-frame attention temporal coherence high-fidelity synthesis temporal consistency data augmentation equivariant affine transformations