ReVideo: Remake a Video with Motion and Content Control
Despite significant advancements in video generation and editing using
diffusion models, achieving accurate and localized video editing remains a
substantial challenge. Additionally, most existing video editing methods
primarily focus on altering visual content, with limited research dedicated to
motion editing. In this paper, we present a novel attempt to Remake a Video
(ReVideo) which stands out from existing methods by allowing precise video
editing in specific areas through the specification of both content and motion.
Content editing is facilitated by modifying the first frame, while the
trajectory-based motion control offers an intuitive user interaction
experience. ReVideo addresses a new task involving the coupling and training
imbalance between content and motion control. To tackle this, we develop a
three-stage training strategy that progressively decouples these two aspects
from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion
module to integrate content and motion control across various sampling steps
and spatial locations. Extensive experiments demonstrate that our ReVideo has
promising performance on several accurate video editing applications, i.e., (1)
locally changing video content while keeping the motion constant, (2) keeping
content unchanged and customizing new motion trajectories, (3) modifying both
content and motion trajectories. Our method can also seamlessly extend these
applications to multi-area editing without specific training, demonstrating its
flexibility and robustness.
664ff85b7ef3fd6ce2fc0292
Chong Mou
admin_assigned
2024-05-24T10:27:47.312Z
63ec78f4c81b6a52391e1291
/avatars/d13b35964c895f7aec69287390036a79.svg
ChongMou
user
Adapter
664ff85b7ef3fd6ce2fc0293
Mingdeng Cao
admin_assigned
2024-05-24T10:26:12.188Z
6374a02d0856ac905bfc6113
/avatars/2cbe75c9cc818a647ca6e416f129c96f.svg
Mingdeng Cao
user
Ljzycmd
664ff85b7ef3fd6ce2fc0294
Xintao Wang
admin_assigned
2024-05-24T10:26:18.902Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
664ff85b7ef3fd6ce2fc0295
Zhaoyang Zhang
admin_assigned
2024-05-24T10:26:53.594Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
664ff85b7ef3fd6ce2fc0296
Ying Shan
664ff85b7ef3fd6ce2fc0297
Jian Zhang
ReVideo: Remake a Video with Motion and Content Control
Despite significant advancements in video generation and editing using
diffusion models, achieving accurate and localized video editing remains a
substantial challenge. Additionally, most existing video editing methods
primarily focus on altering visual content, with limited research dedicated to
motion editing. In this paper, we present a novel attempt to Remake a Video
(ReVideo) which stands out from existing methods by allowing precise video
editing in specific areas through the specification of both content and motion.
Content editing is facilitated by modifying the first frame, while the
trajectory-based motion control offers an intuitive user interaction
experience. ReVideo addresses a new task involving the coupling and training
imbalance between content and motion control. To tackle this, we develop a
three-stage training strategy that progressively decouples these two aspects
from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion
module to integrate content and motion control across various sampling steps
and spatial locations. Extensive experiments demonstrate that our ReVideo has
promising performance on several accurate video editing applications, i.e., (1)
locally changing video content while keeping the motion constant, (2) keeping
content unchanged and customizing new motion trajectories, (3) modifying both
content and motion trajectories. Our method can also seamlessly extend these
applications to multi-area editing without specific training, demonstrating its
flexibility and robustness.
ReVideo is a novel approach to video editing that precisely modifies both content and motion through a three-stage training strategy and spatiotemporal adaptive fusion module.
diffusion models video editing content editing motion editing trajectory-based motion control three-stage training strategy spatiotemporal adaptive fusion module