Trajectory Attention for Fine-grained Video Motion Control
Recent advancements in video generation have been greatly driven by video
diffusion models, with camera motion control emerging as a crucial challenge in
creating view-customized visual content. This paper introduces trajectory
attention, a novel approach that performs attention along available pixel
trajectories for fine-grained camera motion control. Unlike existing methods
that often yield imprecise outputs or neglect temporal correlations, our
approach possesses a stronger inductive bias that seamlessly injects trajectory
information into the video generation process. Importantly, our approach models
trajectory attention as an auxiliary branch alongside traditional temporal
attention. This design enables the original temporal attention and the
trajectory attention to work in synergy, ensuring both precise motion control
and new content generation capability, which is critical when the trajectory is
only partially available. Experiments on camera motion control for images and
videos demonstrate significant improvements in precision and long-range
consistency while maintaining high-quality generation. Furthermore, we show
that our approach can be extended to other video motion control tasks, such as
first-frame-guided video editing, where it excels in maintaining content
consistency over large spatial and temporal ranges.
674d3226810801f2fa57b8ed
Zeqi Xiao
admin_assigned
2024-12-02T09:39:04.354Z
650e37cc11f3210cf7910501
/avatars/dab5c9d647cfa97c59f5170216673a20.svg
zeqixiao
user
zeqixiao
674d3226810801f2fa57b8ee
Wenqi Ouyang
admin_assigned
2024-12-02T09:38:53.298Z
6487d9415d164d8c7f905bcf
/avatars/1c01dd4b88f88a9863a12c09a9a28bed.svg
Wenqi Ouyang
user
Vicky0522
674d3226810801f2fa57b8ef
Yifan Zhou
claimed_verified
2024-12-02T12:52:33.458Z
64a63f9449b08110f761cd73
/avatars/61860202fc818b105ef24e74dd4f7d3c.svg
Yifan Zhou
user
SingleZombie
674d3226810801f2fa57b8f0
Shuai Yang
admin_assigned
2024-12-02T09:38:33.890Z
62a54d0410334c1d024e2f59
https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
Shuai Yang
user
PKUWilliamYang
674d3226810801f2fa57b8f1
Lei Yang
claimed_verified
2024-12-02T12:52:31.720Z
6626a471430a124253f197c8
/avatars/f5747fdbe495d1296fed9d16d8c95857.svg
yl-1993
user
yl-1993
674d3226810801f2fa57b8f2
Jianlou Si
admin_assigned
2024-12-02T09:38:05.905Z
6409693abf3e9a4bb2e64dfd
/avatars/9ddeceec9f4dae3a813165016eb6f4d3.svg
Jianlou
user
Jianlou
674d3226810801f2fa57b8f3
Xingang Pan
admin_assigned
2024-12-02T09:37:47.316Z
646758d6a9b4610868a138ff
https://cdn-avatars.huggingface.co/v1/production/uploads/646758d6a9b4610868a138ff/fNBlK5g4g7vlPAR1bQimZ.png
Xingang Pan
user
Xingang-Pan
Trajectory Attention for Fine-grained Video Motion Control
Recent advancements in video generation have been greatly driven by video
diffusion models, with camera motion control emerging as a crucial challenge in
creating view-customized visual content. This paper introduces trajectory
attention, a novel approach that performs attention along available pixel
trajectories for fine-grained camera motion control. Unlike existing methods
that often yield imprecise outputs or neglect temporal correlations, our
approach possesses a stronger inductive bias that seamlessly injects trajectory
information into the video generation process. Importantly, our approach models
trajectory attention as an auxiliary branch alongside traditional temporal
attention. This design enables the original temporal attention and the
trajectory attention to work in synergy, ensuring both precise motion control
and new content generation capability, which is critical when the trajectory is
only partially available. Experiments on camera motion control for images and
videos demonstrate significant improvements in precision and long-range
consistency while maintaining high-quality generation. Furthermore, we show
that our approach can be extended to other video motion control tasks, such as
first-frame-guided video editing, where it excels in maintaining content
consistency over large spatial and temporal ranges.
Trajectory attention is introduced to enhance video generation by incorporating motion control along pixel trajectories, improving precision and consistency.
video diffusion models camera motion control trajectory attention temporal attention first-frame-guided video editing