FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
Diffusion model has demonstrated remarkable capability in video generation,
which further sparks interest in introducing trajectory control into the
generation process. While existing works mainly focus on training-based methods
(e.g., conditional adapter), we argue that diffusion model itself allows decent
control over the generated content without requiring any training. In this
study, we introduce a tuning-free framework to achieve trajectory-controllable
video generation, by imposing guidance on both noise construction and attention
computation. Specifically, 1) we first show several instructive phenomenons and
analyze how initial noises influence the motion trajectory of generated
content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that
enables trajectory control by modifying noise sampling and attention
mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger
video generation with controllable trajectories. Equipped with these designs,
users have the flexibility to provide trajectories manually or opt for
trajectories automatically generated by the LLM trajectory planner. Extensive
experiments validate the efficacy of our approach in enhancing the trajectory
controllability of video diffusion models.
667b9209835224aefa4346bc
Haonan Qiu
claimed_verified
2024-06-26T08:03:17.268Z
63023b6ab002e9a4a2152890
/avatars/cae8ba0a8d61fb4e576934431f43991b.svg
Haonan Qiu
user
MoonQiu
667b9209835224aefa4346bd
Zhaoxi Chen
admin_assigned
2024-06-26T12:49:17.574Z
62fc8cf7ee999004b5a8b982
/avatars/6c5dda9e58747054a989f077a078f3dc.svg
Zhaoxi Chen
user
FrozenBurning
667b9209835224aefa4346be
Zhouxia Wang
admin_assigned
2024-06-26T12:49:24.235Z
64db92a5858f8a41c11669b7
/avatars/e834d8f1d4781e3bb0b5d6d25b3b3505.svg
Zhouxia Wang
user
wzhouxiff
667b9209835224aefa4346bf
Yingqing He
admin_assigned
2024-06-26T12:49:32.974Z
630231bf7e137e3d6b3b0645
/avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
He
user
Yingqing
667b9209835224aefa4346c0
Menghan Xia
admin_assigned
2024-06-26T12:49:39.656Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
667b9209835224aefa4346c1
Ziwei Liu
admin_assigned
2024-06-26T12:49:45.579Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
Diffusion model has demonstrated remarkable capability in video generation,
which further sparks interest in introducing trajectory control into the
generation process. While existing works mainly focus on training-based methods
(e.g., conditional adapter), we argue that diffusion model itself allows decent
control over the generated content without requiring any training. In this
study, we introduce a tuning-free framework to achieve trajectory-controllable
video generation, by imposing guidance on both noise construction and attention
computation. Specifically, 1) we first show several instructive phenomenons and
analyze how initial noises influence the motion trajectory of generated
content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that
enables trajectory control by modifying noise sampling and attention
mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger
video generation with controllable trajectories. Equipped with these designs,
users have the flexibility to provide trajectories manually or opt for
trajectories automatically generated by the LLM trajectory planner. Extensive
experiments validate the efficacy of our approach in enhancing the trajectory
controllability of video diffusion models.
A tuning-free framework called FreeTraj enhances trajectory control in video diffusion models by modifying noise sampling and attention mechanisms.
diffusion model video generation trajectory control conditional adapter noise construction attention computation FreeTraj LLM trajectory planner