title: FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
summary: Diffusion model has demonstrated remarkable capability in video generation,
  which further sparks interest in introducing trajectory control into the
  generation process. While existing works mainly focus on training-based methods
  (e.g., conditional adapter), we argue that diffusion model itself allows decent
  control over the generated content without requiring any training. In this
  study, we introduce a tuning-free framework to achieve trajectory-controllable
  video generation, by imposing guidance on both noise construction and attention
  computation. Specifically, 1) we first show several instructive phenomenons and
  analyze how initial noises influence the motion trajectory of generated
  content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that
  enables trajectory control by modifying noise sampling and attention
  mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger
  video generation with controllable trajectories. Equipped with these designs,
  users have the flexibility to provide trajectories manually or opt for
  trajectories automatically generated by the LLM trajectory planner. Extensive
  experiments validate the efficacy of our approach in enhancing the trajectory
  controllability of video diffusion models.
paper_authors._id: 667b9209835224aefa4346bc
paper_authors.name: Haonan Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:17.268Z
paper_authors._id: 63023b6ab002e9a4a2152890
paper_authors.avatarUrl: /avatars/cae8ba0a8d61fb4e576934431f43991b.svg
paper_authors.fullname: Haonan Qiu
paper_authors.type: user
paper_authors.user: MoonQiu
paper_authors._id: 667b9209835224aefa4346bd
paper_authors.name: Zhaoxi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:49:17.574Z
paper_authors._id: 62fc8cf7ee999004b5a8b982
paper_authors.avatarUrl: /avatars/6c5dda9e58747054a989f077a078f3dc.svg
paper_authors.fullname: Zhaoxi Chen
paper_authors.type: user
paper_authors.user: FrozenBurning
paper_authors._id: 667b9209835224aefa4346be
paper_authors.name: Zhouxia Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:49:24.235Z
paper_authors._id: 64db92a5858f8a41c11669b7
paper_authors.avatarUrl: /avatars/e834d8f1d4781e3bb0b5d6d25b3b3505.svg
paper_authors.fullname: Zhouxia Wang
paper_authors.type: user
paper_authors.user: wzhouxiff
paper_authors._id: 667b9209835224aefa4346bf
paper_authors.name: Yingqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:49:32.974Z
paper_authors._id: 630231bf7e137e3d6b3b0645
paper_authors.avatarUrl: /avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: Yingqing
paper_authors._id: 667b9209835224aefa4346c0
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:49:39.656Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 667b9209835224aefa4346c1
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:49:45.579Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
paper_summary: Diffusion model has demonstrated remarkable capability in video generation,
  which further sparks interest in introducing trajectory control into the
  generation process. While existing works mainly focus on training-based methods
  (e.g., conditional adapter), we argue that diffusion model itself allows decent
  control over the generated content without requiring any training. In this
  study, we introduce a tuning-free framework to achieve trajectory-controllable
  video generation, by imposing guidance on both noise construction and attention
  computation. Specifically, 1) we first show several instructive phenomenons and
  analyze how initial noises influence the motion trajectory of generated
  content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that
  enables trajectory control by modifying noise sampling and attention
  mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger
  video generation with controllable trajectories. Equipped with these designs,
  users have the flexibility to provide trajectories manually or opt for
  trajectories automatically generated by the LLM trajectory planner. Extensive
  experiments validate the efficacy of our approach in enhancing the trajectory
  controllability of video diffusion models.
paper_ai_keywords: diffusion model video generation trajectory control conditional adapter guidance noise construction attention computation initial noises motion trajectory FreeTraj noise sampling attention mechanisms trajectory planner trajectory controllability