title: Tora: Trajectory-oriented Diffusion Transformer for Video Generation
summary: Recent advancements in Diffusion Transformer (DiT) have demonstrated
  remarkable proficiency in producing high-quality video content. Nonetheless,
  the potential of transformer-based diffusion models for effectively generating
  videos with controllable motion remains an area of limited exploration. This
  paper introduces Tora, the first trajectory-oriented DiT framework that
  integrates textual, visual, and trajectory conditions concurrently for video
  generation. Specifically, Tora consists of a Trajectory Extractor~(TE), a
  Spatial-Temporal DiT, and a Motion-guidance Fuser~(MGF). The TE encodes
  arbitrary trajectories into hierarchical spacetime motion patches with a 3D
  video compression network. The MGF integrates the motion patches into the DiT
  blocks to generate consistent videos following trajectories. Our design aligns
  seamlessly with DiT's scalability, allowing precise control of video content's
  dynamics with diverse durations, aspect ratios, and resolutions. Extensive
  experiments demonstrate Tora's excellence in achieving high motion fidelity,
  while also meticulously simulating the movement of the physical world. Page can
  be found at https://ali-videoai.github.io/tora_video.
paper_authors._id: 66aaf2d41703f12f493c90ac
paper_authors.name: Zhenghao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T14:56:55.279Z
paper_authors._id: 64ae66dad15440de82aeba02
paper_authors.avatarUrl: /avatars/34af02b47b3ccbbf85f2092f668bba80.svg
paper_authors.fullname: Zhang Zhenghao
paper_authors.type: user
paper_authors.user: zhenghao123
paper_authors._id: 66aaf2d41703f12f493c90ad
paper_authors.name: Junchao Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T17:36:22.674Z
paper_authors._id: 659fa7cc430ffa77ac38247e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/659fa7cc430ffa77ac38247e/p-MIAnIIp81TPcTFQWe4T.jpeg
paper_authors.fullname: Junchao Liao
paper_authors.type: user
paper_authors.user: Le0jc
paper_authors._id: 66aaf2d41703f12f493c90ae
paper_authors.name: Menghao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T14:57:10.381Z
paper_authors._id: 65d8a7ee9574acb93c95bd30
paper_authors.avatarUrl: /avatars/0a123112ba87139ee109341ebf2ec795.svg
paper_authors.fullname: Menghao Li
paper_authors.type: user
paper_authors.user: sculmh
paper_authors._id: 66aaf2d41703f12f493c90af
paper_authors.name: Long Qin
paper_authors._id: 66aaf2d41703f12f493c90b0
paper_authors.name: Weizhi Wang
paper_title: Tora: Trajectory-oriented Diffusion Transformer for Video Generation
paper_summary: Recent advancements in Diffusion Transformer (DiT) have demonstrated
  remarkable proficiency in producing high-quality video content. Nonetheless,
  the potential of transformer-based diffusion models for effectively generating
  videos with controllable motion remains an area of limited exploration. This
  paper introduces Tora, the first trajectory-oriented DiT framework that
  integrates textual, visual, and trajectory conditions concurrently for video
  generation. Specifically, Tora consists of a Trajectory Extractor~(TE), a
  Spatial-Temporal DiT, and a Motion-guidance Fuser~(MGF). The TE encodes
  arbitrary trajectories into hierarchical spacetime motion patches with a 3D
  video compression network. The MGF integrates the motion patches into the DiT
  blocks to generate consistent videos following trajectories. Our design aligns
  seamlessly with DiT's scalability, allowing precise control of video content's
  dynamics with diverse durations, aspect ratios, and resolutions. Extensive
  experiments demonstrate Tora's excellence in achieving high motion fidelity,
  while also meticulously simulating the movement of the physical world. Page can
  be found at https://ali-videoai.github.io/tora_video.
paper_ai_keywords: Diffusion Transformer (DiT) Trajectory Extractor (TE) Spatial-Temporal DiT Motion-guidance Fuser (MGF) hierarchical spacetime motion patches 3D video compression network motion patches DiT blocks high motion fidelity