Tora: Trajectory-oriented Diffusion Transformer for Video Generation
Recent advancements in Diffusion Transformer (DiT) have demonstrated
remarkable proficiency in producing high-quality video content. Nonetheless,
the potential of transformer-based diffusion models for effectively generating
videos with controllable motion remains an area of limited exploration. This
paper introduces Tora, the first trajectory-oriented DiT framework that
integrates textual, visual, and trajectory conditions concurrently for video
generation. Specifically, Tora consists of a Trajectory Extractor~(TE), a
Spatial-Temporal DiT, and a Motion-guidance Fuser~(MGF). The TE encodes
arbitrary trajectories into hierarchical spacetime motion patches with a 3D
video compression network. The MGF integrates the motion patches into the DiT
blocks to generate consistent videos following trajectories. Our design aligns
seamlessly with DiT's scalability, allowing precise control of video content's
dynamics with diverse durations, aspect ratios, and resolutions. Extensive
experiments demonstrate Tora's excellence in achieving high motion fidelity,
while also meticulously simulating the movement of the physical world. Page can
be found at https://ali-videoai.github.io/tora_video.
66aaf2d41703f12f493c90ac
Zhenghao Zhang
admin_assigned
2024-08-02T14:56:55.279Z
64ae66dad15440de82aeba02
/avatars/34af02b47b3ccbbf85f2092f668bba80.svg
Zhang Zhenghao
user
zhenghao123
66aaf2d41703f12f493c90ad
Junchao Liao
admin_assigned
2024-08-05T17:36:22.674Z
659fa7cc430ffa77ac38247e
https://cdn-avatars.huggingface.co/v1/production/uploads/659fa7cc430ffa77ac38247e/p-MIAnIIp81TPcTFQWe4T.jpeg
Junchao Liao
user
Le0jc
66aaf2d41703f12f493c90ae
Menghao Li
admin_assigned
2024-08-02T14:57:10.381Z
65d8a7ee9574acb93c95bd30
/avatars/0a123112ba87139ee109341ebf2ec795.svg
Menghao Li
user
sculmh
66aaf2d41703f12f493c90af
Long Qin
66aaf2d41703f12f493c90b0
Weizhi Wang
Tora: Trajectory-oriented Diffusion Transformer for Video Generation
Recent advancements in Diffusion Transformer (DiT) have demonstrated
remarkable proficiency in producing high-quality video content. Nonetheless,
the potential of transformer-based diffusion models for effectively generating
videos with controllable motion remains an area of limited exploration. This
paper introduces Tora, the first trajectory-oriented DiT framework that
integrates textual, visual, and trajectory conditions concurrently for video
generation. Specifically, Tora consists of a Trajectory Extractor~(TE), a
Spatial-Temporal DiT, and a Motion-guidance Fuser~(MGF). The TE encodes
arbitrary trajectories into hierarchical spacetime motion patches with a 3D
video compression network. The MGF integrates the motion patches into the DiT
blocks to generate consistent videos following trajectories. Our design aligns
seamlessly with DiT's scalability, allowing precise control of video content's
dynamics with diverse durations, aspect ratios, and resolutions. Extensive
experiments demonstrate Tora's excellence in achieving high motion fidelity,
while also meticulously simulating the movement of the physical world. Page can
be found at https://ali-videoai.github.io/tora_video.
Tora, a trajectory-oriented Diffusion Transformer framework, integrates textual, visual, and trajectory conditions for high-fidelity video generation with controlled motion.
Diffusion Transformer DiT Tora Trajectory Extractor Spatial-Temporal DiT Motion-guidance Fuser 3D video compression network motion patches motion fidelity