title: 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video
    Generation
summary: This paper aims to manipulate multi-entity 3D motions in video generation.
  Previous methods on controllable video generation primarily leverage 2D control
  signals to manipulate object motions and have achieved remarkable synthesis
  results. However, 2D control signals are inherently limited in expressing the
  3D nature of object motions. To overcome this problem, we introduce
  3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D
  space, given user-desired 6DoF pose (location and rotation) sequences of
  entities. At the core of our approach is a plug-and-play 3D-motion grounded
  object injector that fuses multiple input entities with their respective 3D
  trajectories through a gated self-attention mechanism. In addition, we exploit
  an injector architecture to preserve the video diffusion prior, which is
  crucial for generalization ability. To mitigate video quality degradation, we
  introduce a domain adaptor during training and employ an annealed sampling
  strategy during inference. To address the lack of suitable training data, we
  construct a 360-Motion Dataset, which first correlates collected 3D human and
  animal assets with GPT-generated trajectory and then captures their motion with
  12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments
  show that 3DTrajMaster sets a new state-of-the-art in both accuracy and
  generalization for controlling multi-entity 3D motions. Project page:
  http://fuxiao0719.github.io/projects/3dtrajmaster
paper_authors._id: 675962a407803203aef900ec
paper_authors.name: Xiao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T10:09:04.015Z
paper_authors._id: 63aef2cafcca84593e6682db
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672409763337-noauth.jpeg
paper_authors.fullname: Xiao Fu
paper_authors.type: user
paper_authors.user: lemonaddie
paper_authors._id: 675962a407803203aef900ed
paper_authors.name: Xian Liu
paper_authors._id: 675962a407803203aef900ee
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:21:32.060Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 675962a407803203aef900ef
paper_authors.name: Sida Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:21:24.280Z
paper_authors._id: 62986ca2b58e71e2ac9b8f01
paper_authors.avatarUrl: /avatars/83944db5f3dbb6f47c47c46fb2cb2849.svg
paper_authors.fullname: Sida Peng
paper_authors.type: user
paper_authors.user: pengsida
paper_authors._id: 675962a407803203aef900f0
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:20:48.288Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 675962a407803203aef900f1
paper_authors.name: Xiaoyu Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:21:17.084Z
paper_authors._id: 641af5fcf902cc42730b47e2
paper_authors.avatarUrl: /avatars/73ac99dec226f0e814a16d2f1dbfbce8.svg
paper_authors.fullname: Xiaoyu Shi
paper_authors.type: user
paper_authors.user: btwbtm
paper_authors._id: 675962a407803203aef900f2
paper_authors.name: Ziyang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:20:31.083Z
paper_authors._id: 66c15837b186e4f6a0dac80c
paper_authors.avatarUrl: /avatars/a3b75d6945f1608e64a2fcff887a5024.svg
paper_authors.fullname: Ziyang Yuan
paper_authors.type: user
paper_authors.user: ziyangy
paper_authors._id: 675962a407803203aef900f3
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:43:06.988Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: /avatars/dc9486db75869ce902d0a638eea126bd.svg
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 675962a407803203aef900f4
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:20:13.824Z
paper_authors._id: 64bce15bafd1e46c5504ad38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/bQFX1iFbXEBXcQvUNL811.png
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: di-zhang-fdu
paper_authors._id: 675962a407803203aef900f5
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:19:40.425Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video
    Generation
paper_summary: This paper aims to manipulate multi-entity 3D motions in video generation.
  Previous methods on controllable video generation primarily leverage 2D control
  signals to manipulate object motions and have achieved remarkable synthesis
  results. However, 2D control signals are inherently limited in expressing the
  3D nature of object motions. To overcome this problem, we introduce
  3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D
  space, given user-desired 6DoF pose (location and rotation) sequences of
  entities. At the core of our approach is a plug-and-play 3D-motion grounded
  object injector that fuses multiple input entities with their respective 3D
  trajectories through a gated self-attention mechanism. In addition, we exploit
  an injector architecture to preserve the video diffusion prior, which is
  crucial for generalization ability. To mitigate video quality degradation, we
  introduce a domain adaptor during training and employ an annealed sampling
  strategy during inference. To address the lack of suitable training data, we
  construct a 360-Motion Dataset, which first correlates collected 3D human and
  animal assets with GPT-generated trajectory and then captures their motion with
  12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments
  show that 3DTrajMaster sets a new state-of-the-art in both accuracy and
  generalization for controlling multi-entity 3D motions. Project page:
  http://fuxiao0719.github.io/projects/3dtrajmaster
paper_ai_keywords: 3DTrajMaster 3D space 6DoF pose 3D-motion grounded object injector gated self-attention mechanism video diffusion prior domain adaptor annealed sampling strategy 360-Motion Dataset GPT-generated trajectory 12 evenly-surround cameras 3D UE platforms