title: TC4D: Trajectory-Conditioned Text-to-4D Generation
summary: Recent techniques for text-to-4D generation synthesize dynamic 3D scenes
  using supervision from pre-trained text-to-video models. However, existing
  representations for motion, such as deformation models or time-dependent neural
  representations, are limited in the amount of motion they can generate-they
  cannot synthesize motion extending far beyond the bounding box used for volume
  rendering. The lack of a more flexible motion model contributes to the gap in
  realism between 4D generation methods and recent, near-photorealistic video
  generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D
  generation, which factors motion into global and local components. We represent
  the global motion of a scene's bounding box using rigid transformation along a
  trajectory parameterized by a spline. We learn local deformations that conform
  to the global trajectory using supervision from a text-to-video model. Our
  approach enables the synthesis of scenes animated along arbitrary trajectories,
  compositional scene generation, and significant improvements to the realism and
  amount of generated motion, which we evaluate qualitatively and through a user
  study. Video results can be viewed on our website:
  https://sherwinbahmani.github.io/tc4d.
paper_authors._id: 6603909490c62bb38f073051
paper_authors.name: Sherwin Bahmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:47:20.297Z
paper_authors._id: 647ef66c45baf21ad707b291
paper_authors.avatarUrl: /avatars/4cd941cdca6dd829fdc9cb3fb788a99c.svg
paper_authors.fullname: Sherwin Bahmani
paper_authors.type: user
paper_authors.user: sherwinbahmani
paper_authors._id: 6603909490c62bb38f073052
paper_authors.name: Xian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:47:36.756Z
paper_authors._id: 6423f10a3fa01ecba6fca861
paper_authors.avatarUrl: /avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
paper_authors.fullname: Xian Liu
paper_authors.type: user
paper_authors.user: alvinliu0
paper_authors._id: 6603909490c62bb38f073053
paper_authors.name: Yifan Wang
paper_authors._id: 6603909490c62bb38f073054
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:48:10.431Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 6603909490c62bb38f073055
paper_authors.name: Victor Rong
paper_authors._id: 6603909490c62bb38f073056
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:48:20.683Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 6603909490c62bb38f073057
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:48:26.669Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 6603909490c62bb38f073058
paper_authors.name: Jeong Joon Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T17:34:50.085Z
paper_authors._id: 6517d7f84fadbeb643c68269
paper_authors.avatarUrl: /avatars/ad21288123abcd4488ae1bdcb10b54e7.svg
paper_authors.fullname: JJ Park
paper_authors.type: user
paper_authors.user: jjparkcv
paper_authors._id: 6603909490c62bb38f073059
paper_authors.name: Sergey Tulyakov
paper_authors._id: 6603909490c62bb38f07305a
paper_authors.name: Gordon Wetzstein
paper_authors._id: 6603909490c62bb38f07305b
paper_authors.name: Andrea Tagliasacchi
paper_authors._id: 6603909490c62bb38f07305c
paper_authors.name: David B. Lindell
paper_title: TC4D: Trajectory-Conditioned Text-to-4D Generation
paper_summary: Recent techniques for text-to-4D generation synthesize dynamic 3D scenes
  using supervision from pre-trained text-to-video models. However, existing
  representations for motion, such as deformation models or time-dependent neural
  representations, are limited in the amount of motion they can generate-they
  cannot synthesize motion extending far beyond the bounding box used for volume
  rendering. The lack of a more flexible motion model contributes to the gap in
  realism between 4D generation methods and recent, near-photorealistic video
  generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D
  generation, which factors motion into global and local components. We represent
  the global motion of a scene's bounding box using rigid transformation along a
  trajectory parameterized by a spline. We learn local deformations that conform
  to the global trajectory using supervision from a text-to-video model. Our
  approach enables the synthesis of scenes animated along arbitrary trajectories,
  compositional scene generation, and significant improvements to the realism and
  amount of generated motion, which we evaluate qualitatively and through a user
  study. Video results can be viewed on our website:
  https://sherwinbahmani.github.io/tc4d.
paper_ai_keywords: trajectory-conditioned text-to-4D generation deformation models time-dependent neural representations volume rendering rigid transformation spline compositional scene generation