title: Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video
    Generation Control
summary: Diffusion models have demonstrated impressive performance in generating
  high-quality videos from text prompts or images. However, precise control over
  the video generation process, such as camera manipulation or content editing,
  remains a significant challenge. Existing methods for controlled video
  generation are typically limited to a single control type, lacking the
  flexibility to handle diverse control demands. In this paper, we introduce
  Diffusion as Shader (DaS), a novel approach that supports multiple video
  control tasks within a unified architecture. Our key insight is that achieving
  versatile video control necessitates leveraging 3D control signals, as videos
  are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods
  limited to 2D control signals, DaS leverages 3D tracking videos as control
  inputs, making the video diffusion process inherently 3D-aware. This innovation
  allows DaS to achieve a wide range of video controls by simply manipulating the
  3D tracking videos. A further advantage of using 3D tracking videos is their
  ability to effectively link frames, significantly enhancing the temporal
  consistency of the generated videos. With just 3 days of fine-tuning on 8 H800
  GPUs using less than 10k videos, DaS demonstrates strong control capabilities
  across diverse tasks, including mesh-to-video generation, camera control,
  motion transfer, and object manipulation.
paper_authors._id: 677e11b751b4ae6d3c98fdca
paper_authors.name: Zekai Gu
paper_authors._id: 677e11b751b4ae6d3c98fdcb
paper_authors.name: Rui Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:47:05.657Z
paper_authors._id: 64be3a355b8d826146ed2001
paper_authors.avatarUrl: /avatars/85a87270e7c76345e555803ab31c33d1.svg
paper_authors.fullname: ruiyan
paper_authors.type: user
paper_authors.user: ruiyan
paper_authors._id: 677e11b751b4ae6d3c98fdcc
paper_authors.name: Jiahao Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:46:55.864Z
paper_authors._id: 64d3abeb1058ae59738ba8ce
paper_authors.avatarUrl: /avatars/4a9de3db835ade2c20f0d2de678e85c4.svg
paper_authors.fullname: jiahao lu
paper_authors.type: user
paper_authors.user: jiahao97
paper_authors._id: 677e11b751b4ae6d3c98fdcd
paper_authors.name: Peng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:44:29.526Z
paper_authors._id: 634e6bdd2cd84978c48f3985
paper_authors.avatarUrl: /avatars/2f6ae1f50666ca187a7a3c74b5e173f7.svg
paper_authors.fullname: Peng Li
paper_authors.type: user
paper_authors.user: pengHTYX
paper_authors._id: 677e11b751b4ae6d3c98fdce
paper_authors.name: Zhiyang Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:47:13.361Z
paper_authors._id: 645223fb01d7bd9555ea399a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
paper_authors.fullname: Zhiyang Dou
paper_authors.type: user
paper_authors.user: frankzydou
paper_authors._id: 677e11b751b4ae6d3c98fdcf
paper_authors.name: Chenyang Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:47:21.815Z
paper_authors._id: 635f8ed47c05eb9f59963d3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
paper_authors.fullname: ChenyangSi
paper_authors.type: user
paper_authors.user: ChenyangSi
paper_authors._id: 677e11b751b4ae6d3c98fdd0
paper_authors.name: Zhen Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:47:32.762Z
paper_authors._id: 643ba2f725681c3afaa8f05e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
paper_authors.fullname: Zhen Dong
paper_authors.type: user
paper_authors.user: zhendongucb
paper_authors._id: 677e11b751b4ae6d3c98fdd1
paper_authors.name: Qifeng Liu
paper_authors._id: 677e11b751b4ae6d3c98fdd2
paper_authors.name: Cheng Lin
paper_authors._id: 677e11b751b4ae6d3c98fdd3
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:48:08.642Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 677e11b751b4ae6d3c98fdd4
paper_authors.name: Wenping Wang
paper_authors._id: 677e11b751b4ae6d3c98fdd5
paper_authors.name: Yuan Liu
paper_title: Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video
    Generation Control
paper_summary: Diffusion models have demonstrated impressive performance in generating
  high-quality videos from text prompts or images. However, precise control over
  the video generation process, such as camera manipulation or content editing,
  remains a significant challenge. Existing methods for controlled video
  generation are typically limited to a single control type, lacking the
  flexibility to handle diverse control demands. In this paper, we introduce
  Diffusion as Shader (DaS), a novel approach that supports multiple video
  control tasks within a unified architecture. Our key insight is that achieving
  versatile video control necessitates leveraging 3D control signals, as videos
  are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods
  limited to 2D control signals, DaS leverages 3D tracking videos as control
  inputs, making the video diffusion process inherently 3D-aware. This innovation
  allows DaS to achieve a wide range of video controls by simply manipulating the
  3D tracking videos. A further advantage of using 3D tracking videos is their
  ability to effectively link frames, significantly enhancing the temporal
  consistency of the generated videos. With just 3 days of fine-tuning on 8 H800
  GPUs using less than 10k videos, DaS demonstrates strong control capabilities
  across diverse tasks, including mesh-to-video generation, camera control,
  motion transfer, and object manipulation.
paper_ai_summary: Diffusion as Shader (DaS) achieves versatile video control by leveraging 3D tracking signals, enabling tasks like mesh-to-video generation, camera control, motion transfer, and object manipulation.
paper_ai_keywords: diffusion models video generation 3D control signals 3D tracking videos mesh-to-video generation camera control motion transfer object manipulation