Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
<|reference_start|>Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text: Recent advancements in 3D generation have leveraged synthetic datasets with ground truth 3D assets and predefined cameras. However, the potential of adopting real-world datasets, which can produce significantly more realistic 3D scenes, remains largely unexplored. In this work, we delve into the key challenge of the complex and scene-specific camera trajectories found in real-world captures. We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the Cinematographer, to model the distribution of camera trajectories based on textual descriptions. (2) Next, a Gaussian-driven Multi-view Latent Diffusion Model serves as the Decorator, modeling the image sequence distribution given the camera trajectories and texts. This model, fine-tuned from a 2D diffusion model, directly generates pixel-aligned 3D Gaussians as an immediate 3D scene representation for consistent denoising. (3) Lastly, the 3D Gaussians are refined by a novel SDS++ loss as the Detailer, which incorporates the prior of the 2D diffusion model. Extensive experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.<|reference_end|>
@article{li2024director3d:,
    title={Director3D: Real-world Camera Trajectory and 3D Scene Generation from
  Text},
    author={Xinyang Li, Zhangyu Lai, Linning Xu, Yansong Qu, Liujuan Cao,
  Shengchuan Zhang, Bo Dai, Rongrong Ji},
    journal={arXiv preprint arXiv:2406.17601},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17601},
    primaryClass={cs.CV}
}