Towards Realistic Scene Generation with LiDAR Diffusion Models
<|reference_start|>Towards Realistic Scene Generation with LiDAR Diffusion Models: Diffusion models (DMs) excel in photo-realistic image synthesis, but their adaptation to LiDAR scene generation poses a substantial hurdle. This is primarily because DMs operating in the point space struggle to preserve the curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of their representation power. In this paper, we propose LiDAR Diffusion Models (LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to capture the realism of LiDAR scenes by incorporating geometric priors into the learning pipeline. Our method targets three major desiderata: pattern realism, geometry realism, and object realism. Specifically, we introduce curve-wise compression to simulate real-world LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding for a full 3D object context. With these three core designs, our method achieves competitive performance on unconditional LiDAR generation in 64-beam scenario and state of the art on conditional LiDAR generation, while maintaining high efficiency compared to point-based DMs (up to 107$\times$ faster). Furthermore, by compressing LiDAR scenes into a latent space, we enable the controllability of DMs with various conditions such as semantic maps, camera views, and text prompts.<|reference_end|>
@article{ran2024towards,
    title={Towards Realistic Scene Generation with LiDAR Diffusion Models},
    author={Haoxi Ran, Vitor Guizilini, Yue Wang},
    journal={arXiv preprint arXiv:2404.00815},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.00815},
    primaryClass={cs.CV cs.AI cs.RO}
}