Urban Scene Diffusion through Semantic Occupancy Map
<|reference_start|>Urban Scene Diffusion through Semantic Occupancy Map: Generating unbounded 3D scenes is crucial for large-scale scene understanding and simulation. Urban scenes, unlike natural landscapes, consist of various complex man-made objects and structures such as roads, traffic signs, vehicles, and buildings. To create a realistic and detailed urban scene, it is crucial to accurately represent the geometry and semantics of the underlying objects, going beyond their visual appearance. In this work, we propose UrbanDiffusion, a 3D diffusion model that is conditioned on a Bird's-Eye View (BEV) map and generates an urban scene with geometry and semantics in the form of semantic occupancy map. Our model introduces a novel paradigm that learns the data distribution of scene-level structures within a latent space and further enables the expansion of the synthesized scene into an arbitrary scale. After training on real-world driving datasets, our model can generate a wide range of diverse urban scenes given the BEV maps from the held-out set and also generalize to the synthesized maps from a driving simulator. We further demonstrate its application to scene image synthesis with a pretrained image generator as a prior.<|reference_end|>
@article{zhang2024urban,
    title={Urban Scene Diffusion through Semantic Occupancy Map},
    author={Junge Zhang, Qihang Zhang, Li Zhang, Ramana Rao Kompella, Gaowen Liu,
  Bolei Zhou},
    journal={arXiv preprint arXiv:2403.11697},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11697},
    primaryClass={cs.CV}
}