title: SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
    Semantic MPIs
summary: The advancement of autonomous driving is increasingly reliant on high-quality
  annotated datasets, especially in the task of 3D occupancy prediction, where
  the occupancy labels require dense 3D annotation with significant human effort.
  In this paper, we propose SyntheOcc, which denotes a diffusion model that
  Synthesize photorealistic and geometric-controlled images by conditioning
  Occupancy labels in driving scenarios. This yields an unlimited amount of
  diverse, annotated, and controllable datasets for applications like training
  perception models and simulation. SyntheOcc addresses the critical challenge of
  how to efficiently encode 3D geometric information as conditional input to a 2D
  diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
  images (MPIs) to provide comprehensive and spatially aligned 3D scene
  descriptions for conditioning. As a result, SyntheOcc can generate
  photorealistic multi-view images and videos that faithfully align with the
  given geometric labels (semantics in 3D voxel space). Extensive qualitative and
  quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
  effectiveness in generating controllable occupancy datasets that serve as an
  effective data augmentation to perception models.
paper_authors._id: 66fd0dad7fbd5d307648b010
paper_authors.name: Leheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T12:44:45.347Z
paper_authors._id: 6343e37f73b4f9cedab1c846
paper_authors.avatarUrl: /avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
paper_authors.fullname: Leheng Li
paper_authors.type: user
paper_authors.user: lilelife
paper_authors._id: 66fd0dad7fbd5d307648b011
paper_authors.name: Weichao Qiu
paper_authors._id: 66fd0dad7fbd5d307648b012
paper_authors.name: Yingjie Cai
paper_authors._id: 66fd0dad7fbd5d307648b013
paper_authors.name: Xu Yan
paper_authors._id: 66fd0dad7fbd5d307648b014
paper_authors.name: Qing Lian
paper_authors._id: 66fd0dad7fbd5d307648b015
paper_authors.name: Bingbing Liu
paper_authors._id: 66fd0dad7fbd5d307648b016
paper_authors.name: Ying-Cong Chen
paper_title: SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
    Semantic MPIs
paper_summary: The advancement of autonomous driving is increasingly reliant on high-quality
  annotated datasets, especially in the task of 3D occupancy prediction, where
  the occupancy labels require dense 3D annotation with significant human effort.
  In this paper, we propose SyntheOcc, which denotes a diffusion model that
  Synthesize photorealistic and geometric-controlled images by conditioning
  Occupancy labels in driving scenarios. This yields an unlimited amount of
  diverse, annotated, and controllable datasets for applications like training
  perception models and simulation. SyntheOcc addresses the critical challenge of
  how to efficiently encode 3D geometric information as conditional input to a 2D
  diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
  images (MPIs) to provide comprehensive and spatially aligned 3D scene
  descriptions for conditioning. As a result, SyntheOcc can generate
  photorealistic multi-view images and videos that faithfully align with the
  given geometric labels (semantics in 3D voxel space). Extensive qualitative and
  quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
  effectiveness in generating controllable occupancy datasets that serve as an
  effective data augmentation to perception models.
paper_ai_summary: SyntheOcc, a diffusion model using 3D semantic multi-plane images, generates photorealistic and aligned 3D occupancy datasets for autonomous driving perception models.
paper_ai_keywords: diffusion model 3D occupancy prediction photorealistic images geometric-controlled images conditional input 3D semantic multi-plane images 3D voxel space controllable datasets data augmentation nuScenes dataset