title: DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
summary: LiDAR scene generation has been developing rapidly recently. However,
  existing methods primarily focus on generating static and single-frame scenes,
  overlooking the inherently dynamic nature of real-world driving environments.
  In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework
  capable of generating large-scale, high-quality LiDAR scenes that capture the
  temporal evolution of dynamic environments. DynamicCity mainly consists of two
  key models. 1) A VAE model for learning HexPlane as the compact 4D
  representation. Instead of using naive averaging operations, DynamicCity
  employs a novel Projection Module to effectively compress 4D LiDAR features
  into six 2D feature maps for HexPlane construction, which significantly
  enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we
  utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in
  parallel, which improves both network training efficiency and reconstruction
  accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x
  training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model
  for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded
  Rollout Operation is proposed to reorganize all six feature planes of the
  HexPlane as a squared 2D feature map. In particular, various conditions could
  be introduced in the diffusion or sampling process, supporting versatile 4D
  generation applications, such as trajectory- and command-driven generation,
  inpainting, and layout-conditioned generation. Extensive experiments on the
  CarlaSC and Waymo datasets demonstrate that DynamicCity significantly
  outperforms existing state-of-the-art 4D LiDAR generation methods across
  multiple metrics. The code will be released to facilitate future research.
paper_authors._id: 671a297304f808324dfa8237
paper_authors.name: Hengwei Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:25:40.458Z
paper_authors._id: 65990b48ac728bc303537234
paper_authors.avatarUrl: /avatars/717f599381ae845721c6daccb3bb1f76.svg
paper_authors.fullname: Hengwei Bian
paper_authors.type: user
paper_authors.user: hengwei
paper_authors._id: 671a297304f808324dfa8238
paper_authors.name: Lingdong Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:25:25.234Z
paper_authors._id: 62df78222d89ce551ce0f71d
paper_authors.avatarUrl: /avatars/89fba294cff2d2f941d121c1923e4c76.svg
paper_authors.fullname: Lingdong Kong
paper_authors.type: user
paper_authors.user: ldkong
paper_authors._id: 671a297304f808324dfa8239
paper_authors.name: Haozhe Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T12:35:00.035Z
paper_authors._id: 63f47b5321eb234ab739e91a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f47b5321eb234ab739e91a/vWfFNVtMkHl8gieha5PPd.jpeg
paper_authors.fullname: Haozhe Xie
paper_authors.type: user
paper_authors.user: hzxie
paper_authors._id: 671a297304f808324dfa823a
paper_authors.name: Liang Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T16:09:29.882Z
paper_authors._id: 64ef30fb0de94d31b7921175
paper_authors.avatarUrl: /avatars/d4f4c7be460befc1cedec13bbf9db972.svg
paper_authors.fullname: pan
paper_authors.type: user
paper_authors.user: pldeqiushui
paper_authors._id: 671a297304f808324dfa823b
paper_authors.name: Yu Qiao
paper_authors._id: 671a297304f808324dfa823c
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T14:25:47.367Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
paper_summary: LiDAR scene generation has been developing rapidly recently. However,
  existing methods primarily focus on generating static and single-frame scenes,
  overlooking the inherently dynamic nature of real-world driving environments.
  In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework
  capable of generating large-scale, high-quality LiDAR scenes that capture the
  temporal evolution of dynamic environments. DynamicCity mainly consists of two
  key models. 1) A VAE model for learning HexPlane as the compact 4D
  representation. Instead of using naive averaging operations, DynamicCity
  employs a novel Projection Module to effectively compress 4D LiDAR features
  into six 2D feature maps for HexPlane construction, which significantly
  enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we
  utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in
  parallel, which improves both network training efficiency and reconstruction
  accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x
  training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model
  for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded
  Rollout Operation is proposed to reorganize all six feature planes of the
  HexPlane as a squared 2D feature map. In particular, various conditions could
  be introduced in the diffusion or sampling process, supporting versatile 4D
  generation applications, such as trajectory- and command-driven generation,
  inpainting, and layout-conditioned generation. Extensive experiments on the
  CarlaSC and Waymo datasets demonstrate that DynamicCity significantly
  outperforms existing state-of-the-art 4D LiDAR generation methods across
  multiple metrics. The code will be released to facilitate future research.
paper_ai_keywords: VAE HexPlane Projection Module mIoU Expansion & Squeeze Strategy DiT-based diffusion model Padded Rollout Operation trajectory-driven generation command-driven generation inpainting layout-conditioned generation CarlaSC dataset Waymo dataset LiDAR scene generation 4D scene generation