title: Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
summary: Text-to-3D generation has achieved remarkable success via large-scale
  text-to-image diffusion models. Nevertheless, there is no paradigm for scaling
  up the methodology to urban scale. Urban scenes, characterized by numerous
  elements, intricate arrangement relationships, and vast scale, present a
  formidable barrier to the interpretability of ambiguous textual descriptions
  for effective model optimization. In this work, we surmount the limitations by
  introducing a compositional 3D layout representation into text-to-3D paradigm,
  serving as an additional prior. It comprises a set of semantic primitives with
  simple geometric structures and explicit arrangement relationships,
  complementing textual descriptions and enabling steerable generation. Upon
  this, we propose two modifications -- (1) We introduce Layout-Guided
  Variational Score Distillation to address model optimization inadequacies. It
  conditions the score distillation sampling process with geometric and semantic
  constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes,
  we represent 3D scene with a Scalable Hash Grid structure, incrementally
  adapting to the growing scale of urban scenes. Extensive experiments
  substantiate the capability of our framework to scale text-to-3D generation to
  large-scale urban scenes that cover over 1000m driving distance for the first
  time. We also present various scene editing demonstrations, showing the powers
  of steerable urban scene generation. Website: https://urbanarchitect.github.io.
paper_authors._id: 66173c72843718c34d0d9983
paper_authors.name: Fan Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:39:10.807Z
paper_authors._id: 6429759de5f33939cf3a57a6
paper_authors.avatarUrl: /avatars/16c40c87802ed3463a64174e4a66149b.svg
paper_authors.fullname: Fan Lu
paper_authors.type: user
paper_authors.user: FanLu97
paper_authors._id: 66173c72843718c34d0d9984
paper_authors.name: Kwan-Yee Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:26:22.847Z
paper_authors._id: 64b9226b25e5f8a3e0fc1bda
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9226b25e5f8a3e0fc1bda/sU5ZN_a3VeTela--6sX5-.jpeg
paper_authors.fullname: Kwan-Yee Lin
paper_authors.type: user
paper_authors.user: KwanYee
paper_authors._id: 66173c72843718c34d0d9985
paper_authors.name: Yan Xu
paper_authors._id: 66173c72843718c34d0d9986
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:26:49.874Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 66173c72843718c34d0d9987
paper_authors.name: Guang Chen
paper_authors._id: 66173c72843718c34d0d9988
paper_authors.name: Changjun Jiang
paper_title: Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
paper_summary: Text-to-3D generation has achieved remarkable success via large-scale
  text-to-image diffusion models. Nevertheless, there is no paradigm for scaling
  up the methodology to urban scale. Urban scenes, characterized by numerous
  elements, intricate arrangement relationships, and vast scale, present a
  formidable barrier to the interpretability of ambiguous textual descriptions
  for effective model optimization. In this work, we surmount the limitations by
  introducing a compositional 3D layout representation into text-to-3D paradigm,
  serving as an additional prior. It comprises a set of semantic primitives with
  simple geometric structures and explicit arrangement relationships,
  complementing textual descriptions and enabling steerable generation. Upon
  this, we propose two modifications -- (1) We introduce Layout-Guided
  Variational Score Distillation to address model optimization inadequacies. It
  conditions the score distillation sampling process with geometric and semantic
  constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes,
  we represent 3D scene with a Scalable Hash Grid structure, incrementally
  adapting to the growing scale of urban scenes. Extensive experiments
  substantiate the capability of our framework to scale text-to-3D generation to
  large-scale urban scenes that cover over 1000m driving distance for the first
  time. We also present various scene editing demonstrations, showing the powers
  of steerable urban scene generation. Website: https://urbanarchitect.github.io.
paper_ai_summary: Introducing a compositional 3D layout representation and modifications to text-to-3D generation techniques enables scaling to large-scale urban scenes with enhanced interpretability and steerable generation.
paper_ai_keywords: text-to-3D generation text-to-image diffusion models compositional 3D layout representation semantic primitives geometric structures Layout-Guided Variational Score Distillation score distillation geometric constraints semantic constraints Scalable Hash Grid steerable generation