title: Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from
    User's Casual Sketches
summary: 3D Content Generation is at the heart of many computer graphics applications,
  including video gaming, film-making, virtual and augmented reality, etc. This
  paper proposes a novel deep-learning based approach for automatically
  generating interactive and playable 3D game scenes, all from the user's casual
  prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and
  convenient way to convey the user's design intention in the content creation
  process. To circumvent the data-deficient challenge in learning (i.e. the lack
  of large training data of 3D scenes), our method leverages a pre-trained 2D
  denoising diffusion model to generate a 2D image of the scene as the conceptual
  guidance. In this process, we adopt the isometric projection mode to factor out
  unknown camera poses while obtaining the scene layout. From the generated
  isometric image, we use a pre-trained image understanding method to segment the
  image into meaningful parts, such as off-ground objects, trees, and buildings,
  and extract the 2D scene layout. These segments and layouts are subsequently
  fed into a procedural content generation (PCG) engine, such as a 3D video game
  engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can
  be seamlessly integrated into a game development environment and is readily
  playable. Extensive tests demonstrate that our method can efficiently generate
  high-quality and interactive 3D game scenes with layouts that closely follow
  the user's intention.
paper_authors._id: 66b58d530d73f862e38bc6d2
paper_authors.name: Yongzhi Xu
paper_authors._id: 66b58d530d73f862e38bc6d3
paper_authors.name: Yonhon Ng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:04:41.443Z
paper_authors._id: 649a3ced96d5747b35dfa18a
paper_authors.avatarUrl: /avatars/a11440f2a90ad818451fd5a4358f4f45.svg
paper_authors.fullname: Yonhon Ng
paper_authors.type: user
paper_authors.user: yonhonng
paper_authors._id: 66b58d530d73f862e38bc6d4
paper_authors.name: Yifu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:04:46.738Z
paper_authors._id: 655c2847935d0f9a75e75922
paper_authors.avatarUrl: /avatars/a35309aab86e8ef11622eb35d21b6a72.svg
paper_authors.fullname: Yifu Wang
paper_authors.type: user
paper_authors.user: yifuwang
paper_authors._id: 66b58d530d73f862e38bc6d5
paper_authors.name: Inkyu Sa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:39:26.370Z
paper_authors._id: 62d8be84e2b34bed74422dbf
paper_authors.avatarUrl: /avatars/f58b880f2095c7e4282bf1fd5129cbd9.svg
paper_authors.fullname: Inkyu
paper_authors.type: user
paper_authors.user: enddl22
paper_authors._id: 66b58d530d73f862e38bc6d6
paper_authors.name: Yunfei Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:04:57.321Z
paper_authors._id: 640ba6d22f382421bbeb9bcd
paper_authors.avatarUrl: /avatars/6f7ece66c01882dfce71a399fd18564f.svg
paper_authors.fullname: Yunfei Duan
paper_authors.type: user
paper_authors.user: kownse
paper_authors._id: 66b58d530d73f862e38bc6d7
paper_authors.name: Yang Li
paper_authors._id: 66b58d530d73f862e38bc6d8
paper_authors.name: Pan Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T17:43:14.945Z
paper_authors._id: 6444d87e5691ca69b0d8f56a
paper_authors.avatarUrl: /avatars/78d4d2b36d629a8e6ad833e102bb86f7.svg
paper_authors.fullname: Peter Ji
paper_authors.type: user
paper_authors.user: peterji
paper_authors._id: 66b58d530d73f862e38bc6d9
paper_authors.name: Hongdong Li
paper_title: Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from
    User's Casual Sketches
paper_summary: 3D Content Generation is at the heart of many computer graphics applications,
  including video gaming, film-making, virtual and augmented reality, etc. This
  paper proposes a novel deep-learning based approach for automatically
  generating interactive and playable 3D game scenes, all from the user's casual
  prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and
  convenient way to convey the user's design intention in the content creation
  process. To circumvent the data-deficient challenge in learning (i.e. the lack
  of large training data of 3D scenes), our method leverages a pre-trained 2D
  denoising diffusion model to generate a 2D image of the scene as the conceptual
  guidance. In this process, we adopt the isometric projection mode to factor out
  unknown camera poses while obtaining the scene layout. From the generated
  isometric image, we use a pre-trained image understanding method to segment the
  image into meaningful parts, such as off-ground objects, trees, and buildings,
  and extract the 2D scene layout. These segments and layouts are subsequently
  fed into a procedural content generation (PCG) engine, such as a 3D video game
  engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can
  be seamlessly integrated into a game development environment and is readily
  playable. Extensive tests demonstrate that our method can efficiently generate
  high-quality and interactive 3D game scenes with layouts that closely follow
  the user's intention.
paper_ai_keywords: 2D denoising diffusion model isometric projection mode image understanding method procedural content generation (PCG) 3D video game engine (Unity or Unreal)