Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from
  User's Casual Sketches
3D Content Generation is at the heart of many computer graphics applications,
including video gaming, film-making, virtual and augmented reality, etc. This
paper proposes a novel deep-learning based approach for automatically
generating interactive and playable 3D game scenes, all from the user's casual
prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and
convenient way to convey the user's design intention in the content creation
process. To circumvent the data-deficient challenge in learning (i.e. the lack
of large training data of 3D scenes), our method leverages a pre-trained 2D
denoising diffusion model to generate a 2D image of the scene as the conceptual
guidance. In this process, we adopt the isometric projection mode to factor out
unknown camera poses while obtaining the scene layout. From the generated
isometric image, we use a pre-trained image understanding method to segment the
image into meaningful parts, such as off-ground objects, trees, and buildings,
and extract the 2D scene layout. These segments and layouts are subsequently
fed into a procedural content generation (PCG) engine, such as a 3D video game
engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can
be seamlessly integrated into a game development environment and is readily
playable. Extensive tests demonstrate that our method can efficiently generate
high-quality and interactive 3D game scenes with layouts that closely follow
the user's intention.
66b58d530d73f862e38bc6d2
Yongzhi Xu
66b58d530d73f862e38bc6d3
Yonhon Ng
admin_assigned
2024-08-09T10:04:41.443Z
649a3ced96d5747b35dfa18a
/avatars/a11440f2a90ad818451fd5a4358f4f45.svg
Yonhon Ng
user
yonhonng
66b58d530d73f862e38bc6d4
Yifu Wang
admin_assigned
2024-08-09T10:04:46.738Z
655c2847935d0f9a75e75922
/avatars/a35309aab86e8ef11622eb35d21b6a72.svg
Yifu Wang
user
yifuwang
66b58d530d73f862e38bc6d5
Inkyu Sa
claimed_verified
2024-08-13T07:39:26.370Z
62d8be84e2b34bed74422dbf
/avatars/f58b880f2095c7e4282bf1fd5129cbd9.svg
Inkyu
user
enddl22
66b58d530d73f862e38bc6d6
Yunfei Duan
admin_assigned
2024-08-09T10:04:57.321Z
640ba6d22f382421bbeb9bcd
/avatars/6f7ece66c01882dfce71a399fd18564f.svg
Yunfei Duan
user
kownse
66b58d530d73f862e38bc6d7
Yang Li
66b58d530d73f862e38bc6d8
Pan Ji
claimed_verified
2024-08-09T17:43:14.945Z
6444d87e5691ca69b0d8f56a
/avatars/78d4d2b36d629a8e6ad833e102bb86f7.svg
Peter Ji
user
peterji
66b58d530d73f862e38bc6d9
Hongdong Li
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from
  User's Casual Sketches
3D Content Generation is at the heart of many computer graphics applications,
including video gaming, film-making, virtual and augmented reality, etc. This
paper proposes a novel deep-learning based approach for automatically
generating interactive and playable 3D game scenes, all from the user's casual
prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and
convenient way to convey the user's design intention in the content creation
process. To circumvent the data-deficient challenge in learning (i.e. the lack
of large training data of 3D scenes), our method leverages a pre-trained 2D
denoising diffusion model to generate a 2D image of the scene as the conceptual
guidance. In this process, we adopt the isometric projection mode to factor out
unknown camera poses while obtaining the scene layout. From the generated
isometric image, we use a pre-trained image understanding method to segment the
image into meaningful parts, such as off-ground objects, trees, and buildings,
and extract the 2D scene layout. These segments and layouts are subsequently
fed into a procedural content generation (PCG) engine, such as a 3D video game
engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can
be seamlessly integrated into a game development environment and is readily
playable. Extensive tests demonstrate that our method can efficiently generate
high-quality and interactive 3D game scenes with layouts that closely follow
the user's intention.
A novel deep-learning approach uses sketch-based input and pre-trained models to generate interactive 3D game scenes with accurate layouts.
denoising diffusion model isometric projection image understanding procedural content generation PCG engine