HexaGen3D: StableDiffusion is just one step away from Fast and Diverse
  Text-to-3D Generation
Despite the latest remarkable advances in generative modeling, efficient
generation of high-quality 3D assets from textual prompts remains a difficult
task. A key challenge lies in data scarcity: the most extensive 3D datasets
encompass merely millions of assets, while their 2D counterparts contain
billions of text-image pairs. To address this, we propose a novel approach
which harnesses the power of large, pretrained 2D diffusion models. More
specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image
model to jointly predict 6 orthographic projections and the corresponding
latent triplane. We then decode these latents to generate a textured mesh.
HexaGen3D does not require per-sample optimization, and can infer high-quality
and diverse objects from textual prompts in 7 seconds, offering significantly
better quality-to-latency trade-offs when comparing to existing approaches.
Furthermore, HexaGen3D demonstrates strong generalization to new objects or
compositions.
65a76e840f169670d78e92a8
Antoine Mercier
65a76e840f169670d78e92a9
Ramin Nakhli
admin_assigned
2024-01-17T10:09:53.945Z
613a36a1a488863c19207314
/avatars/63520d7e81b08dc0ca468f73047117d6.svg
Ramin Nakhli
user
raminnakhli
65a76e840f169670d78e92aa
Mahesh Reddy
65a76e840f169670d78e92ab
Rajeev Yasarla
65a76e840f169670d78e92ac
Hong Cai
65a76e840f169670d78e92ad
Fatih Porikli
65a76e840f169670d78e92ae
Guillaume Berger
HexaGen3D: StableDiffusion is just one step away from Fast and Diverse
  Text-to-3D Generation
Despite the latest remarkable advances in generative modeling, efficient
generation of high-quality 3D assets from textual prompts remains a difficult
task. A key challenge lies in data scarcity: the most extensive 3D datasets
encompass merely millions of assets, while their 2D counterparts contain
billions of text-image pairs. To address this, we propose a novel approach
which harnesses the power of large, pretrained 2D diffusion models. More
specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image
model to jointly predict 6 orthographic projections and the corresponding
latent triplane. We then decode these latents to generate a textured mesh.
HexaGen3D does not require per-sample optimization, and can infer high-quality
and diverse objects from textual prompts in 7 seconds, offering significantly
better quality-to-latency trade-offs when comparing to existing approaches.
Furthermore, HexaGen3D demonstrates strong generalization to new objects or
compositions.
HexaGen3D fine-tunes a 2D diffusion model to generate high-quality 3D assets from textual prompts using 6 orthographic projections and a latent triplane, offering better quality-to-latency trade-offs than existing methods.
diffusion models text-to-image orthographic projections latent triplane textured mesh per-sample optimization