Diffusion models have demonstrated amazing capabilities
in generating highly detailed images based on an input
prompt or condition [16,20,23]. The use of depth estimates
has previously been used in diffusion models as an addi-
tional condition to perform depth-to-image generation [31].
Later, [24] and [6] showed that monocular depth estima-
tion can also be modeled as a denoising diffusion process
through the use of images as an input condition. In this work
we propose a diffusion model that simultaneously gener-
ates an RGB image and its corresponding depth map given
a text prompt as input. While our proposed model may be
functionally comparable to an image generation and depth
estimation model in cascade, there are several differences,
challenges, and beneﬁts of our proposed combined model.
An adequate monocular depth estimation model requires
large and diverse data [15,19], however, as there is no depth
ground truth available for generated images it is hard for off-
LDM3D- Latent Diffusion Model for 3D