Figure 1. LDM3D overview. Illustrating the training pipeline: the 16-bit grayscale depth maps are packed into 3-channel RGB-like depth
images, which are then concatenated with the RGB images along the channel dimension. This concatenated RGBD input is passed through
the modiﬁed KL-AE and mapped to the latent space. Noise is added to the latent representation, which is then iteratively denoised by
the U-Net model. The text prompt is encoded using a frozen CLIP-text encoder and mapped to various layers of the U-Net using cross-
attention. The denoised output from the latent space is fed into the KL-decoder and mapped back to pixel space as a 6-channel RGBD
output. Finally, the output is separated into an RGB image and a 16-bit grayscale depth map. Blue frame: text-to-image inference pipeline.
Initiating from a Gaussian distributed noise sample in the 64x64x4-dimensional latent space. Given a text prompt, this pipeline generates
an RGB image and its corresponding depth map.
LDM3D- Latent Diffusion Model for 3D