  We propose a novel 3d colored shape reconstruction method from a single RGB
image through diffusion model. Diffusion models have shown great development
potentials for high-quality 3D shape generation. However, most existing work
based on diffusion models only focus on geometric shape generation, they cannot
either accomplish 3D reconstruction from a single image, or produce 3D
geometric shape with color information. In this work, we propose to reconstruct
a 3D colored shape from a single RGB image through a novel conditional
diffusion model. The reverse process of the proposed diffusion model is
consisted of three modules, shape prediction module, color prediction module
and NeRF-like rendering module. In shape prediction module, the reference RGB
image is first encoded into a high-level shape feature and then the shape
feature is utilized as a condition to predict the reverse geometric noise in
diffusion model. Then the color of each 3D point updated in shape prediction
module is predicted by color prediction module. Finally, a NeRF-like rendering
module is designed to render the colored point cloud predicted by the former
two modules to 2D image space to guide the training conditioned only on a
reference image. As far as the authors know, the proposed method is the first
diffusion model for 3D colored shape reconstruction from a single RGB image.
Experimental results demonstrate that the proposed method achieves competitive
performance on colored 3D shape reconstruction, and the ablation study
validates the positive role of the color prediction module in improving the
reconstruction quality of 3D geometric point cloud.

Computer Vision and Pattern Recognition