  Reconstruction of geometric structures from images using supervised learning
suffers from limited available amount of accurate data. One type of such data
is accurate real-world RGB-D images. A major challenge in acquiring such ground
truth data is the accurate alignment between RGB images and the point cloud
measured by a depth scanner. To overcome this difficulty, we consider a
differential optimization method that aligns a colored point cloud with a given
color image through iterative geometric and color matching. In the proposed
framework, the optimization minimizes the photometric difference between the
colors of the point cloud and the corresponding colors of the image pixels.
Unlike other methods that try to reduce this photometric error, we analyze the
computation of the gradient on the image plane and propose a different direct
scheme. We assume that the colors produced by the geometric scanner camera and
the color camera sensor are different and therefore characterized by different
chromatic acquisition properties. Under these multimodal conditions, we find
the transformation between the camera image and the point cloud colors. We
alternately optimize for aligning the position of the point cloud and matching
the different color spaces. The alignments produced by the proposed method are
demonstrated on both synthetic data with quantitative evaluation and real
scenes with qualitative results.

Computer Vision and Pattern Recognition