title: Calculate surface normals from depth image using neighboring pixels cross product
body: <p>As the title says I want to calculate the surface normals of a given depth image by using the cross product of neighboring pixels. I would like to use Opencv for that and avoid using PCL however, I do not really understand the procedure, since my knowledge is quite limited in the subject. Therefore, I would be grateful is someone could provide some hints. To mention here that I do not have any other information except the depth image and the corresponding rgb image, so no <code>K</code> camera matrix information.</p>
  
  <p>Thus, lets say that we have the following depth image:</p>
  
  <p><a href="https://i.stack.imgur.com/Air7G.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Air7G.png" alt="enter image description here"></a></p>
  
  <p>and I want to find the normal vector at a corresponding point with a corresponding depth value like in the following image:</p>
  
  <p><a href="https://i.stack.imgur.com/ISL4l.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ISL4l.png" alt="enter image description here"></a> </p>
  
  <p>How can I do that using the cross product of the neighbouring pixels? I do not mind if the normals are not highly accurate.</p>
  
  <p>Thanks.</p>
  
  <hr>
  
  <p><strong>Update:</strong></p>
  
  <p>Ok, I was trying to follow @timday's answer and port his code to Opencv. With the following code:</p>
  
  <pre><code>Mat depth = &lt;my_depth_image&gt; of type CV_32FC1
  Mat normals(depth.size(), CV_32FC3);
  
  for(int x = 0; x &lt; depth.rows; ++x)
  {
      for(int y = 0; y &lt; depth.cols; ++y)
      {
  
          float dzdx = (depth.at&lt;float&gt;(x+1, y) - depth.at&lt;float&gt;(x-1, y)) / 2.0;
          float dzdy = (depth.at&lt;float&gt;(x, y+1) - depth.at&lt;float&gt;(x, y-1)) / 2.0;
  
          Vec3f d(-dzdx, -dzdy, 1.0f);
          Vec3f n = normalize(d);
  
          normals.at&lt;Vec3f&gt;(x, y) = n;
      }
  }
  
  imshow("depth", depth / 255);
  imshow("normals", normals);
  </code></pre>
  
  <p>I am getting the correct following result (I had to replace <code>double</code> with <code>float</code> and <code>Vecd</code> to <code>Vecf</code>, I do not know why that would make any difference though):</p>
  
  <p><a href="https://i.stack.imgur.com/SYhcY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/SYhcY.png" alt="enter image description here"></a></p>