title: Mismatch between Point Projection and Warped 2-D Image [opencv]
body: <p>I am using 2 different methods to render an image (as an opencv Matrix):</p>
  
  <ol>
  <li><p>an implemented projection function that uses the camera intrinsics (focal length, principal point; distortion is disabled) - this function is used in other software packages and is supposed to work correctly (<a href="https://github.com/ethz-asl/okvis/tree/master/okvis_cv/include/okvis/cameras" rel="nofollow noreferrer">repository</a>)</p></li>
  <li><p>a 2D to 2D image warping (here, I'm determining the intersections of the corner-rays of my camera with my 2D image that should be warped into my camera frame); this backprojection of the corner points is using the same camera model as above</p></li>
  </ol>
  
  <p>Now, I overlay these two images and what should basically happen is that a projected pen-tip (method 1.) should line up with a line that is drawn on the warped image (method 2.). However, this is not happening. </p>
  
  <p>There is a tiny shift in both directions, depending on the orientation of the pen that is writing, and it is reduced when I am shifting the principal point of the camera. Now my question is, since I am not considering the principal point in the 2D-2D image warping, can this be the cause of the mismatch? Or is it generally impossible to align those two, since the image warping is a simplification of the projection process?</p>
  
  <p><a href="https://i.stack.imgur.com/3Gbpa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3Gbpa.png" alt="enter image description here"></a></p>
  
  <ul>
  <li><p>Grey Point: projected origin (should fall in line with the edges of the white area)</p></li>
  <li><p>Blue Reticle: penTip that should "write" the Bordeaux-colored line</p></li>
  <li><p>Grey Line: pen approximation</p></li>
  <li><p>Red Edge: "x-axis" of white image part</p></li>
  <li><p>Green Edge: "y-axis" of white image part</p></li>
  </ul>
  
  <p>EDIT:</p>
  
  <p>I also did the same projection with the origin of the coordinate system, and here, the mismatch grows, the further the origin moves out of the center of the image. (so <code>delta[warp,project]</code> gets larger on the image borders compare to the center)</p>