title: Cannot undistort points using OpenCv undistortPoints method
body: <p>I'm trying to undistort certain points of an image using the "undistort points" method of OpenCV, but without success.
  <a href="https://i.stack.imgur.com/200Xy.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/200Xy.jpg" alt="Here an example where I undistort all the whole picture"></a></p>
  
  <p>These will be my undistorsion coefficients</p>
  
  <pre><code>  optic_camera_matrix: [[710.52285,  0.0,      882.14702],
                          [0.0,        713.9636, 638.8421],
                          [0.0,        0.0,      1.0]],
  
    distorsion_coeffs: [[-0.4176419401669212,
                                   0.15978235598732332,
                                   -8.299875092923166e-05,
                                   -0.001784191694247801,
                                   -0.027396621999692457]],
  </code></pre>
  
  <p>Even though I am capable of undistorting the whole image, in order to optimize the camera processing time, if I undistort just the corner points (red points of the image):</p>
  
  <pre><code>distorted_border_points = np.array([[[584,1415],
                                          [576,457],
                                          [1956,415],
                                          [1996,1422],
                                          [1261,242],
                                          [1281,1594]]],np.float32)
  
  undistorted_points =  cv2.undistortPoints(distorted_border_points, optic_camera_matrix, distorsion_coeffs)
  </code></pre>
  
  <p>I get this in return: </p>
  
  <pre><code>[[[ -6.40190065e-01   1.66883194e+00]
    [ -4.87006754e-01  -2.88225353e-01]
    [ -1.82562262e-01   3.74070629e-02]
    [ -5.28450182e-04  -3.51850584e-04]
    [  8.09574544e-01  -8.40054870e-01]
    [ -5.28259724e-02  -1.22379906e-01]]]
  </code></pre>
  
  <p>If plotted, they don't align in a rectangle, like in the first image.</p>
  
  <p>I believe the undistorsion coefficients are well calcualted (since undistorsion works on the first image), but here I attach the code of the cam</p>
  
  <pre><code>import glob
  import cv2
  import numpy as np
  import os
  import json
  import numpy as np
  
  directory = os.path.dirname(__file__)
  
  
  def get_optic_calibration_parameters(device,config_folder=None):
      if config_folder is None:
          optic_calibration_path = directory + '/../config/' + \
              device + '/optic_calibration.json'
      else:
          optic_calibration_path = config_folder + device + '/optic_calibration.json'
  
      if not os.path.exists(optic_calibration_path):
          os.makedirs(optic_calibration_path[:-22])
  
      with open(optic_calibration_path) as optic_calibration_file:
          optic_calibration = json.load(optic_calibration_file)
  
      optic_camera_matrix = optic_calibration['optic_camera_matrix']
      distorsion_coeffs = optic_calibration['distorsion_coeffs']
      optic_resolution = optic_calibration['optic_resolution']
  
      return optic_camera_matrix, distorsion_coeffs, optic_resolution
  
  
  def _save_calibration_parameters(camera_matrix, distorsion_coeffs, optic_resolution, device, config_folder=None):
  
      if config_folder is None:
          optic_calibration_path = directory + '/../config/' + \
              device + '/optic_calibration.json'
      else:
          optic_calibration_path = config_folder + device + '/optic_calibration.json'
  
      if not os.path.exists(optic_calibration_path):
          os.makedirs(optic_calibration_path[:-22])
  
      optic_calibration_parameters = {'optic_camera_matrix': camera_matrix.tolist(),
                                      'distorsion_coeffs': distorsion_coeffs.tolist(),
                                      'optic_resolution': optic_resolution}
  
      with open(optic_calibration_path, 'wb') as optic_calibration_file:
          json.dump(optic_calibration_parameters, optic_calibration_file)
  
      return
  
  
  def _get_image_points(plot=False, dim=(4, 5), input_dir='calibration_samples',extension='jp'):
      criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
  
      # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
      objp = np.zeros((dim[0] * dim[1], 3), np.float32)
      objp[:, :2] = np.mgrid[0:dim[1], 0:dim[0]].T.reshape(-1, 2)
  
      # Arrays to store object points and image points from all the images.
      objpoints = []  # 3d point in real world space
      imgpoints = []  # 2d points in image plane.
  
      images = glob.glob(input_dir + '*.'+extension+'*')
      for fname in images:
          img = cv2.imread(fname)
          gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
  
          # Find the chess board corners
          ret, corners = cv2.findChessboardCorners(gray, (dim[1], dim[0]), None)
  
          # If found, add object points, image points (after refining them)
          if ret == True:
              objpoints.append(objp)
  
              corners2 = cv2.cornerSubPix(
                  gray, corners, (11, 11), (-1, -1), criteria)
              imgpoints.append(corners2)
  
              # Draw and display the corners
              if plot:
                  img = cv2.drawChessboardCorners(
                      img, (dim[1], dim[0]), corners2, ret)
                  cv2.imshow('img', img)
                  cv2.waitKey(500)
  
      resolution = (img.shape[1], img.shape[0])
  
      return imgpoints, objpoints, resolution
  
  
  def calibrate_camera(optic_resolution, imgpoints, objpoints, device, config_folder=None):
  
      _, camera_matrix, distorsion_coeffs, _, _ = cv2.calibrateCamera(
          objpoints, imgpoints, optic_resolution, None, None)
  
      _save_calibration_parameters(
          camera_matrix, distorsion_coeffs, optic_resolution, device, config_folder=config_folder)
  
      return
  </code></pre>
  
  <p>To execute the function I loaded different images:</p>
  
  <pre><code>imgpoints, objpoints, optic_resolution = _get_image_points(plot=False, dim=(4,5), input_dir=calibration_samples)
  _show_N_chessborders(N=3, dim=(4,5), input_dir=calibration_samples)
  calibrate_camera(optic_resolution, imgpoints, objpoints,device, config_folder=config_folder)
  </code></pre>
  
  <p>And that's how I store the json config files</p>
  
  <p>I would appreciate it if anyoce could help me with a solution. Thank you!</p>