This makes sense! I am quite certain now that my error lies here.
Here is a gist using kornia.warp_affine  for the transformation as they use F.grid_sample under the hood.  (source for kornia.warp_affine 4)
Here is my code:

  
      gist.github.com
  
  
    https://gist.github.com/mowolf/fa1b0c327ddd3add758bac5d990a4638 18
mock_transformation_network.py
import kornia
import numpy as np
import torch.nn.functional as F


def get_M():
    '''
    Provide mock trafo Matrix and original image sizes
    :return: w_original, h_original, M 
    '''
This file has been truncated. show original




  
  
    
    
  
  



How can I now make sure that my gradient is not loosing precision? Must the size of the resulting trafo tensor just be big enough? The interpolation with nearest might also have this problem. Would bilinear fix that or should I make sure that I dont scale down?