I need the gradient of this warping operation.
// - tensor   :  output of my neural network 
//               tensor.requires_grad = True

warped_tensor = F.grid_sample(tensor, 
                              grid,
                              align_corners=True,
                              mode='bilinear',
                              padding_mode='zeros').

This operation returns a gradient, however it seems to be not correct. I used the warped_tensor and just the tensor for my loss and with the warped_tensor my network does not correctly optimise the weights.
Is this an autograd issue or is there some other issue I am not seeing here?

The same happens when I use #vision:kornia
// - tensor   :  output of my neural network 
//               tensor.requires_grad = True
// - M      : transformation Matrix prev created with kornia

warped_tensor: torch.tensor = kornia.warp_affine(tensor, M, dsize=(h_original, w_original))