Hi,
Without looking too much into detail I can tell you that the gradient of mat = F.grid_sample(mat, grid, mode='nearest') is 0 because of mode=nearest. Try with any other option and you should have a non-zero gradient.
Hope this helps.