I guess there might be a mismatch between the x and y coordinates in the image coordinate system and the rows and columns in affine_grid. Could you try to swap the coordinates of the axes and rerun the code?