If I understand the grid_sample and affine_grid method correctly, you are defining a sampling grid, where each position gives you the pixel location of the input image.
If your grid is thus containing values in all positions, it would just fill these using the values.
Thus, if you don’t want to fill the complete output image (i.e. leave the borders empty to maintain the aspect ratio), you would have to change the grid itself.