Yeah, you are right. If you want to use more complex transformations that are not affine, then you need to generate your own custom grid to pass to grid_sample.
If you want, you can always start by calling affine_grid and then modifying the grid that you get to your liking before passing it to grid_sample, but that’s entirely up to you.
Just keep in mind that if you generate your own grid by hand, you need to worry about things like which way you are setting the align_corners parameter in order to make sure that the grid you are producing has the right values to get your intended effect.