Hi Emilien,
I am currently also playing with affine_grid and grid_sample and found your post via the forum search.
When I rotate 3D data in my setup I get a rotated and sheared output.
So far I am still trying to find the root cause but I expect that something unexpected happens in grid_sample I am not taking into account (currently I’m looking into the implications of this github comment 8, but I still have to think about it).
Have you tried to see if you rotation is preserving edge lengths of your cube, i.e., it only rotates and does not shear too?  (I guess a small shear can be quite tricky to see in the 3D visualizations.)
Maybe a more basic setup where you only rotate in one dimension can help you pinpoint the problem better?
Kind regards
Michael