Thanks for your time, I figured it out! It came down to calculating a batch of rotation matrices theta and constructing a batch of affine_grid based on that.
The only problem is that the first call to affine_grid takes a long time. I suspect it’s doing some kind of initialization?