Going from tensors to numpy arrays and viceversa is not recommended, especially when you wan to compute gradients over this tensors.
Have you tried to work only on tensors?
Maybe you can code that Rotation using pytorch functions, or maybe there are some libraries written in pytorch you could use.
For instance, I found this method from PyTorch3D library: pytorch3d.transforms 5