Solved by eqy in post #2 
              
              
                Have you tried alternative rotation implementations (e.g., skimage’s rotate or albumentations’s rotate)? 
Albumentations in particular claims to be very fast for rotation: benchmark.