You could profile your application with the native PyTorch profiler or e.g. Nsight Systems and could try to narrow down where the bottleneck is. Once isolated the improvements depend on the actual root cause. For general recommendation check the performance guide 1.