I would recommend to check the Performance Guide 6 and check if you could enable e.g. cudnn.benchmark or use automatic mixed-precision training.
Also, you could profile the code using the PyTorch Profiler and see where the bottlenecks are (e.g. in the data loading or the model itself).