You could remove all data loading code, create random tensors on the GPU, and profile the training separately. This would give you an idea, if the data loading and processing might be the bottleneck or any other part of your training.