This could indicate that your training loop might be increasing the computation graph and thus not the data loading but the training is slower.
Could you remove the complete model and training, and just time the data loading loop?