It seems you are seeing a slowdown in training and it shouldn’t depend on the training/eval mode of the model.
Could you adapt the data loading measurement from the ImageNet example 19?
data_time should go towards zero if you don’t have a data loading bottleneck.
Also, are you the only user on this machine or could the machine be blocked by other processes during your training?
Do you see any increase in GPU memory usage during training?