This might indicate that you are running into a CPU bottleneck, e.g. from data loading.
You could try to adapt the data loading measurements from the ImageNet example 105 to check, if that’s the case or not.