Probably you have a bottleneck somewhere, so that your GPU is starving.
I assume you using a DataLoader. Could you increase num_workers?
Are you using pin_memory=True? Is your data on an SSD?
Have a look at this line of code 822 from the ImageNet example to check, if your DataLoader is the reason.
Alternatively, you can have a look aat torch.utils.bottleneck 487 for further debugging.