The small peaks in your GPU utilization might come from a bottleneck in your code, e.g. loading and processing the data.
Are you using multiple workers (with pin_memory=True)?
Also, is your data stored on an SSD?
Have a look at these lines of code 13 from the ImageNet example to see how to time the DataLoader.