The issue sounds indeed like a data loading bottleneck similar to this one 1 so you could profile your code to isolate the bottleneck further.
As a quick test, you could replace the data loading with a single random tensor and let the model train on it. If the periodic slowdown is gone, it would point towards the data loading.