Solved by ptrblck in post #2 
              
              
                Are you using multiple workers in your DataLoader? 
If so, could you try to increase the number? 
It looks like your training procedure has to wait sometimes for the DataLoader to provide new batches. Also, is your data locally on an SSD?