Hi-It could be the case that your dataloader is taking more time than model forward or backward
Check the time to load images properly. Use time.per_counter() instead of time.time() and torch.cuda.synchronize() before you do the time reading for eg.
torch.cuda.synchronize()
t0 = time.perf_counter()
ouput = model(input)
loss = criterion(ouput, target)
loss.backward()
torch.cuda.synchronize() 
print('Time taken',time.perf_counter()-t0)