Thanks for the information!
How big is your model? Could you profile it again with torch.utils.bottleneck 597?
Since CUDA operations are run asynchronously, your DataLoader might have to wait for the CUDA op to finish, thus reporting a false number.
An alternative would be to add torch.cuda.synchronize() calls before and after the forward and backward calls, and time it manually.