Thanks for your reply. I think I have found the reason, it’s because the data loader is a little slow, especially the transform functions. The training time is the same whether torch.backends.cudnn.benchmark is True or False, PyTorch may call cudnn automatically.
As your replay, PyTorch and Caffe has no large discrepancies, are there any more details or links?