You can’t “ignore train code”, because it’s what decides how fast will your script run. In our benchmarks we don’t see large discrepancies between PyTorch and Caffe. One thing that might make it faster would be to set torch.backends.cudnn.benchmark = True