Maybe it is because Caffe assume image is in BGR format while PyTorch is assuming Image is in RGB format? So you may need to swap the order of first conv layer weight or you need to make sure Image feed to network is in BGR format in PyTorch.