They still look different to me, after glancing throw the caffe & pytorch code. I don’t know how to fix that, you just have to do the research yourself.
You mentioned pytorch is way faster than caffe, which is weird too, most libraries should have similar speed, so something feel fishy here.
Finally, regarding caffe providing better result, I think it’s just that you are a beginner. And you are copying paste the caffe model from somebody’s and attempt to implement a pytorch model by yourself. If you use a pytorch model proved to work by somebody and compare it with a caffe model you write from scratch yourself, the result could be the other way around