Did you train from scratch?
From what I have heard from people, since pytorch is relatively new it may have some design issues such has initialization. People may have hard time reproducing results from a different environment such as tf or caffe.