Caffe also uses Xavier, the dropout ratio is the same, the weight decay is the same, however , Caffe applies them to all layers, unlike how it is done in Pytorch (Added in the optimizer), the optimization policy is also the same. I also set the batch_size equally in Pytorch.
So basically I tried my best to match everything from Caffe.
My concern is, from a beginner point of view, creating and training an architecture in Caffe seems to provide much better result than in Pytorch. This would not be good if I want to achieve the best possible result !
I want to use Pytorch for everything and outperform all other frameworks.