For SGD, I used 0.01 for both Caffe and Pytorch. 0.1 was too high thats why I reduced it to 0.01.
What is ??? vs Xavier ? I believe Caffe Model also uses Xavier and I also used Xavier in Pytorch .
Concerning decay on all params vs some can you direct me to any Pytorch documentation ?
How can I change this behavior in Pytorch ( Is it even possible? )
Now that you mentioned it, I notice Pytorch seem to devide all image values by 255 while it is not customary in Caffe ( at least I did not do anything when I was training the model in Caffe)