Weight initialisation is done through Xavier’s approach :
m.weight.data.normal_(0, math.sqrt(2. / n)), for each conv module m in the network
m.weight.data.normal_(0, 0.01), for the fc layer on top of the net.
As for momentum, I used the commonly 0.9 value although I do not know it that suits the learning rate I have since this one is really low.
I have to admit I am not really familiar with the relation between learning rate and momentum since both of them seem to impact the weight update on different aspects.
I was looking for more documentation about these two hyper-parameters when I came across weight decay. I understood that it aims at reducing the magnitude increase of the weights. Is that correct ?
Not sure how to tune momentum (and weight decay ?) so as to get a decent learning rate.