Thanks for all the tips about BatchNorm, weight inits, and the thought about the 2x GPU. I didn’t realize SGD was so sensitive without these considerations.
Last question: is there a succinct explanation for why Adam optimizes Alexnet handily given many different hyper-params (which I’ve tried) while SGD is stubborn? I’ve looked over the Adam paper and understand it estimates “exponential moving averages of the gradient and the squared gradient”, but I don’t understand how this takes care of an issue which, like you say, BatchNorm and other nuances about the batchsize could solve.