jpeg729:

I tried again with ELU and SGD, and got 2% success. Tuning the learning rate does nothing much, but adding momentum=.9 gives 98% success with SGD.


That’s probably because not zeroing the gradient has the same effect as adding momentum: it retains some of the gradient from the previous training step.