bheinzerling:

@jpeg729, you’re missing optimizer.zero_grad() at the start of the training loop.


Oh. Oops. And I’m the one who pointed out that it was missing in the original code. 
I tried again with ELU and SGD, and got 2% success. Tuning the learning rate does nothing much, but adding momentum=.9 gives 98% success with SGD.
I suppose all this goes to show the vast number of ways we can alter our models and get widely differing results.