Exactly what I thought. Though the fact that not zeroing the gradients actually worked, suggests that with ELU activation and MSELoss the problem was nicely convex, so that repeatedly taking steps in roughly the same direction pretty nearly always led to the solution.