Solved by Dawid_S in post #5 
              
              
                Well, eventually I was able to train an almost sensible neural net using Adam with 0.0001 or 0.00001 lr, I don’t remember. It was still clearly worse than SGD so I abandoned it, but I was comfortable with the fact that it’s probably possible, so maybe I don’t have any NN bugs