I gave my code more time and it started to learn something. But it trains much slower and a final loss is much higher then it should be (if you compare with other implementations 3, a similar number of parameters, architecture, and learning rate)
@mathematicsofpaul