Small update. I figured out that the gradient problem is mainly because of Adam storing momentum parameters that should also decrease in size. However, hacking around with that seems suboptimal. Any suggestions?