Interesting about Adam being the go-to, I will remember that. No problem and thanks for the help, last thing, would you be able to direct me to anywhere that can educate me on the “nuance” of these optimizers (beyond the basics; I’ve taken a look at the docs and the Adam paper but didn’t learn much practical knowledge to tackle whatever issue I’m seeing)?

With that being said, you might want to adapt the learning rate in SGD and see, if this could help.

Also, I tried this, as well as other tweaks to the SGD optimizer, to no avail  I’m currently baffled