punto:

Interesting about Adam being the go-to, I will remember that.


Yeah, but also be careful about it and don’t limit yourself, as I’m usually not working on training an entire model end2end, but just debug some convergence issues. Adam might not be the go-to anymore.
You should check the current literature on optimizers in ML and also refer to knowingly working repositories to see, what they are using.
@rwightman would know a lot more about successfully training state of the art CV models.