Glad you found this useful. In my own work I’ve never found Adam actually outperforming SGD but many people use Adam / AdamW for their training so I’m not sure if there are any more tricks that I’m missing.