Awesome work. There are a few other notable optimizers that may be worth comparing your work to, although I’m no expert:


AdaBelief: [2010.07468] AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients 6  (fast convergence/stability) (2020)

Adafactor: [1804.04235] Adafactor: Adaptive Learning Rates with Sublinear Memory Cost 5 (saves memory/fast training) (2018)

Fromage: [2002.03432] On the distance between two neural networks and the stability of learning 5 (no learning rate tuning/works on GANs and Transformers) (2020)

LAMB: [1904.00962] Large Batch Optimization for Deep Learning: Training BERT in 76 minutes 3 (consistent training performance/Transformers & ResNets)

SM3 (Square-root of Minima of Sums of Maxima of Squared-gradients Method): [1901.11150] Memory-Efficient Adaptive Optimization 6 (memory efficient & adaptive/designed to decrease memory overhead especially with large models like Transformers-based BERT, etc.) (2019)

Yogi: http://www.sanjivk.com/yogi_nips2018.pdf 7 (similar to Adam/convergence and generalisation/focuses on Adam and RMSprop’s issues) (2020)