  The optimization algorithm and its hyperparameters can significantly affect
the training speed and resulting model accuracy in machine learning
applications. The wish list for an ideal optimizer includes fast and smooth
convergence to low error, low computational demand, and general applicability.
Our recently introduced continual resilient (CoRe) optimizer has shown superior
performance compared to other state-of-the-art first-order gradient-based
optimizers for training lifelong machine learning potentials. In this work we
provide an extensive performance comparison of the CoRe optimizer and nine
other optimization algorithms including the Adam optimizer and resilient
backpropagation (RPROP) for diverse machine learning tasks. We analyze the
influence of different hyperparameters and provide generally applicable values.
The CoRe optimizer yields best or competitive performance in every investigated
application, while only one hyperparameter needs to be changed depending on
mini-batch or batch learning.

Machine Learning;Optimization and Control;Computational Physics