  This report provides an introduction to the ensmallen numerical optimization
library, as well as a deep dive into the technical details of how it works. The
library provides a fast and flexible C++ framework for mathematical
optimization of arbitrary user-supplied functions. A large set of pre-built
optimizers is provided, including many variants of Stochastic Gradient Descent
and Quasi-Newton optimizers. Several types of objective functions are
supported, including differentiable, separable, constrained, and categorical
objective functions. Implementation of a new optimizer requires only one
method, while a new objective function requires typically only one or two C++
methods. Through internal use of C++ template metaprogramming, ensmallen
provides support for arbitrary user-supplied callbacks and automatic inference
of unsupplied methods without any runtime overhead. Empirical comparisons show
that ensmallen outperforms other optimization frameworks (such as Julia and
SciPy), sometimes by large margins. The library is available at
https://ensmallen.org and is distributed under the permissive BSD license.

Mathematical Software;Machine Learning;Software Engineering;Optimization and Control