Beyond automatic differentiation
> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting).I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity of SGD is important for performance! There is no reason to expect results with unrealistic assumptions like these to generalize to the cases people actually care about.> at the cost of a single additional forward pass that increases the wall time for each step by a small factor (about 2x in the example above).So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps.