I am interested in understanding the precise purpose of the max_iter kwarg in the LBFGS optimizer. As far as I can tell from inspecting the code, it has no effect on the number of gradients that are used to construct the limited memory Hessian, and similarly it has no direct effect on the number of linesearch calls or any other internal workings of the LBFGS algorithm. It seems that the only function of max_iter is to group a certain number of LBFGS steps together within one outwardly exposed step of the Optimizer class.
If there is something more to max_iter I would be very grateful if someone could explain it.
If this is truly the only function of max_iter, then why is the default value set to 20? This seems to be a very strange choice (and a very strange functionality to have in general) when none of the first-order optimizers (Adam, SGD, etc) have anything similar to this, which implicitly makes their permanent value of max_iter set to 1. The effect of max_iter > 1 in LBFGS just makes the algorithm appear to run extremely slow (compared to the first-order methods), but have crazy good convergence relative to the number of optimizer steps (compared to the first-order methods). Of course, compared on equal footing (i.e. max_iter = 1) , LBFGS with no linesearch is only a little bit slower than Adam and converges only a little bit faster per step. I would think that it would be easier for users to understand the true differences in performance between the two algorithms if the default value of max_iter was set to 1 instead of 20.
Thanks!