The internal workings of some optimization algorithms (some examples are Adam, its variants, BFGS, and its variants) have “memory-based” components that generally improve their performance over very naive direct gradient descent procedures. Adam uses running averages over quantities computed in previous iterations, and L-BFGS stores previous gradients in order to compute an approximate Hessian (which improves the ability of the algorithms to “step” in the correct direction).
When using these types of algorithms, it is detrimental to stop the optimization and then initialize a completely new optimizer, (even if it is of the same algorithm) because the “memory” effects are thrown away.
I’m not an expert on saving the state of an optimizer, but with my moderate experience using L-BFGS I do believe that initializing a new optimizer using the state of the original optimizer will allow you to retain the “memory” from the old one. This should hopefully lead to very similar performance to the case in which you keep the old optimizer running.
These are just general guidelines for how the algorithms themselves work. I’m not sure exactly how they apply to the specific problem instance that you posted above.