Stochastic Hessian Fittings with Lie Groups
<|reference_start|>Stochastic Hessian Fittings with Lie Groups: This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterative solutions, using Hessian-vector products or stochastic gradients only, with Hessian fittings in the Euclidean space, the manifold of symmetric positive definite (SPL) matrices, to a variety of Lie groups. The most intriguing discovery is that the Hessian fitting itself as an optimization problem is strongly convex under mild conditions with a specific yet general enough Lie group. This discovery turns Hessian fitting into a well behaved optimization problem, and facilitates the designs of highly efficient and elegant Lie group sparse preconditioner fitting methods for large scale stochastic optimizations.<|reference_end|>
@article{li2024stochastic,
    title={Stochastic Hessian Fittings with Lie Groups},
    author={Xi-Lin Li},
    journal={arXiv preprint arXiv:2402.11858},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11858},
    primaryClass={stat.ML cs.LG math.OC}
}