H-Fac: Memory-Efficient Optimization with Factorized Hamiltonian Descent
<|reference_start|>H-Fac: Memory-Efficient Optimization with Factorized Hamiltonian Descent: In this study, we introduce a novel adaptive optimizer, H-Fac, which incorporates a factorized approach to momentum and scaling parameters. Our algorithm demonstrates competitive performances on both ResNets and Vision Transformers, while achieving sublinear memory costs through the use of rank-1 parameterizations for moment estimators. We develop our algorithms based on principles derived from Hamiltonian dynamics, providing robust theoretical underpinnings. These optimization algorithms are designed to be both straightforward and adaptable, facilitating easy implementation in diverse settings.<|reference_end|>
@article{nguyen2024memory-efficient,
    title={Memory-Efficient Optimization with Factorized Hamiltonian Descent},
    author={Son Nguyen, Lizhang Chen, Bo Liu, Qiang Liu},
    journal={arXiv preprint arXiv:2406.09958},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09958},
    primaryClass={cs.LG}
}