Strong convexity-guided hyper-parameter optimization for flatter losses
<|reference_start|>Strong convexity-guided hyper-parameter optimization for flatter losses: We propose a novel white-box approach to hyper-parameter optimization. Motivated by recent work establishing a relationship between flat minima and generalization, we first establish a relationship between the strong convexity of the loss and its flatness. Based on this, we seek to find hyper-parameter configurations that improve flatness by minimizing the strong convexity of the loss. By using the structure of the underlying neural network, we derive closed-form equations to approximate the strong convexity parameter, and attempt to find hyper-parameters that minimize it in a randomized fashion. Through experiments on 14 classification datasets, we show that our method achieves strong performance at a fraction of the runtime.<|reference_end|>
@article{yedida2024strong,
    title={Strong convexity-guided hyper-parameter optimization for flatter losses},
    author={Rahul Yedida, Snehanshu Saha},
    journal={arXiv preprint arXiv:2402.05025},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.05025},
    primaryClass={cs.LG}
}