Sharpness-Aware Minimization for Evolutionary Feature Construction in Regression
<|reference_start|>Sharpness-Aware Minimization for Evolutionary Feature Construction in Regression: In recent years, genetic programming (GP)-based evolutionary feature construction has achieved significant success. However, a primary challenge with evolutionary feature construction is its tendency to overfit the training data, resulting in poor generalization on unseen data. In this research, we draw inspiration from PAC-Bayesian theory and propose using sharpness-aware minimization in function space to discover symbolic features that exhibit robust performance within a smooth loss landscape in the semantic space. By optimizing sharpness in conjunction with cross-validation loss, as well as designing a sharpness reduction layer, the proposed method effectively mitigates the overfitting problem of GP, especially when dealing with a limited number of instances or in the presence of label noise. Experimental results on 58 real-world regression datasets show that our approach outperforms standard GP as well as six state-of-the-art complexity measurement methods for GP in controlling overfitting. Furthermore, the ensemble version of GP with sharpness-aware minimization demonstrates superior performance compared to nine fine-tuned machine learning and symbolic regression algorithms, including XGBoost and LightGBM.<|reference_end|>
@article{zhang2024sharpness-aware,
    title={Sharpness-Aware Minimization for Evolutionary Feature Construction in
  Regression},
    author={Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang},
    journal={arXiv preprint arXiv:2405.06869},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.06869},
    primaryClass={cs.LG cs.NE}
}