Making Robust Generalizers Less Rigid with Soft Ascent-Descent
<|reference_start|>Making Robust Generalizers Less Rigid with Soft Ascent-Descent: While the traditional formulation of machine learning tasks is in terms of performance on average, in practice we are often interested in how well a trained model performs on rare or difficult data points at test time. To achieve more robust and balanced generalization, methods applying sharpness-aware minimization to a subset of worst-case examples have proven successful for image classification tasks, but only using deep neural networks in a scenario where the most difficult points are also the least common. In this work, we show how such a strategy can dramatically break down under more diverse models, and as a more robust alternative, instead of typical sharpness we propose and evaluate a training criterion which penalizes poor loss concentration, which can be easily combined with loss transformations such as CVaR or DRO that control tail emphasis.<|reference_end|>
@article{holland2024making,
    title={Making Robust Generalizers Less Rigid with Soft Ascent-Descent},
    author={Matthew J. Holland, Toma Hamada},
    journal={arXiv preprint arXiv:2408.03619},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.03619},
    primaryClass={cs.LG}
}