Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
<|reference_start|>Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance: Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios.<|reference_end|>
@article{lu2024sharpness-diversity,
    title={Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance},
    author={Haiquan Lu, Xiaotian Liu, Yefan Zhou, Qunli Li, Kurt Keutzer, Michael
  W. Mahoney, Yujun Yan, Huanrui Yang, Yaoqing Yang},
    journal={arXiv preprint arXiv:2407.12996},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12996},
    primaryClass={stat.ML cs.LG}
}