Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift
<|reference_start|>Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift: Natural distribution shift causes a deterioration in the perception performance of convolutional neural networks (CNNs). This comprehensive analysis for real-world traffic data addresses: 1) investigating the effect of natural distribution shift and weather augmentations on both detection quality and confidence estimation, 2) evaluating model performance for both classification and object localization, and 3) benchmarking two common uncertainty quantification methods - Ensembles and different variants of Monte-Carlo (MC) Dropout - under natural and close-to-natural distribution shift. For this purpose, a novel dataset has been curated from publicly available autonomous driving datasets. The in-distribution (ID) data is based on cutouts of a single object, for which both class and bounding box annotations are available. The six distribution-shift datasets cover adverse weather scenarios, simulated rain and fog, corner cases, and out-of-distribution data. A granular analysis of CNNs under distribution shift allows to quantize the impact of different types of shifts on both, task performance and confidence estimation: ConvNeXt-Tiny is more robust than EfficientNet-B0; heavy rain degrades classification stronger than localization, contrary to heavy fog; integrating MC-Dropout into selected layers only has the potential to enhance task performance and confidence estimation, whereby the identification of these layers depends on the type of distribution shift and the considered task.<|reference_end|>
@article{diet2024prediction,
    title={Prediction Accuracy & Reliability: Classification and Object
  Localization under Distribution Shift},
    author={Fabian Diet, Moussa Kassem Sbeyti, Michelle Karg},
    journal={arXiv preprint arXiv:2409.03543},
    year={2024},
    doi={10.1007/978-3-031-66842-5_9},
    number={This is a preprint of the following chapter: F. Diet, M. Kassem
  Sbeyti & M. Karg, Prediction Accuracy & Reliability: Classification and
  Object Localization Under Distribution Shift, published in Machine Learning
  and Granular Computing: A Synergistic Design Environment, edited by W.
  Pedrycz & SM. Chen, 2024, Springer Nature Switzerland reproduced with
  permission of Springer Nature Switzerland. The final authenticated version is
  available online at: http://dx.doi.org/10.1007/978-3-031-66842-5_9},
    archivePrefix={arXiv},
    eprint={2409.03543},
    primaryClass={cs.CV cs.AI cs.LG}
}