summaries: 	This paper discussed how data properties (e.g., label noise, label imbalance, data size) affects calibration error. The author designed experiments on varying computer vision datasets (i.e., cifar10, cifar100, eurosat and iNaturalist) qualitatively: 1) calibration error for various individual classes under class-imbalance situation; 2) calibration error for different scale of label noise; 3) calibration error under non-uniform noise; 4) calibration error under various scale of dataset size; 5) Calibration error under different combinations of data augmentations. The experimental results show that poor calibration performance accompanies with large noisy label rate, large imbalance ratio and small dataset size. For the reason of small dataset size causing poor calibration error, this paper provided the theoretical proof.  This work is an empirical survey of the calibration problem with convnets. The authors use several existing benchmark datasets and create synthetic class-imbalance for datasets that are initially balanced. They then extend the well-known results on higher prediction error of minority class, to its calibration error. The work investigates several existing methods that alleviate prediction error in imbalanced datasets and examine their effect on calibration error. At last, the effect of dataset size and data augmentation on calibration error is reported. Later on, the effect of random label noise is also examined. The observations, although not surprising, have not been reported before  In this work, authors demonstrate that dataset properties can significantly affect calibration and suggest that calibration should be measured during dataset curation. In the field of applied AI to real-life problem, we face all the time decision-makings on what is the most effective strategy in the pipeline (eg. sampling, noise, labeling) and this paper present some evidence for those decisions.
abstractText: Neural networks are known to be data-hungry, and collecting large labeled datasets is often a crucial step in deep learning deployment. Researchers have studied dataset aspects such as distributional shift and labeling cost, primarily using downstream prediction accuracy for evaluation. In sensitive real-world applications such as medicine and self-driving cars, not only is the accuracy important, but also the calibration – the extent that model uncertainty reflects the actual correctness likelihood. It has recently been shown that modern neural networks are ill-calibrated. In this work, we take a complementary approach – studying how dataset properties, rather than architecture, affects calibration. For the common issue of dataset imbalance, we show that calibration varies significantly among classes, even when common strategies to mitigate class imbalance are employed. We also study the effects of label quality, showing how label noise dramatically increases calibration error. Furthermore, poor calibration can come from small dataset sizes, which we motive via results on network expressivity. Our experiments demonstrate that dataset properties can significantly affect calibration and suggest that calibration should be measured during dataset curation.
references.authors: Omar Alonso
references.title: Implementing crowdsourcing-based relevance experimentation: an industrial perspective
references.venue: Information retrieval,
references.authors: Peter M Attia Aditya Grover Norman Jin Kristen A Severson Todor M Markov Yang-Hung Liao Michael H Chen Bryan Cheong Nicholas Perkins Zi Yang
references.title: Closed-loop optimization of fast-charging protocols for batteries with machine learning
references.authors: Mariusz Bojarski Davide Del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D Jackel Mathew Monfort Urs Muller Jiakai Zhang
references.title: End to end learning for self-driving cars
references.venue: arXiv preprint arXiv:1604.07316,
references.authors: Mateusz Buda Atsuto Maki Maciej A Mazurowski
references.title: A systematic study of the class imbalance problem in convolutional neural networks
references.venue: Neural Networks,
references.authors: Chris Callison-Burch Mark Dredze
references.title: Creating speech and language data with amazon’s mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s
references.authors: Kaidi Cao Colin Wei Adrien Gaidon Nikos Arechiga Tengyu Ma
references.title: Learning imbalanced datasets with label-distribution-aware margin loss
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Rich Caruana Yin Lou Johannes Gehrke Paul Koch Marc Sturm Noemie Elhadad
references.title: Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission
references.venue: In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining,
references.authors: Mauro Cettolo Jan Niehues Sebastian Stüker Luisa Bentivogli Marcello Federico
references.title: Report on the 11th iwslt evaluation campaign, iwslt 2014
references.venue: In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam,
references.authors: Koby Crammer Michael Kearns Jennifer Wortman
references.title: Learning from data of variable quality
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ekin D Cubuk Barret Zoph Dandelion Mane Vijay Vasudevan Quoc V Le
references.title: Autoaugment: Learning augmentation policies from data
references.venue: arXiv preprint arXiv:1805.09501,
references.authors: Yin Cui Menglin Jia Tsung-Yi Lin Yang Song Serge Belongie
references.title: Class-balanced loss based on effective number of samples
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Chuan Guo Geoff Pleiss Yu Sun Kilian Q Weinberger
references.title: On calibration of modern neural networks
references.venue: arXiv preprint arXiv:1706.04599,
references.authors: Bo Han Jiangchao Yao Gang Niu Mingyuan Zhou Ivor Tsang Ya Zhang Masashi Sugiyama
references.title: Masking: A new perspective of noisy supervision
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Patrick Helber Benjamin Bischke Andreas Dengel Damian Borth
references.title: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification
references.venue: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,
references.authors: Joel Hestness Sharan Narang Newsha Ardalani Gregory Diamos Heewoo Jun Hassan Kianinejad Md Patwary Mostofa Ali Yang Yang Yanqi Zhou
references.title: Deep learning scaling is predictable, empirically
references.venue: arXiv preprint arXiv:1712.00409,
references.authors: Chien-Ju Ho Shahin Jabbari Jennifer Wortman Vaughan
references.title: Adaptive task assignment for crowdsourced classification
references.venue: In International Conference on Machine Learning,
references.authors: Gao Huang Zhuang Liu Kilian Q Weinberger
references.title: Densely connected convolutional networks. corr abs/1608.06993 (2016)
references.venue: arXiv preprint arXiv:1608.06993,
references.authors: Nathalie Japkowicz Shaju Stephen
references.title: The class imbalance problem: A systematic study
references.venue: Intelligent data analysis,
references.authors: David R Karger Sewoong Oh Devavrat Shah
references.title: Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing
references.authors: Melissa G Keith Louis Tay Peter D Harms
references.title: Systems perspective of amazon mechanical turk for organizational research: Review and recommendations
references.venue: Frontiers in Psychology,
references.authors: Alex Kendall Roberto Cipolla
references.title: Modelling uncertainty in deep learning for camera relocalization. corr abs/1509.05909
references.authors: Ashish Khetan Sewoong Oh
references.title: Achieving budget-optimality with adaptive schemes in crowdsourcing
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ranjay Krishna Yuke Zhu Oliver Groth Justin Johnson Kenji Hata Joshua Kravitz Stephanie Chen Yannis Kalantidis Li-Jia Li David A Shamma
references.title: Visual genome: Connecting language and vision using crowdsourced dense image annotations
references.venue: International journal of computer vision,
references.authors: Alex Krizhevsky Geoff Hinton
references.title: Convolutional deep belief networks on cifar-10
references.venue: Unpublished manuscript,
references.authors: Volodymyr Kuleshov Nathan Fenner Stefano Ermon
references.title: Accurate uncertainties for deep learning using calibrated regression
references.venue: arXiv preprint arXiv:1807.00263,
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He Piotr Dollár
references.title: Focal loss for dense object detection
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: John Miller Karl Krauth Benjamin Recht Ludwig Schmidt
references.title: The effect of natural distribution shift on question answering models
references.venue: arXiv preprint arXiv:2004.14444,
references.authors: Adam S Miner Albert Haque Jason A Fries Scott L Fleming Denise E Wilfley G Terence Wilson Arnold Milstein Dan Jurafsky Bruce A Arnow W Stewart Agras
references.title: Assessing the accuracy of automatic speech recognition for psychotherapy
references.venue: npj Digital Medicine,
references.authors: Mahdi Pakdaman Naeini Gregory F Cooper Milos Hauskrecht
references.title: Obtaining well calibrated probabilities using bayesian binning
references.venue: In Proceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence,
references.authors: Alexandru Niculescu-Mizil Rich Caruana
references.title: Predicting good probabilities with supervised learning
references.venue: In Proceedings of the 22nd international conference on Machine learning,
references.authors: Myle Ott Michael Auli David Grangier Marc’Aurelio Ranzato
references.title: Analyzing uncertainty in neural machine translation
references.venue: arXiv preprint arXiv:1803.00047,
references.authors: Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli
references.title: fairseq: A fast, extensible toolkit for sequence modeling
references.venue: In Proceedings of NAACL-HLT 2019: Demonstrations,
references.authors: Giorgio Patrini Alessandro Rozza Aditya Krishna Menon Richard Nock Lizhen Qu
references.title: Making deep neural networks robust to label noise: A loss correction approach
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Genevieve Patterson James Hays
references.title: Sun attribute database: Discovering, annotating, and recognizing scene attributes
references.venue: In 2012 IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg Kilian Q Weinberger
references.title: On fairness and calibration
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Benjamin Recht Rebecca Roelofs Ludwig Schmidt Vaishaal Shankar
references.title: Do imagenet classifiers generalize to imagenet
references.venue: arXiv preprint arXiv:1902.10811,
references.authors: Ye Ren PN Suganthan N Srikanth
references.title: Ensemble methods for wind and solar power forecasting—a state-of-the-art review
references.venue: Renewable and Sustainable Energy Reviews,
references.authors: Marco Tulio Ribeiro Sameer Singh Carlos Guestrin
references.title:  why should i trust you?” explaining the predictions of any classifier
references.venue: In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining,
references.authors: Hao Su Jia Deng Li Fei-Fei
references.title: Crowdsourcing annotations for visual object detection
references.venue: In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence,
references.authors: Santosh K Suram Yexiang Xue Junwen Bai Ronan Le Bras Brendan Rappazzo Richard Bernstein Johan Bjorck Lan Zhou R Bruce van Dover Carla P Gomes
references.title: Automated phase mapping with agilefd and its application to light absorber discovery in the v–mn–nb oxide system
references.venue: ACS combinatorial science,
references.authors: Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jon Shlens Zbigniew Wojna
references.title: Rethinking the inception architecture for computer vision
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Bart Thomee David A Shamma Gerald Friedland Benjamin Elizalde Karl Ni Douglas Poland Damian Borth Li-Jia Li
references.title: Yfcc100m: The new data in multimedia research
references.venue: Communications of the ACM,
references.authors: Dimitris Tsipras Shibani Santurkar Logan Engstrom Andrew Ilyas Aleksander Madry
references.title: From imagenet to image classification: Contextualizing progress on benchmarks
references.venue: arXiv preprint arXiv:2005.11295,
references.authors: Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona Serge Belongie
references.title: The inaturalist species classification and detection dataset
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in neural information processing systems,
references.authors: Yu-Xiong Wang Deva Ramanan Martial Hebert
references.title: Learning to model the tail
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Chulhee Yun Suvrit Sra Ali Jadbabaie
references.title: Small relu networks are powerful memorizers: a tight analysis of memorization capacity
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Bianca Zadrozny Charles Elkan
references.title: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers
references.venue: In Icml,
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.venue: arXiv preprint arXiv:1611.03530,
references.authors: Yuchen Zhang Xi Chen Dengyong Zhou Michael I Jordan
references.title: Spectral methods meet em: A provably optimal algorithm for crowdsourcing
references.venue: In Advances in neural information processing systems,
references.authors: Yudian Zheng Guoliang Li Yuanbing Li Caihua Shan Reynold Cheng
references.title: Truth inference in crowdsourcing: Is the problem solved
references.venue: Proceedings of the VLDB Endowment,
sections.heading: 1 INTRODUCTION
sections.text: Neural networks often require large amounts of labeled data to perform well, making data curation a crucial but costly aspect of deployment. Thus, researchers have studied dataset properties such as distributional shift (Miller et al., 2020) and the bias in crowd-sourced computer vision datasets (Tsipras et al., 2020) among others. Often, the evaluation criteria in such studies is downstream prediction accuracy. However, neural networks are increasingly deployed in sensitive real-world applications such as medicine (Caruana et al., 2015), self-driving cars (Bojarski et al., 2016), and scientific analysis (Attia et al., 2020), where not only accuracy matters but also calibration. Calibration is the extent to which model certainty reflects the actual correctness likelihood. Calibration can be important when costs of false positives and false negatives are asymmetric, e.g., for a deadly disease with cheap treatment, doctors might initiate treatment when the probability of being sick exceeds 10%. Beyond simple classification, calibration can be important for beam-search in NLP (Ott et al., 2018) and algorithmic fairness (Pleiss et al., 2017). Calibration in machine learning has been studied by e.g Zadrozny & Elkan (2001); Naeini et al. (2015). Niculescu-Mizil & Caruana (2005) have shown that small scale neural networks can yield well-calibrated predictions. However, it has recently been observed by (Guo et al., 2017) that modern neural networks are ill-calibrated, whereas the now primitive Lenet (LeCun et al., 1998) achieves good calibration.
  In this work, we take a complementary approach; instead of focusing on network architecture, we study how calibration is influenced by dataset properties. We primarily focus on computer vision and perform extensive experiments across common benchmarks and more exotic datasets such as satellite images (the eurosat dataset (Helber et al., 2019)) and species detection (the iNaturalist dataset (Van Horn et al., 2018)). We consistently find that dataset properties can significantly affect calibration, causing effects comparable to network architecture. For example, we consider the ubiquitous problem of class imbalanced datasets, a common issue in practice (Van Horn et al., 2018; Krishna et al., 2017; Thomee et al., 2016). For such datasets, the miscalibration is not uniform but instead varies across the different classes. This problem persists even when common strategies to mitigate class imbalanced are employed. Another practical concern is generating high-quality labels via e.g. crowdsourcing (Karger et al., 2011). We demonstrate how labeling quality affects calibration, with noisier labels resulting in worse calibration. Additionally, we show that just the size of the dataset
  has a strong effect on calibration. This also holds when one artificially increases the dataset size by data augmentation. We motivate our findings by considering the geometry of the cross-entropy loss and utilizing recent results on network expressivity (Yun et al., 2019). If the dataset is sufficiently small compared to the number of parameters, we argue that the lack of minimizer for the cross-entropy loss biases the network to high confidence and poor calibration. Our results highlight an underappreciated aspect of calibration and suggest that for sensitive applications, one should measure calibration during dataset curation.
sections.heading: 2 BACKGROUND
sections.text: Calibration. Calibration has a traditional place in machine learning (Zadrozny & Elkan, 2001; Naeini et al., 2015). Before the advent of modern deep learning, Niculescu-Mizil & Caruana (2005) showed that neural networks can yield well-calibrated predictions for classification. However, Guo et al. (2017) showed that modern neural networks are ill-calibrated. Modern neural networks are modeled as e.g. resnet (He et al., 2016) or densenets (Huang et al., 2016). It is important to note that accuracy and calibration do not necessarily follow each other, but can move independently – modern neural networks are ill-calibrated, but still yield excellent accuracy. Beyond image classification, the importance of calibration in NLP has further been studied by Ott et al. (2018) and its relationship to fairness by Pleiss et al. (2017).
  Metrics for Calibration. We let {xi} ∈ Rn×dx be a dataset of n datapoints with dx features and take {yi} to be the labels. Following Guo et al. (2017), we assume that a neural network h outputs h(xi) = (p̂i, ŷi), where ŷi is the predicted class and p̂i is the estimated probability that the prediction is correct. For evaluating calibration, we divide the interval [0, 1] into M equally sized bins and assign predictions to bins based upon p̂. Within each bin Bm we define the accuracy as acc(Bm) =
  1 |Bm| ∑ i∈Bm 1(ŷi = yi). Similarly, we define the confidence as conf(Bm) = 1 |Bm| ∑ i∈Bm p̂i. For a well-calibrated model, we would expect the confidence and accuracy of each bin to be close to each other. Calibration error can be measured by their difference, evaluated on the test set. The
  resulting metric is known as the expected calibration error (Naeini et al., 2015), often abbreviated as ece. Mathematically, it is defined as follows:
  ece = M∑
  m=1
  |Bm| n ∣∣∣∣acc(Bm)− conf(Bm)∣∣∣∣ (1)
sections.heading: 3 EXPERIMENTS
sections.text: Experimental setup. We consider the following computer vision datasets. Cifar10 & Cifar100 (Krizhevsky & Hinton, 2010) which contains 50,000 RGB images spanning ten or hundred classes respectively. Classes are balanced. Eurosat (Helber et al., 2019), which is a dataset of satellite images over continental Europe; there are ten balanced classes and 27,000 images in total. iNaturalist (Van Horn et al., 2018) which is a dataset for species detection. We use the FGVC6 version (FGVC6, 2019), compromising over 260,000 images and an imbalanced hierarchical class system compromising e.g., species and phylum. We perform classification at the ”class” level, resulting in nine classes. Across all datasets, we use the same architecture, Resnet50 (He et al., 2016). We use hyperparameters from the original resnet paper (He et al., 2016): cross-entropy loss optimized with SGD using a learning rate at 0.1 and decreased by a factor 0.1 after 50 % and 75 % of the training, a batch size of 128, a weight decay of 0.0001, and momentum of 0.9. For the cifar/eurosat/inaturalist, networks are trained over 62/30/331 × 103 gradient steps, corresponding to 160 epochs for each dataset. We use randomized cropping and random horizontal flipping for data augmentation, see Appendix A for data preprocessing. Experiments are repeated five times, with mean and standard deviations reported. Calibration error is measured in expected calibration error(ece) as in eq. (1), using M = 15 and evaluated on the test set.
  Inbalanced dataset. A common problem in practice, not necessarily found in benchmark datasets, is class imbalance (i.e., the number of available samples varies between classes), see Van Horn et al. (2018); Krishna et al. (2017); Thomee et al. (2016). Here we study how imbalanced datasets affect
  the calibration error for individual classes. Whereas the iNaturalist dataset is naturally imbalanced, the cifar and eurosat datasets are not. For these datasets, we simulate long-tailed class imbalance following Cao et al. (2019). We randomly reorder the classes from = 1 to n, and only keep a αi−1 fraction of examples for class i. Given some desired ratio ρ between the class with the most and fewest samples, one picks α such that αn−1 = ρ. Following Buda et al. (2018), we also consider a step-imbalance, where half of the classes are downsampled by a factor ρ. We consider ρ = 100 as done by e.g. Cao et al. (2019) and keep the test set balanced. For cifar/eurosat, we randomly chose what classes to subsample to eliminate class-specific properties. Since iNaturalist is already imbalanced, we keep it as it is, but note that the class-specific properties are correlated with class-specific imbalance. After this procedure, we train the DNNs as normal and give the average calibration error for individual classes. The results are shown in Figure 1. Generally, classes with fewer examples have significantly higher ECE, showing how imbalance can have a significant effect on model calibration. For the iNaturalist dataset, we have some outlier classes which is likely due to class-specific effects, e.g., the class with the most labels might be unusually hard to calibrate.
  Methods for Imbalanced Datasets. As class imbalanced is a problem of practical importance, there is ample work on mitigating this issue. One common strategy is to sample the dataset unevenly when generating mini-batches, attempting to obtain a roughly balanced dataset. One can both oversample minority classes (Buda et al., 2018) and undersample the majority class (Japkowicz & Stephen, 2002). Another strategy is instead to weight the objective function to give all classes
  approximately the same weight in the objective function. A common strategy is weight classes inversely proportional to their frequency (Wang et al., 2017). Recently, Cui et al. (2019) has proposed to reweight based upon the ”effective” number of samples, which is defined per a mathematical formula. We here investigate if the calibration issues of an imbalanced dataset persist when using such mitigation strategies. Thus, we consider the standard cross-entropy (original), resampling inversely proportional to the frequency (sampling), reweighting inversely proportional to the frequency (weighted), the weighting scheme of Cui et al. (2019) (CB), and the focal loss of Lin et al. (2017) (focal). Additionally we consider label smoothing (Szegedy et al., 2016) (label smooth). We construct imbalanced datasets as in the previous section. Due to limited computational resources, we only consider the three smallest datasets for these experiments. Calibration errors are given in Table 1, and we give the largest and smallest calibration error among classes and the average ratio of the two. We see that issues of imbalanced calibration errors, while sometimes improving, still persist. Standard deviations are given in Table 2 in Appendix A.
  Label Quality. When collecting labels, for example via crowdsourcing, a common issue is label quality (Patterson & Hays, 2012; Su et al., 2012; Callison-Burch & Dredze, 2010). For example, workers might have poor incentives to perform well or lack the necessary skills for quality labeling. To study the effects of potentially mislabeled data, we artificially inject symmetric noise into the training set. This is done by selecting a random subset of the training set corresponding to some fixed fraction, and then shuffling the labels of this set. This setup follows conventions in label noise literature (Patrini et al., 2017; Han et al., 2018). Given these noisy labels, we train the networks and evaluate them on the original test-set (which has no noise). We consider five levels of label noise in increments of 0.1, starting at 0.0. The resulting calibration errors for various noise levels are given in Figure 2, where we see that label noise increases the calibration error across all datasets. Additionally, we consider the effects of non-uniform noise, studied by e.g. Crammer et al. (2006). For class i, we linearly increase a noise level pi from 0.0 to 0.5. Classes are randomly ordered. For each image with original class i, with probability pi we assign it to a new random class. The reassignment probability to class i is proportional to pi. Results are given in Figure 3, where noisier classes suffer from worse calibration. This underscores how label quality control in e.g. crowdsourcing (Su et al., 2012) can be not only important for accuracy, but also for calibration of downstream models.
  Dataset size. The perhaps most common concern for collecting data is the dataset size, and model accuracy typically grows with this size (Hestness et al., 2017). Crowdsourcing labels and bounding boxes for images is common practice, with many researchers investigating strategies to reduce needed queries (Su et al., 2012) In practice, dataset size can be limited by costs of labeling, but also by obtaining the actual data (Suram et al., 2017). Motivated by this, we study the effect of dataset size upon the calibration error. We simply subsample the training sets of the datasets uniformly at random and thereafter train on them, comparing different sizes of the resulting dataset. We consider subsampled sizes, measured in fractions of the original size, from 1.0 to 0.2 in increments of 0.2, and also consider 0.1. The test set is not subsampled. The results of these experiments are given in Figure 4. We see that smaller datasets have substantially larger calibration errors, demonstrating the dramatic effect that dataset size can have not only on accuracy, but also on calibration error.
  Augmentations. Beyond actual dataset size, it is common to artificially increase the size of the dataset by augmenting it, e.g., randomly cropping the images or slightly shifting the color balance (Cubuk et al., 2018). We have seen that the size of the dataset influences calibration, and now consider the effect of increasing the effective dataset size via augmentations. We use both randomized cropping and horizontal flipping for our training and consider removing these components while keeping other training parameters fixed. The outcome of this experiment is shown in Figure 5, and we see that removing data augmentations significantly increases the calibration error. Viewing data augmentation as a strategy of extending the training set, we again see how smaller training sets increase the calibration error, just as in Figure 4.
  NLP. While we primarily focus on computer vision, we here consider experiments in NLP. Language models often generate text via beam-search, and it has been observed that calibration is important for this process (Ott et al., 2018). Here we investigate the effect of dataset size on calibration in NLP. We considering translation of the IWSLT’14 German to English dataset (Cettolo et al., 2014) using transformer architectures (Vaswani et al., 2017) with code and default hyperparameters from the publicly available fairseq codebase (Ott et al., 2019). As before, we simply subsample the training set uniformly at random for variable sizes and train the transformer with all other training parameters fixed. Figure 6 shows how the mean calibration error, with standard deviations as error bars, varies during training. Again we see how the dataset size influences the calibration. It is natural to guess that there might be word-level calibration issues too, e.g., rare words might have worse calibration.
sections.heading: 4 THEORETICAL MOTIVATION
sections.text: Figure 4 and Figure 5 show that the size of the dataset affects calibration, with smaller datasets resulting in worse calibration. To explain this, let us consider the cross-entropy loss. We will let lij denote the logit for image i and class j. Furthermore, let ci be the index of the correct class for image i. The soft-max cross-entropy loss function is then defined as
  ` = ∑ i `i = − ∑ i log exp(lici)∑ j exp(lij) = − ∑ i ( lici + log (∑ j exp(lij) ))
  (2)
  We note that this loss function decreases monotonically as the logit lici increases. This implies that there is no global minimizer, but instead that if the other logits are fixed, we have `i → 0 as lici →∞. The logit tending to infinity implies that the confidence of the prediction tends to 1. The lack of minimizer for soft-max cross-entropy is in stark contrast with e.g. label smoothing, which penalizes large confidence, see Figure 7. Let us imagine that the network has infinite capacity. If we optimize it for a sufficient amount of time, we would expect ` to tend to zero, which implies that the logits tend to infinity. This corresponds to the confidence on the training set tending to 100%, which most likely implies overconfidence and poor calibration. We can formalize this observation, but we first need to state some assumptions.
  Assumption 1 Let {xi} ∈ Rn×dx be a dataset of n datapoints with dx features each. Let {yi} ∈ {0, 1}n×c be a one-hot label encoding that assigns each image one out of c classes, where c is a constant. We assume that all datapoints {xi} are distinct, i.e. xi 6= xi,∀i 6= j.
  Under such assumptions, recent results in network expressivity say that one can essentially memorize a training set (Yun et al., 2019) if the width is at least of order O( √ n). In an idealized setting, where we optimize the function without computational considerations, such expressivity means that the loss function can be optimized towards its minimizer 0. This means train set confidence growing to 100%, likely translating to poor calibration. We formalize this line of argument in Theorem 1.
  Theorem 1 Let Assumption 1 hold. Let f be a Relu networks with four or more layers and with width at least Ω( √ n) and parameters w. Let ` be equal to the loss function in eq. (2). Then (i) minw `(f(w)) no global minima; (ii) the confidence tends to 1 as `→ 0.
  The formal proof is given in Appendix B, we here provide some intuition. To prove (i) and (ii), it suffices that the network can fit the training set with 1.0 accuracy. This is typically the condition in practice (Zhang et al., 2016), and whereas we consider an idealized argument without computation costs, the conclusions agree with our experimental results. For the sake of contradiction, let us assume that we are in a global minima with parameters w and 100% accuracy. Now consider `i when we scale the final layer by (1 + α) for α > 0. The network output is then (1 + α)lij , and `i is − log ( exp((1 + α)lici)/ ∑ j exp((1 + α)lij) = log(1 + ∑ j 6=i exp ( (1 + α)(lij − lici) ) . The fact that we have perfect train accuracy means that (lij − lici) < 0 ∀j 6= ci. Thus, the loss must shrink, as ∑ j 6=i exp ( (1 + α)(lij − lici) ) decreases with α and as log is monotone. By contradiction, we are not in such a global minima. The results of Yun et al. (2019) say that we can always find weights which achieves perfect accuracy using O( √ n) parameters, and thus that there are no issues with fitting that dataset that prevents the loss from tending to 0. Thus, if the dataset is small compared to the number of parameters, we expect overconfidence and poor calibration. This conclusion agrees with observations of Guo et al. (2017), who show that depth and width increases miscalibration.
sections.heading: 5 RELATED WORK
sections.text: There is much recent work on how datasets influence the behavior of neural networks. Tsipras et al. (2020) shows how the process used to collect labels for imagenet can introduce bias into the resulting dataset. Miller et al. (2020) studies how the shift between different datasets can influence the performance of question and answering systems. Recht et al. (2019) construct new test sets for imagenet and cifar10, and observe differences in generalization compared to the original test sets. Imbalanced datasets is a common issue when applying machine learning in practice (Van Horn et al., 2018; Krishna et al., 2017; Thomee et al., 2016), and researcher often describe the ”heavytail” of class labels (Cui et al., 2019). Traditional work on class imbalance includes Japkowicz & Stephen (2002) which investigates different sampling strategies, applicable to most machine learning models. For models of empirical risk minimization, one can instead reweight samples. A relatively recent reweighting scheme is proposed by Cui et al. (2019), where one uses the effective number of samples, which can be calculated from a simple formula.
  For generating datasets, a common strategy is to employ crowdsourcing, where one lets ordinary people assign labels in a large-scale automated fashion, commonly via Amazon’s Mechanical Turk system (Keith et al., 2017). Typical applications of crowdsourcing include analyzing images and providing bounding boxes (Patterson & Hays, 2012; Su et al., 2012), providing linguistic annotations for natural language (Callison-Burch & Dredze, 2010), or evaluating the relevance of search engines results (Alonso, 2013). Another application is machine learning debugging Ribeiro et al. (2016). The idea of eliciting and aggregating crowdsourced labels efficiently has inspired much algorithmic work (Khetan & Oh, 2016; Zhang et al., 2014). Common issues include finding tasks that result in high-quality labels, dealing with inconsistent labels (Karger et al., 2011; Zheng et al., 2017) and heterogenous workers (Ho et al., 2013).
  Calibration in machine learning has been studied for a long time (Zadrozny & Elkan, 2001; Naeini et al., 2015) due to its practical implications. For neural networks, Caruana et al. (2015) demonstrated that shallow neural networks can yield well-calibrated predictions on classification tasks. In contrast, Guo et al. (2017) show how modern neural networks are ill-calibrated, with width and depth resulting in worse calibration scores, and investigate mitigation strategies. Neural network calibration has implications in NLP (Ott et al., 2018), fairness (Pleiss et al., 2017) and reinforcement learning (Kuleshov et al., 2018). For applications such as medicine (Miner et al., 2020), meteorology (Ren et al., 2015) and autonomous vehicles (Bojarski et al., 2016) it can be important for performance. Reliable uncertainty estimates also allow one to integrate DNNS with other probabilistic models, incorporating e.g. camera information (Kendall & Cipolla, 2015).
sections.heading: 6 CONCLUSIONS
sections.text: We have investigated the effects that datasets can have on network calibration. By generating label noise and class imbalance synthetically, we show how calibration error increases with label noise and few samples. We also study how calibration changes with dataset size. Our work points towards the importance of high-quality dataset curation for generating well-calibrated predictions, and highlight issues that are relevant in high-stakes applications such as autonomous vehicles and medical applications. These calibration issues can potentially be mitigated both at dataset curation time and training time; we defer such studies to future work. A practical takeaway from this work is that for sensitive applications, one should evaluate calibration when collecting datasets.
sections.heading: A APPENDIX
sections.text: Data Preprocessing. For the cifar10 and cifar100 datasets, we use the original dataset size of 32- by-32 pixels. Cropping is performed by first padding with 4 pixels on each side, and thereafter performing a random 32-by-32 crop. For the eurosat dataset, the images are subsampled to 32-by32 pixels. Since the eurosat dataset does not have a dedicated train/test split, we split it ourselves, using a fixed random tenth of the dataset for testing across all experiments. Random cropping is performed as for cifar. For the inaturalist dataset, the images are resized into 64-by-64 images, and we pad by 8 pixels on each side before extracting a random 64-by-64 crop.
sections.heading: B PROOF
sections.text: The results of Yun et al. (2019) mean that under Assumptions 1, there exists a four-layer Relu network with Ω( √ n) parameters that can memorize the dataset, perfectly matching the one-hot encoding of the labels. For details, we refer to Yun et al. (2019).
  Proof Sketch for Theorem 1. We start with part (i). Assume for contradiction that we are in a global minima with loss `0 and perfect accuracy. Now, consider `i when we scale the final layer by (1 +α) for α > 0. The network output is then (1 + α)lkj , and `i is − log ( exp((1 + α)lici)/ ∑ j exp((1 +
  α)lij) = log(1+ ∑ j 6=i exp ( (1+α)(lij− lici) ) . The fact that we have perfect train accuracy means
  that (lij − lici) < 0 ∀j 6= ci. Thus the loss must shrink, as ∑ j 6=i exp ( (1 +α)(lij − lici) ) decreases with α and as log is monotone. This contradicts us being in such a global minima. If we do not have perfect accuracy, there must be some example k such that `k must be larger than log 2 (since there is another class with larger logit). Results of Yun et al. (2019) says that we can find network with perfect accuracy and thereafter scale the weight to less than log 2n . (ii). The confidence for a single example is exp(lici )∑
  j exp(lj) = exp(−`i). We have noted how ` → 0, since ` =
  ∑ i `i we have `i → 0.
  This implies that exp(−`i)→ 1 which means that the confidence approaches 1.