summaries: This paper leverages differential privacy’s stability properties to investigate its use for improved anomaly and backdoor attack detection. Under an assumption (called “uniformly asymptotic empirical risk minimization”), the authors show that difference between the expected loss of a differentially private learning algorithm on an outlier (where the expectation is taken over the randomness of the learning algorithm) and the expected loss of the same algorithm on data from the underlying distribution (expectation taken over data & randomness of the algorithm) is lower bounded by a (possibly/hopefully) non-negative quantity with high probability. The authors then conduct a set of experiments to show that differential privacy improves the performance of outliers, novel examples, and backdoor attack detection.

summaries: This paper proposes the idea of using differential privacy (DP) to improve the performance of outlier and novelty detection. Differential privacy was proposed as a privacy metric which limits the contribution of a single data point in the training set to the output. This property naturally controls how poisoned data would affect the output of the learned model. Under the assumption that a well-trained model would incur a higher loss on the outliers, the paper gives a theoretic bound on how this loss will decrease if there are poisoned samples in the training set.

abstractText: Outlier detection and novelty detection are two important topics for anomaly detection. Suppose the majority of a dataset are drawn from a certain distribution, outlier detection and novelty detection both aim to detect data samples that do not fit the distribution. Outliers refer to data samples within this dataset, while novelties refer to new samples. In the meantime, backdoor poisoning attacks for machine learning models are achieved through injecting poisoning samples into the training dataset, which could be regarded as “outliers” that are intentionally added by attackers. Differential privacy has been proposed to avoid leaking any individual’s information, when aggregated analysis is performed on a given dataset. It is typically achieved by adding random noise, either directly to the input dataset, or to intermediate results of the aggregation mechanism. In this paper, we demonstrate that applying differential privacy can improve the utility of outlier detection and novelty detection, with an extension to detect poisoning samples in backdoor attacks. We first present a theoretical analysis on how differential privacy helps with the detection, and then conduct extensive experiments to validate the effectiveness of differential privacy in improving outlier detection, novelty detection, and backdoor attack detection.

authors:
Martin Abadi
Andy Chu
Ian Goodfellow
H Brendan McMahan
Ilya Mironov
Kunal Talwar
Li Zhang

title: Deep learning with differential privacy

venue: In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security,

authors:
Charu C Aggarwal
Philip S Yu

title: Outlier detection for high dimensional data

venue: In ACM Sigmod Record,

authors:
Eugene Bagdasaryan
Vitaly Shmatikov

title: Differential privacy has disparate impact on model accuracy

venue: arXiv preprint arXiv:1905.12101,

authors:
Gilles Blanchard
Gyemin Lee
Clayton Scott

title: Semi-supervised novelty detection

venue: Journal of Machine Learning Research,

authors:
Olivier Bousquet
André Elisseeff

title: Stability and generalization

venue: Journal of machine learning research,

authors:
Guilherme O Campos
Arthur Zimek
Jörg Sander
Ricardo JGB Campello
Barbora Micenková
Erich Schubert
Ira Assent
Michael E Houle

title: On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study

venue: Data Mining and Knowledge Discovery,

authors:
Nicholas Carlini
Chang Liu
Jernej Kos
Úlfar Erlingsson
Dawn Song

title: The secret sharer: Measuring unintended neural network memorization & extracting secrets

venue: arXiv preprint arXiv:1802.08232,

authors:
Frank De Morsier
Devis Tuia
Maurice Borgeaud
Volker Gass
Jean-Philippe Thiran

title: Semisupervised novelty detection using svm entire solution path

venue: IEEE Transactions on Geoscience and Remote Sensing,

authors:
Min Du
Feifei Li
Guineng Zheng
Vivek Srikumar

title: Deeplog: Anomaly detection and diagnosis from system logs through deep learning

venue: In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security,

authors:
Lian Duan
Lida Xu
Ying Liu
Jun Lee

title: Cluster-based outlier detection

venue: Annals of Operations Research,

title: Differential privacy: A survey of results. In Theory and Applications of Models of Computation—TAMC, volume 4978 of Lecture Notes in Computer Science

title: Differential privacy

venue: Encyclopedia of Cryptography and Security, pp

authors:
Cynthia Dwork
Vitaly Feldman
Moritz Hardt
Toniann Pitassi
Omer Reingold
Aaron Leon Roth

title: Preserving statistical validity in adaptive data analysis

venue: In Proceedings of the forty-seventh annual ACM symposium on Theory of computing,

authors:
Justin Gottschlich
Abdullah Muzahid

title: Autoperf: A generalized zero-positive learning system to detect software performance anomalies

venue: arXiv preprint arXiv:1709.07536,

authors:
Tianyu Gu
Brendan Dolan-Gavitt
Siddharth Garg

title: Badnets: Identifying vulnerabilities in the machine learning model supply chain

venue: arXiv preprint arXiv:1708.06733,

authors:
Victoria Hodge
Jim Austin

title: A survey of outlier detection methodologies

venue: Artificial intelligence review,

title: Kernel pca for novelty detection

venue: Pattern recognition,

authors:
Shiva Prasad Kasiviswanathan
Homin K Lee
Kobbi Nissim
Sofya Raskhodnikova
Adam Smith

title: What can we learn privately

venue: SIAM Journal on Computing,

authors:
Mathias Lecuyer
Vaggelis Atlidakis
Roxana Geambasu
Daniel Hsu
Suman Jana

title: Certified robustness to adversarial examples with differential privacy

authors:
Jian-Guang Lou
Qiang Fu
Shengqi Yang
Ye Xu
Jiang Li

title: Mining invariants from console logs for system problem detection

venue: In USENIX Annual Technical Conference,

authors:
Markos Markou
Sameer Singh

title: Novelty detection: a review—part 1: statistical approaches

venue: Signal processing,

title: Novelty detection: a review—part 2:: neural network based approaches

authors:
Rina Okada
Kazuto Fukuchi
Jun Sakuma

title: Differentially private analysis of outliers

venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,

authors:
F. Pedregosa
G. Varoquaux
A. Gramfort
V. Michel
B. Thirion
O. Grisel
M. Blondel
P. Prettenhofer
R. Weiss
V. Dubourg
J. Vanderplas
A. Passos
D. Cournapeau
M. Brucher
M. Perrot
E. Duchesnay

title: Scikit-learn: Machine learning in Python

authors:
NhatHai Phan
Ruoming Jin
My T Thai
Han Hu
Dejing Dou

title: Preserving differential privacy in adversarial learning with provable robustness

authors:
Congzheng Song
Thomas Ristenpart
Vitaly Shmatikov

title: Machine learning models that remember too much

title: URL http://iiis.tsinghua.edu.cn/~weixu/ sospdata.html. [Online; accessed 3-May-2019

venue: Hdfs log dataset,

authors:
Huan Xu
Constantine Caramanis
Sujay Sanghavi

title: Robust pca via outlier pursuit

venue: In Advances in Neural Information Processing Systems,

authors:
Wei Xu
Ling Huang
Armando Fox
David Patterson
Michael I Jordan

title: Detecting large-scale system problems by mining console logs

venue: In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles,

authors:
Samuel Yeom
Irene Giacomelli
Matt Fredrikson
Somesh Jha

title: Privacy risk in machine learning: Analyzing the connection to overfitting

venue: IEEE 31st Computer Security Foundations Symposium (CSF),

authors:
Arthur Zimek
Erich Schubert
Hans-Peter Kriegel

title: A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analysis and Data Mining: The ASA Data

venue: Science Journal,

authors:
Arthur Zimek
Ricardo JGB Campello
Jörg Sander

title: Ensembles for unsupervised outlier detection: challenges and research questions a position paper

venue: Acm Sigkdd Explorations Newsletter,

heading: 1 INTRODUCTION

text:
Given a dataset where most of the samples are from a certain distribution, outlier detection aims to detect the minorities in the dataset that are far from the distribution, while the goal of novelty detection is to detect newly observed data samples that do not fit the distribution. On the other hand, poisoning examples that are intentionally added by attackers to achieve backdoor attacks could be treated as one type of “outliers” in the training dataset. Using machine learning for outlier/novelty detection is typically to train a model that learns the distribution where the training data samples are drawn from, and the final trained model could give a high anomaly score for the outliers/novelties that deviate from the same distribution. In both cases, the machine learning model is not supposed to learn from the outliers in the training dataset. Unfortunately, deep learning models that contain millions of parameters tend to remember too much (Song et al. [2017]), and can easily overfit to rare training samples (Carlini et al. [2018]).
Protecting data privacy has been a major concern in many applications, because sensitive data are being collected and analyzed. Differential privacy has been proposed to “hide” certain input data from the output; that is, by looking at the statistical results calculated from input data, one cannot tell whether the input data contain a certain record or not. The way of applying differential privacy is to add random noise to the input data or the data analysis procedure, such that the output difference caused by the input difference can be hidden by the noise. A known fact is that differential privacy implies stability (Kasiviswanathan et al. [2011]). Particularly, a differentially private learning algorithm is stable in the sense that the model learned by the algorithm is insensitive to the removal or replacement of an arbitrary point in the training dataset (Bousquet & Elisseeff [2002]). When the training dataset contains a handful of outliers, the output model of a stable learning algorithm should be close to the one trained on the clean portion of the training set. Intuitively, compared with the model trained on contaminated dataset, the one trained on clean data could be better at distinguishing outliers from normal data. Therefore, differential privacy can potentially be leveraged to improve the
identification of outliers. This motivates us to apply differential privacy to anomaly detection and defense against backdoor attacks.
Our contribution. First, we present a theoretical explanation on why differential privacy can help to detect outliers from a training and testing dataset, as well as an analysis on the relationship between the number of outliers to detect and the amount of random noise to apply. Second, to demonstrate the effectiveness, we apply differential privacy to an autoencoder network trained on a constructed MNIST dataset with injected outliers, for both outlier detection and novelty detection, to show how much the utility could be improved with different amount of outliers and noise. Third, we apply differential privacy to a real-world task - Hadoop file system log anomaly detection. System log anomaly detection is an important topic in computer security. Our proposed method greatly improves upon the state-of-the-art system in this field. The results indicate that differential privacy is able to eliminate almost all the false negatives, and achieve significantly improved overall utility, compared with the current state-of-the-art work DeepLog (Du et al. [2017]). Finally, via a proof-of-concept experiment using MNIST dataset with injected poisoning samples, we show that the idea of outlier detection could be extended to backdoor attack detection, and that differential privacy is able to further improve the performance.

heading: 2 PRELIMINARY

text:
Given an input dataset and an aggregation mechanism, differential privacy (Dwork [2011]) aims to output the requested aggregation results, which are guaranteed not to reveal the participation of any individual data record. Formally, differential privacy is defined as below:
Definition 1 (Differential privacy). A randomized mechanismM applied to data domain D is said to be ( , δ)-differentially private if for any adjacent datasets d, d′ in D, and any subset of outputs S ⊆ Range(M), it holds that
Pr[M(d) ∈ S] ≤ e Pr[M(d′) ∈ S] + δ,
where stands for the privacy bound, and δ stands for the probability to break this bound.
The adjacent datasets d, d′ could be understood as two databases, where only one record differs, i.e., ‖d − d′‖1 = 1. Differential privacy guarantees that the difference between d and d′ are not revealed through inspecting the outputsM(d) andM(d′). Clearly, the closer is to 0, the more indistinguishableM(d) andM(d′) are, and hence the stronger the privacy guarantee is. A common approach to enforcing differential privacy for a function f : D→ R, is to add random Gaussian noise N (0, σ2) to perturb the output in R. The intuition is that, for given adjacent datasets d and d′, one cannot tell whether the difference between f(d) and f(d′) is incurred by the single record that differs in d and d′, or by the random noise being applied. The magnitude of Gaussian noise needs to be tailored to the maximum difference between f(d) and f(d′), which is formally defined as L2-sensitivity. Definition 2 (L2-sensitivity). The L2-sensitivity for a function f : D→ R is:
∆ = max d,d′∈D ‖d−d′‖1=1
‖f(d)− f(d′)‖2
The noise scale σ to apply can be calculated as below (Dwork et al. [2014]).
Theorem 1. To perturb a function with sensitivity ∆ under ( , δ) - differential privacy, the minimum noise scale σ of Gaussian mechanism is given by: σ = ∆ · √ 2 ln 1.25δ , where ∈ (0, 1).
Deep learning with differential privacy (Abadi et al. [2016]) The procedure of deep learning model training is to minimize the output of a loss function, through numerous stochastic gradient descent (SGD) steps. Abadi et al. [2016] proposed a differentially private SGD algorithm that works as follows. At each SGD step, a fixed number of randomly selected training samples are used as a mini batch. For each mini batch training, the following two operations are performed to enforce differential privacy: 1) clip the norm of the gradient for each example, with a clipping bound C, to
limit the sensitivity of gradient; 2) sum the clipped per-example gradients and add Gaussian noise N (0, σ2), before updating the model parameters. Abadi et al. [2016] further proposed a moment accounting mechanism which calculates the aggregate privacy bound when performing SGD for multiple steps. Differential privacy is immune to post-processing. Therefore, the output of the trained model for any queries enjoys the same privacy guarantee as the above SGD-based training process.

heading: 3 THE CONNECTION BETWEEN DIFFERENTIAL PRIVACY AND OUTLIER DETECTION

text:
By definition, random noise added into model training for differential privacy hides the influence of a single record on the learned model. Intuitively, if applying differential privacy to the training process, the contribution of rare training examples will be hidden by random noise, resulting in a model that underfits the outliers. Such model will facilitate novelty and outlier detection because it will be less confident in predicting the atypical examples. In this section, we first present a theorem to precisely characterize the above intuition, and then analyze the relationship between the number of outliers in the training dataset and the amount of noise to apply.
Notations LetZ be the sample space andH be the hypothesis space. The loss function l : H×Z → R measures how well the hypothesis h ∈ H explains a data instance z ∈ Z . A learning algorithm A : Zn → H learns some hypothesis A(S) given a set S of n samples. For instance, in supervised learning problems, Z = X × Y , where X is the feature space and Y is the label space; H is a collection of models h : X → Y; and l(h, z) measures how well h predicts the feature-label relationship z = (x, y).
Let S = {z1, . . . , zn} be a set of independent samples drawn from an unknown distribution D on Z . For a given distribution D, an oracle hypothesis is the one that minimizes the expected loss:
h∗ = arg min h Ez∼D[l(h, z)] (1)
We define an outlier as a data instance that has significantly different loss from the population under the oracle hypothesis. Definition 3. We say z̃ is an outlier with regard to distribution D if
l(h∗, z̃)− Ez∼D[l(h∗, z)] ≥ T (2) where T is a constant that depends only on D.
We will prove the usefulness of differential privacy to detect outliers for the classes of learning algorithms that produce hypotheses converging to the optimal hypothesis asymptotically pointwise. We define such learning algorithms to be uniformly asymptotic empirical risk minimization (UAERM). Definition 4. A (possibly randomized) learning algorithm A is UAERM with rate ξ(n,A) if for any distribution D defined on the domain Z , it holds that
∀z |ES∼DnEh∼A(S)l(h, z)− l(h∗, z)| ≤ ξ(n,A) (3)
In the definition, we make it explicit that the rate ξ(n,A) depends on the learning algorithm A. For instance, if A is a differentially private learning algorithm, the rate will depend on the privacy parameters. In that case, with slight abuse of notation, we will denote the rate for a ( , δ)-differentially private learning algorithm trained on n data instances by ξ(n, , δ).
Due to the nonconvexity of their loss functions, neural networks may not enjoy a useful, tight characterization of the learning rate. Thus, we will empirically verify that using noisy SGD to learn differentially private neural networks is UAERM. Moreover, as we will show in the experiment, ξ(n, , δ) grows as privacy parameters and δ become smaller. Intuitively, this is because larger noise is required to ensure stronger privacy guarantees, which, on the other hand, slows down the convergence of the learning algorithm.
Without loss of generality, we assume that 0 ≤ l(h, z) ≤ 1. The following theorem exhibits how the prediction performance of differentially private models on normal data will differ from outliers and connects the difference to the privacy parameters of the learning algorithm and the amount of outliers in the training data.
Theorem 2. Suppose that a learning algorithm A is ( , δ)-differentially private and UAERM with the rate ξ(n, , δ). Let S′ = S ∪ U , where S ∼ Dn and U contains c arbitrary outliers. Then
Eh∼A(S′)l(h, z̃)− Eh∼A(S′)Ez∼Dl(h, z) ≥ T − 2 ( ξ(n, , δ) + √ n(e − 1 + δ)2
2 log
2 γ + ec − 1 + cec δ
) (4)
with probability at least 1− γ.
The two terms in the left-hand side of (4) represent the model’s prediction loss on outliers and normal test data drawn from D, respectively. Due to the stochasticity of differentially private learning algorithms, the difference is characterized by the expectation taken over the randomness of the learned models. The theorem establishes a lower bound on the prediction performance difference between normal and outlier data. A larger difference indicates that identifying outliers will be easier.
The impact of privacy parameters on the lower bound manifests itself in two aspects. On one hand, stronger privacy guarantees (i.e., smaller and δ) will require higher noise to be added into the training process, which increases the learning rate ξ(n, , δ). On the other hand, increasing privacy level will improve the stability of the learning algorithm; the resulting models tend to ignore the outliers in the training set and become closer to the ones trained on completely clean data, thus making the outlier detection more effective. The second aspect is embodied by the fact that the terms except ξ(n, , δ) in the parenthesis of the lower bound grow with and δ. Therefore, the privacy parameters cannot be too large or too small in order to ensure optimal anomaly detection performance.
Moreover, the relationship between the right-hand side of (4) and c indicates that the anomaly detection problem will be more difficult with more outliers in the training set. Dissecting the righthand side of (4), we further observe that c appears always in tandem with . This implies that for larger number of outliers in the training set (i.e., c is larger), we will need to tune down and δ to maintain the same novelty detection performance.
Last but not least, the definition of outliers in our paper is quite general—it does not make any assumptions about how the outliers are generated. Also, we do not make assumptions about whether these outliers are in training or test data. Therefore, our analysis can shed light on detecting various types of anomalies, including but not limited to outlier/novelty detection, backdoor detection, and noisy label detection. In the following experimental section, we will focus our evaluation on outlier/novelty detection and defense against backdoor attacks.

heading: 4 EXPERIMENTS

text: This section empirically evaluates the effectiveness of differential privacy in improving anomaly detection and backdoor attack detection. We call an outlier/novelty or a poisoning example as a positive, and other normal data samples as negatives. The metrics measured by each experiment include: false positive (FP), false negative (FN), Precision = TP / (TP+FP), Recall = TP / (TP+FN), Area under the receiver operating characteristic curve (AUROC) which is the area under the TPRFPR curve, Area under the Precision - Recall curve (AUPR) which summarizes the Precision - Recall curve, as well as F-measure =2× Precision× Recall / ( Precision + Recall ). The detailed explanations of all these measures could be found in (Wikipedia [2019b;a]; scikit-learn [2017a;b]).

heading: 4.1 OUTLIER DETECTION AND NOVELTY DETECTION WITH AUTOENCODERS

text:
Autoencoder is a type of neural network that has been widely used for outlier detection and novelty detection. It contains an encoder network which reduces the dimension of the input data, and a decoder network which aims to reconstruct the input. Hence, the learning goal of autoencoders is to minimize the reconstruction error, which is consequently the loss function. Because the dimensionality reduction brings information loss, and the learning goal encourages to preserve the information that is common to most training samples, outliers that contain rare information could be identified by measuring model loss. In this section, with a varying amount of outliers and noise scale, we show how differential privacy would improve the utility of anomaly detection with autoencoders.
Datasets. We utilize MNIST dataset composed by handwritten digits 0-9, and notMNIST dataset (Kaggle [2017]), which contains letters A-J with different fonts. The original MNIST data contain 60, 000 training images, and 10, 000 test images, which we refer to as MNIST-train and MNIST-test respectively. The notMNIST data contain 10, 000 training images and 1, 000 test images, denoted as notMNIST-train and notMNIST-test. Based on these datasets, we intentionally construct training datasets with varying amount of injected outliers. Specifically, each training dataset is constructed with a particular outlier ratio ro, such that the resulted dataset MNIST-OD-train(ro) contains 60, 000 images in total, where a percentage of 1− ro are from MNIST-train, and ro are from notMNIST-train. For each training dataset MNIST-OD-train(ro), a set of autoencoder models are trained with varying noise scale σ applied for differential privacy. For an autoencoder model trained on dataset MNISTOD-train(ro), outlier detection is thus to detect the ro × 60, 000 outliers from MNIST-OD-train(ro). For novelty detection, we further construct a test dataset MNIST-ND-test, which is composed by the entire MNIST-test dataset and notMNIST-test dataset, a total of 11, 000 images. The goal of novelty detection is to identify the 1, 000 notMNIST-test images as novelties.
Evaluation metrics. To check whether a data sample is an outlier/novelty using autoencoders, the standard practice is to set a loss threshold based on training statistics, and any sample having a loss above this threshold is an outlier/novelty. To measure the performance under different thresholds, we use the AUPR score which is a threshold-independent metric. Compared with other metrics such as the AUROC score, the AUPR score is more informative when the positive/negative classes are highly unbalanced, e.g., for outlier detection where the ratio of outliers is extremely low. More experiment settings with both AUPR and AUROC metrics are in appendix which present similar observations.
Set-up. For autoencoders, the encoder network contains 3 convolutional layers with max pooling, while the decoder network contains 3 corresponding upsampling layers. For differential privacy, we use a clipping bound C = 1 and δ = 10−5, and vary the noise scale σ as in (Abadi et al. [2016]). All models are trained with a learning rate of 0.15, a mini-batch size of 200 and for a total of 60 epochs.
Validation of UAERM for Noisy SGD. To begin with, we first conduct experiments to empirically validate our assumption in Theorem 2. While a rigorous verification of the assumption is intractable as it requires the knowledge of underlying data distribution and computing expected loss over randomness of both data distribution and differentially private algorithms, our experiments provide a sanity check of the assumption by replacing the expectation by the empirical average of a large number of data samples. For this set of experiments, we only utilize MNIST data for training, while the test dataset contains all available MNIST and notMNIST test samples. The oracle hypothesis is trained on all available training data, while each differentially private model is trained with varying privacy level , and training data size. For a fixed training set and , we perform training for multiple times to accommodate the randomness of differentially private training. Further, we train on multiple randomly selected training sets of the same size. We measure the loss of each resulting model on the test set, and calculate the average testing loss across different runs of differentially private training and different randomly selected training sets of the same size. We then compute the largest difference between the averaged test loss and the test loss associated with the oracle hypothesis. The results are shown in Figure 1. Each data point in the figure is an average of 9 differentially private models
trained on 3 randomly sampled subsets of the training data, and 3 random training processes for each sampled subset. As in Figure 1, the larger the training data size, and the larger is, the closer of the randomized model to the oracle hypothesis, validating our assumption in Theorem 2 that noisy SGD is UAERM.
Detection results Table 1 shows the outlier detection (OD) results on dataset MNIST-OD-train, as well as the novelty detection (ND) results on dataset MNIST-ND-test. OD mimics the unsupervised anomaly detection case. ND mimics the case where the autoencoder model is supposed to be trained on normal data, to detect unforeseen anomalies, while the training dataset is noisy. The first row where σ =N/A is for the baseline model without differential privacy applied. It performs well when ro = 0.1%, but drops significantly when ro reaches 0.5%. That’s because for a mini-batch size of 200 that we use, an outlier ratio of 0.5% in training data results in an average of one outlier in each mini-batch, which could be learned by the baseline model. Note that the clipping bound C=1 also restricts the contribution of outliers in SGD steps. We conduct an ablation study which only clips the per-example gradients with C without adding any noise in each gradient descent step. The results are shown as σ = 0 in Table 1. As an intermediate step to bound the sensitivity in differential privacy, clipping itself is able to slightly improve the anomaly detection results in most cases. Still, we show that increasing the noise scale could further improve the utility. We highlight one of the best results in each column, and find that the trend follows our analysis in Theorem 2. Specifically, the more outliers in the training dataset, the larger noise scale is needed for the best improvement. As explained for (4), our theory shows that the privacy parameters cannot be too large or too small to ensure optimal anomaly detection performance, which coincides with the experimental results in Table 1. Although it could be challenging to select the desired noise level for training, we note that as shown in Table 1, applying differential privacy effectively improves the anomaly detection performance in most cases, except when σ is too big to ruin the model parameters completely (e.g., σ=50) . Therefore, it is generally safe and almost always helpful to apply a small amount of differential privacy noise for anomaly detection. The noise scale could be increased further as long as the model converges. However, it should be noted that applying differential privacy makes the model training much slower than the baseline. In our experience utilizing NVIDIA Tesla V100 SXM2 GPU cards, the training time for each epoch could be up to 80 times longer. Finally, a training data portion as high as 10% might not be “outliers”, but could be part of the input pattern that should be learned by the model. We show in this case, a relatively large noise scale could effectively improve the anomaly detection results (e.g., σ=10), but it’s up to the requirement of the application whether to apply this.

heading: 4.2 HADOOP FILE SYSTEM LOG ANOMALY DETECTION WITH LSTM

text:
In this section, we use a real-world example for Hadoop file system log anomaly detection, to show how anomaly detection with differential privacy outperforms the current state-of-the-art results.
Dataset The Hadoop file system (HDFS) log dataset (Wei Xu [2009]) is generated through running Hadoop map-reduce jobs for 48 hours on 203 Amazon EC2 nodes. This dataset contains over 11 million log entries, which could be further grouped into 575, 059 block sessions by the block identifier each log has. Each block is associated with a normal/abnormal label provided by domain experts. Over the past decade this log dataset has been extensively used for research in system log anomaly detection (Xu et al. [2009]; Lou et al. [2010]; Du et al. [2017]). The state-of-the-art results are achieved by DeepLog (Du et al. [2017]), which we use as the baseline model. As in DeepLog, our training dataset contains 4, 855 normal block sessions, while the test dataset includes 553, 366 normal sessions and 16, 838 abnormal sessions.
Baseline model and metrics DeepLog utilizes LSTM neural networks to learn system log sequences. Note that system log messages are textual logs, e.g., “Transaction A finished on server B.”. Before applying LSTM, a log parsing step first maps each log message into its corresponding log printing statement in the source code, e.g., “print(’Transaction %s finished on server %s.”%(x,y))". Since there are only a constant number (e.g., N ) of log printing statements in the source code, each one could be mapped to a discrete value from a fixed vocabulary set (e.g., having size N ). With that, a block session of log messages could be parsed to a sequence of discrete values, e.g. “22 5 5 5 11 9 11 9 11 9 26 26 26”. Leveraging the fact that hidden execution paths written in source code restrict the possibilities of how one system log follows another, DeepLog trains an LSTM model
on normal discrete sequences, which learns to predict the next discrete value given its history. In detection, the LSTM model predicts a probability distribution on all possible values that may appear at a given time step. The real executed value is detected as abnormal if it’s unlikely to happen based on LSTM prediction. The criteria presented in DeepLog is to first sort the predicted values based on the assigned probabilities, e.g., for a prediction “{5: 0.2, 9: 0.08, 11: 0.01, 26: 0.7, ...}”, the order would be 26, 5, 9, 11, .... The given value to detect is checked against the sorted top k predictions, and is detected as abnormal if it’s not one of them. For anomaly detection metrics, we want to highlight that applying differential privacy significantly reduces false negatives, without introducing many false positives. Therefore, we’ll focus on the comparison over the number of false positives and false negatives, while also presenting measurements that indicate the overall detection performance.
Set up For the baseline model DeepLog, we train an LSTM model for 100 epochs, and use the final model as the anomaly detection model. The model related parameters are: 2 layers, 256 units per layer, 10 time steps, and a batch size of 256. We call the DeepLog model with differential privacy as DeepLog+DP. For differential privacy, we use a clipping bound C = 1, δ = 10−5, and vary the noise scale σ. All other model related settings for DeepLog+DP are the same as DeepLog.
Results Figure 2a shows the comparison of FP and FN under different thresholds k, with the increase of noise scale σ. For clarity, we only show the following two cases for baseline model DeepLog: k = 10 which has the maximum FP and the minimum FN , and k = 18 which has the minimum FP and the maximum FN . Note that y axis is plotted as log scale. It is clear that applying DP noise significantly reduces FN in all cases, from over a thousand in DeepLog, to hundreds or even zero in DeepLog+DP. Also, the larger noise being added, the more FN are reduced. Although more FP could be introduced in some cases, we note that in system anomaly detection, the merit of fewer false negatives in fact worth the cost of more false positives. Reported false positives could be further checked by system admin, and then fed into the model for incremental learning. However, a false negative may never be found out, until a more disastrous event occurs due to the un-discovery of it.
The F-measure measurements are plotted in Figure 2b. For DeepLog model, F-measure ranges from 90.38% (k = 20) to 93.81% (k = 10). For DeepLog+DP, the best F-measure scores include 96.29% (σ = 0.25, k = 15) and 96.28% (σ = 1, k = 18), which show clear improvements over DeepLog model. Note that the best FN and FP measurements reported in DeepLog (Du et al. [2017]) are 619 and 833 respectively, while DeepLog+DP achieves FN =383, FP =762 at the F-measure of 96.28% (σ = 1, k = 18); and FN =123, FP =1040 at the F-measure of 96.29% (σ = 0.25, k = 15), showing its ability to significantly reduce false negatives without introducing many false positives. As shown in the figure, DeepLog performs better when k is smaller, while DeepLog+DP benefits from larger ks. This scenario could also be explained by the addition of differential privacy noise. Since the trained model does not overfit to outliers, it assigns to anomalies much lower probabilities, so that anomalies are ranked much lower than that in the DeepLog model. Meanwhile, normal execution logs are also possibly predicted with lower probabilities because of the uncertainty brought by the noise. As a result, the ideal threshold k for DeepLog+DP is higher than that of DeepLog. We also
note that a large noise scale could hurt the overall performance, as shown by the downward trend when σ increases from 1.75 to 2.0.

heading: 4.3 BACKDOOR ATTACK DETECTION

text:
Since poisoning examples for backdoor attacks are essentially “outlier” training samples injected by attackers, this section conducts proof-of-concept experiments to examine whether measuring model loss as for outliers works to detect poisoning examples, and whether differential privacy is able to further improve the performance. This detection scenario is particularly useful for backdoor attacks injected in the crowdsourcing scenario, where the model trainer gathers training data from untrusted individuals. In this case, the model trainer does not have control over the data quality but does have control over the model training process. Our proposal of adding DP noise is useful for detecting backdoor attacks and training more robust models in such a scenario.
Dataset and set up MNIST dataset as described in Section 4.1 is used in this set of experiments. We refer the original 60,000 training images as CLEAN-train and the 10,000 test images as CLEANtest. We construct the backdoor attacks as described in (Gu et al. [2017]), Section 3.1.2. Specifically, each poisoning example is generated by reversing 4 pixel values in the bottom right corner of a clean image having label i, and assigning backdoor label (i+ 1)%10. From CLEAN-train, we randomly sample a poisoning ratio of rp images to be poisoning examples, resulting in a poisoned training dataset POISONED-train(rp). To demonstrate the effectiveness of the poisoning attacks, we use the CLEAN-test dataset, as well as POISONED-test dataset which is constructed by poisoning all images in CLEAN-test. For image classification model, we use convolutional neural network (CNN) containing 2 convolutional layers with max pooling, and a softmax layer to output desired labels. The differentially private models are trained with the same configurations as in Section 4.1 unless otherwise noted.
Metrics We first evaluate the effectiveness of the constructed backdoor attack. A successful backdoor attack should have high image classification accuracy on CLEAN-test, which we refer to as benign accuracy, as well as high accuracy on POISONED-test with poisoned labels, which indicates the success rate. To investigate whether measuring the classification model loss is able to detect poisoning examples, and whether differential privacy is able to improve the detection performance, we leverage metrics AUPR and AUROC as described at the beginning of Section 4.
Results We first evaluate the backdoor attack effectiveness and the detection performance with varying poisoning ratio rp, under different noise scale σ, and fixed clipping bound C = 1. The results are summarized in Table 2. σ =N/A indicates classification models trained without differential privacy. Benign accuracy remains high on clean data. Backdoor success rate is only around half at a poisoning ratio of 0.5%, and shows successful (97.12% success rate) at a poisoning ratio of 1%. Detecting poisoning examples by measuring model loss shows some level of effectiveness when the poisoning ratio is low (e.g., 0.5%). Furthermore, applying differential privacy to the model training process is able to significantly improve the detection performance. Similar as in Table 1, the higher the poisoning ratio, the larger the noise level (smaller ) to achieve the best improvement. Another interesting observation is that, a differentially private model is naturally robust to backdoor attacks. As indicated in Table 2, differential privacy effectively limits the success of backdoor attacks, reducing the success rate below 0.5% in all cases. In comparison, the utility downgrade on benign accuracy is little.
To further evaluate the applicability of using the same CNN model for both anomaly detection and image classification, seeking to co-optimize the performance of the model for both tasks, we collect measurements with varying clipping bound C, and fixed noise scale σ = 0.5, as a complement to Table 2. The results are summarized in Table 3. Note that a small C may hurt model performance as more model parameters are clipped. When C is greater than most parameter values, the effect of increasing C is similar to that of increasing σ (Abadi et al. [2016]). From Table 3, we can observe that the best model for anomaly detection could have a similar set of parameters with the best model for image classification. However, in general, as shown in Table 2, classification accuracy and robustness are often two conflicting desiderata; model trainers can tune the privacy parameter in order to meet the task-specific requirements for accuracy and robustness.

heading: 5 RELATED WORK

text:
To the best of our knowledge, this paper is the first one that proposes to improve outlier/novelty detection with differential privacy, and further extends it to backdoor attack detection. Note that this is not the first work that combines outlier detection and differential privacy together. Okada et al. [2015] aim to preserve input data privacy while detecting outliers. The two tasks are contradicting in this case as the identification of outliers (part of input data) implies certain privacy leakage, so Okada et al. [2015] try to find a balance. In contrast, we focus on improving anomaly detection performance with differential privacy, which is only applied to the model training stage, but no privacy protection is provided for the input data in detection stage when the outliers are actually being detected.
Outlier detection and novelty detection are closely related to each other and often addressed together (Hodge & Austin [2004]; scikit-learn [2017c]; Pedregosa et al. [2011]). Outlier detection is the process of identifying rare items in a dataset that significantly differ from the majority (Aggarwal & Yu [2001]), while novelty detection is to detect new observations that lie in the low density area of the existing dataset (Markou & Singh [2003a;b]). Previous work mostly achieves outlier detection using unsupervised learning methods (Zimek et al. [2012; 2014]; Campos et al. [2016]), while novelty detection typically assumes a normal dataset is available for training, and is realized by semi-supervised learning (Blanchard et al. [2010]; De Morsier et al. [2013]). In both cases, it involves summarizing a distribution that the majority of training data are drawn from. Traditional methods such as clustering (Duan et al. [2009]) and principal component analysis (PCA) (Xu et al. [2010]; Hoffmann [2007]) have been frequently used. In this paper, we leverage deep learning based detection methods including autoencoders (Gottschlich et al. [2017]) and LSTM (Du et al. [2017]) as the baselines, and further extend the idea of measuring model loss to backdoor attack detection.
Proposed by Dwork [2008], differential privacy has been a powerful tool to protect input data privacy. Kasiviswanathan et al. [2011] show that differential privacy implies stability on the output statistical results. Further, Dwork et al. [2015] point out that the empirical average of the output of a differentially private algorithm on a random dataset is close to the true expectation with high probability. Differential privacy has been utilized to train machine learning models that are robust to adversarial examples (Phan et al. [2019]; Lecuyer et al. [2018]), and to bound the success of inference attacks (Yeom et al. [2018]). In this paper, we utilize the property of differential privacy to improve anomaly detection and privacy is ensured via the technique proposed in Abadi et al. [2016].
Lastly, we note that a recent paper by Bagdasaryan & Shmatikov [2019] showed that accuracy of differentially private models drops much more for the underrepresented classes and subgroups. Intrinsically, our paper exploited the same phenomenon to improve anomaly detection. Bagdasaryan & Shmatikov [2019] studied the phenomenon empirically, while our work provides a theoretical analysis, which, for the first time, precisely characterizes the dependence of the performance gap between
the majority and the underrepresented group on the privacy parameters. Moreover, Bagdasaryan & Shmatikov [2019] mainly considered the implication of differential privacy to the fairness of machine learning models; by contrast, our paper focuses on anomaly detection and backdoor attacks and exhibits strong empirical evidence for the efficacy of differential privacy in these two application domains.

heading: 6 CONCLUSION

text: In this paper, inspired by the fact that differential privacy implies stability, we apply DP noise to improve the performance of outlier detection and novelty detection, with an extension to backdoor attack detection. We first provide the theoretical basis for the efficacy of differential privacy for identifying anomalies, connecting the hardness of the identification problem to privacy parameters. Our theoretical results are useful to explain various experimental findings, including how the anomaly detection performance varies with privacy parameters and the number of outliers in the training set. We perform extensive experiments to demonstrate the effectiveness of differential privacy for anomaly detection. To fully evaluate the effectiveness of DP in anomaly detection with different amount of outliers and noisee, we first construct a contaminated dataset based on MNIST and train autoencoder anomaly detection models with varying noise scale applied. We then evaluate the performance using a real-world task, Hadoop file system log anomaly detection, by applying DP noise to DeepLog, the current state-of-the-art detection model. The evaluation results show that DP noise is effective towards reducing the number of false negatives, and further improving the overall utility. Finally, we generalize the idea of measuring model loss for outlier detection to backdoor attack detection and further improve the performance via differential privacy.

heading: A APPENDIX

text:
A.1 PROOF OF THEOREM 2
The following result will be used for reasoning about the performance gap of the differentially private learned models between regular data points and outliers.
Theorem 3 (McDiarmid, 1989). Let S be a set of n points and Si be the set with the ith element in S replaced by z′i, let F : Zn → R be any measurable function for which there exits constants ci (i = 1, . . . , n) such that
sup S∈Zn,z′i∈Z
|F (S)− F (Si)| ≤ ci (5)
then
PS [F (S)− ES [F (S)] ≥ ] ≤ e−2 2/
∑n i=1 c 2 i (6)
Moreover, we will need the following lemma on group differential privacy.
Lemma 1. If A is ( , δ)-differentially private with respect to one change in the database, then A is (c , cec δ)-differentially private with respect to c changes in the database.
Now, we are ready to present the proof of Theorem 2.
Theorem 2. Suppose that a learning algorithm A is ( , δ)-differentially private and UAERM with the rate ξ(n, , δ). Let S′ = S ∪ U , where S ∼ Dn and U contains c arbitrary outliers. Then
Eh∼A(S′)l(h, z̃)− Eh∼A(S′)Ez∼Dl(h, z) ≥ T − 2 ( ξ(n, , δ) + √ n(e − 1 + δ)2
2 log
2 γ + ec − 1 + cec δ
) (4)
with probability at least 1− γ.
Proof. Let the probability density/mass defined by A(S′) and A(S) be p(h) and p′(h), respectively. Using Lemma 1, for any z ∈ Z we have
Eh∼A(S)l(h, z) = ∫ 1
0
Ph∼A(S)[l(h, z) > t]dt (7)
≤ ∫ 1
0
(ec Ph∼A(S′)[l(h, z) > t] + ce c δ)dt (8)
= ec Eh∼A(S′)[l(h, z)] + cec δ (9)
It follows that
Eh∼A(S)l(h, z)− Eh∼A(S′)[l(h, z)] ≤ (ec − 1)Eh∼A(S′)[l(h, z)] + cec δ (10) ≤ ec − 1 + cec δ (11)
By symmetry, it also holds that
Eh∼A(S′)l(h, z)− Eh∼A(S)[l(h, z)] ≤ (ec − 1)Eh∼A(S)[l(h, z)] + cec δ (12) ≤ ec − 1 + cec δ (13)
Thus, we have the following bound:
|Eh∼A(S)l(h, z)− Eh∼A(S′)[l(h, z)]| ≤ ec − 1 + cec δ (14)
Moreover, let Si be the set with the ith element in S replaced by z′i. Then, by the same argument above, we have
|Eh∼A(S)l(h, z)− Eh∼A(Si)[l(h, z)]| ≤ e − 1 + δ (15)
for all i = 1, . . . , n. Then, using Theorem 3 that relates the first order differences of random functions to their variance, we obtain
PS∼D[|Eh∼A(S)l(h, z)− ESEh∼A(S)l(h, z)| ≥ τ ] ≤ 2e − 2τ2 n(e −1+δ)2 (16)
Hence,
PS∼D[|Eh∼A(S)l(h, z)− ESEh∼A(S)l(h, z)| ≥
√ n(e − 1 + δ)2
2 log
2 γ ] ≤ γ (17)
Combining (14) with (17), we have
PS∼D[|Eh∼A(S′)l(h, z)− ESEh∼A(S)l(h, z)| ≤
√ n(e − 1 + δ)2
2 log
2 γ + ec − 1 + cec δ]
≥ 1− γ (18)
Since A is UAERM, the following inequality holds with probability at least 1− γ:
|Eh∼A(S′)l(h, z)− l(h∗, z)| ≤ ξ(n, , δ) +
√ n(e − 1 + δ)2
2 log
2 γ + ec − 1 + cec δ (19)
for all z ∈ Z . For a given outlier z̃, it satisfies l(h∗, z̃)−Ez∼D[l(h∗, z)] ≥ T by definition. Combining it with (19), we get
Eh∼A(S′)l(h, z̃)− Ez∼DEh∼A(S′)l(h, z) ≥ T − 2 ( ξ(n, , δ) + √ n(e − 1 + δ)2
2 log
2 γ + ec − 1 + cec δ
) (20)
A.2 ADDITIONAL EXPERIMENT RESULTS
We list additional experimental results in this section. With more parameters being tested and more metrics being collected, these extensive results further validate our observations presented in the main paper body: differential privacy could improve anomaly detection and backdoor attack detection; and the higher ratio of outliers in the training data, the more noise (smaller privacy bound ) is needed to achieve the best improvement.
A.2.1 AUTOENCODER ANOMALY DETECTION
Besides the AUPR scores presented in Table 1 for outlier detection and novelty detection using autoencoders, we present additional results with more parameters being tested in Table 4. The observations are similar. As an intermediate step in differential privacy to bound the sensitivity, clipping itself without adding any noise is able to improve the performance of outlier detection and novelty detection. Adding various amounts of random Gaussian noise is able to further improve the utility, except when the amount of noise is too big (e.g., σ = 50 and 100) to ruin the model. We also indicate the privacy bound as accumulated by the moments accountant mechanism in (Abadi et al. [2016]). Interestingly, many values are too big to provide any meaningful privacy guarantee, but they are still able to improve the anomaly detection performance.
Besides AUPR scores, we further present AUROC scores in Table 5 for the same set of experiments. Autoencoders, as validated by many previous works (Gottschlich et al. [2017]), present great effectiveness in detecting outliers and novelties, especially when the outlier ratio in training dataset is slow (e.g., below 1%). Although not as obvious as AUPR scores, the improvements brought by differential privacy follow a similar trend, where the improvement is more significant with larger noise (smaller ) being applied to models trained with more outliers.
A.2.2 IMPROVEMENTS OVER DEEPLOG
To compare the differentially private models with DeepLog, we utilize the same anomaly detection criteria, i.e., Top-k based anomaly detection as what’s presented in (Du et al. [2017]). Nevertheless, as a direct extension of the idea in measuring model loss for anomaly detection, we also tested the classification probability as one type of threshold for anomaly detection. In particular, if an actual system log entry is predicted with a probability lower than some threshold Tp, we treat this log entry as a detected anomaly. While the baseline DeepLog results are not as good as the Top-k based detection, we show that similarly, differential privacy is able to significantly reduce the number of false negatives, without introducing too many new false positives.
Probability-based detection. As a preliminary result, we use probability-based anomaly detection to demonstrate the effectiveness of differential privacy noise in reducing FN . Table 6 shows FN and FP for DeepLog and DeepLog+DP with increasing noise levels, under different probability thresholds Tp. It is clear that differential privacy noise could effectively reduce FN , and the larger noise being added, the more false negatives are reduced. We also note that when σ = 0.25, the privacy bound = 90.5. It is often thought that a privacy bound > 20 is completely useless in terms of protecting privacy. Here we indicate that a small amount of noise may be enough to reduce FN . Although more false positives are incurred because of differential privacy noise, the drop in TPR is negligible, considering the large volume of normal data. For example, when Tp = 2 × 10−6 and σ = 1, the FN drop from 1261 to 183 indicates a TNR increase of 8% (91.7%→ 98.8%), while the FP increase from 2291 to 3734 only shows a TPR decrease of 0.3% (99.59%→ 99.32%).
AUROC score To evaluate the overall performance of DeepLog+DP compared with DeepLog under different thresholds, we further compute the AUROC score of DeepLog and DeepLog+DP with different noise scale σ. As shown in Table 7, DeepLog already achieves excellent AUROC score, considering the large amount of normal data and the significantly fewer anomalies. However, an adequate amount of differential privacy noise is still able to improve the performance.
Privacy bound Table 7 also indicates the privacy bound . Note that < 10 is often considered as usable and < 1 is a tight bound that well protects privacy. Considering all the cases, σ = 1 gives the best anomaly detection utility as well as a tight privacy bound to protect training data privacy.
A.2.3 BACKDOOR ATTACK DETECTION
In this section, we evaluate more parameters for the experiment set up described in Section 4.3 BACKDOOR ATTACK DETECTION, and measure benign accuracy, success rate, AUPR score and AUROC score as explained in Section 4.3 for each experiment setting. Similar as the observations in Section 4.3, a differentially private trained machine learning model is naturally more robust to backdoor attacks. The evidence is that the benign accuracy (Table 8) is affected little by differential privacy except when the noise scale is too big to ruin the model parameters, compared with the significant downgrade (e.g., 98.1% to 0.3%) in backdoor success rate (Table 9). Also, as shown in Table 10 and Table 11, measuring model loss to detect poisoning examples could be useful when the
poisoning ratio is low. Nevertheless, applying differential privacy is able to significantly improve the detection performance for a poisoning ratio as high as 45%.

title: ROBUST ANOMALY DETECTION AND BACKDOOR AT- TACK DETECTION VIA DIFFERENTIAL PRIVACY


summaries: This paper focuses on the calibration for the regression problem. First, it investigates the shortcomings of a recently proposed calibration metric [1] for regression, and show theoretically where this metric can be fooled. Later, it introduces new metrics for measuring the calibration in regression problem which are instantiated from similar idea as the ECE calibration metric [2] used for classification problem. The paper defines the uncertainty as the mean square error and like the ECE idea, it divides the samples into different uncertainty bins and for each bin, it calculates RMSE of network output uncertainty. The RMSE versus variance of estimated uncertainty is depicted as the reliability diagram.

summaries: This paper is concerned with uncertainty calibration diagnostics and re-calibration methods, applied to neural network regression. It is motivated by a flaw in the diagnostic proposed by Kuleshov+ 2018 (abreviated K2018 below), as explained around eq4, and proposes a replacement diagnostic for uncertainty calibration quality (sec 3 before 3.1). It then specialises by considering the class of uncertainty prediction schemes qualified as "direct" uncertainty modeling (defined sec1), in which the network predicts the parameters of a parametric distribution over the target output, typically mean and variance of a Gaussian. For this class of schemes, it proposes a recalibration method (sec 3.1), which consists, as shown eq12, of rescaling (with a single parameter $s$) the square root of the variance predicted by the neural network. It then presents experiments (sec4) to demonstrate that the motivating flaw can be evidenced by their diagnostic, and fixed by their recalibration method where it makes sense (ie where predicted uncertainties are not random, i.e. statistically independent of the empirical uncertainty).

abstractText: Predicting not only the target but also an accurate measure of uncertainty is important for many applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for calibration of a regression uncertainty (Kuleshov et al., 2018) has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach inspired by reliability diagrams used in classification tasks. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple scaling-based calibration that preforms well in our experimental tests. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO (Lin et al., 2014) and KITTI (Geiger et al., 2012) datasets.

title: Multiple hypothesis tracking for multiple target tracking

venue: IEEE Aerospace and Electronic Systems Magazine,

authors:
Jifeng Dai
Yi Li
Kaiming He
Jian Sun

title: R-fcn: Object detection via region-based fully convolutional networks

venue: In Proceedings of the 30th International Conference on Neural Information Processing Systems,

authors:
Morris H. DeGroot
Stephen E. Fienberg

title: The comparison and evaluation of forecasters

venue: The Statistician: Journal of the Institute of Statisticians,

authors:
David Eigen
Christian Puhrsch
Rob Fergus

title: Depth map prediction from a single image using a multi-scale deep network

venue: Advances in Neural Information Processing Systems

authors:
Di Feng
Lars Rosenbaum
Klaus Dietmayer

title: Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection

title: Uncertainty in deep learning

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

venue: In Proceedings of the 33rd International Conference on Machine Learning

authors:
Andreas Geiger
Philip Lenz
Raquel Urtasun

title: Are we ready for autonomous driving? the kitti vision benchmark suite

venue: In Conference on Computer Vision and Pattern Recognition (CVPR),

title: Fast r-cnn

venue: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV),

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q. Weinberger

title: On calibration of modern neural networks

venue: In ICML,

authors:
Trevor Hastie
Robert Tibshirani
Jerome Friedman

title: The Elements of Statistical Learning

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Yihui He
Xiangyu Zhang
Marios Savvides
Kris Kitani

title: Softer-nms: Rethinking bounding box regression for accurate object detection

authors:
Alex Kendall
Yarin Gal

title: What uncertainties do we need in bayesian deep learning for computer vision

authors:
Volodymyr Kuleshov
Nathan Fenner
Stefano Ermon

title: Accurate uncertainties for deep learning using calibrated regression

venue: In International Conference on Machine Learning (ICML),

authors:
Balaji Lakshminarayanan
Alexander Pritzel
Charles Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C. Lawrence Zitnick

title: Microsoft coco: Common objects in context

authors:
Mahdi Pakdaman Naeini
Gregory F. Cooper
Milos Hauskrecht

title: Obtaining well calibrated probabilities using bayesian binning

venue: In AAAI,

authors:
Alexandru Niculescu-Mizil
Rich Caruana

title: Predicting good probabilities with supervised learning

venue: In Proceedings of the 22Nd International Conference on Machine Learning,

authors:
D.A. Nix
A.S. Weigend

title: Estimating the mean and variance of the target probability distribution

venue: IEEE International Conference on Neural Networks (ICNN’94),

authors:
Buu Phan
Rick Salay
Krzysztof Czarnecki
Vahdat Abdelzad
Taylor Denouden
Sachin Vernekar

title: Calibrating uncertainties in object localization

venue: task. CoRR,

title: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

venue: In ADVANCES IN LARGE MARGIN CLASSIFIERS,

title: On the infinite divisibility of the lognormal distribution

venue: Scandinavian Actuarial Journal,

authors:
Juan Manuel Vera-Diaz
Daniel Pizarro
Javier Macías Guarasa

title: Towards end-to-end acoustic localization using deep learning: from audio signal to source position

venue: coordinates. CoRR,

authors:
Bianca Zadrozny
Charles Elkan

title: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers

venue: In Proceedings of the Eighteenth International Conference on Machine Learning,

heading: 1 INTRODUCTION

text:
Regression problems arise in many real-world machine learning tasks. To name just a few: Depth from a single image (Eigen et al., 2014), Object localization and Acoustic localization (Vera-Diaz et al., 2018). Many of these tasks are solved by deep neural networks used within decision making pipelines which require the machine learning block not only to predict the target but to also output its confidence in the prediction. For example, the commonly used Kalman-Filter tracking algorithm (Blackman, 2004) requiring variance estimation for the observed object’s location estimation. In addition, we may want the system to output a final uncertainty, reflecting real-world empirical
probabilities, to allow a safety-critical system such as a self-driving car agent to take appropriate actions when confidence drops. In practice, using the confidence in the localization of objects has been shown to improve the non-maximal suppression stage and consequently the overall detection performance (He et al., 2018). Similarly, (Feng et al., 2018) describe a probabilistic 3D vehicle detector for Lidar point clouds that can model both classification and spatial uncertainty.
To provide uncertainty estimation, each prediction produced by the machine learning module during inference should be a distribution over the target domain. There are several approaches for achieving this, most common are Bayesian neural networks (Gal, 2016; Gal & Ghahramani, 2016), ensembles (Lakshminarayanan et al., 2017) and outputting a parametric distribution directly (Nix & Weigend, 1994). Bayesian neural networks place a probability distribution over the network parameters, which is translated to an uncertainty in the prediction, providing a technically sound approach but with overhead at inference time. In the direct approach, outputs of the network represent the parameters of the output distribution for either discrete (Niculescu-Mizil & Caruana, 2005) or continuous (Nix & Weigend, 1994) distributions. Note that the direct approach naturally captures the aleatoric uncertainty (inherent observation noise), but captures less the epistemic uncertainty (uncertainty in the model) (Kendall & Gal, 2017). We chose as a test case for our calibration method, the direct approach for producing uncertainty: we transform the network output from a single scalar to a Gaussian distribution by taking the scalar as the mean and adding a branch that predicts the standard deviation (STD) as in (Lakshminarayanan et al., 2017). While this is probably the simplest form, it is commonly used in practice, and our analysis is applicable to more complex distributions as well as other approaches.
Adjusting the output distributions to match the observed empirical ones via a post process is called uncertainty calibration. It was shown that modern deep networks tend to be over confident in their predictions (Guo et al., 2017). The same study revealed that for classification, Platt Scaling (Platt, 1999), a simple scaling of the pre-activation of the last layer, achieves well calibrated confidence estimates (Guo et al., 2017). In this paper we show that a similar simple scaling strategy, applied to the standard deviations of the output distributions, can calibrate regression algorithms as well.
One major question is how to define calibration for regression, where the model outputs a continuous distribution over possible predictions. In recent work (Kuleshov et al., 2018) suggested a definition based on credible intervals where if we take the p percentiles of each predicted distribution the output should fall below them for exactly p percent of the data. Based on this definition the authors further suggested a calibration evaluation metric and re-calibration method. While this seems very sensible and has the advantage of considering the entire distribution, we found serious flaws in this definition. The main problem arises from averaging over the whole dataset. We show, both empirically and analytically, that one can calibrate using this evaluation metric practically any output distribution, even one which is entirely uncorrelated with the empirical uncertainty as can be seen in Fig. 1. We elaborate on this property of the evaluation method described in (Kuleshov et al., 2018) in Section 2 and show empirical evidence in Section 4.
We propose a new simple definition for calibration for regression, which is closer to the standard one for classification. Calibration for classification can be viewed as expecting the output for every single data point to correctly predict its error, in terms of misclassification probability. In a similar fashion, we define calibration for regression by simply replacing the misclassification probability with the mean square error. Based on this definition, we propose a new calibration evaluation metric similar to the Expected Calibration Error (ECE) (Naeini et al., 2015), which groups examples into interval bins with similar uncertainty, and then measures the discrepancy between each bin’s parameters and the parameters of the empirical distribution within the bin. An additional dispersion measure completes our set of diagnostic tools by revealing cases where the individual uncertainty outputs are uninformative as they all return similar values.
Finally, we propose a calibration method where we re-adjust the predicted uncertainty, in our case the outputted Gaussian variance, by minimizing the negative-log-likelihood (NLL) on a separate re-calibration set. We show good calibration results on a real-world dataset using a simple parametric model which scales the uncertainty by a constant factor. As opposed to (Kuleshov et al., 2018), we show that our approach cannot calibrate predicted uncertainty that is uncorrelated with the real uncertainty, as one would expect.
To summarize, our main contributions are:
• Revealing the fundamental flaws in the current definition of calibrated regression uncertainty (Kuleshov et al., 2018)
• A new proposed definition for calibrated uncertainty in regression tasks
• A simple scaling method that can reduce the calibration error similar to temperature scaling for classification (Guo et al., 2017), evaluated on large scale real world vision datasets.

heading: 1.1 RELATED WORK

text:
While shallow neural networks are typically well-calibrated (Niculescu-Mizil & Caruana, 2005), modern, deep networks, albeit superior in accuracy, are no-longer calibrated (Guo et al., 2017). Uncertainty calibration for classification is a relatively studied field. Calibration plots or Reliability diagrams provide a visual representation of uncertainty prediction calibration (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) by plotting expected sample accuracy as a function of confidence. Confidence values are grouped into interval bins to allow computing the sample accuracy. A perfect model corresponds to the plot of the identity function. The Expected Calibration Error (ECE) (Naeini et al., 2015) summarizes the reliability diagram by averaging the error (gap between confidence and accuracy) in each bin, producing a single value measure of the calibration. Similarly, the Maximum Calibration Error (MCE) (Naeini et al., 2015) measures the maximal gap. Negative Log Likelihood (NLL) is a standard measure of a model’s fit to the data (Hastie et al., 2001) but combines both accuracy of the model and its uncertainty estimation in one measure. Based on these measures, several calibration methods were proposed, which transform the network’s confidence output to one that will produce a calibrated prediction. Non-parametric transformations include Histogram Binning (Zadrozny & Elkan, 2001), Bayesian Binning into Quantiles (Naeini et al., 2015) and Isotonic Regression (Zadrozny & Elkan, 2001) while parametric transformations include versions of Platt Scaling (Platt, 1999) such as Matrix Scaling and Temperature Scaling (Guo et al., 2017). In (Guo et al., 2017) it is demonstrated that the simple Temperature Scaling, consisting of a one scalingparameter model which multiplies the last layer logits, suffices to produce excellent calibration on many classification data-sets.
In comparison with classification, calibration of uncertainty prediction in regression, has received little attention so far. As already described, (Kuleshov et al., 2018) propose a practical method for evaluation and calibration based on confidence intervals and isotonic regression. The proposed method is applied in the context of Bayesian neural networks. In recent work (Phan et al., 2018), the authors follow (Kuleshov et al., 2018) definition and method of calibration for regression, but use a standard deviation vs. MSE scatter plot, somewhat similar to our approach, as a sanity check.

heading: 2 CONFIDENCE-INTERVALS BASED CALIBRATION

text:
We next review the method for regression uncertainty calibration proposed in (Kuleshov et al., 2018) which is based on confidence intervals, and highlight its shortcomings. We refer to this method in short as the “interval-based” calibration method. We start by introducing basic notations for uncertainty calibration used throughout the paper.
Notations. Let X,Y ∼ P be two random variables jointly distributed according to P and X × Y their corresponding domains. A dataset {(xt, yt)}Tt=1 consists of i.i.d samples of X,Y . A forecaster H : X → P(Y) outputs per example xt a distribution pt ≡ H(xt) over the label space, where P(Y ) is the set of all distributions over Y . In classification tasks, Y is discrete and pt is a multinomial distribution, and in regression tasks in which Y is a continuous domain, pt is usually a parametric probability density function, e.g. a Gaussian. For regression, we denote by Ft : Y → [0, 1] the CDF corresponding to pt.
According to (Kuleshov et al., 2018) a forecaster in a regression setting H is calibrated if:∑T t=1 I{yt ≤ F −1 t (p)}
T
T→∞−−−−→ p,∀p ∈ [0, 1] (1)
.
Intuitively this means that the yt is smaller than F−1t (p) with probability approximately p, or that the predicted CDF matches the empirical one as the dataset size goes to infinity. This is equivalent to
PX,Y
( Y ≤ [F (X)]−1 (p) ) = p, ∀p ∈ [0, 1] (2)
Where F (X) represents the CDF corresponding to H(X). This notion is translated by (Kuleshov et al., 2018) to a practical evaluation and calibration methodology. A re-calibration dataset S = {(xt, yt)}Tt=1 is used to compute the empirical CDF value for each predicted CDF value p ∈ Ft (yt):
P̂ (p) = |{yt|Ft (yt) ≤ p, t = 1 . . . T}|
T (3)
The calibration consists of fitting a regression function R (i.e. isotonic regression) , to the set of points {(p, P̂ (p))}Tt=1. For diagnosis the authors suggest a calibration plot of {(p, P̂ (p))} at equally spaced values of p.
We start by intuitively explaining the basic limitation of this methodology. From Eq. 3 P̂ is nondecreasing and therefore isotonic regression finds a perfect fit. Therefore, the modified CDF R ◦ Ft will satisfy P̂ (p) = p on the re-calibration set, and the new forecaster is calibrated up to sampling error. This means that perfect calibration is possible no matter what the CDF output is, even for output CDFs which are statistically independent of the actual empirical uncertainty. We note that this might be acceptable when the uncertainty prediction is degenerate, e.g. all output distributions are Gaussian with the same variance, but this is not the case here. We also note that the issue is with the calibration definition not the re-calibration, as we show with the following analytic example.
We next present a concise analytic example in which the output distribution and the ground truth distribution are independent, yet fully calibrated according to Eq. 2. Consider the case where the target has a normal distribution yt ∼ N (0, 1) and the network outputH(xt) has a Cauchy distribution with zero location parameter and random scale parameter γt independent of xt and yt, defined as:
zt ∼ N (0, 1) (4) γt = |zt|
H(xt) = Cauchy(0, γt)
Following a known equality for Cauchy distributions, the CDF output of the network Ft(y) = F ( y γt ) , where F is the CDF of a Cauchy distribution with zero location and 1 scale parameters. First we note that ytγt and yt zt
, i.e. with and without the absolute value, have the same distribution due to symmetry. Next we recall the well known fact that the ratio of two independent normal random variables is distributed as Cauchy with zero location and 1 scale parameters (i.e. ytzt ∼ Cauchy(0, 1)). This means that probability that Ft(yt) ≡ F ( ytγt ) ≤ p is exactly p (recall that F is a Cauchy(0, 1) CDF). In other words, the prediction is perfectly calibrated according to the definition in Eq. 2, even though the scale parameter was random and independent of the distribution of yt.
While the Cauchy distribution is a bit unusual due to the lack of mean and variance, the example does not depend on it and it was chosen for simplicity of exposition. It is possible to prove the existence of a distribution whose product of two independent samples is Gaussian (Thorin, 1977) and replace the Cauchy with a Gaussian, but it is an implicit construction and not a familiar distribution.

heading: 3 OUR METHOD

text:
We present a new definition for calibration for regression, as well as several evaluation measures and a reliability diagram for calibration diagnosis, analogous to the ones used for classification (Guo et al., 2017). The basic idea is that for each value of uncertainty, measured through standard deviation σ, the expected mistake, measured in mean square error (MSE), matches the predicted error σ2. This is similar to classification with MSE replacing the role of mis-classification error. More formally, if µ(x) and σ(x)2 are the predicted mean and variance respectively then we consider a regressor well calibrated if:
∀σ : Ex,y [ (µ(x)− y)2|σ(x)2 = σ2 ] = σ2. (5)
In contrast to to (Kuleshov et al., 2018) this does not average over points with different values of σ2 (at least in the definition, for practical measures some binning is needed), but only considers the mean and variance and not the entire distribution. We claim that this captures the desired meaning of calibration, i.e. for each individual example you can correctly predict the expected mistake.
Since we can expect each exact value of σ2 in our dataset to appear exactly once, we evaluate eq. 3 empirically using binning, same as for classification. Formally, let σt be the standard deviation of predicted output PDF pt and assume without loss of generality that the examples are ordered by increasing values of σt. We also assume for notation simplicity that the number of bins, N , divides the number of examples, T . We divide the indices of the examples to N bins, {Bj}Nj=1, such that: Bj = {(j− 1) · TN + 1, . . . , j · T N }. Each resulting bin therefore represents an interval in the standard deviation axis: [mint∈Bj{σt},maxt∈Bj{σt}]. The intervals are non-overlapping and their boundary values are increasing.
To evaluate how calibrated the forecaster is, we compare per bin j two quantities as follows. The root of the mean variance:
mVAR(j) = √√√√ 1 |Bj | ∑ t∈Bj σ2t (6)
And the empirical root mean square error:
RMSE(j) = √√√√ 1 |Bj | ∑ t∈Bj (yt − ŷt)2 (7)
where ŷt is the mean of the predicted PDF (pt)
For diagnosis, we propose a reliability diagram which plots the RMSE as function of the mVAR as shown in Figure 4. The idea is that for a calibrated forecaster per bin the mVAR and the observed RMSE should be approximately equal, and hence the plot should be close to the identity function. Apart from this diagnosis tool which as we will show is valuable for assessing calibration, we propose additional scores for evaluation.
Expected Normalized Calibration Error (ENCE). For summarizing the error in the calibration we propose the following measure:
ENCE = 1
N N∑ j=1 |mVAR(j)−RMSE(j)| mVAR(j)
(8)
This score averages the calibration error in each bin, normalized by the bin’s mean predicted variance, since for larger variance we expect naturally larger errors. This measure is analogous to the expected calibration error (ECE) used in classification.
STDs Coefficient of variation (CV ). In addition to the calibration error we would like to measure the dispersion of the predicted uncertainties. If for example the forecaster predicts a single homogeneous uncertainty measure for each example, which matches the empirical uncertainty of the predictor for the entire population, then the ENCE would be zero, but the uncertainty estimation per example would be uninformative. Therefore, we complement the ENCE measure with the Coefficient of Variation (cv) for the predicted STDs which measures their dispersion:
cv =
√∑T t=1(σt−µσ)2
T−1
µσ (9)
where µσ = 1T ∑T t=1 σt. Ideally the cv should be high indicating a disperse uncertainty estimation over the dataset. We propose using the ENCE as the primary calibration measure and the cv as a secondary diagnostic tool.

heading: 3.1 CALIBRATION

text:
To understand the need for calibration, let us start by considering a trained neural network for regression, which has very low mean squared error (MSE) on the train data. We now add a separate branch that predicts uncertainty as standard deviation, which together with the original network output interpreted as the mean, defines a Gaussian distribution per example. In this case, the NLL loss on the train data can be minimized by lowering the standard deviation of the predictions, without changing the MSE on train or test data. On test data however, MSE will be naturally higher. Since the predicted STDs remain low on test examples, this will result in higher NLL and ENCE values for the test data. This type of miss-calibration is defined as over-confidence, but opposite or mixed cases can occur depending on how the model is trained.
Negative log-likelihood. NLL is a standard measure for a probabilistic model’s quality (Hastie et al., 2001). When training the network to output classification confidence or a regression distribution, it is commonly used as the objective function to minimize. It is defined as:
NLL = − T∑ t=1 log ([H(xt)](yt)) (10)
We propose using the NLL on the re-calibration set as our objective for calibration, and the reliability diagram, together with its summary measures (ENCE , cv) for diagnosis of the calibration. In the most general setting a calibration function maps predicted PDFs to calibrated PDFs: R(Θ) : P(Y)→ P(Y) where θ is the set of parameters defining the mapping. Optimizing calibration over the re-calibration set is obtained by finding θ yielding minimal NLL:
arg min θ
( −
T∑ t=1 log ([R(pt; Θ)] (yt))
) . (11)
To ensure the calibration generalization, the diagnosis should be made on a separate validation set. Multiple choices exist for the family of functions R belongs to. We propose using STD Scaling, (in analogy to Temperature Scaling (Guo et al., 2017)), which essentially multiplies the STD of each predicted distribution by a constant scaling factor s. If the predicted PDF is that of a Gaussian distribution,N (µ, σ2), then the re-calibrated PDF isN (µ, (s·σ)2). Hence, in this case the calibration objective (Eq. 11) is:
arg min s
( −
T∑ t=1 log
( 1√
2πsσt e
(yt−µt)2
2s2σ2t )) = arg min
s
( T
2 log(s)− T∑ t=1 (yt − µt)2 2s2σ2t
) (12)
If the original predictions are overconfident, as common in neural networks, then the calibration should set s > 1. This is analogous to Temperature Scaling in classification: a single multiplicative parameter is tuned to fix over or under-confidence of the model, and it does not modify the model’s final prediction since µt remains unchanged.
More complex calibration methods. Histogram binning and Isotonic Regression applied to the STDs can be also used as calibration methods. We chose STD scaling since: (a) it is less prone to overfit the validation set, (b) it does not enforce minimal and maximal STD values, (c) it is easy to implement and (d) empirically, it produced good calibration results.

heading: 4 EXPERIMENTAL RESULTS

text: We next show empirical results of our approach on two tasks: a controlled synthetic regression problem and object detection bounding box regression. In both tasks we examine the effect of outputting trained and random uncertainty on the calibration process. In all training and optimization stages we use an SGD optimizer with learning rate 0.001 and 0.9 momentum.

heading: 4.1 SYNTHETIC REGRESSION PROBLEM

text:
Experimenting with a synthetic regression problem enables us to control the target distribution Y and to validate our method. We randomly generate T = 50, 000 input samples {xt, yt}Tt=1. We sample xt
from X ∼ Uniform[0.1, 1] and yt from Y ∼ N (xt, x2t ). This way, the target standard deviation of sample xt is xt. We train a fully-connected network with four layers and a ReLU activation function on the generated training set using the smooth L1 loss function. We then add a separate branch with its own four layers to predict uncertainty.
Per example xt, The original network output is considered the mean of a Gaussian distribution (µt) and the additional output as its standard deviation (σt). For numerical stability, as suggested by (Kendall & Gal, 2017), the network outputs log(σ2). In the random uncertainty experiment, per example, the standard deviation representing the uncertainty is randomly drawn from Uniform[1, 10]. For the predicted uncertainty experiment, the uncertainty branch is optimized using the NLL loss (Eq. 10) while the rest of the network weights are fixed. By fixing the remaining weights, the predicted mean (µt) remains unchanged making sure we do not calibrate using over-confident predictions. We then re-calibrate as described in Sec. 3.1 on a separate re-calibration set consisting of 6, 000 samples.
As one can see in Fig. 2 the confidence interval method can almost perfectly calibrate the random independent uncertainty estimation, as the expected and observed confidence level match and we get the desired identity curve. This phenomenon is extremely undesirable for safety critical applications where falsely relying on uninformative uncertainty can lead to severe consequences. It is important to note that the perfect calibration did not arise from giving the same fixed σ for each prediction, which would be acceptable, as the isotonic regression modifies the probabilities directly and not the outputted standard deviations. In contrast you can see how our method can only marginally improve the calibration and one can clearly see, both from the ENCE value and visually from the graph, that the predictions are not calibrated. In the trained experiment, in which uncertainty is predicted by the network, we can see in Fig. 3 that the network almost perfectly learns the correct uncertainty, as expected from the problem simplicity and the high data availability. In this case both methods do not change the calibration results much. The important thing to note is that our calibration and evaluation
method can easily differentiate between both cases, the random and predicted uncertainty, while they are almost exactly the same after calibrating with (Kuleshov et al., 2018).

heading: 4.2 BOUNDING BOX REGRESSION FOR OBJECT DETECTION

text:
In computer vision, an object detector outputs per input image a set of bounding boxes, each commonly defined by 5 outputs: classification confidence and four positional outputs (tx, ty, tw, th) representing its (x,y) position, width and height. We show results on each positional output as an independent regression task using the R-FCN detector (Dai et al., 2016). To this architecture we add an additional uncertainty branch predicting the corresponding STDs for each regression output, (σx, σy, σw, σh). Thus, the network outputs a Gaussian distribution per regression task. For training the network weights we use the entire Common objects in context (COCO) dataset (Lin et al., 2014). For uncertainty calibration and validation we use two separate subsets of the KITTI (Geiger et al., 2012) object detection benchmark dataset, which consists of road scenes. Training the uncertainty output on one dataset and performing calibration on a different one reduces the risk of over-fitting and increases the calibration validity. See Appendix A for further details.
We initially train the network without the additional uncertainty branch as in (Dai et al., 2016), while the uncertainty branch weights are randomly initialized. Therefore, in this state which we refer to as untrained uncertainty, random uncertainties are assigned to each example. We then train the uncertainty branch by minimizing the NLL loss (Eq. 10) on the training set, freezing all network weights but the uncertainty head for 1K training iterations with 6 images per iteration. Freezing the rest of the network ensures that the additional uncertainty estimation does not sacrify accuracy. The result of this stage is the network with predicted uncertainty. Finally, we train the NLL loss for 1K additional training iterations on the re-calibration set, to optimize the single scaling parameter s, and obtain the calibrated uncertainty.
Figure 4 shows the resulting reliability diagrams before calibration (predicted uncertainty) and after (calibrated uncertainty) for all four positional outputs, on the validation set. As can be observed from the monotonously increasing curve before calibration, the output uncertainties are indeed correlated with the empirical ones. Additionally, since the curves are entirely above the ideal one, the predictions are over confident. Using the learned scaling factor s which varies between 1.1 and 1.2, the ENCE is significantly reduced as shown in table 1. The cv remains unchanged after calibration since it is invariant to uniform scaling of the output STDs (Eq. 9). For untrained uncertainty, Fig. 1 shows that after calibration, just as with the synthetic dataset, using the interval-based method, uncertainty is almost perfectly calibrated. In contrast, our method reveals the lack of correlation between the predictions and empirical uncertainties before/ after applying calibration (See results in Appendix A).

heading: 5 CONCLUSIONS

text:
Calibration, and more generally uncertainty prediction, are critical parts of machine learning especially in safety-critial applications. In this work we exposed serious flaws in the current approach to define and evaluate calibration for regression problem. We also proposed an alternative approach and showed that even a very simple re-calibration method can lead to significant improvement in real-world applications. Our proposed method for calibration effectively takes into consideration the first two moments when comparing the output and real distributions. An interesting direction for future
research would be extending our method to handle more complex distributions using, for example, higher-order moments.

heading: A BOUNDING BOX REGRESSION FOR OBJECT DETECTION: ADDITIONAL DETAILS AND RESULTS

text:
In this section we provide additional details on the network architecture and datasets used for bounding box prediction as well as additional results using random predictions. As our base architecture we use the R-FCN detector (Dai et al., 2016) with a ResNet-101 backbone (He et al., 2016). The R-FCN regression branch outputs per region candidate a 4-d vector that parametrizes the bounding box as tb = (tx, ty, tw, th) following the accepted parametrization in (Girshick, 2015). We use these outputs in our experiments as four seperate regression outputs. To this architecture we add an uncertainty branch, identical in structure to the regression branch, that outputs a 4-d vector (u1, u2, u3, u4) ≡ (log(σ2x).log(σ 2 y), log(σ 2 w), log(σ 2 h)), each representing the log variance of the Gaussian distributions of the corresponding output. As before, the original regression output represents the Gaussian mean (i.e. µx = tx).
For training the network weights we use the entire Common objects in context (COCO) dataset (Lin et al., 2014). As stated previously we use a two-stage training approach. We first train the original RFCN network, and then freeze all weights and train only the additional uncertainty prediction branch. In this way we train uncertainty prediction without sacrificing the network’s accuracy. Note however that our method completely holds if the entire network is trained at once (e.g. if confidence estimation importance is such that accuracy may be marginally sacrified). For uncertainty calibration we use the KITTI (Geiger et al., 2012) object detection benchmark dataset, which consists of road scenes. We divide the KITTI dataset into a re-calibration set used for training the calibration parameters (∼ 6K images), and a validation set (∼ 1.5K images, 37K object instances). The classes in the KITTI dataset represent a small subset of the classes in the COCO dataset, and therefore we reduce our model training on COCO to the 9 relevant classes (e.g. car, person) and map them accordingly to the KITTI classes.
Figure 5 shows the reliability diagrams for the four bounding box regression outputs with untrained uncertainty before and after we apply our calibration method. As with the synthetic dataset, the graphs immediately reveal the disconnect between the random values and the empirical uncertainties. In all the cases the calibration results in a highly non-calibrated uncertainty according to our metrics.