summaries: This paper introduces a taxonomy of OODs and proposed an integrated approach to detect different types of OODs. Their taxonomy classifies OOD on the nature of their uncertainty and they show that no single state-of-the-art approach detects all these OOD types. Motivated by this observation, they combine multiple existing OOD detection methods to detect various types of OODs.

summaries: This paper introduces a novel taxonomy for OOD outliers. The authors analyze current OOD detection approaches and uncover their limitations. They propose to fuse several existing approaches into a combined one and extensively evaluate it on various data sets (CIFAR,10, SVNH, MNIST, STL10, ImageNet, etc.). The proposed integrated OOD detection approach clearly shows superior performance.

summaries: The authors explore the different kinds of outliers and show that the methods previously proposed detect different kinds of OOD and not a single one can detect them all. The authors propose an interesting study of the different kind of outlier on synthetic data which  illustrates well the different characteristics of the outlier types. The authors then propose to combine different methods to increase the OOD detection rate. Experiments are conducted on 3 images classification datasets using different deep neural networks. For each dataset, samples from other databases are introduced as outliers and must be detected. The combination method yield better detection rates than baseline methods in almost all configurations.

abstractText: Deep neural networks (DNNs) are known to produce incorrect predictions with very high confidence on out-of-distribution (OOD) inputs. This limitation is one of the key challenges in the adoption of deep learning models in high-assurance systems such as autonomous driving, air traffic management, and medical diagnosis. This challenge has received significant attention recently, and several techniques have been developed to detect inputs where the model’s prediction cannot be trusted. These techniques use different statistical, geometric, or topological signatures. This paper presents a taxonomy of OOD outlier inputs based on their source and nature of uncertainty. We demonstrate how different existing detection approaches fail to detect certain types of outliers. We utilize these insights to develop a novel integrated detection approach that uses multiple attributes corresponding to different types of outliers. Our results include experiments on CIFAR10, SVHN and MNIST as in-distribution data and Imagenet, LSUN, SVHN (for CIFAR10), CIFAR10 (for SVHN), KMNIST, and F-MNIST as OOD data across different DNN architectures such as ResNet34, WideResNet, DenseNet, and LeNet5.

authors:
Jinwon An
Sungzoon Cho

title: Variational autoencoder based anomaly detection using reconstruction probability

venue: Special Lecture on IE,

authors:
Mariusz Bojarski
Davide Del Testa
Daniel Dworakowski
Bernhard Firner
Beat Flepp
Prasoon Goyal
Lawrence D Jackel
Mathew Monfort
Urs Muller
Jiakai Zhang

title: End to end learning for self-driving cars

venue: arXiv preprint arXiv:1604.07316,

authors:
Tarin Clanuwat
Mikel Bober-Irizar
Asanobu Kitamoto
Alex Lamb
Kazuaki Yamamoto
David Ha

title: Deep learning for classical japanese literature

venue: arXiv preprint arXiv:1812.01718,

authors:
Adam Coates
Andrew Ng
Honglak Lee

title: An analysis of single-layer networks in unsupervised feature learning

venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,

authors:
Jeffrey De Fauw
Joseph R Ledsam
Bernardino Romera-Paredes
Stanislav Nikolov
Nenad Tomasev
Sam Blackwell
Harry Askham
Xavier Glorot
Brendan O’Donoghue
Daniel Visentin

title: Clinically applicable deep learning for diagnosis and referral in retinal disease

venue: Nature medicine,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Georgia Gkioxari
Ross Girshick
Jitendra Malik

title: Contextual action recognition with r* cnn

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: arXiv preprint arXiv:1412.6572,

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q Weinberger

title: On calibration of modern neural networks

venue: arXiv preprint arXiv:1706.04599,

authors:
Awni Hannun
Carl Case
Jared Casper
Bryan Catanzaro
Greg Diamos
Erich Elsen
Ryan Prenger
Sanjeev Satheesh
Shubho Sengupta
Adam Coates

title: Deep speech: Scaling up end-to-end speech recognition

venue: arXiv preprint arXiv:1412.5567,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: arXiv preprint arXiv:1610.02136,

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

venue: In International Conference on Learning Representations,

authors:
Dan Hendrycks
Mantas Mazeika
Saurav Kadavath
Dawn Song

title: Using self-supervised learning can improve model robustness and uncertainty

venue: In Advances in Neural Information Processing Systems,

title: Kernel pca for novelty detection

venue: Pattern recognition,

title: Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management

venue: Reliability Engineering & System Safety,

authors:
Gao Huang
Zhuang Liu
Laurens Van Der Maaten
Kilian Q Weinberger

title: Densely connected convolutional networks

authors:
Eyke Hüllermeier
Willem Waegeman

title: Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction

venue: arXiv preprint arXiv:1910.09457,

authors:
Heinrich Jiang
Been Kim
Melody Guan
Maya Gupta

title: To trust or not to trust a classifier

venue: In Advances in neural information processing systems,

authors:
Kyle D Julian
Mykel J Kochenderfer

title: Neural network guidance for uavs

venue: In AIAA Guidance, Navigation, and Control Conference,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Kimin Lee
Honglak Lee
Kibok Lee
Jinwoo Shin

title: Training confidence-calibrated classifiers for detecting out-of-distribution samples

venue: arXiv preprint arXiv:1711.09325,

authors:
Kimin Lee
Kibok Lee
Honglak Lee
Jinwoo Shin

title: A simple unified framework for detecting out-of-distribution samples and adversarial attacks

authors:
Shiyu Liang
Yixuan Li
Rayadurgam Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

venue: arXiv preprint arXiv:1706.02690,

authors:
Weitang Liu
Xiaoyun Wang
John Owens
Sharon Yixuan Li

title: Energy-based out-of-distribution detection

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Navonil Majumder
Soujanya Poria
Alexander Gelbukh
Erik Cambria

title: Deep learning-based document modeling for personality detection from text

venue: IEEE Intelligent Systems,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

authors:
Nicolas Papernot
Patrick McDaniel

title: Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning

venue: arXiv preprint arXiv:1803.04765,

authors:
Jie Ren
Peter J Liu
Emily Fertig
Jasper Snoek
Ryan Poplin
Mark Depristo
Joshua Dillon
Balaji Lakshminarayanan

title: Likelihood ratios for out-of-distribution detection

authors:
Jihoon Tack
Sangwoo Mo
Jongheon Jeong
Jinwoo Shin

title: Csi: Novelty detection via contrastive learning on distributionally shifted instances

authors:
Fisher Yu
Ari Seff
Yinda Zhang
Shuran Song
Thomas Funkhouser
Jianxiong Xiao

title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

venue: arXiv preprint arXiv:1506.03365,

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

venue: arXiv preprint arXiv:1605.07146,

heading: 1 INTRODUCTION

text:
Deep neural networks (DNNs) have achieved remarkable performance-levels in many areas such as computer vision (Gkioxari et al., 2015), speech recognition (Hannun et al., 2014), and text analysis (Majumder et al., 2017). But their deployment in the safety-critical systems such as self-driving vehicles (Bojarski et al., 2016), aircraft collision avoidance (Julian & Kochenderfer, 2017), and medical diagnoses (De Fauw et al., 2018) is hindered by their brittleness. One major challenge is the inability of DNNs to be self-aware of when new inputs are outside the training distribution and likely to produce incorrect predictions. It has been widely reported in literature (Guo et al., 2017a; Hendrycks & Gimpel, 2016) that deep neural networks exhibit overconfident incorrect predictions on inputs which are outside the training distribution. The responsible deployment of deep neural network models in high-assurance applications necessitates detection of out-of-distribution (OOD) data so that DNNs can abstain from making decisions on those.
Recent approaches for OOD detection consider different statistical, geometric or topological signatures in data that differentiate OODs from the training distribution. For example, the changes in the softmax scores due to input perturbations and temperature scaling have been used to detect OODs (Hendrycks & Gimpel, 2016; Liang et al., 2017; Guo et al., 2017b). Papernot & McDaniel (2018) use the conformance among the labels of the nearest neighbors while Tack et al. (2020) use cosine similarity (modulated by the norm of the feature vector) to the nearest training sample for the detection of OODs. Lee et al. (2018) consider the Mahalanobis distance of an input from the in-distribution data to detect OODs. Several other metrics such as reconstruction error (An & Cho, 2015), likelihood-ratio between the in-distribution and OOD samples (Ren et al., 2019), trust scores (ratio of the distance to the nearest class different from the predicted class and the distance to the predicted class) (Jiang et al., 2018), density function (Liu et al., 2020; Hendrycks et al., 2019a), probability distribution of the softmax scores (Lee et al., 2017; Hendrycks et al., 2019b; Tack et al., 2020; Hendrycks et al., 2019a) have also been used to detect OODs. All these methods attempt to develop a uniform approach with a single signature to detect all OODs accompanied by empirical
evaluations that use datasets such as CIFAR10 as in-distribution data and other datasets such as SVHN as OOD.
Our study shows that OODs can be of diverse types with different defining characteristics. Consequently, an integrated approach that takes into account the diversity of these outliers is needed for effective OOD detection. We make the following three contributions in this paper:
• Taxonomy of OODs. We define a taxonomy of OOD samples that classify OODs into different types based on aleatoric vs epistemic uncertainty (Hüllermeier & Waegeman, 2019), distance from the predicted class vs the distance from the tied training distribution, and uncertainty in the principal components vs uncertainty in non-principal components with low variance.
• Incompleteness of existing uniform OOD detection approaches. We examine the limitations of the state-of-the-art approaches to detect various types of OOD samples. We observe that not all outliers are alike and existing approaches fail to detect particular types of OODs. We use a toy dataset comprising two halfmoons as two different classes to demonstrate these limitations.
• An integrated OOD detection approach. We propose an integrated approach that can detect different types of OOD inputs. We demonstrate the effectiveness of our approach on several benchmarks, and compare against state-of-the-art OOD detection approaches such as the ODIN (Liang et al., 2017) and Mahalanobis distance method (Lee et al., 2018).

heading: 2 OOD TAXONOMY AND EXISTING DETECTION METHODS

text:
DNNs predict the class of a new input based on the classification boundaries learned from the samples of the training distribution. Aleatory uncertainty is high for inputs which are close to the classification boundaries, and epistemic uncertainty is high when the input is far from the learned distributions of all classes (Hora, 1996; Hüllermeier & Waegeman, 2019). Given the predicted class of a DNN model on a given input, we can observe the distance of the input from the distribution of this particular class and identify it as an OOD if this distance is high. We use this top-down inference approach to detect this type of OODs which are characterized by an inconsistency in model’s prediction and input’s distance from the distribution of the predicted class. Further, typical inputs to DNNs are high-dimensional and can be decomposed into principal and non-principal components based on the direction of high variation; this yields another dimension for classification of OODs. We, thus, categorize an OOD using the following three criteria.
1. Is the OOD associated with higher epistemic or aleatoric uncertainty, i.e., is the input away from in-distribution data or can it be confused between multiple classes?
2. Is the epistemic uncertainty of an OOD sample unconditional or is it conditioned on the class predicted by the DNN model?
3. Is the OOD an outlier due to unusually high deviation in the principal components of the data or due to small deviation in the non-principal (and hence, statistically invariant) components?
Figure 1 demonstrates different types of OODs which differ along these criteria. Type 1 OODs have high epistemic uncertainty and are away from the indistribution data. Type 2 OODs have high epistemic uncertainty with respect to each of the 3 classes even though approximating all in-distribution (ID) data using a single Guassian distribution will miss these outliers. Type 3 OODs have high aleatoric uncertainty as they are close to the decision boundary between class 0 and class 1. Type 4 and 5 have high epistemic uncertainty with respect to their closest classes. While Type 4 OODs are far from the distribution along the principal axis, Type 5 OODs vary along a relatively invariant axis where even a small
deviation indicates that the sample is an OOD.
Limitations of Existing Detection Methods. We empirically demonstrate the limitations of existing OOD detection methods on a two-dimensional (2D) half-moon dataset with two classes. As shown in Figure 2, we consider three clusters of OOD samples: cluster A (black), B (brown) and C(red). Figure 2 (right) shows the 2D penultimate features of the classifier.
Different approaches differ in their ability to detect different OOD types as illustrated in Figure 3.
• Figure 3(a) shows that the Mahalanobis distance (Lee et al., 2018) from the mean and tied covariance of all the training data in the feature space cannot detect OODs in the clusters B and C corresponding to class-conditional epistemic uncertainty and aleatoric uncertainty, respectively. It attains the overall true negative rate (TNR) of 39.09% at the 95% true positive rate (TPR).
• Figure 3(b) shows that the softmax prediction probability (SPB) (Hendrycks & Gimpel, 2016) cannot detect the OODs in cluster A corresponding to high epsitemic uncertainty. The TNR ( at 95% TPR) reported by the SPB technique is 60.91%.
• Figure 3(c) shows that class-wise Principal Component Analysis (PCA) (Hoffmann, 2007) cannot detect OODs in cluster C corresponding to high aleatoric uncertainty. We performed PCA of the two classes separately in the feature space and used the minimum reconstruction error to detect OODs. This obtained overall TNR of 80.91% (at 95% TPR).
• Figure 3(d) shows that K-Nearest Neighbor (kNN) (Papernot & McDaniel, 2018) nonconformance in the labels of the nearest neighbors cannot detect OODs in clusters A and B with high epistemic uncertainty. The overall TNR (at 95% TPR) reported by this technique is 15%.
These observations can be explained by the focus of different detection techniques on measuring different forms of uncertainty. This motivates our integrated OOD detection method.

heading: 3 INTEGRATED OOD DETECTION METHOD

text:
Complementary information about different OOD types can be used to detect a wider range of OODs. Figure 4 shows the improvement in the TNR of the OOD detector composed with information about different classes of OODs on the two half-moons dataset. Non-conformity in the labels of the nearest neighbors captures OODs in cluster C. Mahalanobis distance from the tied in-distribution detects OODs in cluster A. Reconstruction error from the PCA of the 2 class distributions captures OODs in cluster B. Softmax scores further strengthens the OOD detection by reporting OODs in cluster C that are undetected by the other three methods.
The integrated OOD detection approach, thus, uses the following attributes, each specialized in detecting a specific type (or a combination of types) of OODs:
1. Mahalanobis distance from the in-distribution density estimate that considers either tied (Lee et al., 2018) or class-wise covariance estimate. This attribute captures the overall or classconditional epistemic uncertainty of an OOD. Our refinement to also use class-wise covariance significantly improves detection of OODs when coupled with PCA approach described below.
2. Conformance measure among the variance of the Annoy (Bernhardsson, 2018) nearest neighbors calculated as the Mahalanobis distance of the input’s conformance to the closest class conformance. Our experiments found this to be very effective in capturing aleatoric uncertainty. This new attribute is a fusion of nearest-neighbor and Mahalanobis distance methods in literature.
3. Prediction confidence of the classifier as the maximum softmax score on the perturbed input where the perturbation used is the same as ODIN approach (Liang et al., 2017). This boosts the detection of high aleatoric uncertainty by sharpening the class-wise distributions.
4. Reconstruction error using top 40% of PCA components where the components are obtained via class conditional PCA of the training data. This boosts the detection of high class-wise epistemic uncertainty by eliminating irrelevant features.
This fusion of attributes from existing state-of-the-art detection methods and new attributes was found to be the most effective integrated appraoch capable of detecting the different types of OODs. We evaluated it on several benchmarks as discussed in Section 4 with ablation study in Appendix.

heading: 4 EXPERIMENTAL RESULTS

text:
Attributes forming the signature of the OOD detector used in the experiments The signature of the OOD detector used in the experiments is the weighted sum of four attributes, one from each of the following four categories:
1. Distance from the in-distribution density estimate: We use mahalanobis distance of the input with respect to the closest class conditional distribution. The parameters of this distance are chosen from one of the following two categories:
• empirical class means and tied empirical covariance of training samples • empirical class means and empirical class covariance of training samples
2. Reconstruction error: We perform class conditional PCA empirically from the training samples. We use minimum reconstruction error of the input from the top 40% eigen vectors of the class conditional eigen spaces.
3. Prediction confidence of the classifier: We use maximum value of the temperature scaled softmax scores (S) on the perturbed input. Perturbations to the input (x) are made according to the following equation (Liang et al., 2017)
x̃ = x− sign(−∇xlogSŷ(x;T )) (1) The values of the magnitude of noise ( ) and the temperature scaling parameter (T ) are chosen from one of the following three categories:
• = 0 and T = 0 • = 0 and T = 10 • = 0.005 and T = 10
4. Conformance measure among the nearest neighbors: We compute an m-dimensional feature vector to capture the conformance among the input’s nearest neighbors in the training samples, where m is the dimension of the input. We call this m-dimensional feature vector as the conformance vector. The conformance vector is calculated by taking the mean deviation along each dimension of the nearest neighbors from the input. We hypothesize that this deviation for the in-distribution samples would vary from the OODs due to aleatory uncertainty.
The value of the conformance measure is calculated by computing mahalanobis distance of the input’s conformance vector to the closest class conformance distribution. Similar to the distance for the in-distribution density estimate, the parameters of this mahalanobis distance are chosen from the following two categories:
• empirical class means and tied empirical covariance on the conformance vectors of the training samples
• empirical class means and empirical class covariance on the conformance vectors of the training samples
The value of the number of the nearest neighbors is chosen from the set {10, 20, 30, 40, 50} via validation. We used Annoy (Approximate Nearest Neighbors Oh Yeah) (Bernhardsson, 2018) to compute the nearest neighbors.
The weights of the four attributes forming the signature of the OOD detector are generated in the following manner. We use a small subset (1000 samples) of both the in-distribution and the generated OOD data to train a binary classifier using the logistic loss. The OOD data used to train the classifier is generated by perturbing the in-distribution data using the Fast Gradient Sign attack (FGSM) (Goodfellow et al., 2014). The trained classifier (or OOD detector) is then evaluated on the real OOD dataset at the True Positive Rate of 95%. The best result, in terms of the highest TNR on the validation dataset (from the training phase of the OOD detector), from the twelve combinations of the aforementioned sub-categories (one from each of the four attributes) are then reported on the test (or real) OOD datasets.
Datasets and metrics. We evaluate the proposed integrated OOD detection on benchmarks such as CIFAR10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011). We consider standard metrics (Hendrycks & Gimpel, 2016; Liang et al., 2017; Lee et al., 2018) such as the true negative rate (TNR) at 95% true positive rate (TPR), the area under the receiver operating characteristic curve (AUROC), area under precision recall curve (AUPR), and the detection accuracy (DTACC) to evaluate our performance.
DNN-based classifier architectures. To demonstrate that the proposed approach generalizes across various network architectures, we consider a wide range of DNN models such as , ResNet (He et al., 2016), WideResNet (Zagoruyko & Komodakis, 2016), and DenseNet (Huang et al., 2017).
Comparison with the state-of-the-art. We compare our approach with the three state-of-the-art approaches: SPB (Hendrycks & Gimpel, 2016), ODIN (Liang et al., 2017), and Mahalanobis (Lee et al., 2018). For the ODIN method, the perturbation noise is chosen from the set {0, 0.0005, 0.001, 0.0014, 0.002, 0.0024, 0.005, 0.01, 0.05, 0.1, 0.2}, and the temperature T is chosen from the set {1, 10, 100, 1000}. These values are chosen from the validation set of the adversarial samples of the in-distribution data generated by the FGSM attack. For the Mahalanobis method, we consider their best results obtained after feature ensemble and input preprocessing with the hyperparameters of their OOD detector tuned on the in-distribution and adversarial samples generated by the FGSM attack. The magnitude of the noise used in pre-processing of the inputs is chosen from the set {0.0, 0.01, 0.005, 0.002, 0.0014, 0.001, 0.0005}. CIFAR10. With CIFAR10 as in-distribution, we consider SVHN (Netzer et al., 2011), TinyImagenet (Deng et al., 2009), and LSUN (Yu et al., 2015) as the OOD datasets. For CIFAR10, we consider two DNNs: ResNet50, and WideResNet. Table 1 shows the results.
SVHN. With SVHN as in-distribution, we consider CIFAR10, Imagenet, and LSUN and as the OOD datasets. For SVHN, we use the DenseNet classifier. Table 1 shows the results.
Key observations. We do not consider pre-processing of the inputs in our integrated OOD detector. Even without input pre-processing and with the exception of CIFAR10 OOD dataset for SVHN indistribution trained on DenseNet, we could perform equally well (and even out-perform in most of the cases) as the Mahalanobis method on its best results generated after pre-processing the input.
We also consider a Subset-CIFAR100 as OODs for CIFAR10. Specifically, from the CIFAR100 classes, we select sea, road, bee, and butterfly as OODs which are visually similar to the ship, automobile, and bird classes in the CIFAR10, respectively. Thus, there can be numerous OOD samples due to aleatoric and class-conditional epistemic uncertainty which makes the OOD detection challenging. Figure 5 shows the t-SNE (Maaten & Hinton, 2008) plot of the penultimate features from
the ResNet50 model trained on CIFAR10. We show 4 examples of OODs (2 due to epistemic and 2 due to aleatoric uncertainty) from Subset-CIFAR100. These OODs were detected by our integrated approach but missed by the Mahalanobis approach.
These observations justify the effectiveness of integrating multiple attributes to detect OOD samples.
Additional experimental results in the appendix. We also compare the performance of the integrated OOD detector with the SPB, ODIN and Mahalanobis detector in supervised settings, as reported by the Mahalanobis method for OOD detection (Lee et al., 2018). These results include experiments on CIFAR10, SVHN and MNIST as in-distribution data and Imagenet,LSUN, SVHN (for CIFAR10), CIFAR10 (for SVHN), KMNIST, and F-MNISTas OOD data across different DNN architectures such as ResNet34, WideResNet,DenseNet, and LeNet5. All these results, along with the ablation studies on OOD detectors with single attributes are included in the Appendix. In almost all of the reported results in the Appendix, our OOD detector could outperform the compared state-of-the-art methods with improvements of even 2X higher TNR at 95% TPR in some cases.

heading: 5 DISCUSSION AND FUTURE WORK

text:
Recent techniques propose refinement in the training process of the classifiers for OOD detection. Some of these techniques include fine-tuning the classifier’s training with an auxiliary cost function for OOD detection (Hendrycks et al., 2019a; Liu et al., 2020). Other techniques make use of selfsupervised models for OOD detection (Tack et al., 2020; Hendrycks et al., 2019b). We perform preliminary experiments to compare the performance of these techniques with our integrated OOD detector that makes use of the feature space of the pre-trained classifiers to distinguish in-distribution samples from OODs. Our approach does not require modification of the training cost function of the original task. These results are reported in the Appendix. We consider making use of the feature space of in our OOD detection technique as a promising prospective future work. Another direction of the future work is to explore the score functions used in these refined training processes for OOD detection (Liu et al., 2020; Hendrycks et al., 2019a; Tack et al., 2020; Hendrycks et al., 2019b) as attributes (or categories of attributes) forming the signature of the integrated OOD detector. Another
avenue of future work is to explore OOD generation techniques other than adversarial examples generated by the FGSM attack for training of the integrated OOD detector.

heading: 6 CONCLUSION

text: We introduced a taxonomy of OODs and proposed an integrated approach to detect different types of OODs. Our taxonomy classifies OOD on the nature of their uncertainty and we demonstrated that no single state-of-the-art approach detects all these OOD types. Motivated by this observation, we formulated an integrated approach that fuses multiple attributes to target different types of OODs. We have performed extensive experiments on a synthetic dataset and several benchmark datasets (e.g., MNIST, CIFAR10, SVHN). Our experiments show that our approach can accurately detect various types of OODs coming from a wide range of OOD datasets such as KMNIST, FashionMNIST, SVHN, LSUN, and Imagenet. We have shown that our approach generalizes over multiple DNN architectures and performs robustly when the OOD samples are similar to in-distribution data.

heading: A APPENDIX

heading: A.1 DEFINING OODS DUE TO EPISTEMIC AND ALEATORIC UNCERTAINTY

text: In general, let there be k classes c1, c2, . . . , ck and the distribution of training data for each class is p(x|ci). The overall training distribution is denoted by p(x). Now, given a new input x̂ to the trained DNN model M , let ĉ =M(x̂) denote the predicted class. The flowchart in Figure 6 shows different sources of uncertainty that could make x̂ an OOD.

heading: A.2 ADDITIONAL EXPERIMENTAL RESULTS

text: We first present preliminary results for comparison with the OOD detection techniques based on fine-tuning of the classifiers (Hendrycks et al., 2019a; Liu et al., 2020; Tack et al., 2020; Hendrycks et al., 2019b). We then present our results on various vision datasets and different architectures of the pre-trained DNN based classifiers for these datasets in comparison to the ODIN, the Mahalanobis and the SPB methods in supervised settings. Finally, we then report results from the ablation study on OOD detection with individual attributes and compare it with our integrated approach on OOD detection.

heading: A.2.1 COMPARISON WITH THE OOD DETECTION TECHNIQUES BASED ON REFINEMENT OF THE TRAINING PROCESS FOR CLASSIFIERS

text:
Recent techniques propose refinement in the training process of the classifiers for OOD detection. Some of these techniques include fine-tuning the training of classifiers with a trainable cost function for OOD detection (Hendrycks et al., 2019a; Liu et al., 2020), self-supervised training of the classifiers to enhance OOD detection (Tack et al., 2020; Hendrycks et al., 2019b) etc.
We perform preliminary experiments to compare the performance of these techniques with our integrated OOD detector that uses features of the pre-trained classifiers to distinguish in-distribution samples from OODs. Table 2 compares TNR (at 95% TPR), AUROC and AUPR for the energy based OOD detector (Liu et al., 2020) and our integrated OOD detector on CIFAR10 with pretrained WideResNet model. The integrated OOD detector was trained on in-distribution and adversarial samples generated by FGSM attack. Table 3 compares the results of the WideResNet model trained on CIFAR10 and fine-tuned with the outlier exposure from the 80 Million Tiny Images with our OOD detector that uses features from the pre-trained WideResNet model trained on CIFAR10. Since 80 Million Tiny Images dataset is no longer available for use, we used a small subset of ImageNet (treated as OOD dataset for CIFAR10 and SVHN datasets (Lee et al., 2018)) for generating OODs for training of the integrated OOD detector. Table 4 compares the OOD detection performance of the self-supervised training based OOD detector with our method. We trained our OOD detector with the in-distribution CIFAR10 as in-distribution samples and adversarial samples generated by FGSM attack from the test dataset of CIFAR10 as OODs. The trained OOD detector was then tested on LSUN as OODs and the results are reported in Table 4. With ResNet-50 as the classifier for CIFAR10, we trained our OOD detector with the in-distribution CIFAR10 as in-distribution samples and adversarial samples generated by FGSM from the test dataset of CIFAR10 as OODs. The trained OOD detector was then tested on SVHN as OODs and these results are compared with the contrastive based learning for OOD detection (Tack et al., 2020) in table 5.

heading: A.2.2 COMPARISON WITH THE STATE-OF-THE-ART OOD DETECTION METHODS IN SUPERVISED SETTINGS ON PRE-TRAINED CLASSIFIERS

text:
We compare our results with the state-of-the-art methods in supervised settings, as reported by the Mahalanobis method for OOD detection (Lee et al., 2018). In supervised settings, a small subset of the real OOD dataset is used in the training of the OOD detector.
Datasets and metrics. We evaluate the proposed integrated OOD detection on benchmarks such as MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky et al., 2009), and SVHN (Netzer et al., 2011). We consider standard metrics (Hendrycks & Gimpel, 2016; Liang et al., 2017; Lee et al., 2018) such as the true negative rate (TNR) at 95% true positive rate (TPR), the area under the receiver operating characteristic curve (AUROC), area under precision recall curve (AUPR) with both in-distribution and OODs as positive samples (AUPR IN and AUPR OUT respectively), and the detection accuracy (DTACC) to evaluate our performance.
DNN-based classifier architectures. To demonstrate that the proposed approach generalizes across various network architectures, we consider a wide range of DNN models such as Lenet (LeCun et al., 1998) , ResNet (He et al., 2016) , and DenseNet (Huang et al., 2017).
Comparison with the state-of-the-art. We compare our approach with the three state-of-the-art approaches: SPB (Hendrycks & Gimpel, 2016), ODIN (Liang et al., 2017) and Mahalanobis (Lee et al., 2018). Since, these experiments are performed in supervised settings, we fix T = 10 and = 0.005 for generating results from the ODIN method. For Mahalanobis distance, we consider the distance in the penultimate layer feature space as well as features from all the layers of the DNN without preprocessing of the input in either settings.
MNIST. With MNIST as in-distribution, we consider KMNIST (Clanuwat et al., 2018) and FashionMNIST(F-MNIST) (Xiao et al., 2017) as OOD datasets. For MNIST, we use the LeNet5 (LeCun et al., 1998) DNN. Results in terms of TNR (at 95% TPR), AUROC, and DTACC are reported in tables 6, 7, and 8. Table 6 shows the results with the features from the penultimate layer in comparison to the ODIN and Mahalanobis methods. Table 7 shows the results with the features from all the layers in comparison to the Mahalanobis method. Table 8 shows the results with the features from the penultimate layer in comparison to the SPB method. In all these settings, our approach outperforms the state-of-the-art approaches for both the OOD datasets. Results in comparison to AUPR IN and AUPR OUT are shown in table 9. Here also, our technique out-performs all the three OOD detectors on all the test cases.
CIFAR10. With CIFAR10 as in-distribution, we consider STL10 (Coates et al., 2011), SVHN (Netzer et al., 2011), Imagenet (Deng et al., 2009), LSUN (Yu et al., 2015), and a subset of CIFAR100 (SCIFAR100) (Krizhevsky et al., 2009) as OOD datasets. For CIFAR10, we consider three DNNs: DenseNet, ResNet34, and ResNet50. Results in terms of TNR (at 95% TPR), AUROC, and DTACC are reported in tables 6, 7, and 8. Table 6 shows the results with the features from the penultimate layer in comparison to the ODIN and Mahalanobis methods. Table 7 shows the results with the features from all the layers in comparison to the Mahalanobis method. Table 8 shows the results with the features from the penultimate layer in comparison to the SPB method. Results in comparison to AUPR IN and AUPR OUT are shown in tables 10, 11, and 12. Here also, the integrated OOD detection technique could out-perform the other three detectors on most of the test cases.
Note that images from STL10 and the subset of CIFAR100 are quite similar to CIFAR10 images. Furthermore, from the CIFAR100 classes, we select sea, road, bee, and butterfly as OODs which are visually similar to the ship, automobile, and bird classes in the CIFAR10, respectively.
SVHN. With SVHN as in-distribution, we consider STL10, CIFAR10, Imagenet, LSUN and, SCIFAR100 as OOD datasets. For SVHN, we consider two DNNs: DenseNet and ResNet34. Results in terms of TNR (at 95% TPR), AUROC, and DTACC are reported in tables 6, 7, and 8. Table 6 shows the results with the features from the penultimate layer in comparison to the ODIN and Mahalanobis methods. Table 7 shows the results with the features from all the layers in comparison to the Mahalanobis method. Table 8 shows the results with the features from the penultimate layer in comparison to the SPB method. Results in comparison to AUPR IN and AUPR OUT are shown in tables 13, and 14. Here also, the integrated OOD detection technique could out-perform the other three detectors on most of the test cases.
Key observations. As shown in Table 6 and Table 7, and Table 8, our approach outperforms the state-of-the-art on all three datasets and with various DNN architectures. On CIFAR10, in terms of the TNR metric, our approach with Resnet50 outperforms Mahalanobis by 56% when SVHN is OOD and our approach with Resnet34 outperforms ODIN by 36% when LSUN is OOD.
While considering STL10 and Subset-CIFAR100 as OODs for CIFAR10, the images from both these datasets are quite similar to CIFAR-10 images. Thus, there can be numerous OOD samples due to aleatoric and class-conditional epistemic uncertainty which makes detection challenging. Although our performance is low on the STL10 dataset, it still outperforms the state-of-the-art. For instance, the proposed approach achieves a 27% better TNR score than the Mahalanobis using ResNet50. On SVHN, in terms of the TNR metric, our approach outperforms ODIN and Mahalanobis by 63% and 13%, respectively on SCIFAR100 using ResNet34. The above observations justify the effectiveness of integrating multiple attributes to detect OOD samples.

heading: A.2.3 ABLATION STUDY

text:
We report ablation study on OOD detection with individual attributes and compare it with our integrated approach on the penultimate feature space of the classifier in the supervised settings as described in the previous section. We call the OOD detector with Mahalanobis distance estimated on class mean and tied covariance (Lee et al., 2018) as Mahala-Tied. Detector based on Mahalanobis distance estimated on class mean and class covariance is referred as Mahala-Class. Similarly conformance among the K-nearest neighbors (KNN) measured by Mahala-Tied and Mahala-Class is referred as KNN-Tied and KNN-Class respectively in these experiments. Results for this study on CIFAR10 with DenseNet architecture, SVHN with DenseNet and ResNet34 architectures are shown in Tables 15, 16 and 17 respectively.
The integrated approach could out-perform all the single attribute based OOD detector in all the tested cases due to detection of diverse OODs. An important observation made from these experiments is that the performance of the single attribute based methods could depend on the architecture of the classifier. For example, while the performance of PCA was really bad in case of DenseNet (for both CIFAR10 and SVHN) as compared to all other methods, it could out-perform all but the integrated approach for SVHN on ResNet34.


summaries: The paper claims that high quality of generated samples and SOTA bpds are achievable by VAEs if the model is deep enough (deep in terms of the number of stochastic layers). The authors explain the architecture that resemblances the U-net architecture, and explain its building blocks. Interestingly, they are able to learn VAEs with up to 78 stochastic layers, and achieve SOTA bpds on CIFAR-10, ImageNet-32, ImageNet-64, FFHQ-256 (5-bit), and setting a great result on FFHQ-1024 (8bit).

summaries: the paper puts forward an idea that deep-enough VAE should perform at least as well as autoregressive models. Authors explore this in the context of image generation, and construct VAE model that is a generalisation of typical autoregressive architectures. They use several tricks to ensure stable training of very deep VAEs and show that final performance exceeds all autoregressive models. This experimentally supports their claim that very deep VAEs encompass autoregressive models.

summaries: This paper shows that deep hierarchical VAEs can outperform state-of-the-art autoregressive models on images. The authors first argue that autoregressive models are special cases of hierarchical VAEs and that hierarchical VAEs are universal approximators. They introduce a simple top-down (LVAE) architecture that scales past 70 layers. Furthermore, the model can be trained without using freebits or KL annealing -- although additional tricks are required (gradient skipping and prior warmup). They demonstrate that likelihood performance is correlated with depth and report state-of-the-art performances on multiple image datasets.

abstractText: We present a hierarchical VAE that, for the first time, generates samples quickly and outperforms the PixelCNN in log-likelihood on all natural image benchmarks. We begin by observing that, in theory, VAEs can actually represent autoregressive models, as well as faster, better models if they exist, when made sufficiently deep. Despite this, autoregressive models have historically outperformed VAEs in loglikelihood. We test if insufficient depth explains why by scaling a VAE to greater stochastic depth than previously explored and evaluating it CIFAR-10, ImageNet, and FFHQ. In comparison to the PixelCNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied to high-resolution images. Qualitative studies suggest this is because the VAE learns efficient hierarchical visual representations. We release our source code and models at https://github.com/openai/vdvae. Low resolution High resolution Figure 1: Selected samples from our very deep VAE on FFHQ-256, and a demonstration of the learned generative process. VAEs can learn to first generate global features at low resolution, then fill in local details in parallel at higher resolutions. When made sufficiently deep, this learned, parallel, multiscale generative procedure attains a higher log-likelihood than the PixelCNN.

authors:
Tom B Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell

title: Language models are few-shot learners

authors:
Mark Chen
Alec Radford
Rewon Child
Jeff Wu
Heewoo Jun
Prafulla Dhariwal
David Luan
Ilya Sutskever

title: Generative pretraining from pixels

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
Xi Chen
Diederik P Kingma
Tim Salimans
Yan Duan
Prafulla Dhariwal
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Variational lossy autoencoder

venue: arXiv preprint arXiv:1611.02731,

authors:
Xi Chen
Nikhil Mishra
Mostafa Rohaninejad
Pieter Abbeel

title: Pixelsnail: An improved autoregressive generative model

venue: arXiv preprint arXiv:1712.09763,

authors:
Rewon Child
Scott Gray
Alec Radford
Ilya Sutskever

title: Generating long sequences with sparse transformers

venue: arXiv preprint arXiv:1904.10509,

authors:
Bin Dai
David Wipf

title: Diagnosing and enhancing vae models

venue: arXiv preprint arXiv:1903.05789,

authors:
Peter Dayan
Geoffrey E Hinton
Radford M Neal
Richard S Zemel

title: The helmholtz machine

venue: Neural computation,

authors:
Prafulla Dhariwal
Heewoo Jun
Christine Payne
Jong Wook Kim
Alec Radford
Ilya Sutskever

title: Jukebox: A generative model for music

venue: arXiv preprint arXiv:2005.00341,

authors:
Laurent Dinh
David Krueger
Yoshua Bengio

title: Nice: Non-linear independent components estimation

venue: arXiv preprint arXiv:1410.8516,

authors:
Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio

title: Density estimation using real nvp

venue: arXiv preprint arXiv:1605.08803,

authors:
Ishaan Gulrajani
Kundan Kumar
Faruk Ahmed
Adrien Ali Taiga
Francesco Visin
David Vazquez
Aaron Courville

title: Pixelvae: A latent variable model for natural images

venue: arXiv preprint arXiv:1611.05013,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Dan Hendrycks
Kevin Gimpel

title: Gaussian error linear units (gelus)

venue: arXiv preprint arXiv:1606.08415,

authors:
Jonathan Ho
Xi Chen
Aravind Srinivas
Yan Duan
Pieter Abbeel

title: Flow++: Improving flowbased generative models with variational dequantization and architecture design

authors:
Jonathan Ho
Ajay Jain
Pieter Abbeel

title: Denoising diffusion probabilistic models

venue: arXiv preprint arxiv:2006.11239,

authors:
Chin-Wei Huang
Ahmed Touati
Laurent Dinh
Michal Drozdzal
Mohammad Havaei
Laurent Charlin
Aaron Courville

title: Learnable explicit density for continuous latent space and variational inference

venue: arXiv preprint arXiv:1710.02248,

authors:
Chin-Wei Huang
David Krueger
Alexandre Lacoste
Aaron Courville

title: Neural autoregressive flows

venue: arXiv preprint arXiv:1804.00779,

authors:
Diederik P Kingma
Max Welling

title: Stochastic gradient vb and the variational auto-encoder

venue: In Second International Conference on Learning Representations, ICLR,

title: An introduction to variational autoencoders

venue: arXiv preprint arXiv:1906.02691,

authors:
Durk P Kingma
Prafulla Dhariwal

title: Glow: Generative flow with invertible 1x1 convolutions

authors:
Durk P Kingma
Tim Salimans
Rafal Jozefowicz
Xi Chen
Ilya Sutskever
Max Welling

title: Improved variational inference with inverse autoregressive flow

authors:
Alexander Kolesnikov
Christoph H Lampert

title: Pixelcnn models with auxiliary variables for natural image modeling

venue: In International Conference on Machine Learning,

authors:
Lars Maaløe
Marco Fraccaro
Valentin Liévin
Ole Winther

title: Biva: A very deep hierarchy of latent variables for generative modeling

authors:
Jacob Menick
Nal Kalchbrenner

title: Generating high fidelity images with subscale pixel networks and multidimensional upscaling

venue: arXiv preprint arXiv:1812.01608,

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
George Papamakarios
Eric Nalisnick
Danilo Jimenez Rezende
Shakir Mohamed
Balaji Lakshminarayanan

title: Normalizing flows for probabilistic modeling and inference

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI Blog,

authors:
Scott Reed
Aäron van den Oord
Nal Kalchbrenner
Sergio Gómez Colmenarejo
Ziyu Wang
Dan Belov
Nando De Freitas

title: Parallel multiscale autoregressive density estimation

venue: arXiv preprint arXiv:1703.03664,

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

title: Stochastic backpropagation and approximate inference in deep generative models

venue: arXiv preprint arXiv:1401.4082,

authors:
Tim Salimans
Durk P Kingma

title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks

authors:
Tim Salimans
Andrej Karpathy
Xi Chen
Diederik P Kingma

title: Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications

venue: arXiv preprint arXiv:1701.05517,

authors:
Casper Kaae Sønderby
Tapani Raiko
Lars Maaløe
Søren Kaae Sønderby
Ole Winther

title: Ladder variational autoencoders

authors:
Emma Strubell
Ananya Ganesh
Andrew McCallum

title: Energy and policy considerations for deep learning in nlp

venue: arXiv preprint arXiv:1906.02243,

authors:
Benigno Uria
Iain Murray
Hugo Larochelle

title: Rnade: The real-valued neural autoregressive density-estimator

authors:
Arash Vahdat
Jan Kautz

title: Nvae: A deep hierarchical variational autoencoder

venue: arXiv preprint arXiv:2007.03898,

authors:
Aaron Van den Oord
Nal Kalchbrenner
Lasse Espeholt
Oriol Vinyals
Alex Graves

title: Conditional image generation with pixelcnn decoders

authors:
Hongyi Zhang
Yann N Dauphin
Tengyu Ma

title: Fixup initialization: Residual learning without normalization

venue: arXiv preprint arXiv:1901.09321,

authors:
Shengjia Zhao
Jiaming Song
Stefano Ermon

title: Learning hierarchical features from generative models

venue: arXiv preprint arXiv:1702.08396,

heading: 1 INTRODUCTION

text:
One potential path to increased data-efficiency, generalization, and robustness of machine learning methods is to train generative models. These models can learn useful representations without human supervision by learning to create examples of the data itself. Many types of generative models have flourished in recent years, including likelihood-based generative models, which include autoregressive models (Uria et al., 2013), variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), and invertible flows (Dinh et al., 2014; 2016). Their objective, the negative log-likelihood, is equivalent to the KL divergence between the data distribution and the model distribution. A wide variety of models can be compared and assessed along this criteria, which corresponds to how well they fit the data in an information-theoretic sense.
Starting with the PixelCNN (Van den Oord et al., 2016), autoregressive models have long achieved the highest log-likelihoods across many modalities, despite counterintuitive modeling assumptions. For example, although natural images are observations of latent scenes, autoregressive models learn dependencies solely between observed variables. That process can require complex function approximators that integrate long-range dependencies (Oord et al., 2016; Child et al., 2019). In contrast, VAEs and invertible flows incorporate latent variables and can thus, in principle, learn a simpler model that mirrors how images are actually generated. Despite this theoretical advantage, on the landmark ImageNet density estimation benchmark, the Gated PixelCNN still achieves higher likelihoods than all flows and VAEs, corresponding to a better fit with the data.
Is the autoregressive modeling assumption actually a better inductive bias for images, or can VAEs, sufficiently improved, outperform autoregressive models? The answer has significant practical stakes, because large, compute-intensive autoregressive models (Strubell et al., 2019) are increasingly used for a variety of applications (Oord et al., 2016; Brown et al., 2020; Dhariwal et al., 2020; Chen et al., 2020). Unlike autoregressive models, latent variable models only need to learn dependencies between latent and observed variables; such models can not only support faster synthesis and higher-dimensional data, but may also do so using smaller, less powerful architectures.
We start this work with a simple but (to the best of our knowledge) unstated observation: hierarchical VAEs should be able to at least match autoregressive models, because autoregressive models are equivalent to VAEs with a powerful prior and restricted approximate posterior (which merely outputs observed variables). In the worst case, VAEs should be able to replicate the functionality of autoregressive models; in the best case, they should be able to learn better latent representations, possibly with much fewer layers, if such representations exist.
We formalize this observation in Section 3, showing it is only true for VAEs with more stochastic layers than previous work has explored. Then we experimentally test it on competitive natural image benchmarks. Our contributions are the following:
• We provide theoretical justification for why greater depth (up to the data dimension D, but also as low as some value K D) could improve VAE performance (Section 3)
• We introduce an architecture capable of scaling past 70 layers, when previous work explored at most 30 (Section 4)
• We verify that depth, independent of model capacity, improves log-likelihood, and allows VAEs to outperform the PixelCNN on all benchmarks (Section 5.1)
• Compared to the PixelCNN, we show the model also uses fewer parameters, generates samples thousands of times more quickly, and can be scaled to larger images. We show evidence these qualities may emerge from the model learning an efficient hierarchical representation of images (Section 5.2)
• We release code and models at https://github.com/openai/vdvae.

heading: 2 PRELIMINARIES

text: We review prior work and introduce some of the basic terminology used in the field.

heading: 2.1 VARIATIONAL AUTOENCODERS

text:
Variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) consist of a generator pθ(x|z), a prior pθ(z), and an approximate posterior qφ(z|x). Neural networks φ and θ are trained end-to-end with backpropagation and the reparameterization trick in order to maximize the evidence lower bound (ELBO):
log pθ(x) ≥ Ez∼qφ(z|x) log pθ(x|z)−DKL[qφ(z|x)||pθ(z)] (1)
See Kingma & Welling (2019) for an in-depth introduction. There are many choices for what networks are used for pθ(x|z), qφ(z|x), and whether pθ(z) is also learned or set to a simple distribution.
We study VAEs with independent pθ(x|z) – that is, where each observed xi is output without conditioning on any other xj . This ensures generation time does not increase linearly with the dimensionality of the data, and requires that these VAEs learn to incorporate the complexity of the data into a rich distribution over latent variables z. It is possible to have autoregressive pθ(x|z) (Gulrajani et al., 2016), but generation is slow for these models. They also sometimes ignore latent variables entirely, becoming equivalent to normal autoregressive models (Chen et al. (2016)).

heading: 2.2 HIERARCHICAL VARIATIONAL AUTOENCODERS

text:
Much of the early work on VAEs incorporate fully-factorized Gaussian qφ(z|x) and pθ(z). This can lead to poor outcomes if the latent variables required for good generation take on a more complex distribution, as is common with independent pθ(x|z). One of the simplest methods of gaining greater expressivity in both distributions is to use a hierarchical VAE, which has several stochastic layers of latent variables. These variables are emitted in groups z0, z1, ...,zN , which are conditionally dependent upon each other in some way. For images, latent variables are typically output in feature maps of varying resolutions, with z0 corresponding to a small number of latent variables at low resolution at the “top” of the network, and zN corresponding to a larger number of latent variables at high resolution at the “bottom”.
One particularly elegant conditioning structure is the top-down VAE, introduced in Sønderby et al. (2016). In this model, both the prior and the approximate posterior generate latent variables in the same order:
pθ(z) = pθ(z0)pθ(z1|z0)...pθ(zN |z<N ) (2) qφ(z|x) = qφ(z0|x)qφ(z1|z0,x)...qφ(zN |z<N ,x) (3)
A diagram of this process appears in Figure 3. A typical implementation of this model has φ first perform a deterministic “bottom-up” pass on the data to generate features, then processes the groups
of latent variables from top to bottom, using feedforward networks to generate features which are shared between the approximate posterior, prior, and reconstruction network pθ(x|z). We adopt this base architecture as it is simple, empirically effective, and has been postulated to resemble biological processes of perception (Dayan et al., 1995).

heading: 3 WHY DEPTH MATTERS FOR HIERARCHICAL VAES

text:
We find that hierarchical VAEs with sufficient depth can not only learn arbitrary orderings over observed variables, but also learn more effective latent variable distributions, if such distributions exist. We present these results below. Definition (N -layer VAE). A deep hierarchical VAE with N stochastic layers, independent p(x|z), and the top-down factorization of the prior and approximate posterior in Equations 2-3. Proposition 1. N-layer VAEs generalize autoregressive models when N is the data dimension Proposition 2. N -layer VAEs are universal approximators of N -dimensional latent densities
Proposition 1 (proof in Appendix, also visualized in Figure 2, left) leads to a possible explanation of why autoregressive models to date have outperformed VAEs: they are deeper, in the sense of statistical dependence. A VAE must be as deep as the data dimension D (3072 layers in the case of 32x32 images) if the images truly require D steps to generate.
Luckily, however, Proposition 2 (proof and further technical requirements in Appendix) suggests that shorter procedures, if they exist, are also learnable. N = D is an extreme case, where the most effective latent variables z ∈ RD may simply be copies of the observed variables. But if for some K < D there exist latent variables z ∈ RK that the generator can use to more efficiently compress the data, Proposition 2 states a K-layer VAE can learn the posterior and prior distribution over those variables.
Such shorter generative paths could emerge in two ways. First, as depicted in Figure 2 (right), if the model discovers that certain variables are conditionally independent given others, the model can generate them in parallel inside a single layer, where qφ(zN |z<N ,x) = ∏ d qφ(z (d) N |z<N ,x). We hypothesize these efficient hierarchies should emerge in images, as they contain many spatially independent textures, and study this in Section 5.2. Second, the model could learn a low-dimensional representation of the data. Dai & Wipf (2019) recently showed that when a VAE is trained on data distributed on a K-dimensional manifold embedded in RD, a VAE will only activate K dimensions in its latent space, meaning that the VAE will require fewer layers unless the manifold dimension is D, which is unlikely to be the case for images.
It is difficult to ascertain the lowest possible value of K for a given dataset, but it may be deeper than most hierarchical VAEs to date. Images have many thousands of observed variables, but early hierarchical VAEs did not exceed 3 layers, until Maaløe et al. (2019) investigated a Gaussian VAE with 15 layers and found it displayed impressive performance along a variety of measures. Kingma et al. (2016) and Vahdat & Kautz (2020) additionally explored networks up to 12 and 30 layers. (These additionally incorporated additional statistical dependencies in the approximate posterior through the usage of inverse autoregressive flow (Kingma et al., 2016), an alternative approach which we contrast with our approach in Section A.4). Nevertheless, given these results we hypothesize that greater depth may improve the performance of VAEs. In the next section, we introduce an architecture capable of scaling to a greater number of stochastic layers. In Section 5.1 we show depth indeed improves performance.

heading: 4 AN ARCHITECTURE FOR VERY DEEP VAES

text: We consider a “very deep” VAE to simply be one with greater depth than has previously been explored (and do not define it to be a specific number of layers). As existing implementations of VAEs did not support many more stochastic layers than they were trained on, we reimplemented a minimal VAE with the sole aim of increasing the number of stochastic layers. This VAE consists only of convolutions, nonlinearities, and Gaussian stochastic layers. It does not exhibit posterior collapse even for large numbers of stochastic layers. We describe key architectural choices here and refer readers to our source code for more details.

heading: 4.1 ARCHITECTURAL COMPONENTS AND INITIALIZATION

text:
A diagram of our network appears in Figure 3. It resembles the ResNet VAE in Kingma et al. (2016), but with bottleneck residual blocks. For each stochastic layer, the prior and posterior are diagonal Gaussian distributions, as used in prior work (Maaløe et al., 2019).
As an alternative to weight normalization and data-dependent initialization (Salimans & Kingma, 2016), we adopt the default PyTorch weight intialization. The one exception is the final convolutional layer in each residual bottleneck block, which we scale by 1√
N , where N is the depth (similar
to Radford et al. (2019); Child et al. (2019); Zhang et al. (2019)). This residual scaling improves stability and performance with many layers, as we show in the Appendix (Table 3).
Additionally, we use nearest-neighbor upsampling for our “unpool” layer, which when paired with our ResNet architecture, allows us to completely remove the “free bits” and KL “warming up” terms that appear in related work. As we detail in the Appendix (Figure 5), when upsampling is done through transposed convolutional layer, the network may ignore layers at low resolution (for instance, 1x1 or 4x4 layers). We found no evidence of posterior collapse in any networks trained with nearest neighbor interpolation.

heading: 4.2 STABILIZING TRAINING WITH GRADIENT SKIPPING

text:
VAEs have notorious “optimization difficulties,” which are not frequently discussed in the literature but nevertheless well-known by practitioners. These manifest as extremely high reconstruction or KL losses and corresponding large gradient norms (up to 1e15). We address this by skipping updates with a gradient norm above a certain threshold, set by hyperparameter. Though we select high thresholds that affect fewer than 0.01% of updates, this technique almost entirely eliminates divergence, and allows networks to train smoothly. We plot the evolution of grad norms and the
values we select in (Figure 6). An alternative approach to stabilizing networks may be the spectral regularization method introduced in Vahdat & Kautz (2020).

heading: 5 EXPERIMENTS

text: We trained very deep VAEs on challenging natural image datasets. All hyperparameters for experiments are available in the Appendix and in our source code.

heading: 5.1 STATISTICAL DEPTH, INDEPENDENT OF CAPACITY, IMPROVES PERFORMANCE

text:
We first tested whether greater statistical depth, independent of other factors, can result in improved performance. We trained a network with 48 layers for 600k steps on ImageNet-32, grouping layers to output variables independently instead of conditioning on each other. If the input for the ith topdown block is xi, we can make K consecutive blocks independent by setting xi+1, ..., xi+K all equal to xi. (Normally, xi+1 = xi + f(block(xi))). This technique reduces the stochastic depth without affecting parameter count. Stochastic depth shows a clear correlation with performance, even up to 48 layers, which is past what previous work has explored (Table 1, left).
We then tested our hypothesis at scale. We trained networks on CIFAR-10, ImageNet-32, and ImageNet-64 with greater numbers of stochastic layers, but with fewer parameters than related work (see Table 2). On CIFAR-10, we trained a model with 45 stochastic layers and only 39M parameters, and found it achieved a test log-likelihood of 2.87 bits per dim (average of 4 seeds). On ImageNet32 and ImageNet-64, we trained networks with 78 and 75 stochastic layers and only approximately 120M parameters, and achieved likelihoods of 3.80 and 3.52.
On all tasks, these results outperform all GatedPixelCNN/PixelCNN++ models, and all nonautoregressive models, while using similar or fewer parameters. These results support our hypothesis that stochastic depth, as opposed to other factors, explains the gap between VAEs and autoregressive models.

heading: 5.2 VERY DEEP VAES LEARN AN EFFICIENT HIERARCHICAL ORDERING

text:
One question that emerges from the analysis in Section 3 is whether VAEs need to be as deep as autoregressive models, or whether they can learn a latent hierarchy of conditionally independent variables which are able to be synthesized in parallel. We qualitatively show this is true in Figure 4. For FFHQ-256 images, the first several layers at low resolution almost wholly determine the global features of the image, even though they only account for less than 1% of the latent variables. The rest of the high-resolution variables appear to be spatially independent, meaning they can be emitted in parallel in a number of layers much lower than the dimensionality of the image. This efficient hierarchical representation may underlie the VAE’s ability to achieve better log-likelihoods than the PixelCNN while simultaneously sampling thousands of times faster. This can be viewed as a learned parallel multiscale generation method, unlike the handcrafted approaches of Kolesnikov & Lampert (2017); Menick & Kalchbrenner (2018); Reed et al. (2017).
Additionally, we found that on all datasets we tested, very deep VAEs used roughly 30% fewer parameters than the PixelCNN (Table 2). One possible explanation is that the learned hierarchical generation procedure involves fewer long-range dependencies, or may otherwise be simpler to learn.
We found that networks in general benefited from more layers at higher resolutions (Table 1, right). This suggests that global features may account for a smaller fraction of information than local details and textures, and that it is important to have many latent variables at high resolution.

heading: 5.2.1 VERY DEEP VAES ARE EASILY SCALED TO HIGH DIMENSIONAL DATA

text:
Scaling autoregressive models to higher resolutions presents several challenges. First, the sampling time and memory requirements of autoregressive models increase linearly with resolution. This scaling makes datasets like FFHQ-256 and FFHQ-1024 intractable for naive approaches. Although clever factorization techniques have been adopted for 256x256 images (Menick & Kalchbrenner, 2018), such factorizations may not be as effective for alternate datasets or higher-resolution images.
Our VAE, in contrast, readily scales to higher resolutions. The same network used for 32x32 images can be applied to 1024x1024 images by introducing a greater number of upsampling layers throughout the network. We found we could train an equal number of steps (1.5M) using a similar number of training resources (32 GPUs for 2.5 weeks) on both 32x32 and 1024x1024 images with few hyperparameter changes (see Appendix for hyperparameters). Samples from both models (displayed in Appendix) require a single forward pass of the model to generate, with only minor differences in runtime. An autoregressive model, on the other hand, would require a thousand times more network evaluations to sample 1024x1024 images and likely require a custom training procedure.

heading: 6 RELATED WORK AND DISCUSSION

text:
Our work is inspired by previous and concurrent work in hierarchical VAEs (Sønderby et al., 2016; Maaløe et al., 2019; Vahdat & Kautz, 2020). Relative to these works, we provide some justification for why deeper networks may perform better, introduce a new architecture, and empirically demonstrate gains in log-likelihood. Many aspects of prior work are complementary with ours and could be combined. Maaløe et al. (2019), for instance, incorporates a “bottom-up” stochastic path that doubles the depth of the approximate posterior, and Vahdat & Kautz (2020) introduces a number of powerful architecture components and improved training techniques. We seek here not to introduce a significantly better method than these alternatives, but to demonstrate that depth is a key overlooked factor in most prior approaches to VAEs.
Diffusion models can be seen as deep VAEs that, like autoregressive models, have a specific analytical posterior. Ho et al. (2020) showed that such models achieve impressive sample quality with great depth, which is in line with our observations that greater depth is helpful for VAEs. One benefit of the VAEs we outline in this work over diffusion models is that our VAEs generate samples with a single network evaluation, whereas diffusion models currently require a large number of network evaluations per sample.
Inverse autoregressive flows (IAF) are also closely related, and we discuss the differences with hierarchical models in Section A.4. The work of Zhao et al. (2017) may also appear to contradict our findings, and we discuss that work in Section A.5.

heading: 7 CONCLUSION

text: We argue deeper VAEs should perform better, introduce a deeper architecture, and show it outperforms all PixelCNN-based autoregressive models in likelihood while being more efficient. We hope this encourages work in further improving VAEs and latent variable models.

heading: ACKNOWLEDGMENTS

text: We thank Aditya Ramesh, Pranav Shyam, Johannes Otterbach, Heewoo Jun, Mark Chen, Prafulla Dhariwal, Alec Radford, Yura Burda, Bowen Baker, Raul Puri, and Ilya Sutskever for helpful discussions. We also thank the anonymous reviewers for helping improve our work.

heading: A APPENDIX

text:
A.1 ABLATIONS OF ARCHITECTURAL COMPONENTS
First, we visualize data that suggests upsampling layers and residual connections have an impact on posterior collapse (Figure 5). Architectural differences may explain why our VAEs do not need “free bits” or KL warmups to avoid posterior collapse.
In Table 3, we show residual initialization leads to smoother and better training of very deep VAEs. Without residual initialization, very deep VAEs encounter a high number of unstable updates and have higher losses.
In Figure 6, we show the max gradient norms experienced throughout training, and show that our skipping criterion avoids a small number of updates that would destabilize the network.
A.2 PROPOSITION 1: N-LAYER VAES GENERALIZE AUTOREGRESSIVE MODELS WHEN N IS THE DATA DIMENSION
Proposition 1 shows that an autoregressive model with an arbitrary ordering over observed variables in x ∈ RN is equivalent to an N -layer VAE with an approximate posterior that simply outputs the observed variables in the given order, and a generator that performs the identity function (see Figure 2).
Without loss of generality, we simplify notation by assuming each vector-valued latent variable zi only has one element, which we write as zi ∈ R. We assume a prior and approximate posterior distribution following Equation 2 and 3.
Proof. Let q(zi = xi|z<i,x) = 1, and p(xi = zi|z) = 1. Then p(z|x) = q(z|x), which is well-known to imply equality in the evidence lower bound (ELBO) of Eq. 1. Since log q(z|x) = log p(x|z) = 0, the ELBO becomes log pθ(x) = log pθ(z) = ∑N i=1 log pθ(zi|z<i) =∑N
i=1 log pθ(xi|x<i), which is equivalent to an autoregressive model over the observed variables.
A.3 PROPOSITION 2: N -LAYER VAES ARE UNIVERSAL APPROXIMATORS OF N -DIMENSIONAL LATENT DENSITIES
Proposition 2 shows that hierarchical VAEs learn depthwise autoregressive flows, and under certain conditions (described in Huang et al. (2017)) can express any density over latent variables of N dimensions, given enough capacity.
Proof. We omit full proof, and refer readers to Huang et al. (2017); Papamakarios et al. (2019), where universality is established for autoregressive flows. Here we only note that the prior and approximate posterior in an N -layer VAE are autoregressive flows: Let pθ(z) be the prior distribution. pθ(z) can be written using the reparameterization trick as a deterministic function of noise drawn from a known base density pN : pθ(z) = pN ( )
∣∣∣det∂f( ,θ)∂ ∣∣∣, where f is a neural network that implements the factorization in Eq. 2. Since f is autoregressive and its Jacobian is lower triangular, pθ(z) can approximate any p(z) that fits the criteria in Huang et al. (2017). The same logic applies to qφ(z|x) and p(z|x). It should be noted that this result depends on f being able to implement the inverse CDF of an arbitrary probability density, and so using Gaussian distributions will restrict the densities the VAE can express in practice. This is a limitation of our architecture that we nevertheless adopt since we hypothesize depth, not the elementwise density, is the more important factor. More discussion on this subject, and options for removing this restriction, are described in Huang et al. (2017) and Huang et al. (2018), and we defer studying more expressive elementwise densities to future work.
A.4 A NOTE ON INVERSE AUTOREGRESSIVE FLOW
Inverse autoregressive flows (IAF, Kingma et al. (2016)) and are similar to very deep VAEs in that they are universal approximators of posterior distributions in VAEs, even with just a single layer and sufficiently expressive univariate density (Huang et al., 2018).
There are several practical differences between IAFs and deep hierarchical VAEs, however, which can result in qualitatively very different behavior. First, the masked autoregressive components in IAF build statistical dependencies spatially, whereas a very deep hierarchical VAE builds dependencies depthwise, and these inductive biases may better suit different domains. Additionally, IAFs spend an equal amount of computation and parameters on each variable. In contrast, a deep VAE can specify a structure, like a hierarchy of global-to-local variables, which have different computational and modeling capacities for each stage. For images, these differences may result in qualitatively different behavior, and it is not clear whether a single layer IAF can readily learn the sort of rich hierarchical decomposition of images that appear with very deep VAEs.
Nevertheless, the two techniques are complementary – IAF was introduced in a deep hierarchical VAE (Kingma et al., 2016), in fact, and it is likely that introducing IAF into our architecture (as in Vahdat & Kautz (2020)) would improve performance.
A.5 A NOTE ON LEARNING HIERARCHICAL FEATURES
The work of Zhao et al. (2017) may appear to contradict our work, by suggesting that additional layers in hierarchical VAEs do not lead to additional expressivity, based off their finding that Gibbs sampling from the last stochastic layer is sufficient to recover the data. For high dimensional data like images, however, the last stochastic layer may have many thousands of variables, and Gibbs sampling may take unacceptably long to converge. A hierarchy of latent variables as in our model allows efficient and tractable sampling from this distribution. Additionally, assumptions regarding global maximization of the ELBO may not apply in practice. Nevertheless, we think further clarifying these contradictory statements would be useful future work.
A.6 BROADER IMPACT
Broadly speaking, any generative model will reflect the biases of the datasets they are trained on. If deployed without careful consideration, generative models (including but not limited to VAEs) trained on research datasets like ImageNet, CIFAR-10, and FFHQ may inadvertently cause harm by propagating or otherwise reinforcing harmful biases in the dataset. Further work is required to improve and debias research benchmark datasets to mitigate this source of negative impact.
Some VAEs are distinguished from other generative models by their fast synthesis of new data examples. Generative models with fast synthesis can allow for realtime synthesis of high dimensional data, such as music, speech, and video. These models could be used to augment human creativity and lead to a number of helpful applications in real-time media applications. Such models could also be used for compression, which could assist in delivering content to bandwidth-constrained regions of the world. They can also be used for spreading disinformation, generally making it less possible to distinguish real from generated data. An additional potential harm is that fast, high quality synthesis of data could end up economically displacing individuals who rely upon creative work, such as musicians, visual artists, and more.
VAEs also are distinguished by their usage of latent variables. Generative models with useful latent variables could have positive impacts in scientific domains, where density estimation could lead to novel insights about chemical, physical, or biological data. Latent variable representations of data could also be helpful in efforts to debias, interpret, or otherwise increase understandibility of models and their representations.