summaries: The work takes inspiration from a recent work on phased LSTM, and proposes to add a Gaussian gate based on time to LSTM cells. With this additional gate, the network can skip updating the states by closing the time-gate, as a result enabling longer memory persistence, and better gradient flow. The authors also propose to add a budget term to force the time-gate to be closed most of the time as a way to save compute. Empirical results suggest the Gaussian-gated LSTMs perform better than regular LSTMs on tasks with long temporal dependencies. The authors also propose to use a curriculum training schedule in which the variance of the gaussian gates is continuously increased to speed up training of LSTMS.

summaries: This paper focuses on the reduction of training time by various mechanisms. By introducing a time gate during training, it controls when a neuron (weights?) can be updated during training. By introducing and additional budget term in the loss function, training costs (number of computations) are reduced by one order of magnitude.

abstractText: Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, thereby allowing the network to preserve memory over long temporal intervals. We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates. The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient flow. This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization. Because the time gate limits the updates of the neuron state, the number of computes needed for the network update is also reduced. By adding a computational budget term to the training loss, we obtain a network which further reduces the number of computes by at least 10×. Finally, we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences.

authors:
Victor Campos
Brendan Jou
Xavier Giro i Nieto
Jordi Torres
Shih-Fu Chang

title: Skip RNN: learning to skip state updates in recurrent neural networks

authors:
Shiyu Chang
Yang Zhang
Wei Han
Mo Yu
Xiaoxiao Guo
Wei Tan
Xiaodong Cui
Michael J. Witbrock
Mark Hasegawa-Johnson
Thomas S. Huang

title: Dilated recurrent neural networks

authors:
Kyunghyun Cho
Bart van Merriënboer
Çalar Gülçehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using RNN Encoder–Decoder for statistical machine translation

venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Tim Cooijmans
Nicolas Ballas
Cesar Laurent
Aaron C. Courville

title: Recurrent batch normalization

authors:
Xavier Glorot
Yoshua Bengio

title: Understanding the difficulty of training deep feedforward neural networks

venue: In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural Comput.,

authors:
Rafal Jozefowicz
Wojciech Zaremba
Ilya Sutskever

title: An empirical exploration of recurrent network architectures

venue: In International Conference on Machine Learning,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Alex Krizhevsky
Vinod Nair
Geoffrey Hinton

title: The CIFAR-10 dataset

venue: online: http://www. cs. toronto. edu/kriz/cifar. html,

authors:
David Krueger
Tegan Maharaj
János Kramár
Mohammad Pezeshki
Nicolas Ballas
Nan Rosemary Ke
Anirudh Goyal
Yoshua Bengio
Aaron Courville
Chris Pal

title: Zoneout: Regularizing RNNs by randomly preserving hidden activations

venue: arXiv preprint arXiv:1606.01305,

authors:
Yann LeCun
Leon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Shuai Li
Wanqing Li
Chris Cook
Ce Zhu
Yanbo Gao

title: Independently recurrent neural network (IndRNN): Building a longer and deeper RNN

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

title: A critical review of recurrent neural networks for sequence learning

authors:
Daniel Neil
Michael Pfeiffer
Shih-Chii Liu

title: Phased LSTM: Accelerating recurrent network training for long or event-based sequences

venue: In Advances in Neural Information Processing Systems,

title: Deep learning in neural networks: An overview

venue: Neural Networks,

title: Distribution-specific hardness of learning neural networks

authors:
Corentin Tallec
Yann Ollivier

title: Can recurrent neural networks warp time

venue: In Proceedings of International Conference on Learning Representation,

authors:
Tijmen Tieleman
Geoffrey Hinton

title: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning

authors:
Trieu H. Trinh
Andrew M. Dai
Thang Luong
Quoc V. Le

title: Learning longer-term dependencies in RNNs with auxiliary losses

authors:
Li Wan
Matthew Zeiler
Sixin Zhang
Yann Le Cun
Rob Fergus

title: Regularization of neural networks using DropConnect

authors:
Adams Wei Yu
Hongrae Lee
Quoc V. Le

title: Learning to skim

venue: text. CoRR,

heading: 1 INTRODUCTION

text:
Numerous methods and architectures have been proposed to mitigate the vanishing gradient problem in RNNs, with LSTMs (Hochreiter & Schmidhuber, 1997) as one of the first prominent solutions doing so by including gating structures in the computation. Although the LSTM has excelled at handling many tasks (Schmidhuber, 2015; Lipton, 2015), it still has difficulties in learning complex and long time dependencies (Neil et al., 2016; Chang et al., 2017; Trinh et al., 2018).
In the last few years, various methods which reduce the state updates of an RNN (LSTM) have been explored to better learn long time dependencies from data. Clockwork RNNs (Koutnik et al., 2014) group the hidden units of the RNN into “modules,” where each module is executed at pre-specified time steps thereby skipping time steps which helps learn longer time dependencies. Recently, various other methods have been proposed which can be characterized by the use of additional “time gates,” kn, that control the information flow from one time step to the next (Krueger et al., 2016; Campos et al., 2017). Phased LSTM (PLSTM) (Neil et al., 2016) learns a parameterized function, kn, from the the time input of the current state and was proven to be successful at learning over long sequences.
The PLSTM time gate, parameterized by period, phase, and ratio parameters for each hidden unit, is defined through a modulo function with an ill-defined gradient. Furthermore, with periodic functions being hard to learn using gradient-based methods (Shamir, 2016) and with kn being periodic, the PLSTM was unable to learn the time gate parameters and hence relied on careful initialization. In order to offset these difficulties, this work proposes a new LSTM variant called the Gaussian-gated LSTM (g-LSTM). Similar to the PLSTM, it is an LSTM model with a parameterized kn but with only two parameters per hidden unit. Unlike the PLSTM which uses a periodic formulation for kn, the g-LSTM uses a Gaussian function.
We show in this work that the g-LSTM can provide a number of possible advantages over the LSTM, in particular, on long sequence tasks that pose convergence problems during training:
• The g-LSTM network can process very long sequences by reducing the time over which the neurons can be updated. It converges faster than the LSTM, especially on sequences that are over 500 steps.
• The “openness” of the neuron for an update can be adapted according to the task during training, even for extreme, non-optimal initializations of the time gate parameters.
• By introducing a computational budget term into the loss function during training, the “openness” of the neuron can be optimized for a reduced computational budget. This reduction can be achieved with little or no degradation to the network performance and is useful for network pruning.
• A “temporal curriculum” training schedule can be set up for the g-LSTM so that it helps to speed up the convergence of a normal LSTM.
The paper is structured as follows: In section 2, we discuss briefly the related work. Then, in section 3, we present the formulation of the g-LSTM, the datasets used in this work and details about the experimental hyperparameters. In section 4, we present experiments demonstrating the usefulness of the g-LSTM with respect to the four claims listed above. We provide gradient analysis in section 5 to further explain the faster convergence results of the g-LSTM. Finally in section 6, we conclude with a brief discussion of the results.

heading: 2 RELATED WORK

text:
There have been a multitude of proposed methods to improve the training of RNNs, especially for long sequences. Apart from incorporating additional gating structures, for example the LSTM and the GRU (Cho et al., 2014), more recently various techniques were proposed to further increase the capabilities of recurrent networks to learn on sequences of length over 1000. Proposed initialization techniques such as the orthogonal initialization of kernel matrices (Cooijmans et al., 2016), chrono initialization of the biases (Tallec & Ollivier, 2018), and diagonal recurrent kernel matrices (e.g. Li et al. (2018)) have demonstrated success. Trinh et al. (2018) propose using truncated backpropagation with an additional auxiliary loss to reconstruct previous events.
Methods that enable more efficient learning on long temporal sequences use solutions that preserve memory over longer timescales. Such solutions were first explored by Koutnik et al. (2014) in the Clockwork RNN (CW-RNN). This network skips state updates by allowing different neurons to be “activated” on different, modulated clock cycles. More recently proposed models for skipping updates include the Phased LSTM (PLSTM) (Neil et al., 2016) which uses a modulo-periodic timing gate to limit state updates; the Zoneout network (Krueger et al., 2016) which skips state updates in a random manner; and the Skip RNN (Campos et al., 2017) which learns a state skipping scheme from the data to shorten the effective sequence length for the task. Additionally, the LSTM-Jump (Yu et al., 2017) uses a reinforcement learning algorithm to learn when to skip state updates, showing a method to more quickly process (long) sequential data with an RNN while maintaining an accuracy comparable to a baseline LSTM.
It has been suggested but not yet demonstrated in the literature that the parameters of the CW-RNN clock cycle and PLSTM timing gate could be learned in training. Currently, the implementation of these networks requires a careful initialization of these parameters. With the Gaussian-gated LSTM (g-LSTM) in this work we present a time gated RNN network that converges on long sequence tasks and also has the ability to learn its time gate parameters even when initialized in a nonoptimal way.

heading: 3 METHODS

heading: 3.1 G-LSTM

text:
The g-LSTM is an LSTM model with an additional time gate (Fig. 1). This time gate is used to regulate the information flow in time. Equations 1 - 3 describe the update equations for the hidden and cell states of the LSTM. Equations 4 and 5 describe the gating mechanism of the time gate, kn.
in = σ(xnWxi + hn−1Whi + bi), fn = σ(xnWxf + hn−1Whf + bf ) (1) c̃n = fn cn−1 + in σ(xnWxg + hn−1Whg + bg) (2)
on = σ(xnWxo + hn−1Who + bo), h̃n = on tanh(c̃n) (3) cn = kn c̃n + (1− kn) cn−1 (4) hn = kn h̃n + (1− kn) hn−1 (5)
In a standard LSTM, the gating functions in, fn, on, represent the input, forget, and output gates respectively at sequence index n. cn is the cell activation vector, and xn and hn represent the input feature vector and the hidden output vector, respectively. The cell state cn is updated with a fraction of the previous cell state that is controlled by fn, and a new input state created from the element-wise (Hadamard) product, denoted by , of in and the candidate cell activation as in Eq. 2. In the g-LSTM, we further control the cell state and the output hidden state through the kn gate which is independent of the input data and hidden states, and is purely dependent on the time input corresponding to the sequence index n. The use of the Hadamard product ensures that each hidden unit is independently controlled by the corresponding time gate unit, thus enabling the different units in the layer to process the input at different time scales.
The time gate kn is defined based on a Gaussian function as: kn = e−(tn−µ) 2/σ2where the mean parameter, µ, defines the time when the hidden unit is “open” and the standard deviation, σ, controls the openness of the time gate for each unit around its corresponding µ. The time inputs t = {t1, t2, ..., tn, ..., tN} for the sequence x = {x1, x2, ..., xn, ..., xN} can correspond to the physical notion of time at the respective sequence input. In the absence of a standard notion of time, we use the sequence indices as the time input, i.e. t = {1, 2, ..., n, ..., N}. In this work, we assume this notion of time by default. The “openness” of kn for a neuron is defined by the parameterization of its Gaussian function.

heading: 3.2 BACK PROPAGATION FOR G-LSTM

text:
An important characteristic of the g-LSTM is reduced gradient flow in back propagation training methods. By having the gating structure as in Eqs. 4 and 5 there are fewer gradient product terms, which reduces the likelihood of vanishing or exploding gradients. In a gradient descent learning scheme for a given loss function, L, when training the recurrent parameters, Wh (from Eqs. 1 - 3), the gradient as in Eq. 6 is used.
∂L
∂Wh =
∂L
∂hN ∂hN ∂Wh
(6)
By the chain rule ∂hN∂Wh expands for all time steps of the sequence, n ∈ {1, ..., N}. Because each output state is gated by the time gate, kn, the gradient terms in the expansion of ∂hN∂Wh are scaled by kn. When the time gate is open less often, i.e. with a small σ value, then there are fewer influential gradient terms. More details are given in Appendix A.

heading: 3.3 DATASETS

text:
The experiments described in the paper are carried out on the adding task and two standard long sequence datasets: the sequential MNIST and the sequential CIFAR-10 datasets.
Adding task: In order to test the long sequence learning capability of the g-LSTM, we use the adding task (Hochreiter & Schmidhuber, 1997). In this task, the network is presented with two sequences of length N , x = {x1, ..., xN}, xn ∼ U(0, 1)) and m = {m1, ...,mN},mn ∈ {0, 1},∑Nt=1mn = 2. The sequence m has exactly two values of 1 and the remaining values of the sequence are 0. The indices of the “1” values are chosen at random. For each pair in the sequence (x,m), the associated label value, y, is the sum of the two values in x corresponding to the “1” values of m. The objective of this task is to minimize the mean squared error between the predicted sum from the network, ŷ, and the labeled sum, y. A new training set of 5000 sequence samples is presented in every epoch during training in order to avoid overfitting. The test set consists of a separate 5000 samples. For N > 1000, it is known that LSTMs have difficulty learning the task and hence we focus on values of N > 1000 in this work.
sMNIST: The sequential MNIST dataset is widely used to analyze the performance of a recurrent model. This dataset consists of 60,000 training samples and 10,000 test samples, each a single vector sequence of length 784 corresponding to the 28 × 28 pixel images in the MNIST dataset (LeCun et al., 1998). We also use permuted MNIST (pMNIST), a permuted variant of the sMNIST dataset where the sequences are processed with a fixed random permutation, making the task harder.
sCIFAR-10: The sequential CIFAR-10 dataset is another long sequence dataset based on CIFAR10 (Krizhevsky et al., 2014) with 10 classes. The 32 × 32 RGB pixel images are reshaped into sequences of length 1024 with 3 dimensional features corresponding the RGB channels at every time step. Like in the sMNIST dataset, the dataset consists of 60,000 training samples and 10,000 test samples.

heading: 3.4 EXPERIMENTAL HYPERPARAMETERS

text: For the adding task, a mean squared error (MSE) loss was used with the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10−3. The g-LSTM time gate parameters were trained using a learning rate of 100. For both sMNIST and sCIFAR-10 datasets, the cross-entropy loss function was used along with the RMSProp optimizer (Tieleman & Hinton, 2012) with a learning rate of 10−3. Decay parameters of 0.5 and 0.9 were used for sMNIST and sCIFAR-10, respectively. The bias of the forget gate is initialized to 1 following (Jozefowicz et al., 2015).

heading: 4 RESULTS

text: Section 4.1 presents results that demonstrate the faster convergence properties of the g-LSTM on long sequence tasks. Section 4.2 shows the trainability of the time gate parameters of the g-LSTM even when the parameters are initialized in a non-optimal way. Section 4.3 presents a modified loss function used during training to reduce the number of computes for the network update and Section 4.4 presents a new “temporal curriculum” learning schedule that allows g-LSTMs to help LSTMs converge faster.

heading: 4.1 FAST CONVERGENCE PROPERTIES OF G-LSTM

text:
First, we look at the convergence properties of the g-LSTM on the long-sequence adding task, the sMNIST task and the sCIFAR-10 task. Table 1, above, details the network architectures used in the experiments in this section. Similar to the architecture from Trinh et al. (2018), the recurrent layer of the sCIFAR-10 network is followed by two 256 unit fully-connected (FC) layers, where DropConnect (Wan et al. (2013)) (p = 0.5) is applied to the second FC layer. The kernel matrices in the LSTM networks were initialized in an orthogonal manner as described in (Cooijmans et al., 2016).
The test performances of these networks during the course of the training on different datasets are shown in Fig. 2, while the corresponding final performance metrics at the end of training are shown in Table 1. From Fig. 2, it is evident that the test loss of the g-LSTM decreases faster in training than the LSTM across all datasets. Further experiments show that this trend is maintained with different training optimizers, LSTM initializations including the bias initialization following Tallec & Ollivier (2018), and network sizes as shown in Appendix D.
Table 2 compares the performance of various networks including the g-LSTM and the baseline LSTM on sMNIST and sCIFAR-10 (from Table 1). The results show that the g-LSTM consistently performs better than the LSTM and has a similar performance to other state-of-the-art networks. Different network sizes were also investigated for the sMNIST task, see Appendix D.

heading: 4.2 TRAINABILITY OF THE TIME GATE PARAMETERS OF G-LSTM

text:
To demonstrate that the g-LSTM can be trained even with non-optimal initializations, we look at the performance of the g-LSTM on the adding task with different time gate parameter initializations. We concern ourselves with sequences of length 1000 that are difficult for the LSTM. The time gate parameters are initialized in a way to temporally constrain the network so that it can only process for a short period of time. For example, a network with time gate parameters initialized with µ = 500 and σ = 40 as in Figure 3 (a) can only process a short period of time around the middle of sequence. It follows that the network would be unable to learn with these parameters because in the adding task the input data is distributed equally across the sequence length (T = 1000). Therefore, in order to learn the task from this initialization, the time gate parameters must learn a distribution such that the gates over all hidden units are open across the entirety of the sequence.
We observe that the time gate parameters do learn, as shown in Figure 3 (b), thereby enabling the network to solve the task. Independent of various time gate initializations, the network reaches an MSE of around 3.9×10−5 at the end of 700 epochs; details of which could be found in Appendix B. The ability of the network to learn the time gate parameters necessary to cover the entire sequence is especially significant because it shows that even with this narrow time window initialization that requires learning of the time gates, the g-LSTM learns the task, whereas the PLSTM does not learn the task as well. An example of this is shown in Appendix C.

heading: 4.3 REDUCTION IN COMPUTATION

text:
Although the formulation of the g-LSTM appears to require more computes, it offers substantial speedup as a large proportion of the neurons can be skipped in a timestep at runtime. We can set a threshold on the time gate so that we skip all corresponding computations for time steps where kn is below this threshold. To further reduce the number of operations, it is preferred that the σ of the kn for different neurons should be small but the network performance should not be significantly degraded. To achieve this goal, we included a “computational budget” loss term during the optimization of the gate parameters, µ and σ. The loss equation for updating the kn parameters is given by:
L = Ldata + λLbudget.
Similar to the Skip RNN network (Campos et al., 2017), a budget loss term which minimizes the average openness of the time gate over time is applied:
Lbudget = E[kn] ≈ N∑ n=1 J∑ j=1 k(j)n
for every neuron j of the g-LSTM. The study was carried out on sMNIST using a network with 110 units, σ initialized to 50, µ initialized uniformly at random between 1 to 784, and a λ value of 1.
The network’s performance of 2.2% LER was comparable to the network’s performance of 1.3% when no additional budget constraint was imposed. The final σ range for the budgeted g-LSTM is much smaller compared to that of the g-LSTM as shown in Fig. 4. There is only a slight increase in LER for the budgeted g-LSTM versus the g-LSTM (see Table 2), even though there is a significant decrease in the average time gate openness across all hidden units.
In order to reduce the number of computes, we set a threshold, vT for kn so that the update steps are carried out only if kn > vT , if kn < vT the previous neuron state can be copied over to the current state. By increasing vT , the number of computes decreases as shown in Fig. 5. In the case of vT = 0.01, only 8.2% of the time gates are open on average across all hidden units and all time steps. Furthermore, the LER increased only slightly to 2.3% from 2.2%.
We give a quantitative estimate for the number of operations (Ops) corresponding to the number of update equations for a g-LSTM. In the estimate, we count a multiply and an add operation as 1 Op and non-linear functions as 5 Ops. For an LSTM, the number of operations is given by
NLSTM = T H(8D + 8H + 29)
where T is the number of time steps, H is the number of hidden units, and D is the dimension of the input data. For a g-LSTM, the number of operations is given by
Ng−LSTM = NLSTM +Ngate
where Ngate = 13T H is the total number of operations for computing the time gate. The total number of operations for the g-LSTM network on the sMNIST dataset is around 80 MOps for N = 110 and T = 784, after thresholding the budgeted g-LSTM this number is reduced to 7.6 MOps. Additional λ hyperparameters were also investigated for the sMNIST task, see Appendix D.

heading: 4.4 TEMPORAL CURRICULUM TRAINING SCHEDULE FOR LSTMS

text:
We demonstrate that it is possible to train an LSTM network to converge faster on a difficult task by using a “temporal curriculum” training schedule for the equivalent g-LSTM network. According to this schedule, the initial σ values of the g-LSTM network are increased continuously throughout the
training period ending up with high values by the end of training. With such high values, the time gates are essentially open, resulting in an LSTM network. At every training epoch, the lowest ρ% of the σ values, σ̂ in the layer are updated as: σ̂ −→ (1 + α) · σ̂. We analyze the impact of this training schedule for training an LSTM network on sCIFAR-10. For the equivalent g-LSTM network with 110 units, µ is initialized uniformly at random between 1 and 1024 and σ is initialized to 50. An α value of 1/6 and ρ value of 15% are chosen. To ensure that the time gate is fully open by the end of training, σ is set to 5000 across all units during the last 10 epochs of training. The learning rate of the time gate parameters is set to 0, i.e. µ and σ are no longer updated. Figure 6 shows that the temporal curriculum training schedule allows for faster convergence of an LSTM network. The final weights of the trained g-LSTM network can then be copied over to a LSTM network for inference.

heading: 5 GRADIENT FLOW

text:
We present results regarding backpropagation flow through the LSTM and g-LSTM networks. Following the hypothesis presented in Section 3.2 on the reduced likelihood of vanishing or exploding gradients in the g-LSTM, we investigate the average gradient norms across time steps, similar to the work in (Krueger et al., 2016). We compute the gradient norms of the loss with respect to the hidden activations, the exact definition is given in Appendix E. Comparing the error propagation of the g-LSTM and LSTM networks on the SMNIST task (as in Sec. 4.1), Figure 7 shows the gradient norms at each time step after training for two different epochs.
Interpreting the gradient flow from higher to lower time steps (right to left), the gradients of the g-LSTM shown in Fig. 7 show higher gradient values in earlier time steps than the LSTM. It is possible that one of the reasons the g-LSTM converges more quickly (as in Fig. 2 (c)) is that this back-propagated gradient information is more consistent across time steps and does not vanish at early time steps.

heading: 6 CONCLUSION

text: This work proposes a novel RNN variant with a time gate which is parameterized by the input in time. The convergence speeds of the g-LSTM and LSTM are similar for short sequence tasks but the g-LSTM shows faster convergence and produces higher accuracies than LSTM networks on long sequence tasks, as demonstrated for adding task sequences which are longer than 1000 timesteps; and on the sMNIST and sCIFAR-10 datasets. We also demonstrate that the time gate parameters of the g-LSTM (unlike those of the PLSTM) are learnable even when the time gates are initialized in an extreme non-optimal manner for the adding task. The time gate of the g-LSTM can reduce the number of computes that is needed for the updates of the LSTM equations and with an additional loss term to reduce the compute budget, the σ values of the time gate are reduced leading to a 10× decrease in the number of actual computes and with little loss in network accuracy, for the sMNIST dataset. The observation that the budgeted g-LSTM has neurons which are closed by the timing gate suggests that this method can be used to prune a network. We also show that our proposed temporal curriculum training schedule for the g-LSTM can help a corresponding LSTM network to converge during training on long sequence tasks. For future work, it will be of interest to investigate whether these properties carry over to larger or domain-specific datasets.

heading: A BACK PROPAGATION IN GAUSSIAN-GATED RNN

text:
For ease of illustration we analyze the gradient of a plain RNN with a Gaussian time gate (Eqs. 7 and 8).
hn = kn · h̃n + (1− kn) · hn−1 (7)
h̃n = f(Wxxn +Whhn−1) (8)
∂hN ∂Wh = ∂h̃0 ∂Wh N∏ n=1 (knWhf ′ n + (1− kn)) + N∑ n=1 (knf ′ nhn−1) N∏ s=n+1 (ksWhf ′ s + (1− ks)) (9)
where ∂h̃0∂Wh = 1, h0 = h̃0 = 0.
From Eq. 9 we can deduce some information about the advantages of the Gaussian time gate in gradient flow for two simple cases of the function kn.
In Case 1 we choose a timing gate openness which corresponds to a very small σ for the Gaussian gate, i.e. the gate is only open for 1 time step.
k5 = 1, kn = 0 ∀ n ∈ {1, ..., N}\{5} ∂hN ∂Wh = f ′5 + f ′ 5f ′ 5h4 = f ′ 5Wh (1 + f ′ 5h4)
In Case 2 we choose a timing gate openness which corresponds to a slightly larger σ for the Gaussian gate, i.e. it is open for 5 time steps.
k2 = 1, k3 = 1, k4 = 1, k5 = 1, k6 = 1, kn = 0 ∀ n ∈ {1, ..., N}\{2, 3, 4, 5, 6}
∂hN ∂Wh = f ′2Whf ′ 3Whf ′ 4Whf ′ 5Whf ′ 6Wh (1 + f ′ 2h1 + f ′ 3h2 + f ′ 4h3 + f ′ 5h4 + f ′ 6h5)
These cases show that there are fewer terms in the gradient for a timing gate that is opened for only a small fraction of the sequence.

heading: B COMPARING VARIOUS G-LSTM INITIALIZATIONS

heading: C COMPARING TIME GATE PARAMETER TRAINABILITY IN G-LSTM AND

heading: D HYPERPARAMETER INVESTIGATION

text:
We look at the network performance for different hyperparameter values, focusing on the sMNIST task.
Network Initialization and Optimizer In Fig. 2(c) of Section 4.1, we show that the g-LSTM network converges faster than the LSTM for the sMNIST task using the RMSProp optimizer and with an orthogonal initialization of LSTM kernels of both networks, as in (Cooijmans et al., 2016). In addition to using this initializer and optimizer we include results using the ADAM initializer (learning rate of 10−3) and a random weight initialization, “Xavier” as in (Glorot & Bengio, 2010). Across all of these different training techniques we consistently observe that the g-LSTM converges more quickly than the LSTM.
We ran further experiments to compare the chrono initialization of the LSTM forget and input biases from (Tallec & Ollivier, 2018). The forget and input biases are set as bf ∼ log(U([1, Tmax − 1])) and bi = −bf where Tmax = 784 for the sMNIST task. The use of the time gate with the g-LSTM shortens the effective sequence length for each unit; to account for this, we also provide the results of using a smaller Tmax value for the chrono initialization, Tmax = σ = 250. The comparison of both g-LSTM and LSTM with the chrono initialization and with the “constant initialization” (bf = 1) in Fig. 12 shows that the g-LSTM with the constant initialization converges the fastest. We hypothesize that the g-LSTM can converge faster when using the constant initialization over the chrono initialization because the time gate’s effect of sequence-length-shortening reduces the necessity for long memory, for which chrono initialization seeks to provide. We see that when we reduce the maximum temporal dependency for the chrono initialization (to Tmax = 250, “chrono-gLSTM-250”) this g-LSTM network converges more quickly, similar to the g-LSTM with a constant bias initialization. This suggests that these two techniques, chrono initialization and a Gaussian time gate, could be used together to improve convergence in LSTM networks.
Network Size Aside from the network size of 110 hidden units, we investigated the training convergence for two additional network sizes: 25 and 220 hidden units. Note that the LSTM for network size 25 is trained for 100 additional epochs until convergence was observed. Across all different network sizes the g-LSTM converges much faster than the LSTM network. With fewer hidden units,
as seen in Fig. 13 (a), an even more dramatic speed-up in convergence is seen for the g-LSTM compared with the LSTM. The final LERs (g-LSTM, LSTM) for each network size are: 25 units (2.79%, 3.58%), 110 units (1.35%, 1.81%) , 220 units (1.10%, 1.34%).
Budgeted g-LSTM We provide additional results of the budgeted network (subsection 4.3) for 2 additional λ values, λ = 0.1 and λ = 10, comparing with the original result, for λ = 1. The final LERs: 2.4% (λ = 0.1), 2.2% (λ = 1), 2.8% (λ = 10). The number of computes used by the network trained with λ = 10 is significantly lower than the network that was trained for both λ = 0.1 and 1.

heading: E AVERAGE GRADIENT NORM DEFINITION

text:
The average gradient norm in Section 5 is defined as:
Γ ∈ RN+ where N is the number of time steps of the sequence (for SMNIST, N = 784).
Γ = 1
K E ∑ (k) ∥∥∥∥ ∂L ∂h (k) n ∥∥∥∥  ≈ 1 LK ∑ (l,k) ∥∥∥∥ ∂L ∂h (k) n ∥∥∥∥ summing over all L samples of the training set and all K hidden units.

title: GAUSSIAN-GATED LSTM: IMPROVED CONVERGENCE BY REDUCING STATE UPDATES


summaries: The authors present a simple algorithm based on the statistics of neural activations of deep networks to detect out-of-distribution samples. The idea is to use the existing running estimate of mean and variance within BatchNorm layers to construct feature representations that are later fed into a simple linear classifier. The authors demonstrate superior performance over the previous state-of-art in the standard evaluation setting and provide fascinating insights and empirical analysis of their method.

summaries: There has been recent interest in using statistics and information summary measures to evaluate what deep nets are trying to do. Following the line of work, the paper suggests to use mean and variance of Z-scores accumulated across all layers/channels as features to distinguish ID and OOD samples. Simple idea but needs some work in its current format.

abstractText: The ability to detect when an input sample was not drawn from the training distribution is an important desirable property of deep neural networks. In this paper, we show that a simple ensembling of first and second order deep feature statistics can be exploited to effectively differentiate in-distribution and out-of-distribution samples. Specifically, we observe that the mean and standard deviation within feature maps differs greatly between in-distribution and out-of-distribution samples. Based on this observation, we propose a simple and efficient plug-and-play detection procedure that does not require re-training, pre-processing or changes to the model. The proposed method outperforms the state-of-the-art by a large margin in all standard benchmarking tasks, while being much simpler to implement and execute. Notably, our method improves the true negative rate from 39.6% to 95.3% when 95% of in-distribution (CIFAR-100) are correctly detected using a DenseNet and the out-of-distribution dataset is TinyImageNet resize. The source code of our method will be made publicly available.

authors:
N. Akhtar
A.S. Mian

title: Threat of adversarial attacks on deep learning in computer vision: A survey

venue: IEEE Access,

authors:
D. Amodei
C. Olah
J. Steinhardt
P. Christiano
J. Schulman
D. Mané

title: Concrete problems in AI safety, June 2016

authors:
D. Bahdanau
K. Cho
Y. Bengio

title: Neural machine translation by jointly learning to align and translate

title: Pattern recognition and machine learning

venue: Information Science and Statistics. Springer-Verlag,

authors:
J. Deng
W. Dong
R. Socher
L.-J. Li
K. Li
L. Fei-Fei

title: ImageNet: A large-scale hierarchical image database

venue: In Computer Vision and Pattern Recognition, Florida,

authors:
I. Goodfellow
J. Pouget-Abadie
M. Mirza
B. Xu
D. Warde-Farley
S. Ozair
A. Courville
Y. Bengio

title: Generative adversarial nets

venue: Advances in Neural Information Processing Systems,

authors:
G. Goswami
N.K. Ratha
A. Agarwal
R. Singh
M. Vatsa

title: Unravelling robustness of deep learning based face recognition against adversarial attacks

venue: In AAAI Conference on Artificial Intelligence,

authors:
C. Guo
G. Pleiss
Y. Sun
K.Q. Weinberger

title: On calibration of modern neural networks

authors:
K.J. Han
A. Chandrashekaran
J. Kim
I. Lane

title: The CAPIO 2017 conversational speech recognition system, December 2018

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep residual learning for image recognition

venue: arXiv preprint arXiv:1512.03385

title: Identity mappings in deep residual networks

venue: In European Conference on Computer Vision,

authors:
D. Hendrycks
K. Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: In International Conference on Learning Representations,

authors:
D. Hendrycks
M. Mazeika
T. Dietterich

title: Deep anomaly detection with outlier exposure, 2018

authors:
G.E. Hinton
L. Deng
D. Yu
G.E. Dahl
A. Mohamed
N. Jaitly
A. Senior
V. Vanhoucke
P. Nguyen
T.N. Sainath
B. Kingsbury

title: Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups

venue: IEEE Signal Processing Magazine,

authors:
G. Huang
Z. Liu
L. van der Maaten
K.Q. Weinberger

title: Densely connected convolutional networks

venue: In Computer Vision and Pattern Recognition, Hawaii,

authors:
S. Ioffe
C. Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

title: Learning multiple layers of features from tiny images

venue: Technical report,

authors:
A. Krizhevsky
I. Sutskever
G.E. Hinton

title: ImageNet classification with deep convolutional neural networks

title: Visualizing high-dimensional data using t-SNE

venue: Journal of Machine Learning Research,

authors:
K. Lee
H. Lee
J. Shin

title: Training confidence-calibrated classifiers for detecting outof-distribution samples

title: A simple unified framework for detecting out-of-distribution samples and adversarial attacks, July 2018b

authors:
S. Liang
Y. Li
R. Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

authors:
Larry M. Manevitz
Malik Yousef

title: One-class SVMs for document classification

authors:
Y. Netzer
Y. Wang
A. Coates
A. Bissacco
B. Wu
A.Y. Ng

title: Reading digits in natural images with unsupervised feature learning

venue: In Workshop on Deep Learning and Unsupervised Feature Learning,

title: An overview of gradient descent optimization algorithms, September 2016

authors:
M. Sabokrou
M. Fayyaz
M. Fathy
Z. Moayed
R. Klette

title: Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes, September 2016

authors:
J. Su
D. Vasconcellos Vargas
S. Kouichi

title: One pixel attack for fooling deep neural networks, October 2017

authors:
A. van den Oord
Y. Li
I. Babuschkin
K. Simonyan
O. Vinyals
K. Kavukcuoglu
G. van den Driessche
E. Lockhart
L.C.C. Rus
F. Stimberg
N. Casagrande
D. Grewe
S. Noury
S. Dieleman
E. Elsen
N. Kalchbrenner
H. Zen
A. Graves
H. King
T. Walters
D. Belov
D. Hassabis

title: Parallel WaveNet: Fast high-fidelity speech synthesis

venue: Technical report, Google Deepmind,

authors:
A. Vyas
N. Jammalamadaka
D. Das X. Zhu
B. Kaul
T.L. Willke

title: Out-of-distribution detection using an ensemble of self supervised leave-out classiers

authors:
P. Xu
K.A. Ehinger
Y. Zhang
A. Finkelstein
S.R. Kulkarni
J. Xiao

title: TurkerGaze: Crowdsourcing saliency with webcam based eye

authors:
F. Yu
Y. Zhang
S. Song
A. Seff
J. Xiao

title: LSUN: Construction of a large-scale image dataset using deep learning with humans

venue: in the loop,

authors:
S. Zagoruyko
N. Komodakis

title: Wide residual networks

venue: In British Machine Vision Conference,

heading: 1 INTRODUCTION

text:
In the past few years, deep neural networks (DNNs) (Goodfellow et al., 2016) have settled as the state-of-the art-techniques in many difficult tasks in a plurality of domains, such as image classification (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012; Han et al., 2018), and machine translation (Bahdanau et al., 2014; van den Oord et al., 2017). This recent progress has been mainly due to their high accuracy and good generalization ability when dealing with realworld data. Unfortunately, DNNs are also highly confident when tested against unseen samples, even if the samples are vastly different from the ones employed during training (Hendrycks & Gimpel, 2017). Moreover, several works have shown that such deep networks are easily fooled by minor perturbations to the input (Goodfellow et al., 2014; Su et al., 2017). Obtaining a calibrated confidence score from a deep neural network is a problem under continuous investigation (Hendrycks & Gimpel, 2017) and a major thread in artificial intelligence (AI) safety (Amodei et al., 2016). In fact, knowing when the model is wrong or inaccurate has a direct impact in many production systems, such as self-driving cars, authentication and disease identification (Akhtar & Mian, 2018; Goswami et al., 2018), to name a few.
Guo et al. (2017) showed that despite producing significantly low classification errors, DNNs confidence scores are not faithful estimates of the true certainty. Their experiments confirmed that depth, width, weight decay, and batch normalization are the main reasons for overconfident scores. Moreover, they demonstrated that a simple and yet powerful method of temperature scaling in the softmax scores is an effective way to improve calibrate a DNNs confidence. While calibrating the classifier’s output to represent a faithful likelihood from the training (in-distribution) data has effective solutions, the problem of detecting whether or not the samples are generated from a known distribution (out-of-distribution), is still an open problem (Hendrycks & Gimpel, 2017).
One straightforward approach to calibrate the classifier’s confidence in order to detect samples whose distribution differs from the training samples distribution is to train a secondary classifier that digests both in-distribution (ID) and out-of-distribution (OOD) data so that anomalies are scored differently from ID samples, as performed in Hendrycks et al. (2018). Re-training a network, however, can be computationally intensive and may even be intractable, since the number of OOD samples is virtually infinite. Other solutions rely on training both classification and generative neural net-
works for OOD using a multi-task loss (Lee et al., 2018a), or using unsupervised fully convolutional networks as done by Sabokrou et al. (2016) to detect OOD in video samples. All these methods, however, have a major drawback: they require re-training a modified model using a different loss function possibly with additional parameters, which increases the computational burden, and demands access to the entire original (and probably huge) training data.
In this work, we propose a novel OOD sample detection method that explores low-level statistics from feature layers. The statistics are obtained directly from the batch normalization layers (Ioffe & Szegedy, 2015), requiring no extra computations during training time, no changes to the network architecture and loss functions, nor preprocessing of the input image. During test time, the proposed method extracts statistics from selected layers and combines them into an OOD-ness score via a linear classifier. Throughout this paper, we observe that the mean and standard deviation of a given channel in a layer differ greatly between ID and OOD samples, which naturally motivates their use as features to be employed by an OOD detector. By selecting the BN layers of a network, we are able to normalize the features according to the learned batch statistics. The effectiveness of the proposed method is validated in two state-of-the-art DNN architectures (DenseNet and ResNet) (Huang et al., 2017; He et al., 2015; 2016; Zagoruyko & Komodakis, 2016) that are trained for image classification tasks in popular datasets. The proposed approach achieves state-of-the-art performance, surpassing all competitors by a large margin in all tested scenarios, while being much more efficient. Notably, our method only requires one forward pass while Liang et al. (2018); Lee et al. (2018b); Vyas et al. (2018) require two forward and one backward passes.
The rest of the paper is organized as follows. Section 2 describes prior work on OOD samples detection. Section 3 introduces the proposed method, whereas Section 4 details all experiments and compares the results with state-of-the-art methods. Finally, we draw our conclusions in Section 5.

heading: 2 PREVIOUS WORK

text:
In this section, we describe recent prior work on OOD detection methods. A summary of all methods described can be seen in Table 1.
Hendrycks & Gimpel (2017) proposed a baseline method based on the posterior distribution (i.e. softmax scores). They showed that well-trained models tend to produce higher scores for ID samples than for OOD ones. Hence their method comprises of applying a threshold on the softmaxnormalized output of a classifier. If the largest score is below the threshold, then the sample is considered OOD.
Liang et al. (2018) continued the aforementioned line of work and proposed the Out-of-Distribution detector for Neural networks (ODIN), which includes a temperature scaling T ∈ R+∗ to the softmax classifier as in Guo et al. (2017). The authors in ODIN argued that a good manipulation of T eases the separation between in- and out-of-distribution samples. Allied to that, they also incorporated small perturbations to the input (inspired by Goodfellow et al. (2014)) whose goal is to increase the softmax score of the predicted class. Liang et al. (2018) found that this kind of perturbation has a stronger effect on ID samples than on OOD ones, increasing the separation between ID and OOD samples. ODIN outperforms the baseline method (Hendrycks & Gimpel, 2017) by a fair
margin; however, it is three times slower due to the two forward and one backward passes needed to preprocess the input, while (Hendrycks & Gimpel, 2017) only requires one forward pass.
Vyas et al. (2018) describes a novel loss function, called margin entropy loss, over the softmax output that attempts to increase the distance between ID and OOD samples. During training, they partition the training data itself into ID and OOD by assigning samples labeled as certain classes as OOD and use the different partitions to train an ensemble of classifiers that are then used to detect OOD samples during test time. They also use the input pre-processing step proposed in Liang et al. (2018), including temperature scaling.
Lee et al. (2018b) is the most recent work on OOD detection that we have knowledge of. They show that the posterior distribution defined by a generative classifier (under Gaussian discriminant analysis) is equivalent to that of the softmax classifier, and the generative classifier eases the separation between in- and out-of-distribution samples. The confidence score is defined using the Mahalanobis distance between the sample and the closest class-conditional Gaussian distribution. They argue that abnormal samples can be better characterized in the DNN feature space rather than the output space of softmax-based posterior distribution as done in previous work (e.g., ODIN). Samples are pre-processed similarly as done in ODIN, but the confidence score is increased instead of the softmax one. To further improve the performance, they also consider intermediate generative classifiers for all layers in the network. The final OOD sample detector is computed as an ensemble of confidence scores, chosen by training a logistic regression on validation samples. This method also shows remarkable results for detection of adversarial attacks and for incremental learning.

heading: 3 PROPOSED SOLUTION

text:
An OOD detector should incorporate information from the training data in a natural manner, without being directly influenced by the loss function, which is intrinsically related to the task which could be well-defined for ID samples but be meaningless for most OOD samples. Moreover, if the OOD method is more dependent on the training distribution, it should be able to be applied to a wide variety networks, and not be designed specifically for a given architecture.
Our method is based on a very simple observation. Input samples with different distributions generate statistically distinct feature spaces in a DNN. In other words, the deep features of an ID sample are distinct from those of an OOD one. Moreover, when using a normalization scheme, such as BN, the features are normalized by the statistics of the ID batch during training, possibly leading to feature statistics that are more similar to the batch statistics, as depicted in Figure 2.
The main problem then becomes how to summarize the feature space distribution for ID samples in a way that best discriminates between ID and OOD samples. In this work we show that using the first and second order statistics within each feature map performs remarkably well for this task.
Figure 1 shows the proposed OOD detector, highlighting the feature statistics that are extracted for each channel at each BN layer. These statistics are then combined through a learned linear decision function, providing a measure of OOD-ness. The way we compute the per-channel statistics
is formalized in Section 3.1. Finally, the linear classifier used to combine the statistics from the different layers is described in Section 3.2.

heading: 3.1 LOW-ORDER FEATURE STATISTICS

text:
In the previous section, we motivated that characterizing the feature-space distributions might lead to a robust OOD detector. As a first approach, one could model these distributions using a nonparametric method to estimate the distribution of the features for each channel, which requires the computation of sufficient statistics using the training data or using a parametric method to fit the distribution (Bishop, 2007), which are both computationally intensive.
Here, we propose to use only the mean and standard deviation computed along the spatial dimension for each channel to summarize the per-channel distribution. As it will be shown, these two statistics are sufficient to distinguish between ID and OOD. Moreover, because the mean and standard deviation of each channel are normalized by the running mean and variance computed during training by the BN layers (Ioffe & Szegedy, 2015), these statistics can be naturally combined within each layer to produce effective features for OOD detection. We describe such a procedure in what follows.
Given the l-th BN layer with input tensor X ∈ RC×W×H , the output BN (l)c,i,j at channel c is
Z (l)c,i,j = X (l)c,i,j − µ (l) c√
[σ (l) c ]2 +
, BN (l)c,i,j = Z (l) c,i,jγ (l) c + β (l) c , (1)
where γ(l)c > 0 and β (l) c are the per channel per layer learned scaling and shifting parameters, > 0 is a small constant value for numerical stability, µ(l)c ∈ R and [σ(l)c ]2 ∈ R+ are the mean and variance estimated through a moving average using the batch statistics during training, and Z(l) ∈ RC×W×H is the normalized feature tensor. It is worth noting that the statistics are calculated independently for each channel c at each layer l.
In this paper, we conjecture that low-order statistics computed from either X (l)c,i,j or Z (l) c,i,j can be used to discriminate between ID and OOD samples. Given the unnormalized input X (l)c,i,j , we can
compute the empirical mean m(l)c and standard deviation σ (l) c features for each channel c as
m(l)c = 1
WH ∑ i,j X (l)c,i,j , [s (l) c ] 2 = 1 WH ∑ i,j [X (l)c,i,j ] 2 − [m(l)c ]2, (2)
and the features for the normalized tensor Z (l)c,i,j are defined as
m̂(l)c = 1
WH ∑ i,j Z (l)c,i,j = m (l) c − µc(l) σ (l) c
[ŝ(l)c ] 2 =
1
WH ∑ i,j [Z (l)c,i,j ] 2 − [m̂(l)c ]2 =
[ ŝ (l) c
σ (l) c
]2 ,
(3)
i.e., the normalized mean feature m̂(l)c represents the difference between the sample mean m (l) c and the BN running mean µ(l)c weighted by the running standard deviation σ (l) c , whereas ŝ (l) c is the normalized standard deviation by the BN running standard deviation σ(l)c .

heading: 3.2 AGGREGATION OF FEATURE STATISTICS

text:
Intra-layers aggregation. The features derived from low-order statistics (equation 3) can be readily used to train a predictor for ID/OOD discrimination. Of course, if they were produced for every feature map in the network, this would result in a feature vector of very high dimension, typically tens of thousands. Instead, we propose to combine these features within each BN layer, so that in the end we obtain one pair of features per layer: average mean and average variance. Taking advantage of the fact that features are approximately normalized by BN’s running statistics, we propose to simply average them for all channels within a layer. Thus, each layer yields the following features, for the normalized case:
m̄(l) = 1
Cl ∑ c m̂(l)c , [s̄ (l)]2 = 1 Cl ∑ c [ŝ(l)c ] 2, (4)
where Cl is the number of channels in layer l. Note that doing this aggregation amounts to computing the mean and standard deviation of all normalized features at the given layer. Using averages of the low-order statistics could lead to issues in deeper layers, where activations are in general concentrated over fewer number of channels. In this case, the mean of the statistics over channels might not be an appropriate data reduction function. Nevertheless, as we show in the experiments section, this did not impact the performance of the proposed method, but more investigation is warranted.
Inter-layers ensemble and final classification. Using all the features in equation 4, i.e., f = (m̄(1), s̄(1), ..., m̄(L), s̄(L)) we fit a simple logistic regression model h(f ;θ) with parameters θ ∈ R2L+1. The parameters of the linear model are learned using a separate validation set formed with ID samples (positive examples) and OOD samples (negative examples).

heading: 4 EXPERIMENTS

text:
In this section, we evaluate the effectiveness of the proposed method in state-of-the-art deep neural architectures, such as DenseNet (Huang et al., 2017) and Wide ResNet (Zagoruyko & Komodakis, 2016), on several computer vision benchmark datasets: CIFAR (Krizhevsky, 2009), TinyImageNet, a subset of ImageNet (Deng et al., 2009), LSUN (Yu et al., 2015), and iSUN (Xu et al., 2015). We also use Gaussian noise and uniform noise as synthetic datasets. This evaluation protocol is the de facto standard in recent OOD detection literature (Hendrycks & Gimpel, 2017; Liang et al., 2018; Vyas et al., 2018; Lee et al., 2018b). All experiments were performed on four models trained from scratch (each one initialized with a different random seed) for each architecture, to account for variance in the model parameters. The code to reproduce all results is publicly available1.
1Code for review avilable at https://www.dropbox.com/s/o652ufpznatvx8u/ ood-codes-review.zip?dl=0.

heading: 4.1 SETUP

text:
Datasets: For backbone training, we use CIFAR-10 and CIFAR-100 datasets which have 10 and 100 classes respectively, both containing 50,000 images in the training set and 10,000 images in the test set. At test time, the test images from CIFAR-10 (CIFAR-100) are considered as ID (positive) samples. For OOD (negative) datasets, we test with datasets containing natural images, such as TinyImageNet resize and crop, LSUN resize and crop, and iSUN, as well as synthetic datasets, such as Gaussian/uniform noise, which is the same dataset setup as in Liang et al. (2018); Hendrycks & Gimpel (2017). This is summarized in Table 2. For all datasets, we did the validation/test set split following the procedure in Liang et al. (2018). Just for reproducibility, 1000 samples from the test set are separated in a validation set used for fitting the logistic regressor and hyper-parameter tuning. The remaining samples (unseen for both backbone model and OOD detector) are used for testing.
Backbone training: Following Liang et al. (2018), we adopt the DenseNet (Huang et al., 2017) and Wide ResNet (Zagoruyko & Komodakis, 2016) architectures as our benchmark networks. For DenseNet, we use depth L = 100, growth rate k = 12, and zero dropout rate (DenseNet-BC-10012). For Wide ResNet, we also follow Liang et al. (2018), with L = 28 and widen factor of 10 (WRN-28-10). All hyperparameters are identical to their original papers. All networks were trained using stochastic gradient descent with Nesterov momentum (Ruder, 2016) and an initial learning rate of 0.1. We train the DenseNet-BC-100-12 for 300 epochs, with batch size 64, momentum 0.9, weight decay of 10−4 and decay the learning rate by a factor of 10 after epochs 150 and 225. We train the WRN-28-10 for 200 epochs, with batch size 128, momentum 0.9, weight decay 0.0005, and decay the learning rate by a factor of 10 after epochs 60, 120, and 160. Table 3 shows each network error rate over 4 independent runs each one initialized with a different random seed.
Logistic regression: The logistic regressor is trained considering only the validation partitions for ID (positive examples) and OOD (negative examples) datasets (see Table 2). Using both mean and standard deviation as input (from equation 4), we have 50 features for WRN-28-10 models, and 198 features for DenseNet-BC-100-12 models. The training was performed using 5-fold cross-validation with the `2 minimization and the regularization factor being chosen as the best one (according to the 5-folds) among 10 values linearly spaced in the range 10−4 and 104.
Evaluation metrics: To evaluate the proposed method, we use the following metrics:
1. True negative rate (TNR) at 95% true positive rate (TPR). Let TP, TN, FP, and FN be the true positive, true negative, false positive, and false negative, respectively. The TNR is defined as TN/(TN+FP) whereas TPR is defined as TP/(TP+FN).
2. Area under the receiver operating characteristic curve (AUROC). AUROC is the area under the FPR=1-TNR against TPR curve.

heading: 4.2 MANIFOLD VISUALIZATION

text: We applied t-SNE (L. van der Maaten, 2008) to visualize our high-dimensional feature space in order to see the similarities between ID/OOD samples. For this, we used one of the WRN-28-10 models trained with CIFAR-10 as ID dataset. We fitted the t-SNE using the ID and all OOD validation samples together using both mean and standard deviation features, and the result is shown in Figure 3 using a perplexity of 30. It is clear from the visualization that the proposed features are concentrated around well-defined clusters. Both synthetic OOD datasets have clear distinct behavior from the natural images ones, and it is straightforward to differentiate them. TinyImageNet (c), LSUN (c) are similar and have some intersection with TinyImageNet (r), LSUN (r). Interestingly, the clustering seems to reflect the dataset generation method (resizing or cropping). Most importantly, one can see that the OOD samples are in different clusters than the ID (CIFAR-10) ones, which indicates that this feature choice is adequate for separating them.

heading: 4.3 PAIRWISE FITTING

text:
In this section, following other methods in the literature, we adjust the linear classifier for each ID/OOD pair. That is, for each pair of ID/OOD datasets, a different OOD classifier is trained using their respective validation samples.
We performed a comparison between the proposed method and four recent methods described in Section 2 (Hendrycks & Gimpel, 2017; Lee et al., 2018b; Liang et al., 2018; Vyas et al., 2018). Since Lee et al. (2018b) did not test using Wide ResNet models and the same datasets as in Liang et al. (2018); Vyas et al. (2018), here we only show the intersection between them: DenseNet BC 100- 12 model, using CIFAR-10 (CIFAR-100) as in-distribution and TinyImageNet (resize) and LSUN (resize) as OOD distribution2. Extended results can be found in the appendix. For both Hendrycks & Gimpel (2017) and Liang et al. (2018) results, we reimplemented the method using their reference implementation3,4. For ODIN (Liang et al., 2018), we employed the same procedure as described by the authors to tune the methods parameters. A detailed description of the procedure can be found in Appendix B.1. For Lee et al. (2018b) and Vyas et al. (2018), we use the values presented on their papers (Table 2 and Table 3, respectively).
The results are compiled in Table 4. Notably, our method outperforms the baseline and ODIN methods by a large margin, and yields better results than Lee et al. (2018b) and Vyas et al. (2018) in all tested cases without requiring any preprocessing, or changing the backbone model. In fact, when setting the OOD-ness threshold to obtain 95% TPR, our method is able to correctly detect all OOD samples from the test partition.
2At the time of submission, Lee et al. (2018b) was still unpublished work 3Baseline: github.com/hendrycks/error-detection 4ODIN: github.com/ShiyuLiang/odin-pytorch

heading: 4.4 TESTING ON UNSEEN OOD DATASETS

text:
We argue that the pairwise fitting scenario presented in the previous section is a limited performance measurement. In fact, many practical applications have OOD samples that do not come from a single distribution mode, and it might be infeasible to collect data from the many different modes of the OOD distribution (in general, some modes are unknown during training). A good OOD detector should be able to correctly identify samples from OOD distibutions for which its parameters have not been adjusted to. With this in mind, we propose a different, harder task in which an OOD detector is fitted to one, or a few, OOD datasets and then it is tested on all OOD datasets available. We note that this is not a standard practice in previous works, like in Liang et al. (2018). We begin by evaluating the generalization ability of our detector in some preliminary experiments, which motivate our decisions in the choice of OOD datasets for fitting the model and feature selection.
Selecting Features: We evaluate the individual impact of each of the proposed features (i.e., layers average mean and standard deviation) by comparing the performance of the classifier with different features as inputs. In this experiment, we fit the linear classifier to a specific OOD validation dataset and test on all of the OOD test datasets available. Our performance metric is the TNR @ 95% TPR averaged over all the tested datasets. Table 5 presents the performance of the classifier for the WRN-28-10, averaged of the four available models, with CIFAR-100 as ID dataset. The classifier fitted using only standard deviation features still achieved very good performance generalizing very well to unknown OOD datasets. Since we are interested in designing an OOD detector that is able to differentiate ID samples from any OOD sample, all our results from now on are presented using the averaged standard deviations per layer as the only features used to train the linear decision function.
Selecting OOD Dataset: To understand to what extent a classifier trained considering one OOD dataset is able to generalize and detect samples from other OOD datasets, we trained the logistic regression considering as positive examples 1000 samples from the ID dataset (CIFAR-100) and as negative examples 1000 samples of a given OOD dataset, using the WRN-28-10 backbone model. As motivated in the previous section, we use the averages of normalized standard deviation features (equation 4). The obtained logistic regressor was then evaluated on the remaining OOD test datasets (unseen by both backbone training and logistic regressor). This procedure was then repeated for each possible OOD dataset, and the results are summarized in Table 6. We see that all classifier fitted using only natural images are capable to generalize well over all other OOD sets, while this is not entirely true when fitting on random noise datasets. Also, fitting to all OODs validation sets (penultimate row), we can achieve even higher TNR scores over all test sets.
Using no OOD Dataset: To further evaluate the effectiveness of the method, we also tested the extreme case where no OOD samples are available for training. To do this, we used an unsupervised
algorithm (one-class SVM (Manevitz & Yousef, 2002) with RBF kernel), and we only fitted to ID samples (i.e., no OOD samples are seen in the training step). The unsupervised results are summarized in the last row of Table 6. As one can see, even the unsupervised method shows reasonable performance; showing again that in the proposed feature space the ID/OOD samples have different behavior. This corroborates the assumption that these features are a good indicator of OOD-ness.
Comparison with ODIN (Liang et al., 2018): We compare the generalization capabilities of our method with the state-of-the-art technique ODIN (Liang et al., 2018), in this new harder task. We fit both OOD classifier to maximize their detection performance on TinyImageNet (c) and Gaussian validation sets (i.e., 2000 OOD samples), and evaluate on all OOD test datasets. For our model, we use standard deviation features as inputs. For ODIN, we tune its hyperparameters using the grid search described in B.1. The results, presented in Table 7, show that our method outperforms ODIN by a large margin, indicating better generalization to samples from unseen OOD datasets.
Effect of feature normalization: We evaluated how much it helps to use the batch statistics computed by BN. As shown in Table 8, normalizing the latent space using the BN statistics before computing the features has clear advantages.
Reduced validation data: We study if our method can correctly detect OOD samples even when a small number of samples is available. Figure 4 shows the TNR @ 95% TPR for WRN 28-10 trained on CIFAR-10 (CIFAR-100), where only 27, 45, 75, 150, 300, 700, 1.5k, 3k (all) test images (ID + TinyImageNet crop + Gaussian, equally divided) are used to fit 25 coefficients of our logistic regressor. Using our method, only 27 images (from each ID and OOD), are necessary to achieve an average of 87.6% of TNR @ 95% TPR.

heading: 5 CONCLUSION

text: Deep neural networks trained to maximize classification performance in a given dataset are extremely adapted to said dataset. The statistics of activations throughout the network for samples from the training distribution (in-distribution) are remarkably stable. However, when a sample from a different distribution (out-of-distribution) is given to the network, its activation statistics depart greatly from those of in-distribution samples. Based on this observation, we propose a very simple yet efficient method to detect out-of-distribution samples. Our method is based on computing averages of low-order statistics at the batch normalization layers of the network, and then use them as features in a linear classifier. This procedure is much simpler and efficient than current stateof-the-art methods, and outperforms them by a large margin in the traditional ID/OOD fitting task (as proposed in previous works). We evaluated all methods in the challenging task of fitting on a single OOD dataset and testing on samples from other (unseen) datasets. In this harder scenario, our method generalizes well to unseen OOD datasets, outperforming ODIN by an even larger margin. Moreover, we show some preliminary results that even in the extreme case where no OOD samples are used for the training (unsupervised) we get reasonable performance.

heading: A SUPPLEMENTARY RESULTS FOR SECTION 3

heading: B SUPPLEMENTARY RESULTS FOR SECTION 4

heading: B.1 PARAMETER FINE-TUNING FOR ODIN

text:
The temperature T and noise magnitude hyperparameters were tuned using 1000 samples from the CIFAR test set as validation and evaluated on the remaining 9000 samples. The hyper-parameter
tuning is carried out on each pair of in- and out-of-distribution samples, which is the same procedure presented in Liang et al. (2018) and Lee et al. (2018b). For reproducibility, a grid search is employed considering T ∈ {1, 1000} and with 21 linearly spaced values between 0 and 0.004 plus [0.00005, 0.0005, 0.0011].

heading: B.2 ADDITIONAL BACKBONE MODEL

text: We also train a Wide ResNet withL = 40 and widen factor of 4 (WRN-40-4) using the same training setup in Section 4, and the results for both training on CIFAR-10 and CIFAR-100 is depicted in Table 9.

heading: B.3 OTHER EVALUATED METRICS

text:
To compare all methods described in Section 2 to our proposed method, we also use the following additional metrics:
1. Area under precision-recall curve (AUPR). The precision is evaluated as TP/(TP+FP) and recall in this case is the TPR. The AUPR-in (AUPR-out) is defined when in-(out-of)distribution samples are considered as the positive ones.

heading: B.4 SELECTING OOD DATASET USING SVHN AS ID

text:
We also tested using the Street View House Number (SVHN) dataset (Netzer et al., 2011) as ID and DenseNet BC 100-12 as backbone model. The pre-trained model is from Lee et al. (2018b), which has a test error rate of 3.58%. The results are displayed in Table 10.
B.5 INTER-LAYER ENSEMBLE.
In this experiment, we evaluate the effect of ensembling across the BN layers, as described in Section 3.2, using a logistic regression. As shown in Figure 7, aggregating confidence scores of intermediate layers indeed improves the detection performance of OOD samples. The left-most bin indicates the TNR at 95% of TPR using only the last BN layer. As we go to the right, we aggregate the shallower BN layers, and the right-most bin indicates that all the BN layers are being used. Another important insight is that deeper layers (on the left of Figure 7) contribute more for the separation between in- and out-of-distribution samples.

heading: B.6 OOD DETECTION COMPARISON

text: We also extended our Table 7 with the extra backbone model (WRN 40-4) and additional metrics. The results are shown in Table 11.

heading: B.7 OOD DETECTION AT DIFFERENT TPR LEVELS

text: We also tested our proposed model on different TPR levels, and the results are depicted in Table 12.

title: DETECTING OUT-OF-DISTRIBUTION SAMPLES USING LOW-ORDER DEEP FEATURE STATISTICS