AN EMPIRICAL ANALYSIS OF DEEP NETWORK LOSS SURFACES
The training of deep neural networks is a high-dimension optimization problem with respect to the loss function of a model. Unfortunately, these functions are of high dimension and non-convex and hence difficult to characterize. In this paper, we empirically investigate the geometry of the loss functions for state-of-the-art networks with multiple stochastic optimization methods. We do this through several experiments that are visualized on polygons to understand how and when these stochastic optimization methods find local minima.
1 INTRODUCTION
Deep neural networks are trained by optimizing an extremely high-dimensional loss function with respect to the weights of the network’s linear layers. The objective function minimized is some measure of the error of the network’s predictions based on these weights compared to training data. This loss function is non-convex and has many local minima. These loss functions are usually minimized using first-order gradient descent (Robbins & Monro, 1951; Polyak, 1964) algorithms such as stochastic gradient descent (SGD) (Bottou, 1991). The success of deep learning critically depends on how well we can minimize this loss function, both in terms of the quality of the local minima found and the time to find them. Understanding the geometry of this loss function and how well optimization algorithms can find good local minima is thus of vital importance.
Several works have theoretically analyzed and characterized the geometry of deep network loss functions. However, to make these analyses tractible, they have relied on simplifications of the network structures, including that the networks are linear (Saxe et al., 2014), or assuming the path and variable independence of the neural networks (Choromanska et al., 2015). Orthogonally, the performance of various gradient descent algorithms has been theoretically characterized (Nesterov, 1983). Again, these analyses make simplifying assumptions, in particular that the loss function is strictly convex, i.e. there is only a single local minimum.
In this work, we empirically investigated the geometry of the real loss functions for state-of-the-art networks and data sets. In addition, we investigated how popular optimization algorithms interact with these real loss surfaces. To do this, we plotted low-dimensional projections of the loss function in subspaces chosen to investigate properties of the local minima selected by different algorithms. We chose these subspaces to address the following questions:
• What types of changes to the optimization procedure result in different local minima? • Do different optimization algorithms find qualitatively different types of local minima?
2 RELATED WORK
2.1 LOSS SURFACES
There have been several attempts to understand the loss surfaces of deep neural networks. Some have studied the critical points of the deep linear neural networks (Baldi, 1989; Baldi & Hornik,
∗Work done during an internship at Janelia Research Campus
1989; Baldi & Lu, 2012). Others further investigated the learning dynamics of the deep linear neural networks (Saxe et al., 2014). More recently, several others have attempted to study the loss surfaces of deep non-linear neural networks (Choromanska et al., 2015; Kawaguchi, 2016; Soudry & Carmon, 2016).
One approach is to analogize the states of neurons as the magnetics dipoles used in spherical spinglass Ising models from statistical physics (Parisi, 2016; Fyodorov & Williams, 2007; Bray & Dean, 2007). Choromanska et al. (2015) attempted to understand the loss function of neural networks through studying the random Gaussian error functions of Ising models. Recent results (Kawaguchi, 2016; Soudry & Carmon, 2016) have provided cursory evidence in agreement with the theory provided by Choromanska et al. (2015) in that they found that that there are no “poor” local minima in neural networks still with strong assumptions.
There is some potential disconnect between these theoretical results and what is found in practice due to several strong assumptions such as the activation of the hidden units and output being independent of the previous hidden units and input data. The work of Dauphin et al. (2014) empirically investigated properties of the critical points of neural network loss functions and demonstrated that their critical points behave similarly to the critical points of random Gaussian error functions in high dimensional space. We will expose further evidence along this trajectory.
2.2 OPTIMIZATION
In practice, the local minima of deep network loss functions are for the most part decent. This implies that we probably do not need to take many precautions to avoid bad local minima in practice. If all local minima are decent, then the task of finding a decent local minimum quickly is reduced to the task of finding any local minimum quickly. From an optimization perspective this implies that solely focusing on designing fast methods are of key importance for training deep networks.
In the literature the common method for measuring performance of optimization methods is to analyze them on nice convex quadratic functions (Polyak, 1964; Broyden, 1970; Nesterov, 1983; Martens, 2010; Erdogdu & Montanari, 2015) even though the problems are applied to non-convex problems. For non-convex problems, however, if two methods converge to different local minima their performance will be dictated on how those methods solve those two convex subproblems. It is challenging to show that one method will beat another without knowledge of the sort of convex subproblems, which is generally not known apriori. What we will explore is whether indeed are some characteristics that can found experimentally. If so, perhaps one could validate where these analytical results are valid or even improve methods for training neural networks.
2.2.1 LEARNING PHASES
One of the interesting empirical observation is that we often observe is that the incremental improvement of optimization methods decreases rapidly even in non-convex problems. This behavior has been discussed as a “transient” phase followed by a “minimization” phase (Sutskever et al., 2013)
where the former finds the neighborhood of a decent local minima and the latter finds the local minima within that neighborhood. The existence of these phases implies that if certain methods are better at different phases one could create novel methods that schedule when to apply each method.
3 EXPERIMENTAL SETUP AND TOOLS
3.1 NETWORK ARCHITECTURES AND DATA SETS
We conducted experiments on three state-of-the-art neural network architectures. Network-inNetwork (NIN) (Lin et al., 2014) and the VGG(Simonyan & Zisserman, 2015) network are feedforward convolutional networks developed for image classification, and have excellent performance on the Imagenet (Russakovsky et al., 2014) and CIFAR10 (Krizhevsky, 2009) data sets. The long short-term memory network (LSTM) (Hochreiter & Schmidhuber, 1997) is a recurrent neural network that has been successful in tasks that take variable-length sequences as input and/or produce variable-length sequences as output, such as speech recognition and image caption generation. These are large networks currently used in many machine vision and learning tasks, and the loss functions minimized by each are highly non-convex.
All results using the feed-forward convolutional networks (NIN and VGG) are on the CIFAR10 image classification data set, while the LSTM was tested on the Penn Treebank next-word prediction data set.
3.2 OPTIMIZATION METHODS
We analyzed the performance of five popular gradient-descent optimization methods for these learning frameworks: Stochastic gradient descent (SGD) (Robbins & Monro, 1951), stochastic gradient descent with momentum (SGDM), RMSprop (Tieleman & Hinton, 2012), Adadelta (Zeiler et al., 2011), and ADAM (Kingma & Ba, 2014). These are all first-order gradient descent algorithms that estimate the gradients based on randomly-grouped minibatches of training examples. One of the major differences between these algorithms is how they select the weight-update step-size at each iteration, with SGD and SGDM using fixed schedules, and RMSprop, Adadelta, and ADAM using adaptive, per-parameter step-sizes. Details are provided in Section A.2.
In addition to these five existing optimization methods, we compare to a new gradient descent method we developed based on the family of Runge Kutta integrators. In our experiments, we tested a second-order Runge-Kutta integrator in combination with SGD (RK2) and in combination with ADAM (ADAM&RK2). Details are provided in Section A.3).
3.3 ANALYSIS METHODS
Several of our empirical analyses are based on the technique of Goodfellow et al. (Goodfellow et al., 2015). They visualize the loss function by projecting it down to one carefully chosen dimension. They plot the value of the loss function along a set of samples along this dimension. The projection space is chosen based on important weight configurations, thus they plot the value of the loss function at linear interpolations between two weight configurations. They perform two such analyses: one in which they interpolate between the initialization weights and the final learned weights, and one in which they interpolate between two sets of final weights, each learned from different initializations.
In this work, we use a similar visualization technique, but choose different low-dimensional subspaces for the projection of the loss function. These subspaces are based on the initial weights as well as the final weights learned using the different optimization algorithms and combinations of them, and are chosen to answer a variety of questions about the loss function and how the different optimization algorithms interact with this loss function. In contrast, Goodfellow et al. only looked at SGDM. In addition, we explore the use of two-dimensional projections of the loss function, allowing us to better visualize the space between local minima. We do this via barycentric and bilinar interpolation for triplets and quartets of points respectively (details in Section A.1).
We refer to the critical points found using these variants of SGD, for which the gradient is approximately 0, as local minima. Our evidence that these are local minima as opposed to saddle points is
similar to that presented in Goodfellow et al. (Goodfellow et al., 2015). If we interpolate beyond the critical point, in this one-dimensional projection, the loss increases (Fig. 10).
3.4 TECHNICAL DETAILS
We used the VGG and NIN implementations from https://github.com/szagoruyko/cifar.torch.git.
The batch size was set to 128 and the number of epochs was set to 200. The learning rate was chosen from the discrete range between [0.2, 0.1, 0.05, 0.01] for SGD and [0.002, 0.001, 0.0005, 0.0001] for adaptive learning methods. We doubled the learning rates when we ran our augmented versions with Runge-Kutta because they required two stochastic gradient computations per epoch. We used batchnormalization and dropout to regularize our networks. All experiments were run on a 6-core Intel(R) Xeon(R) CPU @ 2.40GHz with a TITAN X.
4 EXPERIMENTAL RESULTS
4.1 DIFFERENT OPTIMIZATION METHODS FIND DIFFERENT LOCAL MINIMA
We trained the neural networks described above using each optimization method starting from the same initial weights and with the same minibatching. We computed the value of the loss function for weight vectors interpolated between the initial weights, the final weights for one algorithm, and the final weights for a second algorithm for several pairings of algorithms. The results are shown in the lower triangle of Table 1.
For every pair of optimization algorithms, we observe that the training loss between the final weights for different algorithms shows a sharp increase along the interpolated path. This suggests that each optimization algorithm found a different critical point, despite starting at the same initialization. We investigated the space between other triplets and quadruples of weight vectors (Figure 2 and 3), and even in these projections of the loss function, we still see that the local minima returned by different algorithms are separated by high loss weight parameters.
Deep networks are overparameterized. For example, if we switch all corresponding weights for a pair of nodes in our network, we will obtain effectively the same network, with both the original and permuted networks outputting the same prediction for a given input. To ensure that the weight vectors returned by the different algorithms were functionally different, we compared the outpts of the networks on each example in a validation data set:
dist(θ1, θ2) = √√√√ 1 Ntest Ntest∑ i=1 ‖F (xi, θ1)− F (xi, θ2)‖2,
where θ1 and θ2 are the weights learned by two different optimization algorithms, xi is the input for a validation example, and F (x, θ) is the output of the network for weights θ on input x.
We found that, for all pairs of algorithms, the average distance between the outputs of the networks (Equation 4.1) was approximately 0.16, corresponding to a label disagreement of about 8% (upper triangle of Table 1). Given the generalization error of these networks (approximately 11%, Figure 4), the maximum disagreement we could see was 22%. Thus, these networks disagreed on a large fraction of these test examples – over 13 rd. Thus, the local minima found by different algorithms correspond to effectively different networks, not trivial reparameterizations of the same one.
4.2 DIFFERENT OPTIMIZATION ALGORITHMS FIND DIFFERENT TYPES OF LOCAL MINIMA
Next, we investigated whether the local minima found by the different optimization algorithms had distinguishing properties. To do this, we trained the networks with each optimization algorithm using different initial parameters. We then compared differences between runs of the same algorithm but different initializations to differences between different algorithms.
As shown in Figure 4(a), in terms of training accuracy, we do see some stereotypy for the optima found by different algorithms, with SGD finding local minima with the lowest training accuracy and ADAM, Rmsprop, and Adadelta finding local minima with the highest training accuracy. However, this could be attributed to SGD’s asymtotically slow convergence near local minima due to the gradient diminishing near extrema. Despite this limitation, Figure 4(b) shows that the generalization accuracy of these different local minima on validation data was not significantly different between algorithms. We also did not see a relationship between the weight initialization and the validation accuracy. Thus, while these algorithms fall into different local minima, they are not different in terms of their final quality.
We visualized the loss surface around each of the local minima for the multiple runs. To do this, we plotted the value of the loss function between the initial and final weights for each algorithm (Figure 5(a,c)) for each run of the algorithm from a different initialization. In addition, we plotted
the value of the loss function between the final weights for selected pairs of algorithms for each run (Figure 5(b,d)). We see that the surfaces look strikingly similar for different runs of the same algorithm, but characteristically different for different algorithms. Thus, we found evidence that the different algorithms land in qualitatively different types of local minima.
In particular, we see in Figure 5(a,c) that the size of the basins around the local minima found by ADAM and ADAM&RK2 are larger than those found by SGD and RK2, i.e. that the training loss is small for a wider range of α values. This is a relative measure, and the magnitude of the change in the weight vector is ∆α‖θ1 − θ0‖ for a change of size ∆α, where θ0 is the initial weight vector θ1 is the result found by a given optimization algorithm. In Figure 6, we repeat this analysis, instead showing the loss as a function of the absolute distance in parameter space:
θ(λ) = θ1 + λ θ0 − θ1 ‖θ0 − θ1‖
(1)
We again see that the size of the basin around the local minima varies by optimization algorithm. Note that we evaluate the loss for weight vectors beyond the initial configuration, which had a loss of 2.4.
4.3 ANALYZING LEARNING AFTER “TRANSIENT PERIOD”
Recall that, during optimization, it has been observed that there is a short “transient” phase when the loss decreases rapidly and a “minimization” phase in which the loss decreases slowly (Section 2.2.1 and Figure 1). In this set of experiments, we investigated the effects of switching from one type of optimization method to another at various points during training, in particular at late stages of training when it is thought that a local minimum has been chosen and is only being localized. We switched from one optimization method to another 25%, 50%, and 75% of the way through training. The results are plotted in Figure 7d. We emphasize that we are not switching methods to improve performance, but rather to investigate the shape of the loss function in regions explored during the “minimization” phase of optimization.
We found that, regardless of how late we switch optimization algorithms, as shown in the rightmost column of Figure 7, the local minima found were all different. This directly disagrees with the notion that the local minimum has effectively been chosen before the “minimization” phase, but instead that which local minimum is found is still in flux this late in optimization. It appears that this switch from one local minimum to another happens almost immediately after the optimization method switches, with the training accuracy jumping to the characteristic accuracy for the given method within a few epochs (Figure 7, left column). Interestingly, we also see the distance between the initial and current weight vectors changes drastically after switching from one optimization
method to another, and that this distance is characteristic per algorithm (Figure 7, middle column). While distance increases with training epoch for any single optimization method, it actually starts to decrease when switching from ADAM to SGD.
4.4 EFFECTS OF BATCH-NORMALIZATION
To understand how batch normalization affects the types of local minima found, we performed a set of experiments comparing loss surfaces near local minima found with and without batch normal-
ization for each of the optimization methods. We visualized the surface near these local minima by interpolating between the initial weights and the final weights as well as between pairs of final weights found with different algorithms.
We observed clear qualitative differences between optimization with (Figure 5) and without (Figure 8) batch normalization. We see that, without batch normalization, the quality of local minimum found by a given algorithm is much more dependent on the initialization. In addition, the surfaces between different local minima are more complex in appearance: with batch normalization we see sharp unimodal jumps in performance but without batch normalization we obtain wide bumpy shapes that aren’t necessarily unimodal.
The neural networks are typically initialized with very small parameter values (Glorot & Bengio, 2010; He et al., 2015). Instead, we trained NIN with exotic intializations such as initial parameters drawn from N (−10.0, 0.01) or N (−1.0, 1.0) and observe the loss surface behaviours. The details of results are discussed in Appendix A.5.
5 CONCLUSIONS
In this work, we performed a series of empirical analyses to understand the geometry of the loss functions corresponding to deep neural networks, and how different optimization methods minimize this loss to answer the two questions posed in the introduction.
What types of changes to the optimization procedure result in different local minima?
We found that every type of change to the optimization procedure we tested resulted in a different local minimum. Different local minima were found using the different optimization algorithms from the same initialization (Section 4.1). Even switching the optimization algorithm to another very late in optimization – during the slow “mimimization” portion of learning – resulted in a different local minimum (Section 4.3). The quality of the local minima found, in terms of training and generalization error, is similar. These different local minima were not equivalent, and made mistakes on different test examples (Section 4.1). Thus, they were not trivially different local minima, as would occur if nodes in internal layers of the network were permuted. We observed that the quality of these local minima was only consistently good when we used batch normalization for regularization. Without batch normalization, the quality of the critical points found depended on the initialization, and some solutions found were not as good as others. Our observations are in contrast to the conclusions of Goodfellow et al., i.e. that local minima are not a problem in deep learning because, in the region of the loss function explored by SGD algorithms, the loss function is well-behaved (Goodfellow et al., 2015). Instead, our observations are more consistent with the explanation that the local minima found by popular SGD optimization methods are almost all good (Choromanska et al., 2015; Kawaguchi, 2016; Soudry & Carmon, 2016).
Do different optimization algorithms find qualitatively different types of local minima?
Interestingly, we found that, while the local minima found by the same optimization algorithm from different initializations were different, the shape of the loss function around these local minima was strikingly similar, and was a characteristic of the optimization algorithm. In particular, we found that the size of the basin around ADAM-based optimization was larger than that around vanilla SGD (Section 4.2). A large basin is related to a large margin, as small changes in the weight vector will not affect the training error, and perhaps could have some implications for generalization error. In our experiments, however, we did not observe better generalization error for ADAM than SGD. Questions for potential future research are why the shapes of the loss functions around different local minima found by the same algorithm are so similar, and what the practical implications of this are.
A SUPPLEMANTARY MATERIALS
A.1 3D VISUALIZATION
Goodfellow et al. (2015) introduced the idea of visualizing 1D subspace of the loss surface between the parameters. Here, we propose to visualize loss surface in 3D space through interpolating over three and four vertices.
Linear Interpolation Given two parameters θ1 and θ2, θi = αθ1 + (1− α)θ2, ∀α ∈ [0, 1]. (2)
Bilinear Interpolation Given four parameters θ0, θ1, θ2, and θ3, φi = αθ1 + (1− α)θ2 (3) ϕi = αθ3 + (1− α)θ4 (4) θj = βφi + (1− β)ϕi (5)
for all α ∈ [0, 1] and β ∈ [0, 1].
Barycentric Interpolation Given four parameters θ0, θ1, and θ2, let d1 = θ1 − θ0 and d2 = θ2 − θ0. Then, the formulation of the interpolation is
φi = αd1 + θ0 (6) ϕi = αd2 + θ0 (7) θj = βφi + (1− β)ϕi (8)
for all α ∈ [0, 1] and β ∈ [0, 1].
A.2 OPTIMIZATION METHODS
A.2.1 STOCHASTIC GRADIENT DESCENT
In many deep learning applications both the number of parameters and quantity of input data points can be quite large. This makes the full evaluation of U(θ) be prohibitively expensive. A standard technique for aleviating computational loadis to apply an stochastic approximation to the gradient Robbins & Monro (1951). More precisely, one approximates U by a subset of n data points, denoted by {σj}Nj=1 at each timestep:
Un(θ) = 1
n n∑ j=1 `(θ,xσj ) ' 1 N N∑ i=1 `(θ,xi) = U(θ) (9)
Of course this approximation also carries over to the gradient, which is of vital importance to optimization techniques:
∇Un(θ) = 1 n n∑ j=1 ∇`(θ,xσj ) ' ∇U(θ) (10)
This method is what is commonly called Stochastic Gradient Descent or SGD. So long as the data is distributed nicely the approximation error of Un should be sufficiently small such that not only will SGD still behave like normal GD , but it’s wall clock time for to converge should be significantly lower as well.
Usually one uses the stochastic gradient rather than the true gradient, but the inherent noisiness must be kept in mind. In what follows we will always mean the stochastic gradient.
A.2.2 MOMENTUM
In order to aleviate both noise in the input data as well as noise from stochasticity used in computing quantities one often maintains history of previous evaluations. In order to only require one extra variable one usually stores variables of the form
E[F ]t = αFt + βE[F ]t−1. (11)
where Ft is some value changing over time and E[F ]t is the averaged quantity.
An easy scheme to apply this method to is to compute a rolling weighted average of gradients such as E[g]t = (1− α)gt + αE[g]t−1 but there will be other uses in the future.
A.2.3 PERTINENT METHODS
With the aforementioned tools there are a variety of methods that can be constructed. We choose to view these algorithms as implementations of Explicit Euler on a variety of different vector fields to remove the ambiguity between η and gt. We therefore can define a method by the vector field Xt that explicit Euler is applied to with a single η that is never changed.
SGD with Momentum (SGDM) By simply applying momentum to gt one obtains this stabilized stochastic version of gradient descent:
Xt = −E[g]t. (12) This is the most fundamental method that is used in practice and the basis for everything that follows.
Adagrad Adagrad rescalesXt by summing up the sqaures of all previous gradients in a coefficientwise fashion:
Xt = − gt√∑t
i=1 g 2 i +
. (13)
Here is simply set to some small positive value to prevent division-by-zero. In the future we will neglect this term in denominators because it is always necessary.
The concept is to accentuate variations in gt, but because the denominator is monotonically nondecreasing over time this method is doomed to retard its own progress over time. The denominator can also be seen as a form of momentum where α and β are both set to 1.
Rmsprop A simple generalization of ADAGrad is to simply allow for α and β to be changed from 1. In particular one usually chooses a β less than 1, and presumably α = 1− β. Thus one arrives at a method where the effects of the distance history are diminished:
Xt =− gt√ E[g2]t . (14)
Adadelta Adadelta adds another term to RMSprop in order to guarantee that the magnitude of X is balanced with gt Zeiler et al. (2011). More precisely it maintains
Xt√ E[X 2t ] = − gt√ E[g2t ]
(15)
which results in the following vector field: Xt =− √ E[X 2t ]√ E[g2t ] gt. (16)
and η is set to 1.
ADAM By applying momentum to both gt and g2t one arrives at what is called ADAM. This is often considered a combination of SGDM + RMSprop,
Xt = ct E[g]t√ E[g2]t . (17)
ct =
√ 1−βt2
1−βt1 is the initialization bias correction term with β1, β2 ∈ [0, 1) being the β parameters
used in momentum for g and g2 respectively. Initialization bias is caused by the history of the momentum variable being initially set to zero.
A.3 RUNGE KUTTA
Runge-Kutta methods Butcher (1963) are a broad class of numerical integrators categorized by their truncation error. Because the ordinary differential equations Runge-Kutta methods solve generalize gradient descent, our augmentation is quite straightforward. Although our method applies to all explicit Runge-Kutta methods we will only describe second order methods for simplicity.
The general form of second-order explicit Runge-Kutta on a time-independent vector field is
θt+1 = θt + (a1k1 + a2k2)h (18) k1 = X (θt) (19) k2 = X (θt + q1hk1) (20)
where a1, a2, and q1 are parameters that define a given Runge-Kutta method. Table 3 refers to the parameters used for the different Runge-Kutta variants we use in our experiments.
A.3.1 AUGMENTING OPTIMIZATION WITH RUNGE KUTTA
For a given timestep, explicit integrators can be seen as a morphism over vector fields X → X̄ h. For a gradient gt = ∇θU we can solve a modified RK2 gradient ḡt in the following fashion:
θt+1 = θt + ḡth = Advect rk2 g (θ, h) (21)
rearranged with respect to ḡt
ḡt = Advectrk2g (θ, h)− θt
h (22)
= θt + (a1k1 + a2k2)h− θt
h (23)
= (a1k1 + a2k2). (24) If we simply substitute the gradient gt with ḡt one obtains an RK2-augmented optimization technique.
A.4 EXPERIMENTS WITH RUNGE-KUTTA INTEGRATOR
The results in Figure 9 illustrates that, with the exception of the Midpoint method, stochastic RungeKutta methods outperform SGD. “SGD x2” is the stochastic gradient descent with twice of the learning rate of “SGD”. From the figure, we observe that the Runge-Kutta methods perform even better with half the number of gradient computed by SGD. The reason is because SGD has the accumulated truncated error of O(h) while second-order Runge-Kutta methods have the accumulated truncated error of O(h2).
Unfortunately, ADAM outperforms ADAM+RK2 methods. We speculate that this is because the way how ADAM’s renormalization of input gradients in conjunction with momentum eliminates the value added by using our RK-based descent directions.
A.5 EFFECTS OF BATCH-NORMALIZATION AND EXTREME INITIALIZATIONS
The neural networks are typically initialized with very small parameter values (Glorot & Bengio, 2010; He et al., 2015). Instead, we trained NIN with exotic intializations such as initial parameters drawn from N (−10.0, 0.01) or N (−1.0, 1.0) and observe the loss surface behaviours. The results are shown in Figure 11. We can see that NIN without BN does not train at all with any of these initializations. Swirszcz et al. (2016) mentioned that bad performance of neural networks trained with these initializations are due to finding a bad local minima. However, we see that loss surface region around these initializations are plateau 1 rather than a bad local minima as shown in Figure 11b. On
1We used same initializations as (Swirszcz et al., 2016) but we trained different neural networks with SGD on a different dataset. We used NIN and CIFAR10 and Swirszcz et al. (2016) used smaller neural network and MNIST.
the other hand, NIN with BN does train slowly over time but finds a local minima. This implies that BN redeems the ill-posed loss surface (plateau region). Nevertheless, the local minima it found was not good as when the parameters were initialized with small values. However, it is not totally clear whether this is due to difficulty of training or due to falling in a bad local minima.
A.6 SWITCHING OPTIMIZATION METHODS

The paper is dedicated to better understanding the optimization landscape in deep learning, in particular when explored with different optimization algorithms, and thus it also characterizes the behavior of these algorithms. It heavily re-uses the approach of Goodfellow et al. (2015). I find it hard to understand the contributions of the paper, for example: is it surprising that different algorithms reach different solutions when starting from the same initialization? It would be useful if the authors build such basic intuition in the paper. I also did not receive a clear answer to the question I posed to reviewers regarding clarifying how does the findings of the paper can contribute to future works on optimization in deep learning. And this is what I find fundamentally missing. So for example, there are probably plenty of ways to modify approach of Goodfellow et al. (2015), and similar works, and come up with interesting visualization methods for deep learning - but the question is: how is this helpful in terms of designing better algorithms, gaining more intuition how the optimization surface looks like in general, etc.? This is an interesting paper, though I am fairly confident it is a better fit for the journal than this conference.
It would be interesting and instructive, even for sanity check, to plot the eigenspectra of the solutions recovered by the algorithms to see the order of critical points recovered.


BINING RECENT INSIGHTS FOR LSTMS
LSTMs have become a basic building block for many deep NLP models. In recent years, many improvements and variations have been proposed for deep sequence models in general, and LSTMs in particular. We propose and analyze a series of augmentations and modifications to LSTM networks resulting in improved performance for text classification datasets. We observe compounding improvements on traditional LSTMs using Monte Carlo test-time model averaging, average pooling, and residual connections, along with four other suggested modifications. Our analysis provides a simple, reliable, and high quality baseline model.
1 INTRODUCTION
When exploring a new problem, having a simple yet competitive off-the-shelf baseline is fundamental to new research. For instance, Caruana et al. (2008) showed random forests to be a strong baseline for many high-dimensional supervised learning tasks. For computer vision, off-the-shelf convolutional neural networks (CNNs) have earned their reputation as a strong baseline (Sharif Razavian et al., 2014) and basic building block for more complex models like visual question answering (Xiong et al., 2016). For natural language processing (NLP) and other sequential modeling tasks, recurrent neural networks (RNNs), and in particular Long Short-Term Memory (LSTM) networks, with a linear projection layer at the end have begun to attain a similar status. However, the standard LSTM is in many ways lacking as a baseline. Zaremba (2015), Gal (2015), and others show that large improvements are possible using a forget bias, inverted dropout regularization or bidirectionality. We add three major additions with similar improvements to off-the-shelf LSTMs: Monte Carlo model averaging, embed average pooling, and residual connections. We analyze these and other more common improvements.
2 LSTM NETWORK
LSTM networks are among the most commonly used models for tasks involving variable-length sequences of data, such as text classification. The basic LSTM layer consists of six equations:
it = tanh (Wixt +Riht−1 + bi) (1) jt = σ (Wjxt +Rjht−1 + bj) (2) ft = σ (Wfxt +Rfht−1 + bf ) (3) ot = tanh (Woxt +Roht−1 + bo) (4) ct = it jt + ft ct−1 (5) ht = ot tanh (ct) (6)
Where σ is the sigmoid function, is element-wise multiplication, and vt is the value of variable v at timestep t. Each layer receives xt from the layer that came before it and ht−1 and ct−1 from the previous timestep, and it outputs ht to the layer that comes after it and ht and ct to the next timestep. The c and h values jointly constitute the recurrent state of the LSTM that is passed from one timestep to the next. Since the h value completely updates at each timestep while the c value maintains part of its own value through multiplication by the forget gate f , h and c complement each other very well, with h forming a “fast” state that can quickly adapt to new information and c forming a “slow” state that allows information to be retained over longer periods of time (Zaremba, 2015). While various papers have tried to systematically experiment with the 6 core equations constituting an LSTM (Greff et al., 2015; Zaremba, 2015), in general the basic LSTM equations have proven extremely resilient and, if not optimal, at least a local maximum.
3 MONTE CARLO MODEL AVERAGING
It is common practice when applying dropout in neural networks to scale the weights up at train time (inverted dropout). This ensures that the expected magnitude of the inputs to any given layer are equivalent between train and test, allowing for an efficient computation of test-time predictions. However, for a model trained with dropout, test-time predictions generated without dropout merely approximate the ensemble of smaller models that dropout is meant to provide. A higher fidelity method requires that test-time dropout be conducted in a manner consistent with how the model was trained. To achieve this, we sample k neural nets with dropout applied for each test example and average the predictions. With sufficiently large k this Monte Carlo average should approach the true model average (Srivastava et al., 2014). We show in Figure 1 that this technique can yield more accurate predictions on test-time data than the standard practice. This is demonstrated over a number of datasets, suggesting its applicability to many types of sequential architectures. While running multiple Monte Carlo samples is more computationally expensive, the overall increase is minimal as the process is only run on test-time forward passes and is highly parallelizable. We show that higher performance can be achieved with relatively few Monte Carlo samples, and that this number of samples is similar across different NLP datasets and tasks.
We encountered one ambiguity of Monte Carlo model averaging that to our knowledge remains unaddressed in prior literature: there is relatively little exploration as to where and how the model averaging is most appropriately handled. We investigated averaging over the output of the final recurrent layer (just before the projection layer), over the output of the projection layer (the presoftmax unnormalized logits), and the post-softmax normalized probabilities, which is the approach taken by Gal (2015) for language modeling. We saw no discernible difference in performance between averaging the pre-projection and post-projection outputs. Averaging over the post-softmax probabilities showed marginal improvements over these two methods, but interestingly only for bidirectional models. We also explored using majority voting among the sampled models. This
involves tallying the maximum post-softmax probabilities and selecting the class that received the most votes. This method differs from averaging the post-softmax probabilities in the same way max-margin differs from maximum likelihood estimation (MLE), de-emphasizing the points well inside the decision boundary or the models that predicted a class with extremely high probability. With sufficiently large k, this voting method seemed to work best of the averaging methods we tried, and thus all of our displayed models use this technique. However, for classification problems with more classes, more Monte Carlo samples might be necessary to guarantee a meaningful plurality of class predictions. We conclude that the majority-vote Monte Carlo averaging method is preferable in the case where the ratio of Monte Carlo samples to number of classification labels is large (k/output size).
The Monte Carlo model averaging experiments, shown in Figure 1, were conducted as follows. We drew k = 400 separate test samples for each example, differentiated by their dropout masks. For each sample size p (whose values, plotted on the x-axis, were in the range from 2 to 200 with step-size 2) we selected p of our k samples randomly without replacement and performed the relevant Monte Carlo averaging technique for that task, as discussed above. We do this m = 20 times for each point, to establish the mean and variance for that number of Monte Carlo iterations/samples p. The variance is used to visualize the 90% confidence interval in blue, while the red line denotes the test accuracy computed using the traditional approximation method (inverted dropout at train-time, and no dropout at test-time).
4 EMBED AVERAGE POOLING
Reliably retaining long-range information is a well documented weakness of LSTM networks (Karpathy et al., 2015). This is especially the case for very long sequences like the IMDB sentiment dataset (Maas et al., 2011), where deep sequential models fail to capture uni- and bi-gram occurrences over long sequences. This is likely why n-gram based models, such as a bi-gram NBSVM (Wang and Manning, 2012), outperform RNN models on such datasetes. It was shown by Iyyer et al. (2015) and others that for general NLP classification tasks, the use of a deep, unordered composition (or bagof-words) of a sequence can yield strong results. Their solution, the deep averaging network (DAN), combines the observed effectiveness of depth, with the unreasonable effectiveness of unordered representations of long sequences.
We suspect that the primary advantage of DANs is their ability to keep track of information that would have otherwise been forgotten by a sequential model, such as information early in the sequence for a unidirectional RNN or information in the middle of the sequence for a bidirectional RNN. Our embed average pooling supplements the bidirectional RNN with the information from a DAN at a relatively negligible computational cost.
LSTM
LSTM
LSTM
Softmax
h(1)t
h(2)t
h(3)t
xt… … xt+1xt 1
h(1)t 1 h (1) t
h(2)t 1
h(3)t 1
h(2)t
h(3)t
(a) Res-V1: An illustration of vertical residual connections
LSTM
LSTM
LSTM
Softmax
… …xtxt 1 xt+1
h(1)t 1 h (1) t
h(1)t
h(2)t 1
h(2)t
h(2)t
h(3)t 1
h(3)t
h(3)t
(b) Res-V2: An illustration of vertical and lateral residual connections
Figure 3: An illustration of vertical (ResV) and lateral residual (ResL) connections added to a 3-layer RNN. A model with only vertical residuals is denoted Res-V1, whereas a model with vertical and lateral residuals is denoted “Res-V2”.
As shown in Figure 2, embed average pooling works by averaging the sequence of word vectors and passing this average through an MLP. The averaging is similar to an average pooling layer in a CNN (hence the name), but with the averaging being done temporally rather than spatially. The output of this MLP is concatenated to the final output of the RNN, and the combined vector is then passed into the projection and softmax layer. We apply the same dropout mask to the word vectors when passing them to the RNN as when averaging them, and we apply a different dropout mask on the output of the MLP. We experimented with applying the MLP before rather than after averaging the word vectors but found the latter to be most effective.
5 RESIDUAL CONNECTIONS
For feed-forward convolutional neural networks used in computer vision tasks, residual networks, or ResNets, have obtained state of the art results (He et al., 2015). Rather than having each layer learn a wholly new representation of the data, as is customary for neural networks, ResNets have each layer (or group of layers) learn a residual which is added to the layer’s input and then passed on to the next layer. More formally, if the input to a layer (or group of layers) is x and the output of that layer (or group of layers) is F (x), then the input to the next layer (or group of layers) is x+ F (x), whereas it would be F (x) in a conventional neural network. This architecture allows the training of far deeper models. He et al. (2015) trained convolutional neural networks as deep as 151 layers, compared to 16 layers used in VGGNets (Simonyan and Zisserman, 2014) or 22 layers used in GoogLeNet (Szegedy et al., 2015), and won the 2015 ImageNet Challenge. Since then, various papers have tried to build upon the ResNet paradigm (Huang et al., 2016; Szegedy et al., 2016), and various others have tried to create convincing theoretical reasons for ResNet’s success (Liao and Poggio, 2016; Veit et al., 2016).
We explored many different ways to incorporate residual connections in an RNN. The two most successful ones, which we call Res-V1 and Res-V2 are depicted in Figure 6. Res-V1 incorporates only vertical residuals, while Res-V2 incorporates both vertical and lateral residuals. With vertical residual connections, the input to a layer is added to its output and then passed to the next layer, as is done in feed-forward ResNets. Thus, whereas the input to a layer is normally the ht from the previous layer, with vertical residuals the input becomes the ht + xt from the previous layer. This maintains many of the attractive properties of ResNets (e.g. unimpeded gradient flow across layers, adding/averaging the contributions of each layer) and thus lends itself naturally to deeper networks. However, it can interact unpredictably with the LSTM architecture, as the “fast” state of the LSTM no longer reflects the network’s full representation of the data at that point. To mitigate this unpredictability, Res-V2 also includes lateral residual connections. With lateral residual connections, the input to a layer is added to its output and then passed to the next timestep as the fast state of the LSTM. It is equivalent to replacing equation 6 with ht = ot tanh (ct) + xt. Thus, applying both vertical and lateral residuals ensures that the same value is passed both to the next layer as input and to the next timestep as the “fast” state.
In addition to these two, we explored various other, ultimately less successful, ways of adding residual connections to an LSTM, the primary one being horizontal residual connections. In this architecture, rather than adding the input from the previous layer to a layer’s output, we added the fast state from the previous timestep. The hope was that adding residual connections across timesteps would allow information to flow more effectively across timesteps and thus improve the performance of RNNs that are deep across timesteps, much as ResNets do for networks that are deep across layers. Thus, we believed horizontal residual connections could solve the problem of LSTMs not learning long-term dependencies, the same problem we also hoped to mitigate with embed average pooling. Unfortunately, horizontal residuals failed, possibly because they blurred the distinction between the LSTM’s “fast” state and “slow” state and thus prevented the LSTM from quickly adapting to new data. Alternate combinations of horizontal, vertical, and lateral residual connections were also experimented with but yielded poor results.
6 EXPERIMENTAL RESULTS
6.1 DATASETS
We chose two commonly used benchmark datasets for our experiments: the Stanford Sentiment Treebank (SST) (Socher et al., 2013) and the IMDB sentiment dataset (Maas et al., 2011). This allowed us to compare the performance of our models to existing work and review the flexibility of our proposed model extensions across fairly disparate types of classification datasets. SST contains relatively well curated, short sequence sentences, in contrast to IMDB’s comparatively colloquial and lengthy sequences (some up to 2, 000 tokens). To further differentiate the classification tasks we chose to experiment with fine-grained, five-class sentiment on SST, while IMDB only offered binary labels. For IMDB, we randomly split the training set of 25, 000 examples into training and validation sets containing 22, 500 and 2, 500 examples respectively, as done in Maas et al. (2011).
6.2 METHODOLOGY
Our objective is to show a series of compounding extensions to the standard LSTM baseline that enhance accuracy. To ensure scientific reliability, the addition of each feature is the only change from the previous model (see Figures 4 and 5). The baseline model is a 2-layer stacked LSTM with hidden size 170 for SST and 120 for IMDB, as used in Tai et al. (2015). All models in this paper used publicly available 300 dimensional word vectors, pre-trained using Glove on 840 million tokens of Common Crawl Data (Pennington et al., 2014), and both the word vectors and the subsequent weight matrices were trained using Adam with a learning rate of 10−4.
The first set of basic feature additions were adding a forget bias and using dropout. Adding a bias of 1.0 to the forget gate (i.e. adding 1.0 to the inside of the sigmoid function in equation 3) improves results across NLP tasks, especially for learning long-range dependencies (Zaremba, 2015). Dropout (Srivastava et al., 2014) is a highly effective regularizer for deep models. For SST and IMDB we used grid search to select dropout probabilities of 0.5 and 0.7 respectively, applied to the input of each layer, including the projection/softmax layer. While forget bias appears to hurt performance in Figure
5, the combination of dropout and forget bias yielded better results in all cases than dropout without forget bias. Our last two basic optimizations were increasing the hidden sizes and then adding sharedweight bidirectionality to the RNN. The hidden sizes for SST and IMDB were increased to 800 and 360 respectively; we found significantly diminishing returns to performance from increases beyond this. We chose shared-weight bidirectionality to ensure the model size did not increase any further. Specifically, the forward and backward weights are shared, and the input to the projection/softmax layer is a concatenation of the forward and backward passes’ final hidden states.
All of our subsequent proposed model extensions are described at length in their own sections. For both datasets, we used 60 Monte Carlo samples, and the embed average pooling MLP had one hidden layer and both a hidden dimension and an output dimension of 300 as the output dimension of the embed average pooling MLP. Note that although the MLP weights increased the size of their respective models, this increase is negligible (equivalent to increasing the hidden size for SST from 800 to 804 or the hidden size of IMDB from 360 to 369), and we found that such a size increase had no discernible effect on accuracy when done without the embed average pooling.
6.3 RESULTS
Since each of our proposed modifications operate independently, they are well suited to use in combination as well as in isolation. In Figures 4 and 5 we compound these features on top of the more traditional enhancements. Due to the expensiveness of bidirectional models, Figure 4 also shows these compounding features on SST with and without bidirectionality. The validation accuracy distributions show that each augmentation usually provides some small but noticeable improvement on the previous model, as measured by consistent improvements in mean and median accuracy.
We originally suspected that MC would provide marginal yet consistent improvements across datasets, while embed average pooling would especially excel for long sequences like in IMDB, where n-gram based models and deep unordered compositions have benefited from their ability to retain information from disparate parts of the text. The former hypothesis was largely confirmed. However, while embed average pooling was generally performance-enhancing, the performance boost it yielded for IMDB was not significantly larger than the one it yielded for SST, though that may have been because the other enhancements already encompassed most of the advantages provided by deep unordered compositions.
The only evident exceptions to the positive trend are the variations of residual connections. Which of Res-V1 (vertical only) and Res-V2 (vertical and residual) outperformed the other depended on the dataset and whether the network was bidirectional. The Res-V2 architecture dominated in experiments 4b and 5 while the Res-V1 (only vertical residuals) architecture is most performant in Figure 4a. This
suggests for short sequences, bidirectionality and lateral residuals conflict. Further analysis of the effect of residual connections and model depth can be found in Figure 6. In that figure, the number of parameters, and hence model size, are kept uniform by modifying the hidden size as the layer depth changed. The hidden sizes used for 1, 2, 4, 6, and 8 layer models were 250, 170, 120, 100, and 85 respectively, maintaining ≈ 550, 000 total parameters for all models. As the graph demonstrates,
normal LSTMs (“Vanilla”) perform drastically worse as they become deeper and narrower, while Res-V1 and Res-V2 both see their performance stay much steadier or even briefly rise. While depth wound up being far from a panacea for the datasets we experimented on, the ability of an LSTM with residual connections to maintain its performance as it gets deeper holds promise for other domains where the extra expressive power provided by depth might prove more crucial.
Selecting the best results for each model, we see results competitive with state-of-the-art performance for both IMDB1 and SST, even though many state-of-the-art models use either parse-tree information (Tai et al., 2015), multiple passes through the data (Kumar et al., 2016) or tremendous train and test-time computational and memory expenses (Le and Mikolov, 2014). To our knowledge, our models constitute the best performance of purely sequential, single-pass, and computationally feasible models, precisely the desired features of a solid out-of-the-box baseline. Furthermore, for SST, the compounding enhancement model without bidirectionality, the final model shown in Figure 4b, greatly exceeded the performance of the large bidirectional model (51.6% vs 50.9%), with significantly less training time (Table 1). This suggests our enhancements could provide a similarly reasonable and efficient alternative to shared-weight bidirectionality for other such datasets.
7 CONCLUSION
We explore several easy to implement enhancements to the basic LSTM network that positively impact performance. These include both fairly well established extensions (biasing the forget gate, dropout, increasing the model size, bidirectionality) and several more novel ones (Monte Carlo
1For IMDB, we benchmark only against results obtained from training exclusively on the labeled training set. Thus, we omit results from unsupervised models that leveraged the additional 50, 000 unlabeled examples, such as Miyato et al. (2016).
model averaging, embed average pooling, residual connections). We find that these enhancements improve the performance of the LSTM in classification tasks, both in conjunction or isolation, with an accuracy close to state of the art despite being more lightweight and using less information than the current state of the art models. Our results suggest that these extensions should be incorporated into LSTM baselines.

This paper presents three improvements to the standard LSTM architecture used in many neural NLP models: Monte Carlo averaging, embed average pooling, and residual connections. Each of the modifications is trivial to implement, so the paper is definitely of interest to any NLP researchers experimenting with deep learning.
With that said, I am concerned about the experiments and their results. The residual connections do not seem to consistently help performance; on SST the vertical residuals help but the lateral residuals hurt, and on IMDB it is the opposite. More fundamentally, there need to be more tasks than just sentiment analysis here. I'm not quite sure why the paper's focus is on text classification, as any NLP task using an LSTM encoder could conceivably benefit from these modifications. It would be great to see a huge variety of tasks like QA, MT, etc., which would really make the paper much stronger.
At this point, while the experiments that are included in the paper are very thorough and the analysis is interesting, there need to be more tasks to convince me that the modifications generalize, so I don't think the paper is ready for publication.


BINARY PARAGRAPH VECTORS
Recently Le & Mikolov described two log-linear models, called Paragraph Vector, that can be used to learn state-of-the-art distributed representations of documents. Inspired by this work, we present Binary Paragraph Vector models: simple neural networks that learn short binary codes for fast information retrieval. We show that binary paragraph vectors outperform autoencoder-based binary codes, despite using fewer bits. We also evaluate their precision in transfer learning settings, where binary codes are inferred for documents unrelated to the training corpus. Results from these experiments indicate that binary paragraph vectors can capture semantics relevant for various domain-specific documents. Finally, we present a model that simultaneously learns short binary codes and longer, real-valued representations. This model can be used to rapidly retrieve a short list of highly relevant documents from a large document collection.
1 INTRODUCTION
One of the significant challenges in contemporary information processing is the sheer volume of available data. Gantz & Reinsel (2012), for example, claim that the amount of digital data in the world doubles every two years. This trend underpins efforts to develop algorithms that can efficiently search for relevant information in huge datasets. One class of such algorithms, represented by, e.g., Locality Sensitive Hashing (Indyk & Motwani, 1998), relies on hashing data into short, localitypreserving binary codes (Wang et al., 2014). The codes can then be used to group the data into buckets, thereby enabling sublinear search for relevant information, or for fast comparison of data items.
In this work we focus on learning binary codes for text documents. An important work in this direction has been presented by Salakhutdinov & Hinton (2009). Their semantic hashing leverages autoencoders with sigmoid bottleneck layer to learn binary codes from a word-count bag-ofwords (BOW) representation. Salakhutdinov & Hinton demonstrated that semantic hashing codes used as an initial document filter can improve precision of TF-IDF-based retrieval. Learning from BOW, however, has its disadvantages. First, word-count representation, and in turn the learned codes, are not in itself stronger than TF-IDF. Second, BOW is an inefficient representation: even for moderate-size vocabularies BOW vectors can have thousands of dimensions. Learning fullyconnected autoencoders for such high-dimensional vectors is impractical. Salakhutdinov & Hinton restricted the BOW vocabulary in their experiments to 2000 most frequent words.
Recently several works explored simple neural models for unsupervised learning of distributed representations of words, sentences and documents. Mikolov et al. (2013) proposed log-linear models that learn distributed representations of words by predicting a central word from its context (CBOW model) or by predicting context words given the central word (Skip-gram model). The CBOW model was then extended by Le & Mikolov (2014) to learn distributed representations of documents. Specifically, they proposed Paragraph Vector Distributed Memory (PV-DM) model, in which the central word is predicted given the context words and the document vector. During training, PV-DM learns the word embeddings and the parameters of the softmax that models the conditional probability distribution for the central words. During inference, word embeddings and softmax weights are fixed, but the gradients are backpropagated to the inferred document vector.
In addition to PV-DM, Le & Mikolov studied also a simpler model, namely Paragraph Vector Distributed Bag of Words (PV-DBOW). This model predicts words in the document given only the document vector. It therefore disregards context surrounding the predicted word and does not learn word embeddings. Le & Mikolov demonstrated that paragraph vectors outperform BOW and bagof-bigrams in information retrieval task, while using only few hundreds of dimensions. These models are also amendable to learning and inference over large vocabularies. Original CBOW network used hierarchical softmax to model the probability distribution for the central word. One can also use noise-contrastive estimation (Gutmann & Hyvärinen, 2010) or importance sampling (Cho et al., 2015) to approximate the gradients with respect to the softmax logits.
An alternative approach to learning representation of sentences has been recently described by Kiros et al. (2015). Networks proposed therein, inspired by the Skip-gram model, learn to predict surrounding sentences given the center sentence. To this end, the center sentence is encoded by an encoder network and the surrounding sentences are predicted by a decoder network conditioned on the center sentence code. Once trained, these models can encode sentences without resorting to backpropagation inference. However, they learn representations at the sentence level but not at the document level.
In this work we present Binary Paragraph Vector models, an extensions to PV-DBOW and PV-DM that learn short binary codes for text documents. One inspiration for binary paragraph vectors comes from a recent work by Lin et al. (2015) on learning binary codes for images. Specifically, we introduce a sigmoid layer to the paragraph vector models, and train it in a way that encourages binary activations. We demonstrate that the resultant binary paragraph vectors significantly outperform semantic hashing codes. We also evaluate binary paragraph vectors in transfer learning settings, where training and inference are carried out on unrelated text corpora. Finally, we study models that simultaneously learn short binary codes for document filtering and longer, real-valued representations for ranking. While Lin et al. (2015) employed a supervised criterion to learn image codes, binary paragraph vectors remain unsupervised models: they learn to predict words in documents.
2 BINARY PARAGRAPH VECTOR MODELS
The basic idea in binary paragraph vector models is to introduce a sigmoid nonlinearity before the softmax that models the conditional probability of words given the context. If we then enforce binary or near-binary activations in this nonlinearity, the probability distribution over words will be conditioned on a bit vector context, rather than real-valued representation. The inference in the model proceeds like in Paragraph Vector, except the document code is constructed from the sigmoid activations. After rounding, this code can be seen as a distributed binary representation of the document.
In the simplest Binary PV-DBOW model (Figure 1) the dimensionality of the real-valued document embeddings is equal to the length of the binary codes. Despite this low dimensional representation – a useful binary hash will typically have 128 or fewer bits – this model performed surprisingly well in our experiments. Note that we cannot simply increase the embedding dimensionality in Binary
PV-DBOW in order to learn better codes: binary vectors learned in this way would be too long to be useful in document hashing. The retrieval performance can, however, be improved by using binary codes for initial filtering of documents, and then using a representation with higher capacity to rank the remaining documents by their similarity to the query. Salakhutdinov & Hinton (2009), for example, used semantic hashing codes for initial filtering and TF-IDF for ranking. A similar document
retrieval strategy can be realized with binary paragraph vectors. Furthermore, we can extend the Binary PV-DBOW model to simultaneously learn short binary codes and higher-dimensional realvalued representations. Specifically, in the Real-Binary PV-DBOW model (Figure 2) we introduce a linear projection between the document embedding matrix and the sigmoid nonlinearity. During training, we learn the softmax parameters and the projection matrix. During inference, softmax weights and the projection matrix are fixed. This way, we simultaneously obtain a high-capacity representation of a document in the embedding matrix, e.g. 300-dimensional real-valued vector, and a short binary representation from the sigmoid activations. One advantage of using the Real-Binary
PV-DBOW model over two separate networks is that we need to store only one set of softmax parameters (and a small projection matrix) in the memory, instead of two large weight matrices. Additionally, only one model needs to be trained, rather than two distinct networks.
Binary document codes can also be learned by extending distributed memory models. Le & Mikolov (2014) suggest that in PV-DM, a context of the central word can be constructed by either concatenating or averaging the document vector and the embeddings of the surrounding words. However, in Binary PV-DM (Figure 3) we always construct the context by concatenating the relevant vectors before applying the sigmoid nonlinearity. This way, the length of binary codes is not tied to the dimensionality of word embeddings.
Softmax layers in the models described above should be trained to predict words in documents given binary context vectors. Training should therefore encourage binary activations in the preceding sigmoid layers. This can be done in several ways. In semantic hashing autoencoders Salakhutdinov & Hinton (2009) added noise to the sigmoid coding layer. Error backpropagation then countered the noise, by forcing the activations to be close to 0 or 1. Another approach was used by Krizhevsky & Hinton (2011) in autoencoders that learned binary codes for small images. During the forward pass, activations in the coding layer were rounded to 0 or 1. Original (i.e. not rounded) activations were used when backpropagating errors. Alternatively, one could model the document codes with stochastic binary neurons. Learning in this case can still proceed with error backpropagation, pro-
vided that a suitable gradient estimator is used alongside stochastic activations. We experimented with the methods used in semantic hashing and Krizhevsky’s autoencoders, as well as with the two biased gradient estimators for stochastic binary neurons discussed by Bengio et al. (2013). We also investigated the slope annealing trick (Chung et al., 2016) when training networks with stochastic binary activations. From our experience, binary paragraph vector models with rounded activations are easy to train and learn better codes than models with noise-based binarization or stochastic neurons. We therefore use Krizhevsky’s binarization in our models.
3 EXPERIMENTS
To assess the performance of binary paragraph vectors, we carried out experiments on two datasets frequently used to evaluate document retrieval methods, namely 20 Newsgroups1 and a cleansed version (also called v2) of Reuters Corpus Volume 12 (RCV1). As paragraph vectors can be trained with relatively large vocabularies, we did not perform any stemming of the source text. However, we removed stop words as well as words shorter than two characters and longer than 15 characters. Results reported by Li et al. (2015) indicate that performance of PV-DBOW can be improved by including n-grams in the model. We therefore evaluated two variants of Binary PV-DBOW: one predicting words in documents and one predicting words and bigrams. Since 20 Newsgroups is a relatively small dataset, we used all words and bigrams from its documents. This amounts to a vocabulary with slightly over one million elements. For the RCV1 dataset we used words and bigrams with at least 10 occurrences in the text, which gives a vocabulary with approximately 800 thousands elements.
The 20 Newsgroups dataset comes with reference train/test sets. In case of RCV1 we used half of the documents for training and the other half for evaluation. We perform document retrieval by selecting queries from the test set and ordering other test documents according to the similarity of the inferred codes. We use Hamming distance for binary codes and cosine similarity for real-valued representations. Results are averaged over queries. We assess the performance of our models with precision-recall curves and two popular information retrieval metrics, namely mean average precision (MAP) and the normalized discounted cumulative gain at the 10th result (NDCG@10) (Järvelin & Kekäläinen, 2002). The results depend, of course, on the chosen document relevancy measure. Relevancy measure for the 20 Newsgroups dataset is straightforward: a retrieved document is relevant to the query if they both belong to the same newsgroup. However, in RCV1 each document belongs to a hierarchy of topics, making the definition of relevancy less obvious. In this case we adopted the relevancy measure used by Salakhutdinov & Hinton (2009). That is, the relevancy is calculated as the fraction of overlapping labels in a retrieved document and the query document. Overall, our selection of test datasets and relevancy measures follows Salakhutdinov & Hinton (2009), enabling comparison with semantic hashing codes.
We use AdaGrad (Duchi et al., 2011) for training and inference in all experiments reported in this work. During training we employ dropout (Srivastava et al., 2014) in the embedding layer. To facilitate models with large vocabularies, we approximate the gradients with respect to the softmax logits using the method described by Cho et al. (2015). Binary PV-DM networks use the same number of dimensions for document codes and word embeddings.
Performance of 128- and 32-bit binary paragraph vector codes is reported in Table 1 and in Figure 4. For comparison we also report performance of real-valued paragraph vectors. Note that the binary codes perform very well, despite their far lower capacity: on both test sets the 128-bit Binary PV-DBOW trained with bigrams approaches the performance of the real-valued paragraph vectors. Furthermore, Binary PV-DBOW with bigrams outperforms semantic hashing codes: comparison of precision-recall curves from Figure 4 with Salakhutdinov & Hinton (2009, Figures 6 & 7) shows that on both test sets 128-bit codes learned with this model outperform 128-bit semantic hashing codes. Moreover, the 32-bit codes from this model outperform 128-bit semantic hashing codes on the RCV1 dataset, and on the 20 Newsgroups dataset give similar precision up to approximately 3% recall and better precision for higher recall levels. Note that the difference in this case lies not only in retrieval precision: the short 32-bit Binary PV-DBOW codes are more efficient for indexing than long 128-bit semantic hashing codes.
1Available at http://qwone.com/˜jason/20Newsgroups 2Available at http://trec.nist.gov/data/reuters/reuters.html
We also compared binary paragraph vectors against codes constructed by first inferring short, realvalued paragraph vectors and then using another unsupervised model or hashing algorithm for binarization. When the dimensionality of the paragraph vectors is equal to the size of binary codes, the number of network parameters in this approach is similar to that of Binary PV models. We experimented with an autoencoder with sigmoid coding layer and Krizhevsky’s binarization, with a Gaussian-Bernoulli Restricted Boltzmann Machine (Welling et al., 2004), and with two standard hashing algorithms, namely random hyperplane projection (Charikar, 2002) and iterative quantization (Gong & Lazebnik, 2011). Paragraph vectors in these experiments were inferred using PVDBOW with bigrams. Results reported in Table 2 shows no benefit from using a separate algorithm for binarization. On the 20 Newsgroups dataset an autoencoder with Krizhevsky’s binarization achieved MAP equal to Binary PV-DBOW, while the other three approaches yielded lower MAP. On the larger RCV1 dataset an end-to-end training of Binary PV-DBOW yielded higher MAP than the baseline approaches. Some gain in precision of top hits can be observed for iterative quantization and an autoencoder with Krizhevsky’s binarization. However, it does not translate to an improved MAP, and decreases when models are trained on a larger corpus (RCV1).
Li et al. (2015) argue that PV-DBOW outperforms PV-DM on a sentiment classification task, and demonstrate that the performance of PV-DBOW can be improved by including bigrams in the vocabulary. We observed similar results with Binary PV models. That is, including bigrams in the vocabulary usually improved retrieval precision. Also, codes learned with Binary PV-DBOW provided higher retrieval precision than Binary PV-DM codes. Furthermore, to choose the context size for the Binary PV-DM models, we evaluated several networks on validation sets taken out of the training data. The best results were obtained with a minimal one-word, one-sided context window. This is the distributed memory architecture most similar to the Binary PV-DBOW model.
3.1 TRANSFER LEARNING
In the experiments presented thus far we had at our disposal training sets with documents similar to the documents for which we inferred binary codes. One could ask a question, whether binary paragraph vectors could be used without collecting a domain-specific training set? For example, what if we needed to hash documents that are not associated with any available domain-specific corpus? One solution could be to train the model with a big generic text corpus, that covers a wide variety of domains. It is not obvious, however, whether such model would capture language semantics meaningful for unrelated documents. To shed light on this question we trained Binary PVDBOW with bigrams on the English Wikipedia, and then inferred binary codes for the test parts of the 20 Newsgroups and RCV1 datasets. We used words and bigrams with at least 100 occurrences in the English Wikipedia. The results are presented in Table 3 and in Figure 5. The model trained on an unrelated text corpus gives lower retrieval precision than models with domain-specific training sets, which is not surprising. However, it still performs remarkably well, indicating that the semantics it captured can be useful for different text collections. Importantly, these results were obtained without domain-specific finetuning.
Table 3: Information retrieval results for the Binary PV-DBOW model trained on an unrelated text corpus. Results are reported for 128-bit codes.
MAP NDCG@10 20 Newsgroups 0.24 0.51
RCV1 0.18 0.66
10-2 10-1 100
Recall
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
P re
ci si
o n
training on the 20 Newsgroups training set training on English Wikipedia
(a) 20 Newsgroups
10-2 10-1 100
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
P re ci si o n
training on the RCV1 training set training on English Wikipedia
(b) RCV1
Figure 5: Precision-recall curves for the baseline Binary PV-DBOW models and a Binary PVDBOW model trained on an unrelated text corpus. Results are reported for 128-bit codes.
3.2 RETRIEVAL WITH REAL-BINARY MODELS
As pointed out by Salakhutdinov & Hinton (2009), when working with large text collections one can use short binary codes for indexing and a representation with more capacity for ranking. Following this idea, we proposed Real-Binary PV-DBOW model (Section 2) that can simultaneously learn short binary codes and high-dimensional real-valued representations. We begin evaluation of this model by comparing retrieval precision of real-valued and binary representations learned by it. To this end, we trained a Real-Binary PV-DBOW model with 28-bit binary codes and 300-dimensional real-valued representations on the 20 Newsgroups and RCV1 datasets. Results are reported in Figure 6. The real-valued representations learned with this model give lower precision than PV-DBOW vectors but, importantly, improve precision over binary codes for top ranked documents. This justifies their use alongside binary codes.
Using short binary codes for initial filtering of documents comes with a tradeoff between the retrieval performance and the recall level. For example, one can select a small subset of similar documents by using 28–32 bit codes and retrieving documents within small Hamming distance to the query. This will improve retrieval performance, and possibly also precision, at the cost of recall. Conversely, short codes provide a less fine-grained hashing and can be used to index documents within larger Hamming distance to the query. They can therefore be used to improve recall at the cost of retrieval performance, and possibly also precision. For these reasons, we evaluated Real-Binary PV-DBOW models with different code sizes and under different limits on the Hamming distance to the query. In general, we cannot expect these models to achieve 100% recall under the test settings. Furthermore, recall will vary on query-by-query basis. We therefore decided to focus on the NDCG@10 metric in this evaluation, as it is suited for measuring model performance when a short list of relevant documents is sought, and the recall level is not known. MAP and precision-recall curves are not applicable in these settings.
Information retrieval results for Real-Binary PV-DBOW are summarized in Table 4. The model gives higher NDCG@10 than 32-bit Binary PV-DBOW codes (Table 1). The difference is large when the initial filtering is restrictive, e.g. when using 28-bit codes and 2-bit Hamming distance limit. Real-Binary PV-DBOW can therefore be useful when one needs to quickly find a short list of relevant documents in a large text collection, and the recall level is not of primary importance.
If needed, precision can be further improved by using plain Binary PV-DBOW codes for filtering and standard DBOW representation for raking (Table 4, column C). Note, however, that PV-DBOW model would then use approximately 10 times more parameters than Real-Binary PV-DBOW.
4 CONCLUSION
In this article we presented simple neural networks that learn short binary codes for text documents. Our networks extend Paragraph Vector by introducing a sigmoid nonlinearity before the softmax that predicts words in documents. Binary codes inferred with the proposed networks achieve higher retrieval precision than semantic hashing codes on two popular information retrieval benchmarks. They also retain a lot of their precision when trained on an unrelated text corpus. Finally, we presented a network that simultaneously learns short binary codes and longer, real-valued representations.
The best codes in our experiments were inferred with Binary PV-DBOW networks. The Binary PVDM model did not perform so well. Li et al. (2015) made similar observations for Paragraph Vector models, and argue that in distributed memory model the word context takes a lot of the burden of predicting the central word from the document code. An interesting line of future research could,
therefore, focus on models that account for word order, while learning good binary codes. It is also worth noting that Le & Mikolov (2014) constructed paragraph vectors by combining DM and DBOW representations. This strategy may proof useful also with binary codes, when employed with hashing algorithms designed for longer codes, e.g. with multi-index hashing (Norouzi et al., 2012).
ACKNOWLEDGMENTS
This research is supported by the Polish National Science Centre grant no. DEC-2013/09/B/ST6/01549 “Interactive Visual Text Analytics (IVTA): Development of novel, user-driven text mining and visualization methods for large text corpora exploration.” This research was carried out with the support of the “HPC Infrastructure for Grand Challenges of Science and Engineering” project, co-financed by the European Regional Development Fund under the Innovative Economy Operational Programme. This research was supported in part by PL-Grid Infrastructure.

The method in this paper introduces a binary encoding level in the PV-DBOW and PV-DM document embedding methods (from Le & Mikolov'14). The binary encoding consists in a sigmoid with trained parameters that is inserted after the standard training stage of the embedding.

For a document to encode, the binary vector is obtained by forcing the sigmoid to output a binary output for each of the embedding vector components. The binary vector can then be used for compact storage and fast comparison of documents.

Pros:

- the binary representation outperforms the Semantic hashing method from Salakhutdinov & Hinton '09

- the experimental approach sound: they compare on the same experimental setup as Salakhutdinov & Hinton '09, but since in the meantime document representations improved (Le & Mikolov'14), they also combine this new representation with an RBM to show the benefit of their binary PV-DBOW/PV-DM

Cons:

- the insertion of the sigmoid to produce binary codes (from Lin & al. '15) in the training process is incremental

- the explanation is too abstract and difficult to follow for a non-expert (see details below)

- a comparison with efficient indexing methods used in image retrieval is missing. For large-scale indexing of embedding vectors, derivations of the Inverted multi-index are probably more interesting than binary codes. See eg. Babenko & Lempitsky, Efficient Indexing of Billion-Scale Datasets of Deep Descriptors, CVPR'16

Detailed comments:

Section 1: the motivation for producing binary codes is not given. Also, the experimental section could give some timings and mem usage numbers to show the benefit of binary embeddings

figure 1, 2, 3: there is enough space to include more information on the representation of the model: model parameters + training objective + characteristic sizes + dropout. In particular, in fig 2, it is not clear why "embedding lookup" and "linear projection" cannot be merged in a single smaller lookup table (presumably because there is an intermediate training objective that prevents this).

p2: "This way, the length of binary codes is not tied to the dimensionality of word embeddings." -> why not?

section 3: This is the experimental setup of  Salakhutdinov & Hinton 2009. Specify this and whether there is any difference between the setups.

"similarity of the inferred codes": say here that codes are compared using Hamming distances.

"binary codes perform very well, despite their far lower capacity" -> do you mean smaller size than real vectors?

fig 5: these plots could be dropped if space is needed.

section 3.1: one could argue that "transferring" from Wikipedia to anything else cannot be called transferring, since Wikipedia's purpose is to include all topics and lexical domains

section 3.2: specify how the 300D real vectors are compared. L2 distance? inner product?

fig4: specify what the raw performance of the large embedding vectors is (without pre-filtering with binary codes), or equivalently, the perf of (code-size, Hamming dis) = (28, 28), (24, 24), etc.