LOW-RANK PASSTHROUGH NEURAL NETWORKS
Deep learning consists in training neural networks to perform computations that sequentially unfold in many steps over a time dimension or an intrinsic depth dimension. For large depths, this is usually accomplished by specialized network architectures that are designed to mitigate the vanishing gradient problem, e.g. LSTMs, GRUs, Highway Networks and Deep Residual Networks, which are based on a single structural principle: the state passthrough. We observe that these "Passthrough Networks" architectures enable the decoupling of the network state size from the number of parameters of the network, a possibility that is exploited in some recent works but not thoroughly explored. In this work we propose simple, yet effective, low-rank and low-rank plus diagonal matrix parametrizations for Passthrough Networks which exploit this decoupling property, reducing the data complexity and memory requirements of the network while preserving its memory capacity. We present competitive experimental results on several tasks, including a near state of the art result on sequential randomly-permuted MNIST classification, a hard task on natural data.
1 OVERVIEW
Deep neural networks can perform non-trivial computations by the repeated the application of parametric non-linear transformation layers to vectorial (or, more generally, tensorial) data. This staging of many computation steps can be done over a time dimension for tasks involving sequential inputs or outputs of varying length, yielding a recurrent neural network, or over an intrinsic circuit depth dimension, yielding a deep feed-forward neural network, or both. Training these deep models is complicated by the exploding and vanishing gradient problems (Hochreiter, 1991; Bengio et al., 1994).
Various network architectures have been proposed to ameliorate the vanishing gradient problem in the recurrent setting, such as the LSTM (Hochreiter & Schmidhuber, 1997; Graves & Schmidhuber, 2005), the GRU (Cho et al., 2014b), etc. These architectures led to a number of breakthroughs in different tasks in NLP, computer vision, etc. (Graves et al., 2013; Cho et al., 2014a; Bahdanau et al., 2014; Vinyals et al., 2014; Iyyer et al., 2014). Similar methods have also been applied in the feed-forward setting with architectures such as Highway Networks (Srivastava et al., 2015), Deep Residual Networks (He et al., 2015), and so on. All these architectures are based on a single structural principle which, in this work, we will refer to as the state passthrough. We will thus refer to these architectures as Passthrough Networks.
Another difficulty in training neural networks is the trade-off between the network representation power and its number of trainable parameters, which affects its data complexity during training in addition to its implementation memory requirements. On one hand, the number of parameters can be thought as the number of tunable "knobs" that need to be set to represent a function, on the other hand, it also constrains the size of the partial results that are propagated inside the network. In typical fully connected networks, a layer acting on a n-dimensional state vector has O(n2) parameters stored in one or more matrices, but there can be many functions of practical interest that are simple enough to be represented by a relatively small number of bits while still requiring some sizable amount of memory to be computed. Therefore, representing these functions on a fully connected neural network
∗Work partially done while affiliated with University of Pisa.
can be wasteful in terms of number of parameters. The full parameterization implies that, at each step, all the information in each state component can affect all the information in any state component at the next step. Classical physical systems, however, consist of spatially separated parts with primarily local interactions, long-distance interactions are possible but they tend to be limited by propagation delays, bandwidth and noise. Therefore it may be beneficial to bias our model class towards models that tend to adhere to these physical constraints by using a parametrization which reduces the number of parameters required to represent them. This can be accomplished by imposing some constraints on the n × n matrices that parametrize the state transitions. One way of doing this is to impose a convolutional structure on these matrices (LeCun et al., 2004; Krizhevsky et al., 2012), which corresponds to strict locality and periodicity constraints as in a cellular automaton. These constraints work well in certain domains such as vision, but may be overly restrictive in other domains.
In this work we observe that the state passthrough allows for a systematic decoupling of the network state size from the number of parameters: since by default the state vector passes mostly unaltered through the layers, each layer can be made simple enough to be described only by a small number of parameters without affecting the overall memory capacity of the network, effectively spreading the computation over the depth or time dimension of the network, but without making the network "thin". This has been exploited by some convolutional passthrough architectures (Srivastava et al., 2015; He et al., 2015; Kaiser & Sutskever, 2015), or architectures with addressable read-write memory (Graves et al., 2014; Danihelka et al., 2016).
In this work we propose simple but effective low-dimensional parametrizations that exploit this decoupling based on low-rank or low-rank plus diagonal matrix decompositions. Our approach extends the LSTM architecture with a single projection layer by Sak et al. (2014) which has been applied to speech recognition, natural language modeling (Józefowicz et al., 2016), video analysis (Sun et al., 2015), etc. We provide experimental evaluation of our approach on GRU and LSTM architectures on various machine learning tasks, including a near state of the art result for the hard task of sequential randomly-permuted MNIST image recognition (Le et al., 2015).
2 MODEL
A neural network can be described as a dynamical system that transforms an input u into an output y over multiple time steps T . At each step t the network has a n-dimensional state vector x(t) ∈ Rn defined as
x(t) = { in(u, θ) if t = 0 f(x(t− 1), t, u, θ) if t ≥ 1 (1)
where in is a state initialization function, f is a state transition function and θ ∈ Rk is vector of trainable parameters. The output y = out(x(0 : T ), θ) is generated by an output function out, where x(0 : T ) denotes the whole sequence of states visited during the execution. In a feed-forward neural network with constant hidden layer width n, the input u ∈ Rm and the output y ∈ Rl are vectors of fixed dimension m and l respectively, T is a model hyperparameter. In a recurrent neural network the input u is typically a list of T m-dimensional vectors u(t) ∈ Rm for t ∈ 1, . . . , T where T is variable, the output y is either a single l-dimensional vector or a list of T such vectors. Other neural architectures, such as "seq2seq" transducers without attention (Cho et al., 2014a), can be also described within this framework.
2.1 PASSTHROUGH NETWORKS
Passthrough networks can be defined as networks where the state transition function f has a special form such that, at each step t the state vector x(t) (or a sub-vector x̂(t)) is propagated to the next step modified only by some (nearly) linear, element-wise transformation.
Let the state vector x(t) ≡ (x̂(t), x̃(t)) be the concatenation of x̂(t) ∈ Rn̂ and x̃(t) ∈ Rñ with n̂+ ñ = n (where ñ can be equal to zero). We define a network to have a state passthrough on x̂ if x̂ evolves as
x̂(t) = fπ(x(t− 1), t, u, θ) fτ (x(t− 1), t, u, θ) + x̂(t− 1) fγ(x(t− 1), t, u, θ) (2) where fπ is the next state proposal function, fτ is the transform function, fγ is the carry function and denotes element-wise vector multiplication. The rest of the state vector x̃(t), if present, evolves
according to some other function f̃ . In practice x̃(t) is only used in LSTM variants, while in other passthrough architectures x̂(t) = x(t).
As concrete example, we can describe a fully connected Highway Network as
fπ(x(t− 1), t, u, θ) = g(θ(Wπ)t · x(t− 1) + θ (bπ) t )
fτ (x(t− 1), t, u, θ) = σ(θ(Wτ )t · x(t− 1) + θ (bτ ) t ) fγ(x(t− 1), t, u, θ) = 1⊗n − fτ (x(t− 1), t, u, θ)
(3)
where g is an element-wise activation function, usually the ReLU (Glorot et al., 2011) or the hyperbolic tangent, σ is the element-wise logistic sigmoid, and ∀t ∈ 1, . . . , T , the parameters θ(Wπ)t and θ(Wτ )t are matrices inRn×n and θ (bπ) t and θ (bπ) t are vectors inRn. Dependence on the input u occurs only through the initialization function, which is model-specific and is omitted here, as is the output function.
2.2 LOW-RANK PASSTHROUGH NETWORKS
In fully connected architectures there are n × n matrices that act on the state vector, such as the θ (Wπ) t and θ (Wτ ) t matrices of the Highway Network of eq. 3. Each of these matrices has n
2 entries, thus for large n, the entries of these matrices can make up the majority of independently trainable parameters of the model. As discussed in the previous section, this parametrization can be wasteful. We impose a low-rank constraint on these matrices. This is easily accomplished by rewriting each of these matrices as the product of two matrices where the inner dimension d is a model hyperparameter.
For instance, in the case of the Highway Network of eq. 3 we can redefine ∀t ∈ 1, . . . , T
θ (Wπ) t = θ (Lπ) t · θ (Rπ) t
θ (Wτ ) t = θ (Lτ ) t · θ (Rτ ) t
(4)
where θ(Lπ)t , θ (Lτ ) t ∈ Rn×d and θ (Rπ) t , θ (Rτ ) t ∈ Rd×n. When d < n/2 this result in a reduction of the number of trainable parameters of the model.
Even when n/2 ≤ d < n, while the total number of parameter increases, the number of degrees of freedom of the model still decreases, because low-rank factorization are unique only up to arbitrary d× d invertible matrices, thus the number of independent degrees of freedom of a low-rank layer is
2nd−d2. However, we don’t know whether the training optimizers can exploit this kind of redundancy, thus in this work we restrict to low-rank parametrizations where the number of parameters is strictly reduced.
This low-rank constraint can be thought as a bandwidth constraint on the computation performed at each step: the R matrices first project the state into a smaller subspace, extracting the information needed for that specific step, then the L matrices project it back to the original state space, spreading the selected information to all the state components that need to be updated. A similar approach has been proposed for the LSTM architecture by Sak et al. (2014), although they force the R matrices to be the same for all the functions of the state transition, while we allow each parameter matrix to be parametrized independently by a pair of R and L matrices.
Low-rank passthrough architectures are universal in that they retain the same representation classes of their parent architectures. This is trivially true if the inner dimension d is allowed to be O(n) in the worst case, and for some architectures even if d is held constant. For instance, it is easily shown that for any Highway Network with state size n and T hidden layers and for any > 0, there exist a Low-rank Highway Network with d = 1, state size at most 2n and at most nT layers that computes the same function within an margin of error.
2.3 LOW-RANK PLUS DIAGONAL PASSTHROUGH NETWORKS
As we show in the experimental section, on some tasks the low-rank constraint may prove to be excessively restrictive if the goal is to train a model with fewer parameters than one with arbitrary matrices. A simple extension is to add to each low-rank parameter matrix a diagonal parameter matrix, yielding a matrix that is full-rank but still parametrized in a low-dimensional space. For instance, for the Highway Network architecture we modify eq. 4 to
θ (Wπ) t = θ (Lπ) t · θ (Rπ) t + θ (Dπ) t
θ (Wτ ) t = θ (Lτ ) t · θ (Rτ ) t + θ (Dτ ) t
(5)
where θ(Dπ)t , θ (Dτ ) t ∈ Rn×n are trainable diagonal parameter matrices.
It may seem that adding diagonal parameter matrices is redundant in passthrough networks. After all, the state passthrough itself can be considered as a diagonal matrix applied to the state vector, which is then additively combined to the new proposed state computed by the fπ function. However, since the state passthrough completely skips over all non-linear activation functions, these formulations are not equivalent. In particular, the low-rank plus diagonal parametrization may help in recurrent neural networks which receive input at each time step, since they allow each component of the state vector to directly control how much input signal is inserted into it at each step. We demonstrate the effectiveness of this model in the sequence copy and sequential MNIST tasks described in the experiments section.
3 EXPERIMENTS
The main content of this section reports several experiments on Low-rank and Low-rank plus diagonal GRUs, and an experiment using these parametrizations on a LSTM for language modeling.
A preliminary experiment on Low-rank Highway Networks on the MNIST dataset is reported in appendix A.1.
We applied the Low-rank and Low-rank plus diagonal GRU architectures to a subset of sequential benchmarks described in the Unitary Evolution Recurrent Neural Networks article by Arjovsky et al. (2015), specifically the memory task, the addition task and the sequential randomly permuted MNIST task. For the memory tasks, we also considered two different variants proposed by Danihelka et al. (2016) and Henaff et al. (2016) which are hard for the uRNN architecture. We chose to compare against the uRNN architecture because it set state of the art results in terms of both data complexity and accuracy and because it is an architecture with similar design objectives as low-rank passthrough architectures, namely a low-dimensional parametrization and the mitigation of the vanishing gradient problem, but it is based on quite different principles.
The GRU architecture (Cho et al., 2014b) is a passthrough recurrent neural network defined as
in(u, θ) = θin
fω(x(t− 1), t, u, θ) = σ(θUω · u(t) + θ(Wω) · x(t− 1) + θ(bω)) fγ(x(t− 1), t, u, θ) = σ(θUγ · u(t) + θ(Wγ) · x(t− 1) + θ(bγ)) fτ (x(t− 1), t, u, θ) = 1⊗n − fγ(x(t− 1), t, u, θ) fπ(x(t− 1), t, u, θ) = tanh(θUπ · u(t) + θ(Wπ) · (x(t− 1) fω(x(t− 1), t, u, θ)) + θ(bπ)) (6)
We turn this architecture into the Low-rank GRU architecture by redefining each of the θW matrices as the product of two matrices with inner dimension d. For the memory tasks, which turned out to be difficult for the low-rank parametrization, we also consider the low-rank plus diagonal parametrization. We also applied the low-rank plus diagonal parametrization in the sequential permuted MNIST task and a character-level language modeling task on the Penn Treebank corpus. For the language modeling task, we also experimented with Low-rank plus diagonal LSTMs. Refer to appendix A.2 for model details.
3.0.1 MEMORY TASK
The input of an instance of this task is a sequence of T = N + 20 discrete symbols in a ten symbol alphabet ai : i ∈ 0, . . . 9, encoded as one-hot vectors. The first 10 symbols in the sequence are "data" symbols i.i.d. sampled from a0, . . . , a7, followed by N − 1 "blank" a8 symbols, then a distinguished "run" symbol a9, followed by 10 more "blank" a8 symbols. The desired output sequence consists of N + 10 "blank" a8 symbols followed by the 10 "data" symbols as they appeared in the input sequence. Therefore the model has to remember the 10 "data" symbol string over the temporal gap of size N , which is challenging for a recurrent neural network when N is large. In our experiment we set N = 500, which is the hardest setting explored in the uRNN work. The training set consists of 100, 000 training examples and 10, 000 validation/test examples. The architecture is described by eq. (6), with an additional output layer with a dense n× 10 matrix followed a (biased) softmax. We train to minimize the cross-entropy loss.
We were able to solve this task using a GRU with full recurrent matrices with state size n = 128, learning rate 1× 10−3, mini-batch size 20, initial bias of the carry functions (the "update" gates) 4.0, however this model has many more parameters, nearly 50, 000 in the recurrent layer only, than the uRNN work which has about 6, 500, and it converges much more slowly than the uRNN. We were not able to achieve convergence with a pure low-rank model without exceeding the number of parameters of the fully connected model, but we achieved fast convergence with a low-rank plus diagonal model with d = 50, with other hyperparameters set as above. This model has still more parameters (39, 168 in the recurrent layer, 41, 738 total) than the uRNN model and converges more slowly but still reasonably fast, reaching test cross-entropy < 1× 10−3 nats and almost perfect classification accuracy in less than 35, 000 updates.
In order to obtain a fair comparison, we also train a uRNN model with state size n = 721, resulting in approximately the same number of parameters as the low-rank plus diagonal GRU models. This model very quickly reaches perfect accuracy on the training set in less than 2, 000 updates, but overfits w.r.t. the test set.
We also consider two variants of this task which are difficult for the uRNN model. For both these tasks we used the same settings as above except that the task size parameter is set at N = 100 for consistency with the works that introduced these variants. In the variant of Danihelka et al. (2016), the length of the sequence to be remembered is randomly sampled between 1 and 10 for each sequence. They manage to achieve fast convergence with their Associative LSTM architecture with 65, 505 parameters, and slower convergence with standard LSTM models. Our low-rank plus diagonal GRU architecture, which has less parameters than their Associative LSTM, performs comparably or better, reaching test cross-entropy < 1× 10−3 nats and almost perfect classification accuracy in less than 30, 000 updates. In the variant of Henaff et al. (2016), the length of the sequence to be remembered is fixed at 10 but the model is expected to copy it after a variable number of time steps randomly chosen, for each sequence, between 1 and N = 100. The authors achieve slow convergence with a standard LSTM model, while our low-rank plus diagonal GRU architecture achieves fast convergence,
reaching test cross-entropy < 1× 10−3 nats and almost perfect classification accuracy in less than 38, 000 updates, and perfect test accuracy in 87, 000 updates.
We further train uRNN models with state size n = 721 on these variants of the memory task. We found that the uRNN learns faster than the low-rank plus diagonal GRU on the variable length, fixed lag task (Danihelka et al., 2016) but fails to converge within our training time limit on the fixed length, variable lag task (Henaff et al., 2016).
Training the low-rank plus diagonal GRU on these tasks incurs sometimes in numerical stability problems as discussed in appendix A.2. In order to systemically address these issues, we also trained models with weight normalization (Salimans & Kingma, 2016) and weight row max-norm constraints. These models turned out to be more stable and in fact converge faster, performing on par with the uRNN on the variable length, fixed lag task.
Training curves are shown in figure 2 (top and middle left).
3.0.2 ADDITION TASK
For each instance of this task, the input sequence has length T and consists of two real-valued components, at each step the first component is independently sampled from the interval [0, 1] with uniform probability, the second component is equal to zero everywhere except at two randomly chosen time step, one in each half of the sequence, where it is equal to one. The result is a single real value computed from the final state which we want to be equal to the sum of the two elements of the first component of the sequence at the positions where the second component was set at one. In our experiment we set T = 750.
The training set consists of 100, 000 training examples and 10, 000 validation/test examples. We use a Low-rank GRU with 2× n input matrix, n× 1 output matrix and (biased) identity output activation. We train to minimize the mean squared error loss. We use state size n = 128, maximum rank d = 24. This results in approximately 6, 140 parameters in the recurrent hidden layer. Learning rate was set at 1× 10−3, mini-batch size 20, initial bias of the carry functions (the "update" gates) was set to 4. We trained on 14, 500 mini-batches, obtaining a mean squared error on the test set of 0.003, which is a better result than the one reported in the uRNN article, in terms of training time and final accuracy. The training curve is shown in figure 2 (middle right).
3.0.3 SEQUENTIAL MNIST TASK
This task consists of handwritten digit classification on the MNIST dataset with the caveat that the input is presented to the model one pixel value at time, over T = 784 time steps. To further increase the difficulty of the task, the inputs are reordered according to a random permutation (fixed for all the task instances).
We use Low-rank and Low-rank plus diagonal GRUs with 1× n input matrix, n× 10 output matrix and (biased) softmax output activation. Learning rate was set at 5× 10−4, mini-batch size 20, initial bias of the carry functions (the "update" gates) was set to 5.
Results are presented in table 1 and training curves are shown in figure 2 (bottom row). All these models except the one with the most extreme bottleneck (n = 512, d = 4) exceed the reported uRNN test accuracy of 91.4%, although they converge more slowly (hundred of thousands updates vs. tens of thousands of the uRNN). Also note that the low-rank plus diagonal GRU is more accurate than the
full rank GRU with the same state size, while the low-rank GRU is slightly less accurate (in terms of test accuracy), indicating the utility of the diagonal component of the parametrization for this task.
These are on par with more complex architectures with time-skip connections (Zhang et al., 2016) (reported test set accuracy 94.0%). To our knowledge, at the time of this writing, the best result on this task is the LSTM with recurrent batch normalization by Cooijmans et al. (2016) (reported test set accuracy 95.2%). The architectural innovations of these works are orthogonal to our own and in principle they can be combined to it.
3.0.4 CHARACTER-LEVEL LANGUAGE MODELING TASK
This standard benchmark task consist of predicting the probability of the next character in a sentence after having observed the previous charters. Similar to Zaremba et al. (2014), we use the Penn Treebank English corpus, with standard training, validation and test splits.
As a baseline we use a single layer GRU either with no regularization or regularized with Bayesian recurrent dropout (Gal, 2015). Refer to appendix A.2 for details.
In our experiments we consider the low-rank plus diagonal parametrization, both with tied and untied projection matrices. We set the state size and maximum rank to either reduce the total number of parameters compared to the baselines or to keep the number of parameters approximately the same while increasing the memory capacity. Results are shown in table 2.
Our low-rank plus diagonal parametrization reduces the model per-character perplexity (the base-2 exponential of the bits-per-character entropy). Both the tied and untied versions perform equally when the state size is the same, but the tied version performs better when the number of parameters is kept the same, presumably due to the increased memory capacity of the state vector. Our best model has an extreme bottleneck, over a hundred of times smaller than the state size, while the word-level language models trained by Józefowicz et al. (2016) use bottlenecks of four to eight times smaller than the state size. We conjecture that this difference is due to our usage of the "plus diagonal" parametrization. In terms of absolute perplexity, our results are worse than published ones (e.g. Graves (2013)), although they may not be directly comparable since published results generally use different training and evaluation schemes, such as preserving the network state between different sentences.
In order to address these experimental differences, we ran additional experiments using LSTM architectures, trying to replicate the alphabet and sentence segmentation used in Graves (2013), although we could not obtain the same baseline performance even using the Adam optimizer (using SGD+momentum yields even worse results). In fact, we obtained approximately the same perplexity as our baseline GRU model with the same state size.
We applied the Low-rank plus diagonal parametrizations to our LSTM architecture maintaining the same number of parameters as the baseline. We obtained notable perplexity improvements over the baseline. Refer to appendix A.3 for the experimental details.
We performed additional exploratory experiments on word-level language modeling and subwordlevel neural machine translation (Bahdanau et al., 2014; Sennrich et al., 2015) with GRU-based architectures but we were not able to achieve significant accuracy improvements, which is not particularly surprising given that in these models most parameters are contained in the token embedding and output matrices, thus low-dimensional parametrizations of the recurrent matrices have little effect on the total number of parameters. We reserve experimentation on character-level neural machine translation (Ling et al., 2015; Chung et al., 2016; Lee et al., 2016) to future work.
4 CONCLUSIONS AND FUTURE WORK
We proposed low-dimensional parametrizations for passthrough neural networks based on low-rank or low-rank plus diagonal decompositions of the n × n matrices that occur in the hidden layers. We experimentally compared our models with state of the art models, obtaining competitive results including a near state of the art for the randomly-permuted sequential MNIST task.
Our parametrizations are alternative to convolutional parametrizations explored by Srivastava et al. (2015); He et al. (2015); Kaiser & Sutskever (2015). Since our architectural innovations are orthogonal to these approaches, they can be in principle combined. Additionally, alternative parametrizations could include non-linear activation functions, similar to the network-in-network approach of Lin et al. (2013). We leave the exploration of these extensions to future work.
A APPENDIX: EXPERIMENTAL DETAILS
A.1 LOW-RANK HIGHWAY NETWORKS
As a preliminary exploratory experiment, we applied the low-rank and low-rank plus diagonal Highway Network architecture to the classic benchmark task of handwritten digit classification on the MNIST dataset, in its permutation-invariant (i.e. non-convolutional) variant.
We used the low-rank architecture described by equations 3 and 4, with T = 5 hidden layers, ReLU activation function, state dimension n = 1024 and maximum rank (internal dimension) d = 256. The input-to-state layer is a dense 784× 1024 matrix followed by a (biased) ReLU activation and the state-to-output layer is a dense 1024× 10 matrix followed by a (biased) identity activation. We did not use any convolution layer, pooling layer or data augmentation technique. We used dropout (Srivastava et al., 2014) in order to achieve regularization. We further applied L2-regularization with coefficient λ = 1× 10−3 per example on the hidden-to-output parameter matrix. We also used batch normalization (Ioffe & Szegedy, 2015) after the input-to-state matrix and after each parameter matrix in the hidden layers. Initial bias vectors are all initialized at zero except for those of the transform functions in the hidden layers, which are initialized at −1.0. We trained to minimize the sum of the per-class L2-hinge loss plus the L2-regularization cost (Tang, 2013). Optimization was performed using Adam (Kingma & Ba, 2014) with standard hyperparameters, learning rate starting at 3× 10−3 halving every three epochs without validation improvements. Mini-batch size was equal to 100. Code is available online1.
We obtained perfect training accuracy and 98.83% test accuracy. While this result does not reach the state of the art for this task (99.13% test accuracy with unsupervised dimensionality reduction reported by Tang (2013)), it is still relatively close. We also tested the low-rank plus diagonal Highway Network architecture of eq. 5 with the same settings as above, obtaining a test accuracy of 98.64%. The inclusion of diagonal parameter matrices does not seem to help in this particular task.
A.2 LOW-RANK GRUS
In our experiments (except language modeling) we optimized using RMSProp (Tieleman & Hinton, 2012) with gradient component clipping at 1. Code is available online2. Our code is based on the published uRNN code3 (specifically, on the LSTM implementation) by the original authors for the sake of a fair comparison. In order to achieve convergence on the memory task however, we had to slightly modify the optimization procedure, specifically we changed gradient component clipping with gradient norm clipping (with NaN detection and recovery), and we added a small = 1× 10−8 term in the parameter update formula. No modifications of the original optimizer implementation were required for the other tasks.
In order to address the numerical instability issues in the memory tasks, we also consider a variant of our Low-rank plus diagonal GRU where apply weight normalization as described by Salimans & Kingma (2016) to all the parameter matrices except the output one and the diagonal matrices. All
1https://github.com/Avmb/lowrank-highwaynetwork 2https://github.com/Avmb/lowrank-gru 3https://github.com/amarshah/complex_RNN
these matrices have trainable scale parameters, except for the projection matrices. We further apply an hard constraint on the matrices row norms by clipping them at 10 after each update. We disable NaN detection and recovery during training. The rationale behind this approach, in addition to the general benefits of normalization, is that the low-rank parametrization potentially introduces stability issues because the model is invariant to multiplying a row of an R-matrix by a scalar s and dividing the corresponding column of the L-matrix by s, which in principle allows the parameters of either matrix to grow very large in magnitude, eventually resulting in overflows or other pathological behavior. The weight row max-norm constraint can counter this problem. But the constraint alone could make the optimization problem harder by reducing and distorting the parameter space. Fortunately we could counter this by weight normalization which makes the model invariant to the row-norms of the parameter matrices.
In the language modeling experiment, for consistency with existing code, we used a variant of the GRU where the reset gate is applied after the multiplication by the recurrent proposal matrix rather than before. Specifically:
in(u, θ) = θin
fω(x(t− 1), t, u, θ) = σ(θUω · u(t) + θ(Wω) · x(t− 1) + θ(bω)) fγ(x(t− 1), t, u, θ) = σ(θUγ · u(t) + θ(Wγ) · x(t− 1) + θ(bγ)) fτ (x(t− 1), t, u, θ) = 1⊗n − fγ(x(t− 1), t, u, θ) fπ(x(t− 1), t, u, θ) = tanh(θUπ · u(t) + (θ(Wπ) · x(t− 1)) fω(x(t− 1), t, u, θ) + θ(bπ)) (7)
The character vocabulary size if 51, we use no character embeddings. Training is performed with Adam with learning rate 1× 10−3. Bayesian recurrent dropout was adapted from the original LSTM architecture of Gal (2015) to the GRU architecture as in Sennrich et al. (2016).
Our implementation is based on the "dl4mt" tutorial4 and the Nematus neural machine translation system 5. The code is available online 6.
A.3 LOW-RANK LSTMS
For our LSTM experiments, we modified the implementation of LSTM language model with Bayesian recurrent dropout by Gal (2015)7. In order to match the setup of Graves (2013) more closely, we used a vocabulary size of 49, no embedding layer and one LSTM layer. We found no difference on the baseline model with using peephole connections and not using them, therefore we did not use them on the Low-rank plus diagonal model. We use recurrent dropout and the Adam optimizer with learning rate 2× 10−4. The baseline LSTM model is defined by the gates:
in(u, θ) = 0⊗n̂
fω(x(t− 1), t, u, θ) = σ(θUω · u(t) + θ(Wω) · x̃(t− 1) + θ(bω)) fγ(x(t− 1), t, u, θ) = σ(θUγ · u(t) + θ(Wγ) · x̃(t− 1) + θ(bγ)) fτ (x(t− 1), t, u, θ) = σ(θUτ · u(t) + θ(Wτ ) · x̃(t− 1) + θ(bτ )) fπ(x(t− 1), t, u, θ) = tanh(θUπ · u(t) + θ(Wπ) · x̃(t− 1) + θ(bπ))
(8)
with the state components evolving as: x̂(t) = fπ(x(t− 1), t, u, θ) fτ (x(t− 1), t, u, θ) + x̂(t− 1) fγ(x(t− 1), t, u, θ) x̃(t) = fω(x(t− 1), t, u, θ) tanh(x̂(t))
(9)
The low-rank plus diagonal parametrization is applied on the recurrence matrices θW? as in the GRU models.
The code is available online8. 4https://github.com/nyu-dl/dl4mt-tutorial 5https://github.com/rsennrich/nematus 6https://github.com/Avmb/dl4mt-lm/tree/master/lm 7https://github.com/yaringal/BayesianRNN 8https://github.com/Avmb/lowrank-lstm

The author proposes the use of low-rank matrix in feedfoward and RNNs. In particular, they try their approach in a GRU and a feedforward highway network.
Author also presents as a contribution the passthrough framework, which can describe feedforward and recurrent networks. However, this framework seems hardly novel, relatively to the formalism introduced by LSTM or highway networks.
An empirical evaluation is performed on different datasets (MNIST, memory/addition tasks, sequential permuted MNIST and character level penntreebank).
However, there are few problems with the evaluation:
- In the highway network experiment, the author does not compare with a baseline.
We can not assess what it the impact of the low-rank parameterization. Also, it would be interesting to compare the result with a highway network that have this capacity bottleneck across layer  (first layer of size $n$, second layer of size $d$, third layer of size $n$) and not in the gate functions. Also, how did you select the hyperparameter values?.
- It is unfortunate that the character level penntreebank does not use the same experimental setting than previous works as it prevents from direct comparison.
Also the overall bpc perplexity seems relatively high for this dataset. It is therefore not clear how low-rank decomposition would perform on this task applied on a stronger baseline.
-Author claims state-of-art in the memory task. However, their approach uses  more parameters than the uRNN (41K against 6.5K for the memory) which makes the comparison a little bit unfair toward uRNN. It would be informative to see how low-rank RNN performs using overall 6.5K parameters. Generally, it would be good to see what is the impact of the matrix rank given a fix state size.
- It would be informative as well to have the baseline and the uRNN curve in Figure 2 for the memory/addition task.
- it is not clear when to use low-rank or low-rank + diagonal from the experiments.
Overall, the evaluation in its current form in not really convincing, except for the sequential MNIST dataset.


INFERENCE & INTROSPECTION IN DEEP GENERATIVE MODELS OF SPARSE DATA
Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.
1 INTRODUCTION
Deep latent Gaussian models (DLGMs, a.k.a. variational autoencoders; Rezende et al., 2014; Kingma et al., 2014) have led a resurgence in the use of deep generative models for density estimation. DLGMs assume that observed vectors x are generated by applying a nonlinear transformation (defined by a neural network with parameters θ) to a vector of Gaussian random variables z. Learning in DLGMs proceeds by approximately maximizing the average marginal likelihood p(x) ≡∫ z p(z)p(x|z)dz of the observations x. Computing the true marginal likelihood is intractable, so we resort to variational expectation-maximization (Bishop, 2006), an approximation to maximumlikelihood estimation. To learn the parameters θ of the generative model, the procedure needs to find a distribution q(z|x) that approximates the posterior distribution p(z|x) of the latent vector z given the observations x. In the past, such q distributions were fit using iterative optimization procedures (e.g., Hoffman et al., 2013). But Rezende et al. (2014) and Kingma et al. (2014) showed that q(z|x) can be parameterized by a feedforward “inference network” with parameters φ, speeding up learning. This inference network is trained jointly with the generative model; as training proceeds, the inference network learns to approximate posterior inference on the generative model, and the generative model improves itself using the output of the inference network.
Embedded within this procedure, however, lies a potential problem: both the inference network and the generative model are initialized randomly. Early on in learning, the inference network’s q(z|x) distributions will be poor approximations to the true posterior p(z|x), and the gradients used to update the parameters of the generative model will therefore be poor approximations to the gradients of the true log-likelihood log p(x). Previous stochastic variational inference methods (Hoffman et al., 2013) were slower, but suffered less from this problem since for every data-point, a set of variational parameters was optimized within the inner loop of learning. In this work, we investigate blending the two methodologies for learning models of sparse data. In particular, we use the parameters predicted by the inference network as an initialization and optimize them further during learning. When modeling high-dimensional sparse data, we show that updating the local variational parameters yields generative models with better held-out likelihood, particularly for deeper generative models.
What purpose is served by fitting bigger, deeper, more powerful generative models? Breiman (2001) argues that statistical discriminative modeling falls into two schools of thought: the data modeling culture and the algorithmic modeling culture. The former advocates the use of predictive models that assume interpretable, mechanistic processes while the latter advocates the use of black box techniques with an emphasis on prediction accuracy. Breiman’s arguments also ring true about the
z
x θ φ Figure 1: Deep Latent Gaussian Model: The Bayesian network depicted here comprises a single latent variable with the conditional probability p(x|z) defined by a deep neural network with parameter θ. The dotted line represents the inference network parameterized by φ, which is used for posterior inference at train and test time.
divide between deep generative models with complex conditional distributions and simpler, more interpretable statistical models. Consider a classic model such as Latent Dirichlet Allocation (Blei et al., 2003). It is outperformed in held-out likelihood (Miao et al., 2016) by deeper generative models and assumes a simple probabilistic process for data generation that is unlikely to hold in reality. Yet, its generative semantics lend it a distinct advantage: interpretability. The word-topic matrix in the model allows practitioners to read off what the model has learned about the data. Is there a natural way to interpret the generative model when the conditional distributions are parameterized by a deep neural network?
Our second contribution is to introduce a simple, easy to implement method to interpret what is being learned by generative models such as DLGMs whose conditional probabilities are parameterized by deep neural networks. Our hope is to narrow the perceived gulf between a complex generative model’s representational power and its interpretability. We use the Jacobian of the conditional distribution with respect to latent variables in the Bayesian network to form embeddings (or Jacobian vectors) of the observations. We investigate the properties of the Jacobian vectors obtained from deeper, more non-linear generative models.
2 BACKGROUND
Generative Model: We consider learning in generative models of the form shown in Figure 1. We observe a set of D word count vectors x1:D, where xdv denotes the number of times that word index v ∈ {1, . . . , V } appears in document d. We assume we are given the total number of words per document Nd ≡ ∑ v xdv , and that xd was generated via the following generative process: zd ∼ N (0, I); γ(zd) ≡ MLP(zd; θ); µ(zd) ≡ exp{γ(zd)}∑ v exp{γ(zd)v} ; xd ∼ Multinomial(µ(zd), Nd).
(1)
That is, we draw a Gaussian random vector, pass it through a multilayer perceptron (MLP) with parameters θ, pass the resulting vector through the softmax (a.k.a. multinomial logistic) function, and sample Nd times from the resulting distribution over the vocabulary.1
Variational Learning: For ease of exposition notation we drop the subscript on xd to form x referring to a single data point. We need to approximate the intractable posterior distribution p(z|x) during learning. Using the well-known variational principle, we can obtain the lower bound on the log marginal likelihood of the data (or L(x; θ, φ)) in Eq. 2. where the inequality is by Jensen’s inequality.
log p(x; θ) ≥ E qφ(z|x) [log pθ(x|z))]−KL( qφ(z|x)||p(z) ) = L(x; θ, φ), (2)
We leverage an inference network or recognition network (Hinton et al., 1995), a neural network which approximates the intractable posterior, during learning. This is a parametric conditional distribution that is optimized to perform inference. Kingma & Welling (2014); Rezende et al. (2014) use a neural net (with parameters φ) to parameterize qφ(z|x). The challenge in the resulting optimization problem is that the lower bound (2) includes an expectation w.r.t. qφ(z|x), which implicitly depends on the network parameters φ. This difficulty is overcome by using stochastic backpropagation.
With a normal distribution as our variational approximation we have that qφ(z|x) ∼ N (µφ(x),Σφ(x)). µφ(x),Σφ(x) are functions of the observation x, and we denote by ψ(x) :=
1In keeping with common practice, we neglect the multinomial base measure term N ! x1!···xV ! , which amounts to assuming that the words are observed in a particular order.
{µφ(x),Σφ(x)} the local variational parameters predicted by the inference network. A simple transformation allows one to obtain unbiased Monte Carlo estimates of the gradients of Eqφ(z|x) [log pθ(x|z))] with respect to φ. If we assume the prior p(z) is also normally distributed, the KL and its gradients may be obtained analytically. Throughout this paper we will use θ to denote the parameters of the generative model, and φ to denote the parameters of the inference network.
3 METHODOLOGY
Inference with Global Information: Sparse data typically exhibits long tails and learning in the presence of rare features is challenging. Inference networks learn to regress to the optimal posterior parameters for every data point and global information about the relative frequencies of the individual features in the training distribution may present valuable information during learning.
The simplest way to incorporate first order statistics across the training data into the inferential process is to condition on tf-idf (Baeza-Yates et al., 1999) features instead of the raw-counts. tf-idf is one of the most widely used techniques in information retrieval. In the context of building bag-of-words representations for documents, tf-idf re-weight features to increase the influence of rarer words while decreasing the influence of common words appearing in all documents. The tf-idf-transformed word-count vector is x̃dv ≡ xdv log D∑
d′ min{xd′v,1} . After applying transform, the resulting vector x̃
is normalized by its L2 norm. It’s worthwhile to note that leveraging first-order statistics for inference is difficult in the traditional paradigm of tracking variational parameters for each data point but is easy with inference networks.
Optimizing Local Variational Parameters: The inference network initially comprises a randomly initialized neural network. The predictions of the inference network early in optimization are suboptimal variational parameters used to derive gradients of the parameters of the generative model. This induces noise and bias to the gradients used to update the parameters of the generative model; this noise and bias may push the generative model towards a poor local optimum. Previous work has suggested that deep neural networks (which form the conditional probability distributions pθ(x|z)) are sensitive to initialization (Glorot & Bengio, 2010; Larochelle et al., 2009).
To avoid these issues, we only use the local variational parameters ψ(x) predicted by the inference network to initialize an iterative optimizer that maximizes the ELBO with respect to ψ; we use the optimized variational parameters ˆψ(x) to derive gradients for the generative model. We then train the inference network using stochastic backpropagation and gradient descent, holding the parameters of the generative model θ fixed. Our procedure is detailed in Algorithm 1.
Algorithm 1 Pseudocode for Learning: We evaluate expectations in L(x) (see Eq. 2) using a single sample from the variational distribution and aggregate gradients across mini-batches. M = 1 corresponds to performing no additional optimization to the variational parameters We update θ, ψ(x), φ using stochastic gradient descent with adaptive learning rates ηθ, ηψ(x), ηφ obtained via ADAM (Kingma & Ba, 2015)
Inputs: Dataset D := [x1, . . . , xD], Inference Model: qφ(z|x), Generative Model: pθ(x|z), p(z) while notConverged() do
1. Sample datapoint: x ∼ D 2. Estimate local variational parameters ψ(x)1 using qφ(z|x) 3. Estimate ψ(x)M ≈ ˆψ(x) = arg maxψ(x) L(x; θ;ψ(x)) via SGD as:
m = 1, . . . ,M , ψ(x)m+1 = ψ(x)m + ηψ(x) ∂L(x;θ,ψ(x)m)
∂ψ(x)m
4. Update θ as: θ ← θ + ηθ∇θL(x; θ, ψ(x)M ) 5. Update φ as: φ← φ+ ηφ∇φL(x; θ, ψ(x))
end while
Introspection: Linear models are inherently interpretable. Consider linear regression, factor analysis (Spearman, 1904), and latent Dirichlet allocation (LDA; Blei et al., 2003), which (standardizing notation) assume the following relationships:
Regression: E[y|x] = Wx+ b; Factor Analysis: x ∼ N (0, I); E[y|x] = Wx+ b; Latent Dirichlet Allocation: x ∼ Dirichlet(α); E[y|x] = Wx. (3)
In each case, we need only inspect the parameter matrix W to answer the question “what happens to y if we increase xk a little?” The answer is clear—y moves in the direction of the kth row of W . We can ask this question differently and get the same answer: “what is the derivative ∂E[y|x]∂x ?” The answer is simply the parameter matrix W .
For models as in Fig 1, the variability in the training data is assumed to be due to the single latent state z. The relationship between latent variables z and observations x cannot be quickly read off of the parameters θ. But we can still ask what happens if we perturb z by some small dz—this is simply the directional derivative ∂E[x|z]∂z dz. We can interpret this Jacobian matrix in much the same way we would a factor loading matrix, with two main differences. First, the Jacobian matrix ∂E[x|z] ∂z varies with z—the interpretation of z may change significantly depending on context. Second, DLGMs exhibit rotational symmetry—the prior on z is rotationally symmetric, and the MLP can apply arbitrary rotations to z before applying any nonlinearities, so a priori there is no “natural” set of basis vectors for z. For a given Jacobian matrix, however, we can find the most significant directions via a singular value decomposition (SVD).
Jacobian Vectors: We present our method to generate embeddings from Bayesian networks of the form Figure 1. We consider three variants of Jacobian embedding vectors, based on the unnormalized potentials from the MLP, logarithmic probabilities, and linear probabilities respectively:
J (z)pot = ∂γ(z) ∂z
J (z)log = ∂ logµ(z) ∂z
J (z)prob = ∂µ(z) ∂z
(4)
For any z, {J (z)log,J (z)pot,J (z)prob} ∈ RV×K where K is the latent dimension and V is the dimensionality of the observations. It is this matrix that we use to form embeddings. We denote by ui the Jacobian vector obtained from the row of the Jacobian matrix. When not referring to a particular variant, we use J (z) to denote the Jacobian matrix. J (z) is a function of z leaving open the choice of where to evaluate this function. The semantics of our generative model suggest a natural choice: Jmean := Ep(z)[J (z)]. This set of embeddings captures the variation in the output distribution with respect to the latent state across the prior distribution of the generative model. Additionally, one may also evaluate the Jacobian at the approximate posterior corresponding to an observation x. We study how this may be used to obtain contextual word-vectors.
In frameworks that support automatic differentiation (e.g., Theano; Theano Development Team, 2016), J (z) is readily available and we estimate Jmean via Monte-Carlo sampling from the prior. Deriving Jacobian Vectors: For simplicity, we derive the functional form of the Jacobian in a linear model i.e where γ(zd) = Wzd (c.f Eq 1). We drop the subscript d and denote by γi(z), the ith element of the vector γ(z).
p(xi = 1|z) = exp(γi(z))∑ j exp(γj(z))
and γi(z) = wTi z
For linear models, ∇zγi(z) = wi directly corresponds to J (z)pot. Noting that ∇z exp(γi(z)) = exp(γi(z))∇zγi(z) and ∇z ∑ j exp(γj(z)) = ∑ j exp(γj(z))∇zγj(z), we estimate J (z)prob as:
∇zp(xi = 1|z) = ∇z exp(γi(z))∑ j exp(γj(z)) =
∑ j exp(γj(z))∇z exp(γi(z))− exp(γi(z))∇z ∑ j exp(γj(z))
( ∑ j exp(γj(z))) 2
=
∑ j exp(γj(z)) exp(γi(z))wi − exp(γi(z)) ∑ j exp(γj(z))wj
( ∑ j exp(γj(z))) 2
= p(xi = 1|z)wi − p(xi = 1|z) ∑ j p(xj = 1|z)wj
= p(xi = 1|z)(wi − ∑ j p(xj = 1|z)wj)
Similarly, we may compute J (z)log: ∇z log p(xi = 1|z) = wi − ∑ j p(xj = 1|z)wj = ∑ j p(xj = 1|z)(wi − wj) (5)
Denoting a word-pair vector as wi −wj , where wi, wj are columns of the matrix W . If we define the set of all word-pair vectors as S , then Eq 5 captures the idea that the vector representation for a word i lies in the convex hull of S . Furthermore, the word vector’s location in CONV(S) is determined by the likelihood of the pairing word (xj) under the model p(xj = 1|z).
When we use a non-linear conditional probability distribution J (z)log becomes: ∇z log p(xi = 1|z) = ∑ j p(xj = 1|z)(∇zγi(z)−∇zγj(z)) where ∇zγi(z) is a non-linear function of z. To the best of our knowledge, Jacobian Vectors and their properties have not been studied.
4 RELATED WORK
Learning in Deep Generative Models: Salakhutdinov & Larochelle (2010) optimize the local variational parameters obtained from an inference network when learning deep Boltzmann machines. For DLGMs, Hjelm et al. (2016) also consider the optimization of the local variational parameters, though their exposition focuses on deriving an importance-sampling-based bound to use during learning in deep generative models with discrete latent variables. Their experimental results suggest the procedure does not improve performance much on the binarized MNIST dataset. This is consistent with our experience—we found that our secondary optimization procedure helped more when modeling sparse, high-dimensional text data than when modeling MNIST.
Leveraging Gradient Information: The algorithmic procedure for obtaining Jacobian Vectors that we propose resembles that used to derive Fisher Score features. For an data point X under a parameteric distribution p(X; θ), the Fisher scores is defined as UX = ∇θ log p(X; θ). Jaakkola & Haussler (2007) similarly use UX to form a kernel function for subsequent use in a discriminative classifier. The intuition behind such methods is to note that the derivative of the log-probability with respect to the parameters of the generative model encodes all the variability in the input under the generative process. We rely on a related intuition, although our motivations are different; we are interested in characterizing isolated features such as words, not vector observations such as documents. Also, we consider Jacobians with respect to per-observation latent variables z, rather than globally shared parameters θ.
In the context of discriminative modeling, (Erhan et al., 2009) use gradient information to study the patterns with which neurons are activated in a deep neural networks while (Wang et al., 2016) use the spectra of the Jacobian to study the complexity of the functions learned by neural networks.
Introspection via Embeddings: Landauer et al. (1998) proposed latent semantic analysis, one of the earliest works to create vector space representations of documents. Bengio et al. (2003); Mikolov & Dean (2016) propose log-linear models to create word-representations from document corpora in an unsupervised fashion. Rudolph et al. (2016) describe a family of models to create contextual embeddings where the conditional distributions that lie in the exponential family. Finally, Choi et al. (2016) propose a variant of Word2Vec to create representations of diagnosis codes from temporal Electronic Health Record data. The models above explicitly condition the probability of a word on its nearby context. In contrast, our model models the probability of a word as it appears in the document (or rather, conditioned on its global context). Augmenting the generative model in Figure 1 to incorporate local context is a possible direction for future work.
Miao et al. (2016) learn a shallow log-linear model on text data and obtain embeddings for words from the weight matrix that parameterize their generative model. Li et al. (2016) propose a modification to LDA that explicitly models representations for word in addition to modeling the word-topic structure.
5 EVALUATION
Text Data: We study the effect of further optimization of the variational parameters and inference with tf-idf features on the two datasets of varying size: the smaller 20Newsgroups (Lang, 2008) (train/valid/test: 10768/500/7505, V : 2000) and the larger RCV2 (Lewis et al., 2004) dataset (train/valid/test: 789414/5000/10000, V : 10000). We follow the preprocessing procedure defined in (Miao et al., 2016) for both datasets. We also train models on the Wikipedia corpus used in (Huang et al., 2012). We remove stop words, words appearing less than ten times in the dataset. and select our vocabulary to comprise the union of the top 20000 words in the corpus, the words in the WordSim353
(Finkelstein et al., 2001) and the words in the Stanford Contextual Word Similarity Dataset (SCWS) (Huang et al., 2012). The resulting dataset is of size train/valid: 1212781/1000 and V :20253.
EHR Data: We train shallow and deep generative models on a dataset constructed from Electronic Medical Records. The dataset comprises 185000 patients where each patient’s data across time was aggregated to create a bag-of-diagnosis-codes representation of the patient. The vocabulary comprises four different kinds of medical diagnosis codes: ICD9 (diagnosis), LOINC (laboratory tests), NDC (prescription medication), CPT (procedures). For a single patient, we have 51321 diagnosis codes.
Training Procedure: On all datasets, we train shallow log-linear models (γ(z) = Wz + b) and deeper three-layer DLGMs (γ(z) = MLP(z; θ)). We vary the number of secondary optimization steps M = 1, 100 (cf. Algorithm 1) to study the effect of optimization on ψ(x) with ADAM (Kingma & Ba, 2015). We use a mini-batch size of 500, a learning rate of 0.01 for ψ(x) and 0.0008 for θ, φ. The inference network was fixed to a two-layer MLP whose intermediate hidden layer h(x) was used to parameterize the mean and diagonal log-variance µ(x), log Σ(x). To evaluate the quality of the learned generative models, we report an upper bound on perplexity (Mnih & Gregor, 2014) given by exp(− 1N ∑ i 1 Ni
log p(xi)) where log p(xi) is replaced by Eq 2. The notation 3-M100-tfidf indicates a model where the MLP parameterizing γ(z) has three hidden layers, the local variational parameters are updated 100 times before an update of θ and tf-idf features were used in the inference network.
Improving Learning: Table 1 depicts our results on 20newsgroups and RCV2. On the smaller dataset, we find that the deeper models overfit quickly and are outperformed by shallow generative models. On the larger datasets, the deeper models’ capacity is more readily utilized yielding better generalization. The use of tf-idf features always helps learning on smaller datasets. On larger datasets, the benefits are smaller when we also optimize ψ(x). Finally, the optimization of the local variational parameters appears to help most on the larger datasets. To investigate how this occurs, we plot the held-out likelihood versus epochs. For models trained on the larger RCV2 (Figure 2a) and Wikipedia (Figure 2b) datasets, the larger deep generative models converge to better solutions (and in fewer passes through the data) with the additional optimization of ψ(x).
To study the effect of investigate where optimizing ψ(x) is particularly effective on, we train a three layer model on different subsets of the Wikipedia dataset. The subsets are created by selecting the top K most frequently occurring features in the data. Our rationale is that by holding everything fixed and varying the level of sparsity (datasets with smaller values of K are less sparse) in the data, we can begin to understand when our method is most helpful. On held-out data, we compute the difference between the perplexity when the model is trained with M = 1 (denoted PM1) and M = 100 (denoted PM100) and compute the relative decrease in perplexity obtained as PM1−PM100PM100 . The results are depicted in Figure 2c where we see that our method improves learning as a function of the dimensionality of the data.
In Table 5 in the supplementary material, we study the effect of varying the parameters of the inference network. There, we perform a small grid search over the hidden dimension and the number of layers in the inference network and find that optimizing the variational parameters continues to produces models with lower overall perplexity.
Jacobian Vectors: Our first avenue for introspection into the learned generative model is using log-singular values of the Jacobian matrix. Since the Jacobian matrix precisely encodes how sensitive the outputs are with respect to the inputs, the log-singular value spectrum of this matrix directly captures the amount of variance in the data explained by the latent space. Said differently, we can read off the number of active units in the DLGM or VAE by counting the number of log-singular values larger than zero. Furthermore, this method of introspection depends only on the parameters of the generative model. In Figure 2d, 2e, we see that for larger models continuing to optimize the variational parameters allows us to learn models that use many more of the available latent dimensions. This suggests that, when fit to text data, DLGMs may be particularly susceptible to the overpruning phenomenon noted by Burda et al. (2015). In Figure 2, the lower held-out perplexity and the increased utilization of the latent space suggest that the continued optimization of the variational parameters yields more powerful generative models.
We investigate how the Jacobian matrix may be used for model introspection by studying the qualitative properties of J logmean on DLGMs (of type “3-M100-tfidf”) trained on two, diverse sets of data. We form a Monte Carlo estimate of J logmean using 400 samples. The cosine distance is
used to define neighbors of words in the embedding space of the Jacobian and spectral clustering (Von Luxburg, 2007) is used to form clusters.
In Table 2a, we visualize some of the nearest neighbors of words using J logmean obtained from models trained on the Wikipedia dataset. The neighbors are semantically sensible. Instead of evaluating the Jacobian at L points z1:L ∼ p(z), one may instead evaluate it at z1:L ∼ q(z|x) for some x. In Table 2b, we select three polysemous query words alongside “context words” that disambiguate the query’s meaning. For each word-context pair, we create a document comprising a subset of words in the the
(a) Word Embeddings (Nearest Neighbors): We visualize nearest neighbors of word embeddings. We exclude plurals of the query and other words in the neighborhood.
Query Neighborhood intelligence espionage, secrecy, interrogation, counterterrorism zen dharma, buddhism, buddhas, meditation,yoga artificial artificially, molecules, synthetic, soluble military civilian, armys, commanders, infantry
(b) Word Embeddings (Polysemy): We visualize the nearest neighbors under the Jacobian vector induced by the posterior distribution of a document created based on the context word.
Word Context Neighboring Words crane construction lifting, usaaf, spanned, crushed, lift bird erected, parkland, locally, farmland, causeway bank river watershed, footpath, confluence, drains, tributary money banking, government, bankers, comptroller, fiscal fires burn ignition, combustion, engines, fuel, engine
layoff thunderstorm, grassy, surrounded, walkway, burning (c) Medical Embeddings (Nearest Neighbors): We evaluate nearest neighbors of selected diagnosis, drug and procedure codes (ignoring duplicates and shortening some code names). Metformin, Glimepiride, Pioglitazone and Avandia are diabetic drugs. A contour meter is an instrument to track blood glucose. Advair, Albuterol, Proventil and Spiriva are prescribed to patients with chronic obstructive pulmonary disease (COPD)
context’s Wikipedia page. Then, we use the learned inference network to perform posterior inference to evaluate J logmean at the corresponding q(z|x). This yields a set of contextual Jacobian vectors. We display the nearest neighbors for each word under different contextual Jacobian vectors and find that, while not always perfect, they capture different contextually relevant semantics. The take-away here is that by combining posterior inference in this Bayesian network with our methodology of introspecting the model, one obtains different context-specific representations for the observations despite not having been trained to capture this explicitly.
In Table 8b (appendix), we visualize clusters formed from the embeddings of medical diagnosis codes to find that they exhibit topical coherence. In Table 2c, the nearest neighbors of drugs include other drugs prescribed in conjunction with or as a replacement for the query drug. For diagnosis codes such as “Asbestosis”, the nearest neighbors are symptoms and procedures associated with the
disease. Finally, for a qualitative evaluation of Jacobian vectors obtained from a model trained on movie ratings, we refer the reader to the appendix.
The Semantics of Embeddings: We evaluate the vector space representations that we obtain from Jmean on benchmarks (such as WordSim353 (Finkelstein et al., 2001) and SCWS (Huang et al., 2012)) that attempt to measure the similarity of words. The algorithmically derived measure of similarity is compared to a human-annotated score (between one and ten) using the Spearman rank correlation. The models that we compare to primarily use local context, which yields a more precise signal about the meanings of particular words. Closest to us in terms of training procedure is (Huang (G)) in Table 3a, whose model we outperform. Finding ways to incorporate local context is fertile ground for future work on models tailor-made for extracting embeddings.
For medical codes, we follow the method in (Choi et al., 2016). The authors build two kinds of evaluations to estimate whether an embedding space of medical diagnosis codes captures medically related concepts well. MRMNDF-RT (Medical Relatedness Measure under NDF-RT) leverages a database (NDF-RT) to evaluate how good an embedding space is at answering analogical queries between drugs and diseases such as uDiabetes ≈ uMetformin − (uLung Cancer − uTarceva). (Metformin is a diabetic drug and Tarceva is used in the treatment of lung cancer). The evaluation (MRMCCS) measures if the neighborhood of the diagnosis codes is medically coherent using a predefined medical ontology (CCS) as ground truth. The number computed may be thought of as a measure of precision, where a higher number is better. We refer the reader to the appendix for additional details.
Table 4 details the results on evaluating the medical embeddings. Once again, the baselines we compare (Choi et al., 2016) are variants of Word2Vec that maximize the likelihood of the diagnosis codes conditioned on carefully crafted contexts. Our method performs comparably to the baselines, even though it relies exclusively on global context and was not designed with this task in mind. This setting depicts an instance where Jacobian vectors resulting from a deeper, better-trained model outperform those from a shallow model, highlighting the importance of a method of interpretation agnostic to the structure of the conditional probability functions in the generative model.
Between the three choices of Jacobian vectors, we found that all three perform comparably on the word similarity evaluation with J probmean slightly outperforming the others. On the medical data, with we found similar results aside from a few cases where J probmean did not perform well. For deeper models, we found that optimizing ψ(x) improved the quality of the obtained Jacobian vectors on text and medical data. The full versions of Tables 3 and 4 can be found in the appendix.
6 DISCUSSION
We explored techniques to improve inference and learning in deep generative models of sparse non-negative data. We also developed and explored a novel, simple, yet effective method to interpret the structure of the non-linear generative model via embeddings obtained from the Jacobian matrix relating latent variables to observations. The embeddings are evaluated qualitatively and quantitatively, and were seen to exhibit interesting semantic structure across a variety of domains. Studying the effects of varying the priors on the latent variables, conditioning on context, and varying the neural architectures that parameterize the conditional distributions suggest avenues for blending ideas from generative modeling and Bayesian inference into building more powerful embeddings for data.
APPENDIX A RCV2: COMPLEXITY OF THE INFERENCE NETWORK
We study the effect of varying the inference network on the two-layer model trained with the RCV2 data. Folding fixed a three-layer DLGM with stochastic dimension 100 (same architecture as 3−tfidf in Table 1), we learn models on the RCV2 data using M = 1, 100 (we evaluate and report bounds on perplexity using M = 100) and display the results in Table 5. When M = 1 (the standard procedure for training VAEs and DLGMs), increasing the number of layers in the inference network decreases the quality of the model learned. One possible explanation for this is that the already noisy gradients of the inference network must propagate along a longer path in a deeper inference network, slowing down learning of the parameters φ which in turn affects the quality of inference. In contrast, increasing hidden dimension of the inference network improves results. Generally, we obtain better results (in both train and validation error) with M = 100 than training with M = 1 across the various configurations of the inference network that we tried. Furthermore, we find that when M = 100, the inference network architecture is less relevant and all models converge to approximately the same result suggesting that the procedure treats the output of the inference network as a crude initialization for the variational parameters and that the subsequent steps of optimization are primarily responsible for gains in learning.
APPENDIX B NETFLIX: EMBEDDINGS FOR MOVIES
The Netflix dataset Netflix (2009) comprises movie ratings of 500, 000 users. We treat each user’s ratings as a document and model the numbers ascribed to each movie (from 1− 5) as counts drawn from the multinomial distribution parameterized as in Eq. 1. We train a three-layer DLGM on the dataset, evaluate Jmean with 100 samples and consider two distinct methods of evaluating the learned embeddings. We cluster the movie embeddings (using spectral clustering with cosine distance to obtain 100 clusters) and depict some of the clusters in Table 6a. We find that clusters exhibit coherent themes such as documentary films, horror and James Bond movies. Other clusters (not displayed) included multiple seasons of the same show such as Friends, WWE wrestling, and Pokemon. In Table 6b, we visualize the neighbors of some popular films. In the examples we visualize, the nearest neighbors include sequels, movies from the same franchise or, as in the case of 12 Angry Men, other dramatic classics.
To compare the effect of using a model to create embeddings versus using the raw data from a large dataset directly, we evaluated nearest neighbors of movies using a simple baseline. For a query movie, we found all users who gave the movie a rating of 3 or above (nominally, they watched and liked the movie). Then, for all those users, we computed the mean ratings they gave to every other movie in the vocabulary and ranked them based on the mean ratings. We display the top five movies obtained using this approach in Table 6. The query words are the same as in Table 6b. For most of the queries, the difference between the two is evident and we simply end up with popular, well-liked movies rather than relevant movies.
APPENDIX C WIKIPEDIA: EMBEDDINGS FOR WORD
Table 7 is the full version of Table 3 in the main paper. We find that the three variants of the Jacobian vectors perform comparably across the board. The vectors obtained from shallow log-linear models appear to have the edge. The evaluation on the WordSim and SCWS datasets are done by computing the Spearman rank correlation between human annotated rankings between 1 and 10 and an algorithmically derived measures of word-pair similarity. We first compute the distances between all word pairs. Our measure of similarity is obtained by subtracting the distances from the maximal distance across all word pairs.
APPENDIX D EHR DATA: EMBEDDINGS FOR DIAGNOSIS CODES
For EHR data in particular, the bag-of-diagnosis-codes assumption we make is a crude one since (1) we assume the temporal nature of the patient data is irrelevant, and (2) combining patient statistics over time renders it difficult for the generative model to disambiguate the correlations between codes that correspond to multiple diseases a patient may suffer from. Despite this, it is interesting that the Jacobian vectors still capture much of the meaningful structure among the diagnosis codes (c.f Table 2c, 8b). Here we provide additional details surrounding the evaluating of medical embeddings.
MRMCCS(V,G): The Agency for Healthcare Research and Quality’s clinical classification software (CCS) collapses the hierarchical ICD9 diagnosis codes into clinically meaningful categories. The evaluation on CCS checks whether the nearest neighbors of a disease include other diseases related to it (if they are in the same category in the CCS). Using the ICD9 hierarchy, the authors further split the evaluation task into predicting neighbors of fine-grained and coarse grained diagnosis codes.
For a choice of granularity G ∈ {fine,coarse}, V (G) ∈ V denotes the subset of ICD9 codes in the vocabulary. IG(v(i)) is one if the v’s i’th nearest neighbor: v(i) is in the same group as v according to G.
MRMCCS(V,G) = 1 |V (G)| ∑
v∈V (G) 40∑ k=1 IG(v(i)) log2(i+ 1)
(6)
MRMNDF-RT(V,R): The other evaluation uses the National Drug File Reference Terminology (NDF-RT) to evaluate analogical reasoning. The NDF-RT provides two kinds of relationships (R) between drugs and diseases: May-Treat (if the drug may be used to treat the disease) and May-Prevent. Given φA as the embedding for a code A, this test automates the evaluation of analogies such as φDiabetes︸ ︷︷ ︸
r
≈ φMetformin︸ ︷︷ ︸ v −(φLung Cancer − φTarceva︸ ︷︷ ︸ s ). Here v is the query code and s is a representation
of the relationship we seek. (Metformin is a diabetic drug and Tarceva is used in the treatment of lung cancer.) The evaluation we perform reports a number proportional to the number of times
the neighborhood of v − s contains r for the best value of s (computed from the set of all valid drug-disease relationships in the datasets.) Given V ∗ ∈ V (concepts for which NDF-RT has at-least one substance with the given relation), IR ( ∪40i=1(v − s)(i) ) is one if any of the medical concepts in the top-40 neighborhood of the selected medical concept v satisfies relation R.
MRMNDF-RT(V,R) = 1 |V ∗| ∑ v∈V ∗ IR ( ∪40i=1(v − s)(i) ) (7)
In both cases the choice of 40 was adopted to maintain consistency with (Choi et al., 2016). The evaluation is conducted by taking the average result over all possible seeds s and the best possible seed s for a query.
(a) Medical Analogies: We can perform analogical reasoning with embeddings of medical codes. If we know a drug used to treat a disease, we can use their relationship in vector space to find unknown drugs associated with a different disease. The queries are of the form Code 1→Code 2 =⇒ Code 3→?. Sicca syndrome or Sjogren’s disease is an immune disease treated with Evoxac and Methotrexate is commonly used to treat Rheumatoid Arthiritis. “ Leg Varicosity” denotes the presence of swollen veins under the skin. “Ligation of angioaccess arteriovenous fistula” denotes the tying of passage between an artery and a vein.
(b) Medical Embeddings (Clustering): We visualize some topical clusters of diagnosis codes.
Label Diagnosis Codes Thrombosis Hx Venous Thrombosis, Compression Of Vein, Renal Vein Thrombosis Occular Atrophy Optic Atrophy, Retina Layer Separation, Chronic Endophthalmitis Drug Use Opioid Dependence, Alcohol Abuse-Continuous, Hallucinogen Dep

The paper claims improved inference for density estimation of sparse data (here text documents) using deep generative Gaussian models (variational auto-encoders), and a method for deriving word embeddings from the model's generative parameters that allows for a degree of interpretability similar to that of Bayesian generative topic models.
To discuss the contributions I will quickly review the generative story in the paper: first a K-dimensional latent representation is sampled from a multivariate Gaussian, then an MLP (with parameters \theta) predicts unnormalised potentials over a vocabulary of V words, the potentials are exponentiated and normalised to make the parameters of a multinomial from where word observations are repeatedly sampled to make a document. Here intractable inference is replaced by the VAE formulation where an inference network (with parameters \phi) independently predicts for each document the mean and variance of a normal distribution (amenable to reparameterised gradient computation).
The first, and rather trivial, contribution is to use tf-idf features to inject first order statistics (a global information) into local observations. The authors claim that this is particularly helpful in the case of sparse data such as text.
The second contribution is more interesting. In optimising generative parameters (\theta) and variational parameters (\phi), the authors turn to a treatment which is reminiscent of the original SVI procedure. That is, they see the variational parameters \phi as *global* variational parameters, and the predicted mean \mu(x) and covariance \Sigma(x) of each observation x are treated as *local* variational parameters. In the original VAE, local parameters are not directly optimised, instead they are indirectly optimised via optimisation of the global parameters utilised in their prediction (shared MLP parameters). Here, local parameters are optimised holding generative parameters fixed (line 3 of Algorithm 1). The optimised local parameters are then used in the gradient step of the generative parameters (line 4 of Algorithm 1). Finally, global variational parameters are also updated (line 5).
Whereas indeed other authors have proposed to optimise local parameters, I think that deriving this procedure from the more familiar SVI makes the contribution less of a trick and easier to relate to.
Some things aren't entirely clear to me. I think it would have been nice if the authors had shown the functional form of the gradient used in step 3 of Algorithm 1. The gradient step for global variational parameters (line 5 of Algorithm 1) uses the very first prediction of local parameters (thus ignoring the optimisation in step 3), this is unclear to me. Perhaps I am missing a fundamental reason why that has to be the case (either way, please clarify).
The authors argue that this optimisation turns out helpful to modelling sparse data because there is evidence that the generative model p_\theta(x|z) suffers from poor initialisation. Please, discuss why you expect the initialisation problem to be worse in the case of sparse data.
The final contribution is a neat procedure to derive word embeddings from the generative model parameters. These embeddings are then used to interpret what the model has learnt. Interestingly, these word embeddings are context-sensitive once that the latent variable models an entire document.
About Figures 2a and 2b: the caption says that solid lines indicate validation perplexity for M=1 (no optimisation of local parameters) and dashed lines indicate M=100 (100 iterations of optimisation of local parameters), but the legends of the figures suggest a different reading. If I interpret the figures based on the caption, then it seems that indeed deeper networks exposed to more data benefit from optimisation of local parameters. Are the authors pretty sure that in Figure 2b models with M=1 have reached a plateau (so that longer training would not allow them to catch up with M=100 curves)? As the authors explain in the caption, x-axis is not comparable on running time, thus the question.
The analysis of singular values seems like an interesting way to investigate how the model is using its capacity.  However, I can barely interpret Figures 2c and 2d, I think the authors could have walked readers through them.
As for the word embedding I am missing an evaluation on a predictive task. Also, while illustrative, Table 2b is barely reproducible. The text reads "we create a document comprising a subset of words in the the context’s Wikipedia page." which is rather vague. I wonder whether this construct needs to be carefully designed in order to get Table 2b.
In sum, I have a feeling that the inference technique and the embedding technique are both useful, but perhaps they should have been presented separately so that each could have been explored in greater depth.