While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning — leveraging unlabeled examples to learn about the structure of a domain — remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network (“PredNet”) architecture that is inspired by the concept of “predictive coding” from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure.
1 INTRODUCTION
Many of the most successful current deep learning architectures for vision rely on supervised learning from large sets of labeled training images. While the performance of these networks is undoubtedly impressive, reliance on such large numbers of training examples limits the utility of deep learning in many domains where such datasets are not available. Furthermore, the need for large numbers of labeled examples stands at odds with human visual learning, where one or a few views of an object is often all that is needed to enable robust recognition of that object across a wide range of different views, lightings and contexts. The development of a representation that facilitates such abilities, especially in an unsupervised way, is a largely unsolved problem.
In addition, while computer vision models are typically trained using static images, in the real world, visual objects are rarely experienced as disjoint snapshots. Instead, the visual world is alive with movement, driven both by self-motion of the viewer and the movement of objects within the scene. Many have suggested that temporal experience with objects as they move and undergo transformations can serve as an important signal for learning about the structure of objects (Földiák, 1991; Softky, 1996; Wiskott & Sejnowski, 2002; George & Hawkins, 2005; Palm, 2012; O’Reilly et al., 2014; Agrawal et al., 2015; Goroshin et al., 2015a; Lotter et al., 2015; Mathieu et al., 2016; Srivastava et al., 2015; Wang & Gupta, 2015; Whitney et al., 2016). For instance, Wiskott and Sejnowski proposed “slow feature analysis” as a framework for exploiting temporal structure in video streams (Wiskott & Sejnowski, 2002). Their approach attempts to build feature representations that extract
Code and video examples can be found at: https://coxlab.github.io/prednet/
slowly-varying parameters, such as object identity, from parameters that produce fast changes in the image, such as movement of the object. While approaches that rely on temporal coherence have arguably not yet yielded representations as powerful as those learned by supervised methods, they nonetheless point to the potential of learning useful representations from video (Mohabi et al., 2009; Sun et al., 2014; Goroshin et al., 2015a; Maltoni & Lomonaco, 2015; Wang & Gupta, 2015).
Here, we explore another potential principle for exploiting video for unsupervised learning: prediction of future image frames (Softky, 1996; Palm, 2012; O’Reilly et al., 2014; Goroshin et al., 2015b; Srivastava et al., 2015; Mathieu et al., 2016; Patraucean et al., 2015; Finn et al., 2016; Vondrick et al., 2016). A key insight here is that in order to be able to predict how the visual world will change over time, an agent must have at least some implicit model of object structure and the possible transformations objects can undergo. To this end, we have designed a neural network architecture, which we informally call a “PredNet,” that attempts to continually predict the appearance of future video frames, using a deep, recurrent convolutional network with both bottom-up and topdown connections. Our work here builds on previous work in next-frame video prediction (Ranzato et al., 2014; Michalski et al., 2014; Srivastava et al., 2015; Mathieu et al., 2016; Lotter et al., 2015; Patraucean et al., 2015; Oh et al., 2015; Finn et al., 2016; Xue et al., 2016; Vondrick et al., 2016; Brabandere et al., 2016), but we take particular inspiration from the concept of “predictive coding” from the neuroscience literature (Rao & Ballard, 1999; Rao & Sejnowski, 2000; Lee & Mumford, 2003; Friston, 2005; Summerfield et al., 2006; Egner et al., 2010; Bastos et al., 2012; Spratling, 2012; Chalasani & Principe, 2013; Clark, 2013; O’Reilly et al., 2014; Kanai et al., 2015). Predictive coding posits that the brain is continually making predictions of incoming sensory stimuli (Rao & Ballard, 1999; Friston, 2005). Top-down (and perhaps lateral) connections convey these predictions, which are compared against actual observations to generate an error signal. The error signal is then propagated back up the hierarchy, eventually leading to an update of the predictions.
We demonstrate the effectiveness of our model for both synthetic sequences, where we have access to the underlying generative model and can investigate what the model learns, as well as natural videos. Consistent with the idea that prediction requires knowledge of object structure, we find that these networks successfully learn internal representations that are well-suited to subsequent recognition and decoding of latent object parameters (e.g. identity, view, rotation speed, etc.). We also find that our architecture can scale effectively to natural image sequences, by training using car-mounted camera videos. The network is able to successfully learn to predict both the movement of the camera and the movement of objects in the camera’s view. Again supporting the notion of prediction as an unsupervised learning rule, the model’s learned representation in this setting supports decoding of the current steering angle.
2 THE PREDNET MODEL
The PredNet architecture is diagrammed in Figure 1. The network consists of a series of repeating stacked modules that attempt to make local predictions of the input to the module, which is then subtracted from the actual input and passed along to the next layer. Briefly, each module of the network consists of four basic parts: an input convolutional layer (Al), a recurrent representation layer (Rl), a prediction layer (Âl), and an error representation (El). The representation layer, Rl, is a recurrent convolutional network that generates a prediction, Âl, of what the layer input, Al, will be on the next frame. The network takes the difference between Al and Âl and outputs an error representation, El, which is split into separate rectified positive and negative error populations. The error, El, is then passed forward through a convolutional layer to become the input to the next layer (Al+1). The recurrent prediction layerRl receives a copy of the error signalEl, along with top-down input from the representation layer of the next level of the network (Rl+1). The organization of the network is such that on the first time step of operation, the “right” side of the network (Al’s andEl’s) is equivalent to a standard deep convolutional network. Meanwhile, the “left” side of the network (the Rl’s) is equivalent to a generative deconvolutional network with local recurrence at each stage. The architecture described here is inspired by that originally proposed by (Rao & Ballard, 1999), but is formulated in a modern deep learning framework and trained end-to-end using gradient descent, with a loss function implicitly embedded in the network as the firing rates of the error neurons. Our work also shares motivation with the Deep Predictive Coding Networks of Chalasani & Principe (2013); however, their framework is based upon sparse coding and a linear dynamical system with greedy layer-wise training, whereas ours is rooted in convolutional and recurrent neural networks trained with backprop.
While the architecture is general with respect to the kinds of data it models, here we focus on image sequence (video) data. Consider a sequence of images, xt. The target for the lowest layer is set to the the actual sequence itself, i.e. At0 = xt ∀t. The targets for higher layers, Atl for l > 0, are computed by a convolution over the error units from the layer below, Etl−1, followed by rectified linear unit (ReLU) activation and max-pooling. For the representation neurons, we specifically use convolutional LSTM units (Hochreiter & Schmidhuber, 1997; Shi et al., 2015). In our setting, the Rtl hidden state is updated according to R t−1 l , E t−1 l , as well as R t l+1, which is first spatially upsampled (nearest-neighbor), due to the pooling present in the feedforward path. The predictions, Âtl are made through a convolution of the R t l stack followed by a ReLU non-linearity. For the lowest layer, Âtl is also passed through a saturating non-linearity set at the maximum pixel value: SatLU(x; pmax) := min(pmax, x). Finally, the error response, Etl , is calculated from the difference between Âtl and A t l and is split into ReLU-activated positive and negative prediction errors, which are concatenated along the feature dimension. As discussed in (Rao & Ballard, 1999), although not explicit in their model, the separate error populations are analogous to the existence of on-center, off-surround and off-center, on-surround neurons early in the visual system.
The full set of update rules are listed in Equations (1) to (4). The model is trained to minimize the weighted sum of the activity of the error units. Explicitly, the training loss is formalized in Equation 5 with weighting factors by time, λt, and layer, λl, and where nl is the number of units in the lth layer. With error units consisting of subtraction followed by ReLU activation, the loss at each layer is equivalent to an L1 error. Although not explored here, other error unit implementations, potentially even probabilistic or adversarial (Goodfellow et al., 2014), could also be used.
Atl = { xt if l = 0 MAXPOOL(RELU(CONV(Etl−1))) l > 0
(1)
Âtl = RELU(CONV(R t l)) (2)
Etl = [RELU(A t l − Âtl); RELU(Âtl −Atl)] (3) Rtl = CONVLSTM(E t−1 l , R t−1 l ,UPSAMPLE(R t l+1)) (4)
Ltrain = ∑ t λt ∑ l λl nl ∑ nl Etl (5)
Algorithm 1 Calculation of PredNet states Require: xt
1: At0 ← xt 2: E0l , R 0 l ← 0 3: for t = 1 to T do 4: for l = L to 0 do . Update Rtl states 5: if l = L then 6: RtL = CONVLSTM(E t−1 L , R t−1 L ) 7: else 8: Rtl = CONVLSTM(E t−1 l , R t−1 l ,UPSAMPLE(R t l+1))
9: for l = 0 to L do . Update Âtl , Atl , Etl states 10: if l = 0 then 11: Ât0 = SATLU(RELU(CONV(R t 0))) 12: else 13: Âtl = RELU(CONV(R t l )) 14: Etl = [RELU(A t l − Âtl); RELU(Âtl −Alt)] 15: if l < L then 16: Atl+1 = MAXPOOL(CONV(E l t))
The order in which each unit in the model is updated must also be specified, and our implementation is described in Algorithm 1. Updating of states occurs through two passes: a top-down pass where the Rtl states are computed, and then a forward pass to calculate the predictions, errors, and higher level targets. A last detail of note is that Rl and El are initialized to zero, which, due to the convolutional nature of the network, means that the initial prediction is spatially uniform.
3 EXPERIMENTS
3.1 RENDERED IMAGE SEQUENCES
To gain an understanding of the representations learned in the proposed framework, we first trained PredNet models using synthetic images, for which we have access to the underlying generative stimulus model and all latent parameters. We created sequences of rendered faces rotating with two degrees of freedom, along the “pan” (out-of-plane) and “roll” (in-plane) axes. The faces start at a random orientation and rotate at a random constant velocity for a total of 10 frames. A different face was sampled for each sequence. The images were processed to be grayscale, with values normalized between 0 and 1, and 64x64 pixels in size. We used 16K sequences for training and 800 for both validation and testing.
Predictions generated by a PredNet model are shown in Figure 2. The model is able to accumulate information over time to make accurate predictions of future frames. Since the representation neurons are initialized to zero, the prediction at the first time step is uniform. On the second time step, with no motion information yet, the prediction is a blurry reconstruction of the first time step. After further iterations, the model adapts to the underlying dynamics to generate predictions that closely match the incoming frame.
For choosing the hyperparameters of the model, we performed a random search and chose the model that had the lowest L1 error in frame prediction averaged over time steps 2-10 on a validation set. Given this selection criteria, the best performing models tended to have a loss solely concentrated at the lowest layer (i.e. λ0 = 1, λl>0 = 0), which is the case for the model shown. Using an equal loss at each layer considerably degraded predictions, but enforcing a moderate loss on upper layers that was one magnitude smaller than the lowest layer (i.e. λ0 = 1, λl>0 = 0.1) led to only slightly worse predictions, as illustrated in Figure 9 in the Appendix. In all cases, the time loss weight, λt, was set to zero for the first time step and then one for all time steps after. As for the remaining hyperparameters, the model shown has 5 layers with 3x3 filter sizes for all convolutions, max-pooling of stride 2, and number of channels per layer, for bothAl andRl units, of (1, 32, 64, 128, 256). Model weights were optimized using the Adam algorithm (Kingma & Ba, 2014).
Actual
Predicted
Actual
Predicted
Actual
Predicted
Quantitative evaluation of generative models is a difficult, unsolved problem (Theis et al., 2016), but here we report prediction error in terms of meansquared error (MSE) and the Structural Similarity Index Measure (SSIM) (Wang et al., 2004). SSIM is designed to be more correlated with perceptual judgments, and ranges from−1 and 1, with a larger score indicating greater similarity. We compare the PredNet to the trivial solution of copying the last
frame, as well as a control model that shares the overall architecture and training scheme of the PredNet, but that sends forward the layer-wise activations (Al) rather than the errors (El). This model thus takes the form of a more traditional encoder-decoder pair, with a CNN encoder that has lateral skip connections to a convolutional LSTM decoder. The performance of all models on the rotating faces dataset is summarized in Table 1, where the scores were calculated as an average over all predictions after the first frame. We report results for the PredNet model trained with loss only on the lowest layer, denoted as PredNet L0, as well as the model trained with an 0.1 weight on upper layers, denoted as PredNet Lall. Both PredNet models outperformed the baselines on both measures, with the L0 model slightly outperforming Lall, as expected for evaluating the pixel-level predictions.
Synthetic sequences were chosen as the initial training set in order to better understand what is learned in different layers of the model, specifically with respect to the underlying generative model (Kulkarni et al., 2015). The rotating faces were generated using the FaceGen software package (Singular Inversions, Inc.), which internally generates 3D face meshes by a principal component analysis in “face space”, derived from a corpus of 3D face scans. Thus, the latent parameters of the image sequences used here consist of the initial pan and roll angles, the pan and roll velocities, and the principal component (PC) values, which control the “identity” of the face. To understand the information contained in the trained models, we decoded the latent parameters from the representation neurons (Rl) in different layers, using a ridge regression. The Rl states were taken at the earliest possible informative time steps, which, in the our notation, are the second and third steps, respectively, for the static and dynamic parameters. The regression was trained using 4K sequences with 500 for validation and 1K for testing. For a baseline comparison of the information implicitly embedded in the network architecture, we compare to the decoding accuracies of an untrained network with random initial weights. Note that in this randomly initialized case, we still expect above-chance decoding performance, given past theoretical and empirical work with random networks (Pinto et al., 2009; Jarrett et al., 2009; Saxe et al., 2010).
Latent variable decoding accuracies of the pan and roll velocities, pan initial angle, and first PC are shown in the left panel of Figure 3. There are several interesting patterns. First, the trained models learn a representation that generally permits a better linear decoding of the underlying latent factors than the randomly initialized model, with the most striking difference in terms of the the pan rotation speed (αpan). Second, the most notable difference between the Lall and L0 versions occurs with the first principle component, where the model trained with loss on all layers has a higher decoding accuracy than the model trained with loss only on the lowest layer.
The latent variable decoding analysis suggests that the model learns a representation that may generalize well to other tasks for which it was not explicitly trained. To investigate this further, we assessed the models in a classification task from single, static images. We created a dataset of 25 previously unseen FaceGen faces at 7 pan angles, equally spaced between [−π2 , π 2 ], and 8 roll angles, equally spaced between [0, 2π). There were therefore 7 · 8 = 56 orientations per identity, which were tested in a cross-validated fashion. A linear SVM to decode face identity was fit on a model’s representation of a random subset of orientations and then tested on the remaining angles. For each size of the SVM training set, ranging from 1-40 orientations per face, 50 different random splits were generated, with results averaged over the splits.
For the static face classification task, we compare the PredNets to a standard autoencoder and a variant of the Ladder Network (Valpola, 2015; Rasmus et al., 2015). Both models were constructed to have the same number of layers and channel sizes as the PredNets, as well as a similar alternating convolution/max-pooling, then upsampling/convolution scheme. As both networks are autoencoders, they were trained with a reconstruction loss, with a dataset consisting of all of the individual frames from the sequences used to train the PredNets. For the Ladder Network, which is a denoising autoencoder with lateral skip connections, one must also choose a noise parameter, as well as the relative weights of each layer in the total cost. We tested noise levels ranging from 0 to 0.5 in increments of 0.1, with loss weights either evenly distributed across layers, solely concentrated at the pixel layer, or 1 at the bottom layer and 0.1 at upper layers (analogous to the PredNet Lall model). Shown is the model that performed best for classification, which consisted of 0.4 noise and only pixel weighting. Lastly, as in our architecture, the Ladder Network has lateral and top-down streams that are combined by a combinator function. Inspired by (Pezeshki et al., 2015), where a learnable MLP improved results, and to be consistent in comparing to the PredNet, we used a purely convolutional combinator. Given the distributed representation in both networks, we decoded from a concatenation of the feature representations at all layers, except the pixel layer. For the PredNets, the representation units were used and features were extracted after processing one input frame.
Face classification accuracies using the representations learned by the L0 and Lall PredNets, a standard autoencoder, and a Ladder Network variant are shown in the right panel of Figure 3. Both PredNets compare favorably to the other models at all sizes of the training set, suggesting they learn a representation that is relatively tolerant to object transformations. Similar to the decoding accuracy of the first principle component, the PredNet Lall model actually outperformed the L0 variant. Altogether, these results suggest that predictive training with the PredNet can be a viable alternative to other models trained with a more traditional reconstructive or denoising loss, and that the relative layer loss weightings (λl’s) may be important for the particular task at hand.
3.2 NATURAL IMAGE SEQUENCES
We next sought to test the PredNet architecture on complex, real-world sequences. As a testbed, we chose car-mounted camera videos, since these videos span across a wide range of settings and are characterized by rich temporal dynamics, including both self-motion of the vehicle and the motion of other objects in the scene (Agrawal et al., 2015). Models were trained using the raw videos from the KITTI dataset (Geiger et al., 2013), which were captured by a roof-mounted camera on a car driving around an urban environment in Germany. Sequences of 10 frames were sampled from the “City”, “Residential”, and “Road” categories, with 57 recording sessions used for training and 4 used for validation. Frames were center-cropped and downsampled to 128x160 pixels. In total, the training set consisted of roughly 41K frames.
A random hyperparameter search, with model selection based on the validation set, resulted in a 4 layer model with 3x3 convolutions and layer channel sizes of (3, 48, 96, 192). Models were again trained with Adam (Kingma & Ba, 2014) using a loss either solely computed on the lowest layer (L0) or with a weight of 1 on the lowest layer and 0.1 on the upper layers (Lall). Adam parameters were initially set to their default values (α = 0.001, β1 = 0.9, β2 = 0.999) with the learning rate, α, decreasing by a factor of 10 halfway through training. To assess that the network had indeed learned a robust representation, we tested on the CalTech Pedestrian dataset (Dollár et al., 2009), which consists of videos from a dashboard-mounted camera on a vehicle driving around Los Angeles. Testing sequences were made to match the frame rate of the KITTI dataset and again cropped to 128x160 pixels. Quantitative evaluation was performed on the entire CalTech test partition, split into sequences of 10 frames.
Sample PredNet predictions (for the L0 model) on the CalTech Pedestrian dataset are shown in Figure 4, and example videos can be found at https://coxlab.github.io/prednet/. The model is able to make fairly accurate predictions in a wide range of scenarios. In the top sequence of Fig. 4, a car is passing in the opposite direction, and the model, while not perfect, is able to predict its trajectory, as well as fill in the ground it leaves behind. Similarly in Sequence 3, the model is able to predict the motion of a vehicle completing a left turn. Sequences 2 and 5 illustrate that the PredNet can judge its own movement, as it predicts the appearance of shadows and a stationary vehicle as they approach. The model makes reasonable predictions even in difficult scenarios, such as when the camera-mounted vehicle is turning. In Sequence 4, the model predicts the position of a tree, as the vehicle turns onto a road. The turning sequences also further illustrate the model’s ability to “fill-in”, as it is able to extrapolate sky and tree textures as unseen regions come into view. As an additional control, we show a sequence at the bottom of Fig. 4, where the input has been temporally scrambled. In this case, the model generates blurry frames, which mostly just resemble the previous frame. Finally, although the PredNet shown here was trained to predict one frame ahead, it is also possible to predict multiple frames into the future, by feeding back predictions as the inputs and recursively iterating. We explore this in Appendix 5.3.
Quantitatively, the PredNet models again outperformed the CNN-LSTM EncoderDecoder. To ensure that the difference in performance was not simply because of the choice of hyperparameters, we trained models with four other sets of hyperparameters, which were sampled from the initial random search over the number of layers, fil-
ter sizes, and number of filters per layer. For each of the four additional sets, the PredNet L0 had the best performance, with an average error reduction of 14.7% and 14.9% for MSE and SSIM,
respectively, compared to the CNN-LSTM Encoder-Decoder. More details, as well as a thorough investigation of systematically simplified models on the continuum between the PredNet and the CNN-LSTM Encoder-Decoder can be found in Appendix 5.1. Briefly, the elementwise subtraction operation in the PredNet seems to be beneficial, and the nonlinearity of positive/negative splitting also adds modest improvements. Finally, while these experiments measure the benefits of each component of our model, we also directly compare against recent work in a similar car-cam setting, by reporting results on a 64x64 pixel, grayscale car-cam dataset released by Brabandere et al. (2016). Our PredNet model outperforms the model by Brabandere et al. (2016) by 29%. Details can be found in Appendix 5.2. Also in Appendix 5.2, we present results for the Human3.6M (Ionescu et al., 2014) dataset, as reported by Finn et al. (2016). Without re-optimizing hyperparameters, our
model underperforms the concurrently developed DNA model by Finn et al. (2016), but outperforms the model by Mathieu et al. (2016).
To test the implicit encoding of latent parameters in the car-cam setting, we used the internal representation in the PredNet to estimate the car’s steering angle (Bojarski et al., 2016; Biasini et al., 2016). We used a dataset released by Comma.ai (Biasini et al., 2016) consisting of 11 videos totaling about 7 hours of mostly highway driving. We first trained networks for next-frame prediction and then fit a linear fully-connected layer on the learned representation to estimate the steering angle, using a MSE loss. We again concatenate the Rl representation at all layers, but first spatially average pool lower layers to match the spatial size of the upper layer, in order to reduce dimensionality. Steering angle estimation results, using the representation on the 10th time step, are shown in Figure 5. Given just 1K labeled training examples, a simple linear readout on the PredNet L0 representation explains 74% of the variance in the steering angle and outperforms the CNN-LSTM Enc.-Dec. by 35%. With 25K labeled training examples, the PredNet L0 has a MSE (in degrees2) of 2.14. As a point of reference, a CNN model designed to predict the steering angle (Biasini et al., 2016), albeit from a single frame instead of multiple frames, achieve a MSE of ~4 when trained end-to-end using 396K labeled training examples. Details of this analysis can be found in Appendix 8. Interestingly, in this task, the PredNet Lall model actually underperformed the L0 model and slightly underperformed the CNN-LSTM Enc.-Dec, again suggesting that the λl parameter can affect the representation learned, and different values may be preferable in different end tasks. Nonetheless, the readout from the Lall model still explained a substantial proportion of the steering angle variance and strongly outperformed the random initial weights. Overall, this analysis again demonstrates that a representation learned through prediction, and particularly with the PredNet model with appropriate hyperparameters, can contain useful information about underlying latent parameters.
4 DISCUSSION
Above, we have demonstrated a predictive coding inspired architecture that is able to predict future frames in both synthetic and natural image sequences. Importantly, we have shown that learning to predict how an object or scene will move in a future frame confers advantages in decoding latent parameters (such as viewing angle) that give rise to an object’s appearance, and can improve recognition performance. More generally, we argue that prediction can serve as a powerful unsupervised learning signal, since accurately predicting future frames requires at least an implicit model of the objects that make up the scene and how they are allowed to move. Developing a deeper understanding of the nature of the representations learned by the networks, and extending the architecture, by, for instance, allowing sampling, are important future directions.
ACKNOWLEDGMENTS
We would like to thank Rasmus Berg Palm for fruitful discussions and early brainstorming. We would also like to thank the developers of Keras (Chollet, 2016). This work was supported by IARPA (contract D16PC00002), the National Science Foundation (NSF IIS 1409097), and the Center for Brains, Minds and Machines (CBMM, NSF STC award CCF-1231216).
5 APPENDIX
5.1 ADDITIONAL CONTROL MODELS
Table 3 contains results for additional variations of the PredNet and CNN-LSTM Encoder-Decoder evaluated on the CalTech Pedestrian Dataset after being trained on KITTI. We evaluate the models in terms of pixel prediction, thus using the PredNet model trained with loss only on the lowest layer (PredNet L0) as the base model. In addition to mean-squared error (MSE) and the Structural Similarity Index Measure (SSIM), we include calculations of the Peak Signal-To-Noise Ratio (PSNR). For each model, we evaluate it with the original set of hyperparameters (controlling the number of layers, filter sizes, and number of filters per layer), as well as with the four additional sets of hyperparameters that were randomly sampled from the initial random search (see main text for more details). Below is an explanation of the additional control models:
MSE (x 10−3) PSNR SSIM PredNet 3.13 (3.33) 25.8 (25.5) 0.884 (0.878) PredNet (no El split) 3.20 (3.37) 25.6 (25.4) 0.883 (0.878) CNN-LSTM Enc.-Dec. 3.67 (3.91) 25.0 (24.6) 0.865 (0.856) CNN-LSTM Enc.-Dec. (2x Al filts) 3.82 (3.97) 24.8 (24.6) 0.857 (0.853) CNN-LSTM Enc.-Dec. (except pass E0) 3.41 (3.61) 25.4 (25.1) 0.873 (0.866) CNN-LSTM Enc.-Dec. (+/- split) 3.71 (3.84) 24.9 (24.7) 0.861 (0.857) Copy Last Frame 7.95 20.0 0.762
Equalizing the number of filters in the CNN-LSTM Encoder-Decoder (2x Al filts) cannot account for its performance difference with the PredNet, and actually leads to overfitting and a decrease in performance. Passing the error at the lowest layer (E0) in the CNN-LSTM Enc.-Dec. improves performance, but still does not match the PredNet, where errors are passed at all layers. Finally, splitting the activationsAl into positive and negative populations in the CNN-LSTM Enc.-Dec. does not help, but the PredNet with linear error activation (“no El split”) performs slightly worse than the original split version. Together, these results suggest that the PredNet’s error passing operation can lead to improvements in next-frame prediction performance.
5.2 COMPARING AGAINST OTHER MODELS
While our main comparison in the text was a control model that isolates the effects of the more unique components in the PredNet, here we directly compare against other published models. We report results on a 64x64 pixel, grayscale car-cam dataset and the Human3.6M dataset (Ionescu et al., 2014) to compare against the two concurrently developed models by Brabandere et al. (2016)
and Finn et al. (2016), respectively. For both comparisons, we use a model with the same hyperparameters (# of layers, # of filters, etc.) of the PredNet L0 model trained on KITTI, but train from scratch on the new datasets. The only modification we make is to train using an L2 loss instead of the effective L1 loss, since both models train with an L2 loss and report results using L2-based metrics (MSE for Brabandere et al. (2016) and PSNR for Finn et al. (2016)). That is, we keep the original PredNet model intact but directly optimize using MSE between actual and predicted frames. We measure next-frame prediction performance after inputting 3 frames and 10 frames, respectively, for the 64x64 car-cam and Human3.6M datasets, to be consistent with the published works. We also include the results using a feedforward multi-scale network, similar to the model of Mathieu et al. (2016), on Human3.6M, as reported by Finn et al. (2016).
On a dataset similar to KITTI, our model outperforms the model proposed by Brabandere et al. (2016). On Human3.6M, our model outperforms a model similar to (Mathieu et al., 2016), but underperforms Finn et al. (2016), although we note we did not perform any hyperparameter optimization.
5.3 MULTIPLE TIME STEP PREDICTION
While the models presented here were originally trained to predict one frame ahead, they can be made to predict multiple frames by treating predictions as actual input and recursively iterating. Examples of this process are shown in Figure 6 for the PredNet L0 model. Although the next frame predictions are reasonably accurate, the model naturally breaks down when extrapolating further into the future. This is not surprising since the predictions will unavoidably have different statistics than the natural images for which the model was trained to handle (Bengio et al., 2015). If we additionally train the model to process its own predictions, the model is better able to extrapolate. The third row for every sequence shows the output of the original PredNet fine-tuned for extrapolation. Starting from the trained weights, the model was trained with a loss over 15 time steps, where the actual frame was inputted for the first 10 and then the model’s predictions were used as input to the network for the last 5. For the first 10 time steps, the training loss was calculated on the El activations as usual, and for the last 5, it was calculated directly as the mean absolute error with respect to the ground truth frames. Despite eventual blurriness (which might be expected to some extent due to uncertainty), the fine-tuned model captures some key structure in its extrapolations after the tenth time step. For instance, in the first sequence, the model estimates the general shape of an upcoming shadow, despite minimal information in the last seen frame. In the second sequence, the model is able to extrapolate the motion of a car moving to the right. The reader is again encouraged to visit https://coxlab.github.io/prednet/ to view the predictions in video form. Quantitatively, the MSE of the model’s predictions stay well below the trivial solution of copying the last seen frame, as illustrated in Fig 7. The MSE increases fairly linearly from time steps 2-10, even though the model was only trained for up to t+ 5 prediction.
5.4 ADDITIONAL STEERING ANGLE ANALYSIS
In Figure 8, we show the steering angle estimation accuracy on the Comma.ai (Biasini et al., 2016) dataset using the representation learned by the PredNet L0 model, as a function of the number of frames inputted into the model. The PredNet’s representation at all layers was concatenated (after spatially pooling lower layers to a common spatial resolution) and a fully-connected readout was fit using MSE. For each level of the number of training examples, we average over 10 cross-validation splits. To serve as points of reference, we include results for two static models. The first model is an autoencoder trained on single frame reconstruction with appropriately matching hyperparameters. A fully-connected layer was fit on the autoencoder’s representation to estimate the steering angle in the same fashion as the PredNet. The second model is the default model in the posted Comma.ai code (Biasini et al., 2016), which is a five layer CNN. This model is trained end-to-end to estimate
the steering angle given the current frame as input, with a MSE loss. In addition to 25K examples, we trained a version using all of the frames in the Comma dataset (~396K). For all models, the final weights were chosen at the minimum validation error during training. Given the relatively small number of videos in the dataset compared to the average duration of each video, we used 5% of each video for validation and testing, chosen as a random continuous chunk, and discarded the 10 frames before and after the chosen segments from the training set.
As illustrated in Figure 8, the PredNet’s performance gets better over time, as one might expect, as the model is able to accumulate more information. Interestingly, it performs reasonably well after just one time step, in a regime that is orthogonal to the training procedure of the PredNet where there are no dynamics. Altogether, these results again point to the usefulness of the model in learning underlying latent parameters.
5.5 PREDNET Lall NEXT-FRAME PREDICTIONS
Figures 9 and 10 compare next-frame predictions by the PredNet Lall model, trained with a prediction loss on all layers (λ0 = 1, λl>0 = 0.1), and the PredNet L0 model, trained with a loss only on the lowest layer. At first glance, the difference in predictions seem fairly minor, and indeed, in terms of MSE, the Lall model only underperformed the L0 version by 3% and 6%, respectively, for the rotating faces and CalTech Pedestrian datasets. Upon careful inspection, however, it is apparent that the Lall predictions lack some of the finer details of the L0 predictions and are more blurry in regions of high variance. For instance, with the rotating faces, the facial features are less defined and with CalTech, details of approaching shadows and cars are less precise.
Actual
PredNet 𝐿0
PredNet 𝐿𝑎𝑙𝑙
Error 𝐿𝑎𝑙𝑙 - 𝐿0
Actual
PredNet 𝐿0
PredNet 𝐿𝑎𝑙𝑙
Error 𝐿𝑎𝑙𝑙 - 𝐿0
Actual
PredNet 𝐿0
PredNet 𝐿𝑎𝑙𝑙
Error 𝐿𝑎𝑙𝑙 - 𝐿0
Actual
PredNet 𝐿0
PredNet 𝐿𝑎𝑙𝑙
Error 𝐿𝑎𝑙𝑙 - 𝐿0

Paper Summary
This paper proposes an unsupervised learning model in which the network
predicts what its state would look like at the next time step (at input layer
and potentially other layers).  When these states are observed, an error signal
is computed by comparing the predictions and the observations. This error
signal is fed back into the model. The authors show that this model is able to
make good predictions on a toy dataset of rotating 3D faces as well as on
natural videos. They also show that these features help perform supervised
tasks.
Strengths
- The model is an interesting embodiment of the idea of predictive coding
  implemented using a end-to-end backpropable recurrent neural network architecture.
- The idea of feeding forward an error signal is perhaps not used as widely as it could
  be, and this work shows a compelling example of using it.
- Strong empirical results and relevant comparisons show that the model works well.
- The authors present a detailed ablative analysis of the proposed model.
Weaknesses
- The model (esp. in Fig 1) is presented as a generalized predictive model
  where next step predictions are made at each layer. However, as discovered by
running the experiments, only the predictions at the input layer are the ones
that actually matter and the optimal choice seems to be to turn off the error
signal from the higher layers. While the authors intend to address this in future
work, I think this point merits some more discussion in the current work, given
the way this model is presented.
- The network currently lacks stochasticity and does not model the future as a
  multimodal distribution (However, this is mentioned as potential future work).
Quality
The experiments are well-designed and a detailed analysis is provided
in the appendix.
Clarity
The paper is well-written and easy to follow.
Originality
Some deep models have previously been proposed that use predictive coding.
However, the proposed model is most probably novel in the way it feds back the
error signal and implements the entire model as a single differentiable
network.
Significance
This paper will be of wide interest to the growing set of researchers working
in unsupervised learning of time series. This helps draw attention to
predictive coding as an important learning paradigm.
Overall
Good paper with detailed and well-designed experiments. The idea of feeding
forward the error signal is not being used as much as it could be in our
community. This work helps to draw the community's attention to this idea.


TRAINING DEEP NEURAL-NETWORKS USING A NOISE ADAPTATION LAYER
The availability of large datsets has enabled neural networks to achieve impressive recognition results. However, the presence of inaccurate class labels is known to deteriorate the performance of even the best classifiers in a broad range of classification problems. Noisy labels also tend to be more harmful than noisy attributes. When the observed label is noisy, we can view the correct label as a latent random variable and model the noise processes by a communication channel with unknown parameters. Thus we can apply the EM algorithm to find the parameters of both the network and the noise and estimate the correct label. In this study we present a neural-network approach that optimizes the same likelihood function as optimized by the EM algorithm. The noise is explicitly modeled by an additional softmax layer that connects the correct labels to the noisy ones. This scheme is then extended to the case where the noisy labels are dependent on the features in addition to the correct labels. Experimental results demonstrate that this approach outperforms previous methods.
1 INTRODUCTION
The presence of class label noise inherent to training samples has been reported to deteriorate the performance of even the best classifiers in a broad range of classification problems (Nettleton et al. (2010), Pechenizkiy et al. (2006), Zhu & Wu (2004)). Noisy labels also tend to be more harmful than noisy attributes (Zhu & Wu (2004)). Noisy data are usually related to the data collection process. Typically, the labels used to train a classifier are assumed to be unambiguous and accurate. However, this assumption often does not hold since labels that are provided by human judgments are subjective. Many of the largest image datasets have been extracted from social networks. These images are labeled by non-expert users and building a consistent model based on a precisely labeled training set is very tedious. Mislabeling examples have been reported even in critical applications such as biomedical datasets where the available data are restricted (Alon et al. (1999)). A very common approach to noisy datasets is to remove the suspect samples in a preprocessing stage or have them relabeled by a data expert (Brodley & Friedl (1999)). However, these methods are not scalable and may run the risk of removing crucial examples that can impact small datasets considerably.
Variants that are noise robust have been proposed for the most common classifiers such as logisticregression and SVM (Frénay & Verleysen (2014), Jakramate & Kabán (2012), Beigman & Klebanov (2009)). However, classifiers based on label-noise robust algorithms are still affected by label noise. From a theoretical point of view, Bartlett et al. (2006) showed that most loss functions are not completely robust to label noise. Natarajan et al. (2013) proposed a generic unbiased estimator for binary classification with noisy labels. They developed a surrogate cost function that can be expressed by a weighted sum of the original cost functions, and provided asymptotic bounds for performance. Grandvalet & Bengio (2005) addressed the problem of missing labels that can be viewed as an extreme case of noisy label data. They suggested a semi-supervised algorithm that encourages the classifier to predict the non-labeled data with high confidence by adding a regularization term to the cost function. The problem of classification with label noise is an active research area. Comprehensive up-to-date reviews of both the theoretical and applied aspects of classification with label noise can be found in Frénay & Kaban (2014) and Frénay & Verleysen (2014).
In spite of the huge success of deep learning there are not many studies that have explicitly attempted to address the problem of Neural Net (NN) training using data with unreliable labels. Larsen et al. (1998) introduced a single noise parameter that can be calculated by adding a new regularization term and cross validation. Minh & Hinton (2012) proposed a more realistic noise model that depends on the true label. However, they only considered the binary classification case. Sukhbaatar & Fergus (2014) recently proposed adding a constrained linear layer at the top of the softmax layer, and showed that only under some strong assumptions can the linear layer be interpreted as the transition matrix between the true and noisy (observed) labels and the softmax output layer as the true probabilities of the labels. Reed et al. (2014) suggested handling the unreliability of the training data labels by maximizing the likelihood function with an additional classification entropy regularization term.
The correct unknown label can be viewed as a hidden random variable. Hence, it is natural to apply the EM algorithm where in the E-step we estimate the true label and in the M-step we retrain the network. Several variations of this paradigm have been proposed (e.g. Minh & Hinton (2012), Bekker & Goldberger (2016)). However, iterating between EM-steps and neural network training does not scale well. In this study we use latent variable probabilistic modeling but we optimize the likelihood score function within the framework of neural networks. Current noisy label approaches assume either implicitly or explicitly that, given the correct label, the noisy label is independent of the feature vector. This assumption is probably needed to simplify the modeling and derive applicable learning algorithms. However, in many cases this assumption is not realistic since a wrong annotation is more likely to occur in cases where the features are misleading. By contrast, our framework makes it easy to extend the proposed learning algorithm to the case where the noise is dependent on both the correct label and the input features. In the next section we describe a model formulation and review the EM based approach. In Section 3 we described our method which is based on adding another softmax layer to the network and in Section 4 we present our results.
2 A PROBABILISTIC FRAMEWORK FOR NOISY LABELS
Assume we want to train a multi-class neural-network soft-classifier p(y = i|x;w) where x is the feature vector, w is the network parameter-set and i is a member of the class-set {1, ..., k}. We further assume that in the training process we cannot directly observe the correct label y. Instead, we only have access to a noisy version of it denoted by z. Here we follow the probabilistic modeling and the EM learning approach described in Bekker & Goldberger (2016). In this approach noise generation is assumed to be independent of the features and is modeled by a parameter θ(i, j) = p(z = j|y = i). The noise distribution is unknown and we want to learn it as part of the training phase. The probability of observing a noisy label z given the feature vector x is:
p(z = j|x;w, θ) = k∑
i=1
p(z = j|y = i; θ)p(y = i|x;w) (1)
where k is the number of classes. The model is illustrated in the following diagram:
Neural-Network
w
noisy channel
θ
x y z
In the training phase we are given n feature vectors x1, ..., xn with the corresponding noisy labels z1, ..., zn which are viewed as noisy versions of the correct hidden labels y1, ..., yn. The loglikelihood of the model parameters is:
L(w, θ) = n∑ t=1 log( k∑ i=1 p(zt|yt = i; θ)p(yt = i|xt;w)) (2)
Based on the training data, the goal is to find both the noise distribution θ and the Neural Network parameters w that maximize the likelihood function. Since the random variables y1, ..., yn are hidden, we can apply the EM algorithm to find the maximum-likelihood parameter set. In the E-step of
each EM iteration we estimate the hidden true data labels based on the noisy labels and the current parameters:
cti = p(yt = i|xt, zt;w0, θ0), i = 1, ..., k, t = 1, ..., n (3)
where w0 and θ0 are the current parameter estimations. In the M-step we update both the NN and the noisy channel parameters. The updated noise distribution has a closed-form solution.
θ(i, j) = ∑ t cti1{zt=j}∑
t cti , i, j ∈ {1, ..., k} (4)
The k × k matrix θ can be viewed as a confusion matrix between the soft estimates of the true label {cti|i = 1, ..., k} and the observed noisy labels zt. As part of the EM M-step, to find the updated NN parameter w we need to maximize the following function:
S(w) = n∑ t=1 k∑ i=1 cti log p(yt = i|xt;w) (5)
which is a soft-version of the likelihood function of the fully observed case, based on the current estimate of the true labels. The back-propagation derivatives of the function (5) that we maximize in the M-step are:
∂S ∂ui = n∑ t=1 (p(yt = i|xt, zt;w0, θ0)− p(yt = i|xt;w))h(xt) (6)
such that h is the final hidden layer and u1, ..., uk are the parameters of the soft-max output layer.
The method reviewed here is closely related to the work of Minh & Hinton (2012). They addressed the problem of mislabeled data points in a particular type of dataset (aerial images). The main difference is that in their approach they assumed that they do not learn the noise parameter. Instead they assume that the noise model can be separately tuned using a validation set or set by hand. Note that even if the true noise parameters are given, we still need the apply the EM iterative procedure. However, this assumption makes the interaction between the E-step and the NN learning much easier since each time a data-point xt is visited we can compute the p(yt = i|xt, zt) based on the current network parameters and the pre-defined noise parameters. Motivated by the need for model compression, Hinton et al. (2014) introduced an approach to learn a “distilled” model by training a more compact neural network to reproduce the output of a larger network. Using the notation defined above, in the second training stage they actually optimized the cost function: S(w) =∑n
t=1 ∑k i=1 p(yt = i|xt;w0, θ0) log p(yt = i;xt;w) such that w0 is the parameter of the larger network that was trained using the labels z1, ..., zn, w is the parameter of the smaller network and θ0(i, j) in this case is a non-informative distribution (i.e. θ0(i, j) = 1/k).
There are several drawbacks to the EM-based approach described above. The EM algorithm is a greedy optimization procedure that is notoriously known to get stuck in local optima. Another potential issue with combining neural networks and EM direction is scalability. The framework requires training a neural network in each iteration of the EM algorithm. For real-world, large-scale networks, even a single training iteration is a non-trivial challenge. Moreover, in many domains (e.g. object recognition in images) the number of labels is very large, so many EM iterations are likely to be needed for convergence. Another drawback of the probabilistic models is that they are based on the simplistic assumption that the noise error is only based on the true labels but not on the input features. In this study we propose a method for training neural networks with noisy labels that successfully addresses all these problems.
3 TRAINING DEEP NEURAL NETWORKS USING A NOISE ADAPTATION LAYER
In the previous section we utilized the EM algorithm to optimize the noisy-label likelihood function (2). In this section we describe an algorithm that optimizes the same function within the framework of neural networks. Assume the neural network classifier we are using is based on non-linear intermediate layers followed by a soft-max output layer used for soft classification. Denote the non-linear
function applied on an input x by h = h(x) and denote the soft-max layer that predicts the true y label by:
p(y = i|x;w) = exp(u > i h+ bi)∑k
l=1 exp(u > l h+ bl)
, i = 1, ..., k (7)
where w is the network parameter-set (including the softmax layer). We next add another softmax output layer to predict the noisy label z based on both the true label and the input features:
p(z = j|y = i, x) = exp(u>ijh+ bij)∑ l exp(u > ilh+ bil)
(8)
p(z = j|x) = ∑ i p(z = j|y = i, x)p(y = i|x) (9)
We can also define a simplified version where the noisy label only depends on the true label; i.e. we assume that labels flips are independent of x:
p(z = j|y = i) = exp(bij)∑ l exp(bil)
(10)
p(z = j|x) = ∑ i p(z = j|y = i)p(y = i|x) (11)
We denote the two noise modeling variants as the complex model (c-model) (8) and the simple model (s-model) (10). Hereafter we use the notation wnoise for all the parameters of the second softmax layer which can be viewed as a noise adaptation layer.
In the training phase we are given n feature vectors x1, ..., xn with corresponding noisy labels z1, ..., zn which are viewed as noisy versions of the correct hidden labels y1, ..., yn. The loglikelihood of the model parameters is:
S(w,wnoise) = ∑ t log p(zt|xt) = ∑ t log( ∑ i p(zt|yt = i, xt;wnoise)p(yt = i|xt;w)) (12)
Since the noise is modeled by adding another layer to the network, the score S(w,wnoise) can be optimized using standard techniques for neural network training. By setting
p(z = j|y = i) = θ(i, j) = exp(bij)∑ l exp(bil) , (13)
it can easily verified that, by using either the EM algorithm (2) or the s-model neural network scheme (12), we are actually optimizing exactly the same function. Thus the neural network with the s-model noise adaptation layer provides an alternative optimization strategy to the EM algorithm. Instead of alternating between optimizing the noisy model and the network classifier, we consider them as components of the same network and optimize them simultaneously.
Note that in the c-model, where the noise is also dependent on the input features, we can still apply the EM algorithm to learn the parameters of the additional noise layer. However, there is no closedform solution in the M-step for the optimal parameters and we need to apply neural-network training in the M-step to find the noise-layer parameters.
At test time we want to predict the true labels. Hence, we remove the last softmax layer that aims to get rid of the noise in the training set. We compute the true-label softmax estimation p(y = i|x;w) (7). The proposed architecture for training the neural network based on training data with noisy labels is illustrated in Figure 1.
There are degrees of freedom in the two softmax layer model. Hence, a careful initialization of the parameters of the noise adaptation layer is crucial for successful convergence of the network into a good classifier of the correct labels at test time. We used the parameters of the original network to initialize the parameters of the s-model network that contains the noise adaptation level. We can initialize the softmax parameters of the s-model by assuming a small uniform noise:
bij = log((1− )1{i=j} +
k−1 1{i 6=j})
such that k is the number of different classes. A better procedure is to first train the original NN without the noise-adaptation layer, ignoring the fact that the labels are noisy. We can then treat the labels produced by the NN as the true labels and compute the confusion matrix on the train set and used it as an initial value for the bias parameters:
bij = log(
∑ t 1{zt=j}p(yt = i|xt)∑
t p(yt = i|xt) )
such that x1, ..., xn are the feature vectors of the training dataset and z1, ..., zn are the corresponding noisy labels. So far we have concentrated on parameter initialization for the s-model. The strategy that works best to initialize the c-model parameters is to use the parameters that were optimized for the s-model. In other words we set linear terms uij to zero and initialize the bias terms bij with the values that were optimized by the s-model.
The computational complexity of the proposed method is quadratic in the size of the class-set. Suppose there are k classes to predict, in this case the proposed methods require k+1 sets of softmax operations with a size of k each. Hence there are scalability problems when the class set is large. As we explained in the previous paragraph, we initialized the second soft-max layer using the confusion matrix of the baseline system. The confusion matrix is a good estimation of the label noise. Assume the rows of the matrix correspond to the true labels and the matrix columns correspond to the noisy labels. The l largest elements in the i-th row are the most frequent noisy class values when the true class value is i. We can thus connect the i-th element in the first softmax layer only to its l most probable noisy class candidates. Note that if we connect the i-th label in the first softmax only to the i-th label in the second softmax layer, the second softmax layer collapses to identity and we obtain the standard baseline model. Taking the l most likely connections to the second softmax layer, we allow an additional l−1 possible noisy labels for each correct label. We thus obtain a data driven sparsifying of the second softmax layer which solves the scalability problem since the complexity becomes linear in the number of classes instead of quadratic. In the experiment section we show that by using this approach there is not much deference in performance.
Our architecture, which is based on a concatenation of softmax layers, resembles the hierarchical softmax approach Morin & Bengio (2005) that replaces the flat softmax layer with a hierarchical layer that has the classes as leaves. This allowed them to decompose calculating the probability of the class into a sequence of probability calculations, which saves us from having to calculate the expensive normalization over all classes. The main difference between our approach and theirs (apart from the motivation) is that in our approach the true-label softmax layer is fully connected to the noisy-label layer. Sukhbaatar & Fergus (2014) suggested adding a linear layer to handle noisy labels. Their approach is similar to our s-model. In their approach, however, they proposed a different learning procedure.
4 EXPERIMENTS
In this section, we evaluate the robustness of deep learning to training data with noisy labels with and without explicit noise modeling. We first show results on the MNIST data-set with injected label
noise in our experiments. The MNIST is a database of handwritten digits, which consists of 28× 28 images. The dataset has 60k images for training and 10k images for testing. We used a two hidden layer NN comprised of 500 and 300 neurons. The non-linear activation we used was ReLU and we used dropout with parameter 0.5. We trained the network using the Adam optimizer (Kingma & Ba (2014)) with default parameters, which we found to converge more quickly and effectively than SGD. We used a mini-batch size of 256. These settings were kept fixed for all the experiments described below. In addition to a network that is based on fully connected layers, we also applied a network based on a CNN architecture. The results we obtained in the two architectures were similar. The network we implemented is publicly available 1.
We generated noisy data from clean data by stochastically changing some of the labels. We converted each label with probability p to a different label according to a predefined permutation. We used the same permutation as in Reed et al. (2014). The labels of the test data remained, of course, unperturbed to validate and compare our method to the regular approach.
We compared the proposed noise robust models to other model training strategies. The first network was the baseline approach that ignores the fact that the labels of the training data are unreliable. Denote the observed noisy label by z and the softmax decision by q1, ..., qk. The baseline loglikelihood score (for a single input) is:
S = ∑ i 1{z=i} log(qi)
1code available at https://github.com/udibr/noisy_labels
We also implemented two variants of the noise robust approach proposed by Reed et al. (2014). They suggested a soft version
βS − (1− β)H(q) = β ∑ i 1{z=i} log(qi) + (1− β) ∑ i qi log(qi)
and a hard version: βS + (1− β)max
i log(qi)
In their experiments they took β = 0.8 for the hard version and β = 0.95 for the soft version, and observed that the hard version provided better results. Finally we implemented the two variants of our approach; namely, the noise modeling based only on the labels (s-model) and the noise modeling that was also based on the features (c-model).
Figure 2 depicts the comparative test errors results as a function of the fractions of noise. The results are shown for three different sizes of training data i.e. (20%,50%,100%) of the MNIST training subset. Bootstrapping was used to compute confidence intervals around the mean. For 1000 times, N = 10 samples were randomly drawn with repeats from the N available samples and mean was computed. The confidence interval was taken to be the 2.5% and 97.5% percentiles of this process.
The results show that all the methods that are explicitly aware of the noise in the labels are better than the baseline which is the standard training approach. We revalidated the results reported in Reed et al. (2014) and showed that the hard version of their method performs better than the soft version. In all cases our models performed better than the alternatives. In most cases the c-model was better than the s-model. In the case where the entire dataset was used for training, we can see from the results that there was a phase transition phenomenon. We obtained almost perfect classification results until the noise level was high and there was a sudden strong performance drop. Analyzing why this effect occurred is left for future research.
We next show the results on the CIFAR-100 image dataset Krizhevsky & Hinton (2009) which consists of 32 × 32 color images arranged in 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. We used raw images directly without any preprocessing or augmentation. We generated noisy data from clean data by stochastically changing some of the labels. We converted each one of the 100 labels with probability p to a different label according to a predefined permutation. The labels of the test data remained, of course, unperturbed to validate and compare our method to the regular approach. We used a CNN network with two convolutional layers combined with ReLU activation and max-pooling, followed by two fully connected layers. Figure 3 depicts the comparative test errors results as a function of the fractions of noise for three different sizes of training data i.e. (20%,50%,100%) of the CIFAR-100 training
subset. Bootstrapping was used to compute confidence intervals around the mean in the same way as for the MNIST experiment. The results showed that the proposed method works better than the alternatives. The simple model consistently provided the best results but when the noise level was very high the complex method tended to perform better.
We next report experimental results for the sparse variant of our method that remains efficient even when the class set is large. We demonstrate this on the case of the CIFAR-100 dataset which consists of 100 possible classes. For each class we only took the five most probable classes in the confusion matrix which is used to initialize the model parameter (see Section 3). As can be seen in Figure 4, sparsifying the second softmax layer did not not result in a drop in performance
5 CONCLUSION
In this paper we investigated the problem of training neural networks that are robust to label noise. We proposed an algorithm for training neural networks based solely on noisy data where the noise distribution is unknown. We showed that we can reliably learn the noise distribution from the noisy data without using any clean data which, in many cases, are not available. The algorithm can be easily combined with any existing deep learning implementation by simply adding another softmax output layer. Our results encourage collecting more data at a cheaper price, since mistaken data labels can be less harmful to performance. One possible future research direction would be to generalize our learning scheme to cases where both the features and the labels are noisy. We showed results on datasets with small and medium sized class-sets. Future research direction would be to evaluate the performance and efficiency of the proposed method on tasks with large class-sets.
ACKNOWLEDGMENTS
This work is supported by the Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI).

This work address the problem of supervised learning from strongly labeled data with label noise. This is a very practical and relevant problem in applied machine learning.  The authors note that using sampling approaches such as EM isn't effective, too slow and cannot be integrated into end-to-end training. Thus, they propose to simulate the effects of EM by a noisy adaptation layer, effectively a softmax, that is added to the architecture during training, and is omitted at inference time. The proposed algorithm is evaluated on MNIST and shows improvements over existing approaches that deal with noisy labeled data.
A few comments.
1. There is no discussion in the work about the increased complexity of training for the model with two softmaxes.
2. What is the rationale for having consecutive (serialized) softmaxes, instead of having a compound objective with two losses, or a network with parallel losses and two sets of gradients?
3. The proposed architecture with only two hidden layers isn't not representative of larger and deeper models that are practically used, and it is not clear that shown results will scale to bigger networks.
4. Why is the approach only evaluated on MNIST, a dataset that is unrealistically simple.


TREE-STRUCTURED VARIATIONAL AUTOENCODER
Many kinds of variable-sized data we would like to model contain an internal hierarchical structure in the form of a tree, including source code, formal logical statements, and natural language sentences with parse trees. For such data it is natural to consider a model with matching computational structure. In this work, we introduce a variational autoencoder-based generative model for tree-structured data. We evaluate our model on a synthetic dataset, and a dataset with applications to automated theorem proving. By learning a latent representation over trees, our model can achieve similar test log likelihood to a standard autoregressive decoder, but with the number of sequentially dependent computations proportional to the depth of the tree instead of the number of nodes in the tree.
1 INTRODUCTION
A significant amount of recent and ongoing work has explored the use of neural networks for modeling and generating various kinds of data. Newer techniques like the variational autoencoder (Rezende et al., 2014; Kingma & Welling, 2013) and generative-adversarial networks (Goodfellow et al., 2014) enable training of graphical models where the likelihood function is a complicated neural network which normally makes it infeasible to specify and optimize the marginal distribution analytically. Another family of techniques involves choosing an ordering of the dimensions of the data (which is particularly natural for sequences such as sentences) and training a neural network to estimate the distribution over the value of the next dimension given all the dimensions we have observed so far.
These techniques have led to significant advances in modeling images, text, sounds, and other kinds of complicated data. Language modeling with sequential neural models have halved perplexity (roughly, the error at predicting each word) compared to n-gram methods (Jozefowicz et al., 2016). Neural machine translation using sequence-to-sequence methods have closed half of the gap in quality between prior machine translation efforts and human translation (Wu et al., 2016). Generative image models have similarly progressed such that they can generate samples largely indistinguishable from the original data, at least for relatively small and simple images (Gregor et al., 2015; 2016; Kingma et al., 2016; Salimans et al., 2016; van den Oord et al., 2016), although the quality of the model here is harder to measure in an automated way (Theis et al., 2015).
However, many kinds of data we might wish to model are naturally structured as a tree. Computer program code follows a rigorous grammar, and the usual first step in processing it involves parsing it into an abstract syntax tree, which simultaneously discards aspects of the code irrelevant to the semantics such as whitespace and extraneous parentheses, and makes it more convenient to further interpret, analyze, or manipulate. Statements made in formal logic similarly have a hierarchical structure, which determines arguments to predicates and functions, the scoping of variables and quantifiers, and the application of logical connectives. Natural language sentences also contain a latent syntactic structure which is necessary for determining the meaning of the sentence, although
∗Majority of work done while at Google.
most sentences admit many possible parses due to the multiple meanings of each word and ambiguous groupings.
In this paper, we explore how we can adapt the variational autoencoder to modeling tree-structured data. In general, it is possible to treat the tree as a sequence and then use a sequential model. However, a model which follows the structure of the tree may better capture long-range dependencies: recurrent models sometimes have trouble learning to remember and use information from the distant past when it is relevant to the current context, but these distant parts of the input may be close to each other within the tree structure.
In our proposed approach, we decide the identity of each node in the tree by using a top-down recursive neural network, causing the distributed representation which decides the identity of each node in the tree to be computed as a function of the identity and relative location of its parent nodes.
By using the architecture of the variational autoencoder (Rezende et al., 2014; Kingma & Welling, 2013), our model can learn to capture various features of the trees within continuous latent variables, which are added as further inputs into the top-down recursive neural network and conditions the overall generation process. These latent variables allow us to generate different parts of the tree in parallel; specifically, given a parent node n and its children c1 and c2, the generation of (the distribution placed over different values of) c1 and its descendants is independent of the generation of c2 and its descendants (and vice versa), once we condition upon the latent variables. By structuring the model this way, while our model generates one dimension (the identity of each node within the tree) of the data a time, it is not autoregressive as the probability distribution for one dimension is not a function of the previously generated nodes.
We evaluate our model on a variety of datasets, some synthetic and some real. Our experimental results show that it achieves comparable test set log likelihood to autoregressive sequential models which do not use any latent variables, while offering the following properties:
• For balanced trees, generation requires O(log n) rather than O(n) timesteps required for a sequential model because the children of each node can be generated in parallel. • It is straightforward to resample a subtree while keeping the other parts of the tree intact. • The generated trees are syntactically valid by construction. • The model produces a latent representation for each tree, which may prove useful in other
applications.
2 BACKGROUND AND RELATED WORK
2.1 TREE-STRUCTURED MODELS
Recursive neural nets, which processes a tree in a bottom-up way, have been popular in natural language processing for a variety of tasks, such as sentiment analysis (Socher et al., 2013), question answering (Iyyer et al., 2014), and semantic relation extraction (Socher et al., 2012). Starting from the leaves of the tree, the model computes a representation for each node by combining the representations of its child nodes. In the case of natural language processing, each tree typically represents one sentence, with the leaf nodes corresponding to the words in the sentence and the structure of the internal nodes determined by the constituency parse tree for the sentence.
If we restrict ourselves to binary trees (given that it is possible to binarize arbitrary trees in a lossless way), the we compute the k-dimensional representation rn ∈ Rk for a node n by combining the representations of nodes nleft and nright:
rn = f(Wrnleft + V rnright)
where W and V are square matrices (in Rd×d) and f is a nonlinear activation function, applied elementwise. Leaf nodes are represented by embedding the content of the leaf node into a ddimensional vector, by specifying a lookup table from words to embedding vectors for instance.
Variations and extensions of this approach specify more complicated relationships between rn and the childen’s representations rnleft and rnright , or allow internal nodes to have variable numbers of children. For example, Tai et al. (2015) extend LSTMs to tree-structured models by dividing the vector
representation of each node into a memory cell and a hidden state and updating them accordingly with gating.
Neural network models which generate or explore a tree top-down have received less attention, but have been applied to generation and parsing tasks. Zhang et al. (2015) generate natural language sentences along with their dependency parses simultaneously. In their specification of a dependency parse tree, each word is connected to one parent word and has a variable number of left and right children (a depth-first in-order traversal on this tree recovers the original sentence). Their model generates these trees by conditioning the probability of generating each word on its ancestor words and the previously-generated sibling words using various LSTMs. Dyer et al. (2016) generate sentences jointly with a corresponding constituency parse tree. They use a shift-reduce parser architecture where shift is replaced with word-generation action, and so the sentence and the tree can be generated by performing a sequence of actions corresponding to a depth-first pre-order traversal of the tree. Each action in the sequence is predicted based upon the tree constructed so far, the words (the tree’s terminal nodes) generated so far, and the previous actions performed. Dong & Lapata (2016) generate tree-structured logical forms using LSTMs, where the LSTM state branches along with the tree’s structure; they focus on generating these logical forms when conditioned upon a natural-language description of it.
2.2 VARIATIONAL AUTOENCODERS
The variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014), or VAE for short, provides a way to train a generative model with a fixed prior p(z) and a neural network used to specify pθ(x | z). Typically, the prior p(z) is taken to be a standard multivariate normal distribution (mean at 0) with diagonal unit covariance. Naively, in order to optimize log p(x), we need to compute the following integral:
log pθ(x) = log ∫ z pθ(x | z)p(z)dz
which can be tractable when pθ(x | z) is simple but not when we want to use a neural network to represent it. Inference of the posterior p(z | x) also becomes intractable. Instead, we learn a second neural network qφ(z | x) to approximate the true posterior, and use the following variational bound:
log p(x) ≥ −DKL(qφ(z | x) ‖ p(z)) + Eqφ(z|x)[log pθ(x | z)]
where DKL represents the Kullback-Leibler divergence between the two distributions. Given that we represent qφ(z | x) with a neural network which outputs the mean and diagonal covariance for a normal distribution, we can analytically compute the KL divergence term and then use the reparameterization trick:
Eqφ(z|x)[log pθ(x | z)] = Ep( )[log pθ(x | z = µ+ σ · )]
where p( ) is a standard multivariate normal distribution, and µ and σ are outputs of the neural network implementing qφ(z | x). These two techniques combined allow us to compute stochastic gradients (by sampling , treating it as constant, and backpropagating through the model) and use standard neural network training techniques (such as SGD, Adagrad, and Adam) to train the model.
Another interpretation of the variational autoencoder follows from a modification of the regular autoencoder, where we would like to learn a mapping x → z from the data to a more compact representation z, and an inverse mapping z → x. In the VAE, we replace the deterministic x → z with a probabilistic q(z | x), and as a form of regularization, we ensure that this distribution is close to a prior p(z).
3 TREE-STRUCTURED VARIATIONAL AUTOENCODER
In this section, we describe how we combine the variational autoencoder and recursive neural networks in order to build our model.
3.1 TREES
We consider arbitrarily-branching typed trees where each node contains a type, and either child nodes or a terminal value. Each type may be a terminal type or a non-terminal type; nodes of terminal type contain a value, and nodes of non-terminal type have (zero or more) child nodes.
A non-terminal type T comes with a specification for how many children a node NT of type T should have, and the types permissible for each child location. We distinguish three types of child nodes:
• NT may have some number of singular child nodes. For the ith singular child, we specify SINGULARCHILD(T, i) = {T1, · · · , Tn} as the set of types that child node can have. If the singular child node is optional, we denote this by including φ in this set. SINGULARCHILDCOUNT(T ) gives the number of singular child nodes in T .
• NT may have an arbitrary number of repeated child nodes. Each repeated child node must have type belonging within REPEATEDCHILDREN(T ) = {T1, · · · }. If this set is empty, no repeated child nodes are allowed. These children may be of different types.
For each terminal type, we have a list of values that a node of this type can have. We also have a list of types that the root node can have.
The above specification serves as an extension of context-free grammars, which are commonly used to specify formal languages. The main difference is in optional and repeated children, which makes it easier to specify an equivalent grammar with fewer non-terminal types.
As an example, consider the for loop in the C programming language. A node representing this contains three singular children: an initializer expression, the condition expression (evaluated to check whether the loop should continue running), and the iteration statement, which runs at the end of each loop iteration. It also has repeated children, one child per statement in the loop body.
3.2 BUILDING A TREE
Now that we have specified the kinds of trees we consider, let us look at how we might build one. First, we describe the basic building block that we use to create one node, then we will look at how to compose these together to build an entire tree.
Assume that we know the node that we are about to construct should have type T , and that we have a hidden state h ∈ Rk which contains further information about the node.
• If T is a terminal type, we use WHICHTERMINALVALUET (h), producing a probability distribution over the set of possible values, and sample from this distribution to choose the value.
• If T is a non-terminal type, we use the following procedure GENERATENODE(T,h): 1. Compute m = SINGULARCHILDCOUNT(T ) + 1{REPEATEDCHILDREN(T ) 6= ∅}.
In other words, count the number of singular children, and add 1 if the type allows repeated children.
2. Compute g1, · · · ,gm = SPLITT (h). The function SPLITT (h) : Rk → Rk×· · ·×Rk maps the k-dimensional vector h into n separate k-dimensional vectors g1 to gm.
3. For each singular child i: (a) Sample Ti ∼ WHICHCHILDTYPET,i(gi) from a distribution over the types in
REQUIREDCHILD(T, i). (b) If Ti 6= ∅, use GENERATENODE(Ti,gi) to build the child node recursively.
4. If T specifies repeated children: (a) Compute gcur,gnext = SPLITREPEATEDT (gm). (b) Sample s ∼ STOPREPEATT (gcur) from a Bernoulli distribution. If s = 1, stop
generating repeated children. (c) Sample Tchild ∼ WHICHCHILDTYPET,repeated(gcur), a probability distribution
over the types in REPEATEDCHILDREN(T ). (d) Use GENERATENODE(Tchild,gcur) to build this child recursively. (e) Set gm := gnext and repeat this loop.
For building the entire tree starting from the root, we assume that we have an embedding z which encodes information about the entire tree (we describe how we obtain this in the next section). We sample Troot ∼ WHICHROOTTYPE(z), the type for the root node, and then run GENERATENODE(Troot, z).
3.3 ENCODING A TREE
Recall that the variational autoencoder framework involves training two models simultaneously: p(x | z), the generative (or decoding) model, and q(z | x), the inference (or encoding) model. The previous section described how we specify p(x | z), so we now turn our attention to q(z | x). Overall, we build the inference model by inverting the flow of data in the generative model. Specifically, we use ENCODE(n) to encode a node n with type T :
• If T is a terminal type, return EMBEDDING(v) ∈ Rk by performing a lookup of the contained value v within a table.
• If T is a non-terminal type: 1. Compute gi = ENCODE(ni) for each singular child ni of n. If ni is missing, then gi = 0.
2. If T specifies repeated children, set grepeated := 0 and nchild to the last repeated child of n, and then run: (a) Compute gchild = ENCODE(nchild). (b) Set grepeated := MERGEREPEATEDT (grepeated, gchild) ∈ Rk. (c) Move nchild to the previous repeated child, and repeat (until we run out of repeated children). 3. Return MERGET (g1, . . . , gm, goptional, grepeated) ∈ Rk.
Thus hroot = ENCODE(nroot) gives a summary of the entire tree as a k-dimensional embedding. We then construct q(z | x) = N (µ, σ) where µ = Wµhroot and σ = softplus(Wσhroot). Applying softplus(x) = log(1 + ex) as a nonlinearity gives us a way to ensure that σ is positive as required.
3.4 IMPLEMENTING SPLIT, MERGE, AND WHICH FUNCTIONS
In the previous two sections, we described how the model traverses the tree bottom-up to produce an encoding of it (q(z | x)), and how we can generate a tree top-down from the encoding (p(x | z)). In this section, we explicate the details of how we implemented the SPLIT, MERGE, and WHICH functions that we used previously.
COMBINE. We can consider SPLIT : Rk → Rk × · · · × Rk and MERGE : Rk × · · · × Rk → Rk functions to be specializations of a more general function COMBINE : Rk×· · ·×Rk → Rk×· · ·×Rk which takes m inputs and produces n outputs.
A straightforward implementation of COMBINE is the following: y1, . . . ,yn := COMBINE(x1, . . . ,xm)
[y1 · · · yn] = f(W [x1 · · · xm] + b) where we have taken xi and yi to be column vectors Rk, [x1 · · ·xm] stacks the vectors xi vertically, W ∈ Rn·k×m·k and b ∈ Rn·k are the learned weight matrix and bias vector respectively, and f is a nonlinearity applied elementwise.
For WHICH : Rk → Rd, which computes a probability distribution over d choices, we use a specialization of COMBINE with one input and one (d-sized rather than k-sized) output, and use softmax as the nonlinearity f .
While this basic implementation sufficed initially, we discovered that two modifications led to better performance, which we describe subsequently.
Gating. We added a form of multiplicative gating, similar to those used in Gated Recurrent Units (Chung et al., 2014), Highway Networks (Srivastava et al., 2015), and Gated PixelCNN (van den Oord et al., 2016). The multiplicative gate enables the COMBINE function to more easily pass through information in the inputs to the outputs if that is preferable to transforming the input. Furthermore, the multiplicative interactions used in computing the gates may help the neural network learn more complicated functions.
First, we compute candidate values for yi using a linear layer as before: [ŷ1 · · · ŷn] = f(W [x1 · · · xm] + b)
Then we compute multiplicative gates for each ŷi and each (xi,yj) combination, or (m+ 1)n gate variables (recall that m is the number of inputs and n is the number of outputs).
[gy1 · · · gyn ] = σ(Wgy [x1 · · · xm] + bgy )[ g(x1,y1) · · · g(x1,yn) ] = σ(Wg1 [x1 · · · xm] + bg1)
...[ g(xm,y1) · · · g(xm,yn) ] = σ(Wgm [x1 · · · xm] + bgm)
Then we compute the final outputs yi: yi = gyi ŷi + g(x1,yi) x1 + · · ·+ g(xm,yi) xm. σ is the sigmoid function σ(x) = 1/(1 + e−x) and is the elementwise product. We initialized bgi = 1 and bgy = −1 so that gyi would start out as a small value and g(xi,yj) would be large, encouraging copying of the inputs xi to the outputs yi.
Layer normalization. We found that using layer normalization (Ba et al., 2016) also helps stabilize the learning process. For our model, it is difficult to use batch normalization because the connections of each layer (the functions MERGE, SPLIT, WHICH) occur at variable points according to the particular tree we are considering.
Following the procedure in the appendix of Ba et al. (2016), we replace each instance of f(W [x1 · · · xm] + b) with f(LN(W1x1;α1) + · · · + LN(Wmxm;αm) + b) where Wi ∈ Rnk×k are horizontal slices of W and αi ∈ R are learned multiplicative constants. We use LN(z;α) = α · (z − µ)/σ where µ ∈ R is the mean of z ∈ Rk and σ ∈ R is the standard deviation of z.
3.5 WEIGHT SHARING
In the above model, each function with a different name has different weights. For example, if we have two types PLUS and MINUS each with two required children, then SPLITPLUS and SPLITMINUS will have different weights even though they both have the same signature Rk → Rk × Rk. However, this may be troublesome when we have a very large number of types, because in this scheme the amount of weights increases linearly with the number of types. For such cases, we can apply some of the following modifications:
• Replace all instances of SPLITT : Rk → Rk × · · · × Rk and SPLITREPEATED with a single SPLITREC : Rk → Rk×Rk. We can apply SPLITREC recurrently to get the desired number of child embeddings.
• Similarly, replace instances of MERGE with MERGEREC. • Share weights across the WHICH functions: a WHICH function which produces a distribu-
tion over T1, . . . , Tn contains weights and a bias for each Ti. We can share these weights and biases across all WHICH functions where Ti appears.
3.6 VARIABLE-SIZED LATENT STATE
In order to achieve low reconstruction error Eqφ(z|x)[log pθ(x | z)], the encoder and decoder networks must learn how to encode all information about a tree in z and then be able to reproduce the tree from this representation. If the tree is large, it becomes a difficult optimization problem to learn how to do this effectively, and may require higher-capacity networks in order to succeed at all which would require more time to train.
Instead, we can encode the tree with a variable number of latent state vectors. For each node ni in the tree, we specify q(zni | x) = N (µni , σni) where[
µni σni
] = ( id
softplus )[ Wµ Wσ ] ENCODE(ni)
Then when computing GENERATENODE(T,h), we first sample zni ∼ q(zni | x) at training time or zni ∼ p(z) at generation time, and then use ĥ = MERGELATENT(h, zni) in lieu of h. We fixed the prior of each latent vector zi to be the standard multivariate normal distribution with diagonal unit covariance, and did not investigate computing the prior as a function of other samples of zi as in Chung et al. (2015) or Fraccaro et al. (2016) which also used a variable number of latent state vectors.
4 EXPERIMENTS
4.1 TYPE-AWARE SEQUENTIAL MODEL
For purposes of comparison, we implemented a standard LSTM model for generating each node of the tree sequentially with a depth-first traversal, similar to Vinyals et al. (2015). The model receives each non-terminal type, and terminal value, as a separate token. We begin the sequence with a special 〈BOS〉 token. Whenever an optional child is not present, or at the end of a sequence of repeated children, we insert 〈END〉. This allows us to unambiguously specify a tree following a given grammar as a sequence of tokens.
At generation time, we keep track of the partially-generated tree in order to only consider those tokens which would be syntactically allowed to appear at that point. We also tried using this constraint at training time: when computing the output probability distribution, only consider the syntactically allowed tokens and leave the unnormalized log probabilities of the others unconstrained. However, we found that for our datasets, this did not help much with performance and led to overfitting.
4.2 SYNTHETIC ARITHMETIC DATA
To evaluate the performance of our models in a controlled way, we created a synthetic dataset consisting of arithmetic expressions of a given depth which evaluate to a particular value.
Grammar. We have two non-terminal types, PLUS and MINUS, and one terminal type NUMBER. PLUS and MINUS have two required children, left and right, each of which can be any of PLUS, MINUS, or NUMBER. For NUMBER, we allowed terminal values 0 to 9.
Generating the data. We generate trees with a particular depth, defined as the maximal distance from the root to any terminal node. As such, we consider 1 + (2 + 3) and (1 + 2) − (3 + 4) to both have depth 3. To get trees of depth d which evaluate to v, we first sampled 1,000,000 trees
uniformly at random from all binary tree structures up to depth d− 1, and randomly assigning each non-terminal node to PLUS or MINUS and setting each terminal node to a random integer between 0 and 9. Then we randomly pick two such trees, which when combined with PLUS or MINUS, evaluate to v to build a tree of depth d.
As training data, we generated 100,000 trees of depth 5, 7, 9, and 11. Within each set of trees, each quarter evaluates to −10, −5, 5, and 10 respectively. We use a test set of 1,024 trees, which we generated by first sampling a new set of 1,000,000 subtrees independently.
Results. Table 1 shows statistics on the datasets and the experimental results we obtained from training various models. The tree variational autoencoder model achieves better performance on deeper trees. In particular, the sequential model fails to learn well on depth 11 trees. However, it appears that a tree-structured model but with a fixed z performs similarly, although consistently worse than with the VAE.
4.3 FIRST-ORDER LOGIC PROOF CLAUSES
We next consider a dataset derived from Alemi et al. (2016): fragments of automatically-generated proofs for mathematical theorems stated in first-order logic. An automated theorem prover tries to prove a hypothesis given some premises, producing a series of steps until we conclude the hypothesis follows from the premises. Many theorem provers work by resolution; it negates the hypothesis and shows that a contradiction follows. However, figuring out the intermediate steps in the proof is a highly nontrivial search procedure. If we can use machine learning to generate clauses which are likely to appear as a proof step, we may be able to speed up automated theorem proving significantly.
Grammar. We generate first-order logic statements which are clauses, or a disjunction of literals. Each literal either contains one predicate invocation or asserts that two expressions are equal. Each predicate invocation contains a name, which also determines a fixed number of arguments; each argument is an expression. An expression is either a function invocation, a number, or a variable. A function invocation is structurally identical to a predicate invocation.
We consider the set of functions and predicates to be closed. Furthermore, given that each function and predicate has a fixed number of arguments, we made each of these its own type in the grammar. To avoid having a very large number of weights as a consequence, we applied the modifications described in Section 3.5.
Results. Table 2 describes our results on this dataset. We trained on 955,529 trees and again tested on 1,024 trees. The sequential model demonstrates slightly better log likelihood compared to the tree variational autoencoder model. However, on this dataset we observe a significant improvement
in log likelihood by adding the variational autoencoder to the tree model, unlike on the arithmetic datasets.
5 DISCUSSION AND FUTURE WORK
Conditioning on an outside context. In many applications for modeling tree-structured data, we have an outside context that informs which trees are more likely than others. For example, when generating clauses which may appear in a proof, the hypothesis in question greatly influences the content of the clauses. We leave this question to future work.
Scaling to larger trees. Currently, training the model requires processing an entire tree at once, first to encode it into z and then to decode z to reproduce the original tree. This can blow up the memory requirements, requiring an undesirably small batch size. For autoregressive sequence models, truncated backpropagation through time provides a workaround as a partial form of the objective function can be computed on arbitrary subsequences. In our case, adapting methods from Gruslys et al. (2016) and others may prove necessary.
Improving log likelihood. In terms of log likelihood, our model performed significantly better than an autoregressive sequential model only on one of the datasets we tested, and about the same or slightly worse on the others. Adding depth to the tree structure (Irsoy & Cardie, 2014), and a more sophisticated posterior (Rezende & Mohamed, 2015; Kingma et al., 2016; Sønderby et al., 2016), are some modifications which might help with learning a more powerful model. Introducing more dependencies between the dimensions of x during generation is another possibility but one which may reduce the usefulness of of the latent representation (Bowman et al., 2015).
A MODEL HYPERPARAMETERS
We used the Adam optimizer with a learning rate of 0.01, multiplied by 0.98 every 10000 steps. We clipped gradients to have L2 norm 3. For the synthetic arithmetic data, we used a batch size of 64; for the first-order logic proof clauses, we used a batch size of 256. For the tree-structured variational autoencoder, we set k = 256 and used the ELU nonlinearity wherever another one was not explicitly specified. For the sequential models, we used two stacked LSTMs each with hidden state size 256, no dropout. We always unrolled the network to the full length of the sequence during training, and did not perform any bucketing of sequences by length.
B KL DIVERGENCE DYNAMICS DURING TRAINING
Optimizing the variational autoencoder objective turned out to be a significant optimization challenge, as pointed out by prior work (Bowman et al., 2015; Sønderby et al., 2016; Kingma et al., 2016). Specifically, it is easy for the KL divergence term DKL(qφ(z | x) ‖ p(z)) to collapse to zero, which means that qφ(z | x) is equal to the prior and does not convey any information about x. This leads to uninteresting latent representations and reduces the generative model to one that does not use a latent representation at all.
As explained by Kingma et al. (2016), this phenomenon occurs as at the beginning of training it is much easier for the optimization process to move qφ(z | x) closer to the prior p(z) than to improve p(x | z), especially when qφ(z | x) has not yet learned how to convey any useful information. To combat this, we use a combination of two techniques described in the previous work:
• Anneal the weight on the KL cost term slowly from 0 to 1. Similar to Bowman et al. (2015), our schedule was a shifted and horizontally-scaled sigmoid function.1
• Set a floor on the KL cost, i.e. use −max(DKL(qφ(z | x) ‖ p(z)), λ) instead of DKL(qφ(z | x) ‖ p(z)) in the objective (Kingma et al., 2016). This change means that the model receives no penalty for producing a KL divergence below λ, and as the other part of the objective (the reconstruction term) benefits from a higher KL divergence, it naturally learns a more informative qφ(z | x) at least λ in KL divergence.
We found that at least one of these techniques were required to avoid collapse of the KL divergence to 0. However, as shown in Figure 2, we found that different combinations of these techniques could led to different overall results, suggesting that finding the desired equilibrium necessitates a hyperparameter search.
1To anneal from a to b, we used σ ( step − a+b
2
) /10 to weight the KL cost as a function of the number of
optimization steps taken.

This paper introduces a novel extension of the variational autoencoder to arbitrary tree-structured outputs. Experiments are conducted on a synthetic arithmetic expression dataset and a first-order logic proof clause dataset in order to evaluate its density modeling performance.
Pros:
+ The paper is clear and well-written.
+ The tree-structure definition is sufficiently complete to capture a wide variety of tree types found in real-world situations.
+ The tree generation and encoding procedure is elegant and well-articulated.
+ The experiments, though limited in scope, are relatively thorough. The use of IWAE to obtain a better estimate of log likelihoods is a particularly nice touch.
Cons:
- The performance gain over a baseline sequential model is marginal.
- The experiments are limited in scope, both in the datasets considered and in the evaluation metrics used to compare the model with other approaches. Specifically: (a) there is only one set of results on a real-world dataset and in that case the proposed model performs worse than the baseline, and (b) there is no evaluation of the learned latent representation with respect to other tasks such as classification.
- The ability of the model to generate trees in time proportional to the depth of the tree is proposed as a benefit of the approach, though this is not empirically validated in the experiments.
The procedures to generate and encode trees are clever in their repeated use of common operations. The weight sharing and gating operations seem important for this model to perform well but it is difficult to assess their utility without an ablation (in Table 1 and 2 these modifications are not evaluated side-by-side). Experiments in another domain (such as modeling source code, or parse trees conditioned on a sentence) would help in demonstrating the utility of this model. Overall the model seems promising and applicable to a variety of data but the lack of breadth in the experiments is a concern.
* Section 3.1: "We distinguish three types" => two
* Section 3.6: The exposition of the variable-sized latent state is slightly confusing because the issue of how many z's to generate is not discussed.
* Section 4.2-4.3: When generating the datasets, did you verify that the test set is disjoint from the training set?
* Table 1: Is there a particular reason why the variable latent results are missing for the depth 11 trees?