MEMORY-AUGMENTED ATTENTION MODELLING FOR VIDEOS
Recent works on neural architectures have demonstrated the utility of attention mechanisms for a wide variety of tasks. Attention models used for problems such as image captioning typically depend on the image under consideration, as well as the previous sequence of words that come before the word currently being generated. While these types of models have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex. Motivated by these observations, we propose a novel memory-based attention model for video description. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step, similar to the central executive system proposed in human cognition (Baddeley & Hitch, 1974). This allows the model to not only reason about local attention more effectively, it allows it to consider the entire sequence of video frames while generating each word. Evaluation on the challenging and popular MSVD and Charades datasets show that the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results in the video description.
1 INTRODUCTION
Deep neural architectures have led to remarkable progress in computer vision and natural language understanding problems. Image captioning is one such application that has seen the combination of convolutional structures (Krizhevsky et al., 2012; LeCun et al., 1998), which have been shown to be very effective for problems like object detection, with sequential recurrent structures, which have been shown to be very effective for problems like machine translation (Sutskever et al., 2014). One of the key modelling paradigms shared by most models for image captioning is the notion of an attention mechanism that guide the model to attend to certain parts of the image while generating.
The attention models used for problems such as image captioning typically depend on the single image under consideration and the partial output generated so far, jointly capturing one region of an image and the words being generated. However, such models cannot capture the temporal reasoning necessary to effectively produce words that refer to actions and events taking place over multiple frames in a video. For example, in a video depicting “someone waving a hand”, the “waving” action can start from any frame and can continue on for a variable number of following frames. More importantly, it is likely in a given video quite few frames do not contain any useful information or motion in regard to a given task. Given this, it is not surprising that even with recent advancements in image captioning Xu et al. (2015a); Johnson et al. (2016); Vinyals et al. (2015), video captioning has remained challenging.
Motivated by these observations, we introduce a memory-based attention mechanism for video captioning and description. Our model utilizes memories of past attention in the video when reasoning about where to attend to in a current time step. This allows the model to not only effectively leverage local attention, but also to consider the entire video as it generates each word. This mechanism is similar to the proposed central executive system in human cognition, which is thought to permit human performance on two simultaneous tasks (e.g., seeing and saying) using two separate percep-
tual domains (e.g., visual and linguistic) by binding information from both sources into coherent structure that enables coordination, selective attention, and inhibition.
Our work shares the same goals as recent work on attention mechanisms for sequence-to-sequence architectures, such as Rocktäschel et al. (2016) and Yang et al. (2016). However, there are major differences between this work and our current work. Rocktäschel et al. (2016) considers the domain of entailment relations, where the goal is to determine entailment given two input sentences. They propose a soft attention model that is not only focused on the current state, but the previous as well. In our model, we explicitly store all previous attention into memory. In addition, our memory memorizes the encoded version of the input videos conditioned on previously seen words. Even though Yang et al. (2016) and our work both try to solve the problem of locality of attention, our work is different from them in how the memory architecture is modelled. More specifically, they incorporate discriminative supervision into their ”reviewer” mechanism, which is not the case in our model. Further, their model is applied to image caption generation, which is to some extent simpler than video caption generation because there is no temporal structure to model.
We apply our model on the video captioning problem and evaluate it on the MSVD (Chen & Dolan, 2011) and the Charades (Sigurdsson et al., 2016) datasets. Experimental results show that our proposed architecture outperforms all previous methods and leads to new state of the art results. While we have chosen the video captioning problem for our experiment, the model is general enough that it can be applied to other problems where attention models are used.
2 RELATED WORK
One of the primary challenges in learning a mapping from a visual space (i.e., video or image) to a language space is learning a representation that not only effectively represents each of these modalities, but is also able to translate a representation from one space to the other. Rohrbach et al. (2013) developed a model that generates a semantic representation of visual content that can be used as the source language for the language generation module. Venugopalan et al. (2015b) proposed a deep method to translate a video into a sentence where an entire video is represented with a single vector based on the mean pool of frame features. However, representing a video by an average of its frames misses the temporal structure of the video. To address this problem, recent work (Yao et al., 2015; Pan et al., 2016a; Venugopalan et al., 2015a; Andrew Shin, 2016; Pan et al., 2016b; Xu et al., 2015b; Ballas et al., 2016; Yu et al., 2016) proposed methods to model temporal structure of video as well as language.
The majority of these methods are inspired by sequence to sequence (Sutskever et al., 2014) and attention (Bahdanau et al., 2015) models. Sequence learning (Sutskever et al., 2014) was originally proposed to map the input sequence of a source language to a target language. Even though applying this method with a combination of attention to the problem of translating a video to a description shows promising results, there are some shortcomings. First of all, modelling the video content with a fixed-length vector in order to map it to a language space is a much harder problem than mapping from a language to a language given the complexity of visual content. Since not all frames in a video are equally salient for a short description, and an event can happen in multiple frames, it is important for a model to identify which frames are most salient. Further, the model should be able to focus on points of interest within these frames to select what to talk about. Even using a variable-length vector to represent a video using attention (Yao et al., 2015) can have some problems. More specifically, current attention methods are local Yang et al. (2016), since the attention mechanism works in a sequential structure, and lacks the ability to capture global structure. Moreover, combining a video and a language description as a sequence to sequence is usually done by some variant of a recurrent neural network (RNN) (Hochreiter & Schmidhuber, 1997). Given the limited capacity of a recurrent network to model very long sequences, memory networks (Weston et al., 2014; Sukhbaatar et al., 2015) have been introduced to help the RNN memorize sequences. However, one problem these memory networks suffer from is the difficulty in training the model. The model in Weston et al. (2014) requires supervision at each layer which makes training with backpropagation a challenging task. Even though Sukhbaatar et al. (2015) proposed a memory network that can be trained endto-end, working with memory is still a challenging problem in deep learning especially with write operation (Graves et al., 2014).
To address these problems, we propose a memory-based attention sequence-to-sequence model that not only can learn hierarchical attention relationships, but provides a simple and effective memory structure. In the next section, we explain our model in more detail.
3 LEARNING TO ATTEND AND MEMORIZE
Our goal is to design an architecture that learns where to look and what to look for in a video, in order to talk about it in the description. To achieve this goal, we formulate the problem as sequence learning to maximize the probability of generating a correct description given a video:
Θ∗ = arg max Θ ∑ (S,f1,f2,....,fN ) log p(S|f1, f2, ..., fN ; Θ) (1)
where S is the description, f1, f2, ..., fN are the input video frames, and Θ is the model parameter vector. The main modelling challenges in video description are to develop a system that can model the temporal structure of the video, learn to attend to the important parts of a video, how to memorize the video that was described given all the generated words so far, then generate a new word by looking at the entire video. To address these issues, we propose an end-to-end network that has three components (Figure 1): Temporal Model (TEM), Hierarchical Attention/Memory (HAM), and the Decoder. The goal in TEM is to capture the temporal structure and track motion in a video. The HAM component acts as a hierarchical attention or memory between an input video and the description. More specifically, the HAM learns a hierarchical attention structure that learns where to attend in video given all previously generated words and previous states. The HAM can be interpreted as a memory structure as well, where it learns to memorize an encoded version of a video with language. HAM provides the decoder with the ability to look at an entire video plus all the previously generated words before generating any new words. This is important because a single action normally exhibits multiple frames in the input video. By employing the HAM, the model can effectively model the action over these frames. One of the main contributions of this work is to use a global state to generate any new word. This global state aggregates information from previously generated words and all input frames. We will first describe each component of our model, then explain details of training and inference.
3.1 TEMPORAL MODELER (TEM)
One important question is how to encode the temporal structure of the input video for caption generation. Recently, it has been shown that Recurrent Neural Networks (RNN) has the ability to model the temporal structure in sequential data such as video (Ballas et al., 2016; Sharma et al., 2015; Venugopalan et al., 2015a) and speech (Graves & Jaitly, 2014). Moreover, since frame-to-frame temporal variation tend to be local (Brox & Malik, 2011) and critical in the motion modeling (Ballas et al., 2016), it is important to consider a frame representation that can preserve frame-to-frame temporal variation. Even though using features extracted from the fully connected layers of Convolutional Neural Networks (CNNs) have shown state of the art results in image classification and
recognition (Simonyan & Zisserman, 2014; He et al., 2016), these features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016).
To address the temporal modeling and video representation problems, we use an RNN to model the temporal structure of the video where at each time step, a frame encoding with size of RD is used as an input to the RNN. Instead of extracting features from a top layer of the pretrained CNN, intermediate convolutional maps have been extracted for the video frames. Specifically, for a given video, X , with N frames X = [X1, X2, · · · , XN ], N convolutional maps of size RL×D are extracted where D is dimension of a feature corresponding to L locations in the input frame.
In order to let the network selectively focus on these L locations of each frame given the hidden state of RNN, we apply a soft attention model Bahdanau et al. (2015); Xu et al. (2015a); Sharma et al. (2015), called “Location Attention (Latt)”. More specifically, by using a softmax, each hidden state produces L probabilities to specify which part of the input is more important, and then creates an input map for the RNN using these probabilities. The fLatt is defined as follow:
ρtj = exp((ht−1v ) T Wjp)∑L
k=1 exp((h t−1 v ) T Wkp)
(2)
Ft = L∑ j=1 ρtjX t j (3)
where ht−1v ∈ RK is hidden state of RNN at t − 1, Wp ∈ RK×L, and F t ∈ RD. At each time step, TEM learns a vector representation for each frame, looking at the frame convolution map, and applying the location attention on this map conditioned on all previously seen frames.
Ft = fLatt(Xt,h t−1 v ; Wp) (4) htv = fv(F t, ht−1v ; Θv) (5)
where fv can be a vanilla RNN, LSTM, or GRU and Θv is the parameters of the fv. Due to the fact that vanilla RNNs have gradient vanishing and exploding problems (Pascanu et al., 2013), we use gradient clipping to deal with gradient exploding, and an LSTM with the following flow to deal with the gradient vanishing problem:
it = σ(FtWxi + h t−1 v Whi) f t = σ(FtWxf + h t−1 v Whf ) ot = σ(FtWxo + h t−1 v Who) gt = tanh(FtWxg + h t−1 v Whg) ctv = f t ct−1v + it gt
htv = ot tanh(ct)
where Wh∗ ∈ RK×K , Wx∗ ∈ RD×K , and we define Θv = {Wh∗,Wx∗}.
3.2 HIERARCHICAL ATTENTION/MEMORY (HAM)
One problem with using sequence-to-sequence style architectures (Sutskever et al., 2014), to model a task such as video language description, is how to find a mapping from a video space to a language space that can capture the relationship between a word and video, or more specifically, the connection between an entire video and an entire sentence where there might not be a clear alignment between the two sequences, as opposed to machine translation and speech recognition. Furthermore, the model should be able to identify which part of the video is more relevant to the description because captions normally focus on a tiny fraction of the facts present in the video. More importantly, once the model starts generating the description, it should still be reminded with the video frames to generate meaningful descriptions. In order to address these problems, we propose a memory-based attention that encodes a video into memory, built as a function of the state of the language generation network (a.k.a. Decoder) and the state of the TEM network. More specifically, our Hierarchical Attention/Memory can be formulated as two following steps:
• Attention update [ F̂(Θa)]:
QA = tanh(HvWv + H t′−1 g Wg + H t′−1 m Wm) (6)
αt′ = softmax(U TQA) (7)
F̂ = αTt′Hv (8)
• Memory update:
ht ′ m = fm(h t′−1 m , F̂; Θm) (9)
where Hv ∈ RN×K , Wv and Wg are ∈ RK×K , U ∈ RK , Wm ∈ RM×K , and Θa = {Wv,Wg,Wm, U}. N is number of frames in a given video, Hv = [h1v, ..., hNv ], Ht ′−1 g = [ht ′−1 g , ..., h t′−1 g ], and H t′−1 m = [h t′−1 m , ..., h t′−1 m ]. αt′ is the set of probabilities in a given time step that specifies the attention over an input video state (Hv), memory state (Hm), and decoder state (Hg). In order to let the network remember what has been attended before and the temporal structure of a video, we propose fm to memorize the previous attention and encoded version of an input video with language model. Using fm not only enables the network to memorize previous attention and frames, but also to learn multi-layer attention over an input video and corresponding language. The output of the memory-attention is then used as input to the Decoder.
3.3 DECODER
In order to generate a new word conditioned on all previous words and HAM states, a recurrent structure is modelled as follows:
ht ′ g = fg( s t′ , ht ′ m, h t′−1 g ; Θg) (10) ŝt′ = softmax(Weh t′ g ) (11)
where st ′ is a word vector at t′, We ∈ RK×C and C is the vocabulary size. In addition, ŝt′ assigns a probability to each word in the language. We use LSTMs for both fm and fg.
3.4 TRAINING AND OPTIMIZATION
The goal in our network is to predict the next word given all previously seen words and an input video. In order to optimize our network parameters, Θ = {Wp,Θv,Θa,Θm,We}, we minimize a negative log likelihood loss function, formulated as follow:
L(S,X; Θ) = − T∑ j |V |∑ i sj,i log(̂sj,i) + λ ‖ Θ ‖22 (12)
where |V | is the dictionary size. We fully train our network in an end-to-end fashion using first-order stochastic gradient-based optimization method with an adaptive learning rate. More specifically, in order to optimize our network parameters, we use Adam Kingma & Ba (2015) with learning rate 2× 10−5 and set β1, β2 to 0.8 and 0.999, respectively. At the training, we use a batch size of 16.
4 EXPERIMENTS
DATASET We evaluate our proposed model on the Charades (Sigurdsson et al., 2016) dataset and the Microsoft Video Description Corpus (MSVD) (Chen & Dolan, 2011). Charades contains 9,848 videos (in total) and provides 27, 8471 video descriptions, with 7569 training, 1, 863 test, 400 for the validation. We follow the same split (i.e. training and test splits) as Sigurdsson et al. (2016). It is worth noting that one major difference between this dataset and others is that they use a “Hollywood in Homes” approach to collecting the data (Sigurdsson et al., 2016), where “actors” are crowdsourced, yielding a diverse scene and actor videos. One reason that we report results on this
1Only 16087 out of 27, 847 are used as captions for our evaluation since the 27, 847 refers to script of the video as well as captions.
dataset is because each video has a specific action in it and would be a suitable testbed to evaluate our model.
MSVD is a set of Youtube videos that are annotated by a Mechanical Turker,2 who was asked to pick a clip from a video that represents an activity. In this dataset, each clip is annotated by multiple workers with a single sentence. This dataset contains 1, 970 videos and about 80, 000 descriptions, where 1, 200 of the videos are training data, 670 are test data, and the rest (i.e., 100 videos) are assigned for validation. In order to make the results comparable with other papers, we follow the exact training/validation/test split provided by Venugopalan et al. (2015b).
EVALUATION METRICS Below, we report results on the video caption generation task. In order to evaluate captions generated by our model, we use model-free automatic evaluation metrics. We adopt METEOR, BLEU@N, and CIDEr metrics available from the Microsoft COCO Caption Evaluation code3 to score the system.
VIDEO AND CAPTION PREPROCESSING We preprocess the captions for both datasets using the Natural Language Toolkit (NLTK)4. Beyond this, no other type of preprocessing is used.
We extract sample frames for each video and pass each frame through VGGnet (Simonyan & Zisserman, 2014) without any fine-tuning. For the experiments in this paper, we use the feature maps from conv5 3 layer after applying ReLU. The feature map in this layer is 14× 14× 512. Our TEM component operates on the flattened 196 × 512 of this feature cubes. For the ablation studies, features from fully connected layer are used as well where the features in this layer have 4096 dimension.
HYPER-PARAMETER OPTIMIZATION We use random search (Bergstra & Bengio, 2012) on validation set to select hyper-parameters on both datasets. The word-embedding size, hidden layer size (for both TEM and Decoder), and memory size of the best model on Charades are: 237, 1316, and 437, respectively. These values are 402, 1479, and 797 for the model on MSVD dataset. A stack of two LSTMs are used in the Decoder and TEM.
4.1 VIDEO CAPTION GENERATION
We first present an ablation analysis to elucidate the contribution of the different components of our proposed model. Next, we compare the overall performance of our model on video caption generation task to other models.
ABLATION ANALYSIS
We first perform a series of ablation studies in order to show the contributions of the different components of our model. Specifically, we show that the importance of each components in our model in caption generation task on MSVD dataset. One ablation (denoted as Att + No TEM) corresponds to a simpler version of our model in which we remove the TEM component and instead we pass each frame of a video through a CNN and extract features from the last fully-connected hidden layer (e.g., fc7). In addition, we replace our HAM component with a simpler version where the model only memorizes the current step instead of all previous steps. In another ablation (denoted as No HAM + TEM), we remove the HAM component from our model and keep the rest of our model as it is. In the next variation (denoted as HAM + No TEM), we remove the TEM component and calculate features for each frame, similar to Att + No TEM. Finally, the last row in the table is our proposed model (denoted HAM + TEM) with all its components.
Table 1 reports the result of this study. In this experiment, we sample 40 frames per video and use them as the inputs to a network. As the results show, HAM plays a critical role in our proposed model, and removing it causes a drop in performance. On the other hand, removing TEM by itself does not drop performance as much as dropping the HAM. When we put the two together, they complement one another, resulting in better performance.
2https://www.mturk.com/mturk/welcome 3https://github.com/tylin/coco-caption 4http://www.nltk.org/
PERFORMANCE COMPARISON
Next, to extensively evaluate our model, we compare our model with state-of-the-art models and baselines for the video caption generation task on the MSVD dataset. In this experiment, we use 8 frames per video as the inputs to the TEM module. Table 25 shows the results for this experiment. As the results show, our model gets state-of-the-art scores either in BLEU-4 or METEOR, compared to other methods. This is particularly noteworthy because we do not use external features for the video, such as Optical Flow (Brox et al., 2004) (denoted as Flow in table), 3-Dimensional Convolutional Network features (Tran et al., 2015) (denoted as C3D), or fine-tuned CNN features (denoted as FT) on the action recognition task with dataset such as UCF-101. The only exception happens when we compare our model with (Yu et al., 2016), who uses C3D features. In this method, adding C3D features leads to a huge improvement in their results (compare row 4 with 11 in Table 2). On the other hand, our method without using any external features can achieve better results in comparison with all other methods. This is important because our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.
In addition, we report results on the Charades dataset for video caption generation. This dataset is challenging because only a few captions per video (about 2 per video) are available. In this experiment, we use 16 frames per video as the inputs to the TEM module. Table 3 shows the performance of our method on this dataset. Our method can achieve 10% improvement over Venugopalan et al. (2015a) in the caption generation task. It is worth noting that a human can only achieve a score of 24 in METEOR for this dataset, which illustrated the level of difficulty in this dataset.
QUALITATIVE RESULTS
We show some captions generated by our model in 2. The model mostly generates correct captions for cases where content and ground truth captions are consistent. There are some cases in which our model makes some mistakes. For example, in a ‘a dog is on a trampoline’ video, our model generated “a man is washing a bath” as a caption. This is interesting because the ’man’ object only appears in a few frames (1 or 2), but our model can still recognize the man object in the video.
5− in the Table 2 means, that score was not reported by the corresponding paper.
Table 3: Video captioning evaluation on the test set of 1863 videos in Charades.
Method METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr
Human(Sigurdsson et al., 2016) 24 62 43 29 20 53
Sigurdsson et al. (2016) 16 49 30 18 11 14 Our Model 17.6 50 31.1 18.8 11.5 16.7
Our Captions Ground Truth
A group of people are dancing
A group of young children performing together
A person is cutting the vegetable
A woman is cutting garlic
A man is playing a guitar
A man is playing the guitar
A young girl is playing the flute A little girl is talking on a cordless telephone
A woman is pouring eggs into a bowl
A woman is pouring ingredients into a bowl
A man is playing a flute A man is playing a large flute
A woman is applying makeup
A woman is putting on makeup
A man is cutting a gun
A guy is shooting a gun
A man is washing a bath A dog is on a trampoline
A cat is eating Hamsters are eating
Figure 2: Example captions generated by our model on the test video for MSVD. Incorrect caption cases are shown in red.
5 CONCLUSION
We introduce an end-to-end memory-based attention model to describe an input video using natural language description, similar to the central executive system proposed in human cognition. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step. This allows the model to not only reason about local attention more effectively, but also allows it to consider the entire sequence of video frames while generating each word. Our experiments have confirmed that the memory components in our architecture play a significant role in improving the performance of the entire network. It is worth noting that in this paper, we consider the problem of video caption generation, but our architecture can be applied to any sequence learning problem, which we hope to explore in the future.

The paper proposes an attention-based approach for video description. The approach uses three LSTMs and two attention mechanisms to sequentially predict words from a sequence of frames.
In the LSTM-encoder of the frames (TEM), the first attention approach predicts a spatial attention per frame, and computes the weighted average. The second LSTM (HAM) predicts an attention over the hidden states of the encoder LSTM.
The third LSTM which run temporally in parallel to the second LSTM generates the sentence, one word at a time.
Strength:
===============
-	The paper works on a relevant and interesting problem.
-	Using 2 layers of attention in the proposed way have to my knowledge not been used before for video description. The exact architecture is thus novel (but the work claims much more without sufficient attribution, see blow)
-	The experiments are evaluated on two datasets, MSVD and Charades, showing performance on the level of related work for MSVD and improvements for Charades.
Weaknesses:
===============
1.	Claims about the contribution/novelty of the model seem not to hold:
1.1.	One of the main contributions is the Hierarchical Attention/Memory (HAM):
1.1.1.	It is not clear to me how the presented model (Eq 6-8), are significantly different from the presented model in Xu et al / Yao et al. While Xu et al. attends over spatial image locations and Yao et al. attend over frames, this model attends over encoded video representations h_v^i. A slight difference might be that Xu et al. use the same LSTM to generate, while this model uses an additional LSTM for the decoding.
1.1.2.	The paper states in section 3.2 “we propose f_m to memorize the previous attention”, however H_m^{t’-1} only consist of the last hidden state. Furthermore, the model f_m does not have access to the “attention” \alpha. This was also discussed in comments by others, but remains unclear.
1.1.3.	In the discussion of comments the authors claim that “attention not only is a function a current time step but also a function of all previous attentions and network states.”: While it is true that there is a dependency but that is true also for any LSTM, however the model does not have access to the previous network states as H_g^{t’-1} only consist of the last hidden state, as well as H_m^{t’-1} [at least that is what the formulas say and what Figure 1 suggests].
1.1.4.	The authors claim to have multi-layer attention in HAM, however it remains unclear where the multi-layer comes from.
1.2.	The paper states that in section 3.1. “[CNN] features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016).” This suggests that the approach which follows attacks this problem. However, it cannot model motion as attention \rho between frames is not available when predicting the next frame. Also, it is not clear how the model can capture anything “low level” as it operates on rather high level VGG conv 5 features.
2.	Related work: The difference of HAM to Yao et al. and Xu et al. should be made more clear / or these papers should be cited in the HAM section.
3.	Conceptual Limitation of the model: The model has two independent attention mechanisms, a spatial one, and a temporal one. The spatial (within a frame) is independent of the sentence generation. It thus cannot attend to different aspects of the frames for different words which would make sense. E.g. if the sentence is “the dog jumps on the trampoline”, the model should focus on the dog when saying “dog” and on the trampoline when saying “trampoline”, however, as the spatial attention is fixed this is difficult. Also, the encoder model does not have an explicitly way to look at different aspects in the frame during the encoding so might likely get stuck and always predict the same spatial attention for all frames (or it might e.g. always attend to the dog which moves around, but never on the scene).
4.	Eq 11 contradicts Fig 1: How is the model exactly receiving the previous word as input. Eq. 11 suggests it is the softmax. If this is the case, the authors should emphasize this in the text as this is unusual. More common would be to use the ground truth previous word during training (which Fig 11 suggests) and the “hardmax”, i.e. the highest predicted previous word encoded as one-hot vector at test time.
5.	Clarity:
5.1.	It would be helpful if the same notation would be used in Eq 2-5 and 6-9. Why is a different notation required?
5.2.	It would be helpful if Fig 1 could contain more details or additional figures for the corresponding parts would be added. If space is a problem, e.g. the well-known equations for LSTM, softmax (Eq 2), and log likelihood loss (Eq 12) could be omitted or inlined.
6.	Evaluation:
6.1.	The paper claims that the “the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results”.
6.1.1.	For the MSVD dataset this clearly is wrong, even given the same feature representation. Pan et al. (2016 a) in Table 2 achieve higher METEOR (33.10).
6.1.2.	For this strong claim, I would also expect that it outperforms all previous results independent of the features used, which is also wrong again, Yu et al achieve higher performance in all compared metrics.
6.1.3.	For Charades dataset, this claim is also too bold as hardly any methods have been evaluated on this dataset, so at least all the ablations reported in Table 1 should also be reported for the Charades dataset, to make for this dataset any stronger claims.
6.2.	Missing qualitative results of attention: The authors should show qualitative results of the attention, for both attention mechanisms to understand if anything sensible is happening there. How diverse is the spatial and the temporal attention? Is it peaky or rather uniform?
6.3.	Performance improvement is not significant over model ablations: The improvements over Att+No TEM is only 0.5 Meteor, 0.7 Blue@4 and the performance drops for CIDEr by 1.7.
6.4.	Missing human evaluation: I disagree with the authors that a human evaluation is not feasible. 1. An evaluation on a subset of the test data is not so difficult. 2. Even if other authors do not provide their code/model [and some do], they are typically happy to share the predicted sentences which is sufficient and even better for human evaluation [if not I would explicitly mention that some authors did not share sentences, as this seems clearly wrong]. 3. For model ablations the sentences are available to the authors.
7.	Several of the comments raised by reviewers/others have not yet been incorporated in a revised version of the paper and/or are still not clear from the explanations given. E.g. including SPICE evaluation and making fixes seems trivial.
8.	Hyperparameters are inconsistent: Why are the hyperparemters inconsistent between the ablation analysis (40 frames are sampled) and the performance comparison (8 frames)? Should this not be selected on the validation set? What is the performance of all the ablations with 8 frames?
Other (minor/discussion points)
-	Equation 10: what happens with h_m, and h_g, the LSTM formulas provided only handle two inputs. Are h_m and h_g concatenated.
-	There is a section 4.1 but no 4.2.
-	The paper states in section 4.1 “our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.” However, this seems to be true also for many/most other approaches, e.g. [Venugopalan et al. 2015 ICCV]
Summary:
===============
While the paper makes strong claims w.r.t. to the approach and results, the approach lacks novelty and the results are not convincing over related work and ablations. Furthermore, improved clarity and visualizations of the model and attention results would benefit the paper.


TEMPORAL ENSEMBLING FOR SEMI-SUPERVISED LEARNING
In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.
1 INTRODUCTION
It has long been known that an ensemble of multiple neural networks generally yields better predictions than a single network in the ensemble. This effect has also been indirectly exploited when training a single network through dropout (Srivastava et al., 2014), dropconnect (Wan et al., 2013), or stochastic depth (Huang et al., 2016) regularization methods, and in swapout networks (Singh et al., 2016), where training always focuses on a particular subset of the network, and thus the complete network can be seen as an implicit ensemble of such trained sub-networks. We extend this idea by forming ensemble predictions during training, using the outputs of a single network on different training epochs and under different regularization and input augmentation conditions. Our training still operates on a single network, but the predictions made on different epochs correspond to an ensemble prediction of a large number of individual sub-networks because of dropout regularization.
This ensemble prediction can be exploited for semi-supervised learning where only a small portion of training data is labeled. If we compare the ensemble prediction to the current output of the network being trained, the ensemble prediction is likely to be closer to the correct, unknown labels of the unlabeled inputs. Therefore the labels inferred this way can be used as training targets for the unlabeled inputs. Our method relies heavily on dropout regularization and versatile input augmentation. Indeed, without neither, there would be much less reason to place confidence in whatever labels are inferred for the unlabeled training data.
We describe two ways to implement self-ensembling, Π-model and temporal ensembling. Both approaches surpass prior state-of-the-art results in semi-supervised learning by a considerable margin. We furthermore observe that self-ensembling improves the classification accuracy in fully labeled cases as well, and provides tolerance against incorrect labels.
The recently introduced transform/stability loss of Sajjadi et al. (2016b) is based on the same principle as our work, and the Π-model can be seen as a special case of it. The Π-model can also be seen as a simplification of the Γ-model of the ladder network by Rasmus et al. (2015), a previously presented network architecture for semi-supervised learning. Our temporal ensembling method has connections to the bootstrapping method of Reed et al. (2014) targeted for training with noisy labels.
Algorithm 1 Π-model pseudocode. Require: xi = training stimuli Require: L = set of training input indices with known labels Require: yi = labels for labeled inputs i ∈ L Require: w(t) = unsupervised weight ramp-up function Require: fθ(x) = stochastic neural network with trainable parameters θ Require: g(x) = stochastic input augmentation function
for t in [1,num epochs] do for each minibatch B do zi∈B ← fθ(g(xi∈B)) . evaluate network outputs for augmented inputs z̃i∈B ← fθ(g(xi∈B)) . again, with different dropout and augmentation loss ←− 1|B| ∑ i∈(B∩L) log zi[yi] . supervised loss component
+ w(t) 1C|B| ∑ i∈B ||zi − z̃i||2 . unsupervised loss component
update θ using, e.g., ADAM . update network parameters end for
end for return θ
2 SELF-ENSEMBLING DURING TRAINING
We present two implementations of self-ensembling during training. The first one, Π-model, encourages consistent network output between two realizations of the same input stimulus, under two different dropout conditions. The second method, temporal ensembling, simplifies and extends this by taking into account the network predictions over multiple previous training epochs.
We shall describe our methods in the context of traditional image classification networks. Let the training data consist of total of N inputs, out of which M are labeled. The input stimuli, available for all training data, are denoted xi, where i ∈ {1 . . . N}. Let set L contain the indices of the labeled inputs, |L| = M . For every i ∈ L, we have a known correct label yi ∈ {1 . . . C}, where C is the number of different classes.
2.1 Π-MODEL
The structure of Π-model is shown in Figure 1 (top), and the pseudocode in Algorithm 1. During training, we evaluate the network for each training input xi twice, resulting in prediction vectors zi and z̃i. Our loss function consists of two components. The first component is the standard crossentropy loss, evaluated for labeled inputs only. The second component, evaluated for all inputs, penalizes different predictions for the same training input xi by taking the mean square difference
between the prediction vectors zi and z̃i.1 To combine the supervised and unsupervised loss terms, we scale the latter by time-dependent weighting function w(t). By comparing the entire output vectors zi and z̃i, we effectively ask the “dark knowledge” (Hinton et al., 2015) between the two evaluations to be close, which is a much stronger requirement compared to asking that only the final classification remains the same, which is what happens in traditional training.
It is important to notice that, because of dropout regularization, the network output during training is a stochastic variable. Thus two evaluations of the same input xi under same network weights θ yield different results. In addition, Gaussian noise and augmentations such as random translation are evaluated twice, resulting in additional variation. The combination of these effects explains the difference between the prediction vectors zi and z̃i. This difference can be seen as an error in classification, given that the original input xi was the same, and thus minimizing it is a reasonable goal.
In our implementation, the unsupervised loss weighting function w(t) ramps up, starting from zero, along a Gaussian curve during the first 80 training epochs. See Appendix A for further details about this and other training parameters. In the beginning the total loss and the learning gradients are thus dominated by the supervised loss component, i.e., the labeled data only. We have found it to be very important that the ramp-up of the unsupervised loss component is slow enough—otherwise, the network gets easily stuck in a degenerate solution where no meaningful classification of the data is obtained.
Our approach is somewhat similar to the Γ-model of the ladder network by Rasmus et al. (2015), but conceptually simpler. In the Π-model, the comparison is done directly on network outputs, i.e., after softmax activation, and there is no auxiliary mapping between the two branches such as the learned denoising functions in the ladder network architecture. Furthermore, instead of having one “clean” and one “corrupted” branch as in Γ-model, we apply equal augmentation and noise to the inputs for both branches.
As shown in Section 3, the Π-model combined with a good convolutional network architecture provides a significant improvement over prior art in classification accuracy.
2.2 TEMPORAL ENSEMBLING
Analyzing how the Π-model works, we could equally well split the evaluation of the two branches in two separate phases: first classifying the training set once without updating the weights θ, and then training the network on the same inputs under different augmentations and dropout, using the just obtained predictions as targets for the unsupervised loss component. As the training targets obtained this way are based on a single evaluation of the network, they can be expected to be noisy. Temporal ensembling alleviates this by aggregating the predictions of multiple previous network evaluations into an ensemble prediction. It also lets us evaluate the network only once during training, gaining an approximate 2x speedup over the Π-model.
The structure of our temporal ensembling method is shown in Figure 1 (bottom), and the pseudocode in Algorithm 2. The main difference to the Π-model is that the network and augmentations are evaluated only once per input per epoch, and the target vectors z̃ for the unsupervised loss component are based on prior network evaluations instead of a second evaluation of the network.
After every training epoch, the network outputs zi are accumulated into ensemble outputs Zi by updating Zi ← αZi + (1− α)zi, where α is a momentum term that controls how far the ensemble reaches into training history. Because of dropout regularization and stochastic augmentation, Z thus contains a weighted average of the outputs of an ensemble of networks f from previous training epochs, with recent epochs having larger weight than distant epochs. For generating the training targets z̃, we need to correct for the startup bias in Z by dividing by factor (1 − αt). A similar bias correction has been used in, e.g., Adam (Kingma & Ba, 2014) and mean-only batch normalization (Salimans & Kingma, 2016). On the first training epoch, Z and z̃ are zero as no data from previous epochs is available. For this reason, we specify the unsupervised weight ramp-up function w(t) to also be zero on the first training epoch.
1Squared difference gave slightly but consistently better results than cross-entropy loss in our tests.
Algorithm 2 Temporal ensembling pseudocode. Note that the updates of Z and z̃ could equally well be done inside the minibatch loop; in this pseudocode they occur between epochs for clarity. Require: xi = training stimuli Require: L = set of training input indices with known labels Require: yi = labels for labeled inputs i ∈ L Require: α = ensembling momentum, 0 ≤ α < 1 Require: w(t) = unsupervised weight ramp-up function Require: fθ(x) = stochastic neural network with trainable parameters θ Require: g(x) = stochastic input augmentation function Z ← 0[N×C] . initialize ensemble predictions z̃ ← 0[N×C] . initialize target vectors for t in [1,num epochs] do
for each minibatch B do zi∈B ← fθ(g(xi∈B , t)) . evaluate network outputs for augmented inputs loss ←− 1|B| ∑ i∈(B∩L) log zi[yi] . supervised loss component
+ w(t) 1C|B| ∑ i∈B ||zi − z̃i||2 . unsupervised loss component
update θ using, e.g., ADAM . update network parameters end for Z ← αZ + (1− α)z . accumulate ensemble predictions z̃ ← Z/(1− αt) . construct target vectors by bias correction
end for return θ
The benefits of temporal ensembling compared to Π-model are twofold. First, the training is faster because the network is evaluated only once per input on each epoch. Second, the training targets z̃ can be expected to be less noisy than with Π-model. As shown in Section 3, we indeed obtain somewhat better results with temporal ensembling than with Π-model in the same number of training epochs. The downside compared to Π-model is the need to store auxiliary data across epochs, and the new hyperparameter α. While the matrix Z can be fairly large when the dataset contains a large number of items and categories, its elements are accessed relatively infrequently. Thus it can be stored, e.g., in a memory mapped file.
An intriguing additional possibility of temporal ensembling is collecting other statistics from the network predictions zi besides the mean. For example, by tracking the second raw moment of the network outputs, we can estimate the variance of each output component zi,j . This makes it possible to reason about the uncertainty of network outputs in a principled way (Gal & Ghahramani, 2016). Based on this information, we could, e.g., place more weight on more certain predictions vs. uncertain ones in the unsupervised loss term. However, we leave the exploration of these avenues as future work.
3 RESULTS
Our network structure is given in Table 5, and the test setup and all training parameters are detailed in Appendix A. We test the Π-model and temporal ensembling in two image classification tasks, CIFAR-10 and SVHN, and report the mean and standard deviation of 10 runs using different random seeds.
Although it is rarely stated explicitly, we believe that our comparison methods do not use input augmentation, i.e., are limited to dropout and other forms of permutation-invariant noise. Therefore we report the error rates without augmentation, unless explicitly stated otherwise. Given that the ability of an algorithm to extract benefit from augmentation is also an important property, we report the classification accuracy using a standard set of augmentations as well. In purely supervised training the de facto standard way of augmenting the CIFAR-10 dataset includes horizontal flips and random translations, while SVHN is limited to random translations. By using these same augmentations we can compare against the best fully supervised results as well. After all, the fully supervised results should indicate the upper bound of obtainable accuracy.
3.1 CIFAR-10
CIFAR-10 is a dataset consisting of 32× 32 pixel RGB images from ten classes. Table 1 shows a 2.1 percentage point reduction in classification error rate with 4000 labels (400 per class) compared to earlier methods for the non-augmented Π-model.
Enabling the standard set of augmentations further reduces the error rate by 4.2 percentage points to 12.36%. Temporal ensembling is slightly better still at 12.16%, while being twice as fast to train. This small improvement conceals the subtle fact that random horizontal flips need to be done independently for each epoch in temporal ensembling, while Π-model can randomize once per a pair of evaluations, which according to our measurements is ∼0.5 percentage points better than independent flips.
A principled comparison with Sajjadi et al. (2016b) is difficult due to several reasons. They provide results only for a fairly extreme set of augmentations (translations, flipping, rotations, stretching, and shearing) on top of fractional max pooling (Graham, 2014), which introduces random, local stretching inside the network, and is known to improve classification results substantially. They quote an error rate of only 13.60% for supervised-only training with 4000 labels, while our corresponding baseline is 34.85%. This gap indicates a huge benefit from versatile augmentations and fractional max pooling—in fact, their baseline result is already better than any previous semisupervised results. By enabling semi-supervised learning they achieve a 17% drop in classification error rate (from 13.60% to 11.29%), while we see a much larger relative drop of 65% (from 34.85% to 12.16%).
3.2 SVHN
The street view house numbers (SVHN) dataset consists of 32× 32 pixel RGB images of real-world house numbers, and the task is to classify the centermost digit. In SVHN we chose to use only the
official 73257 training examples following Salimans et al. (2016). Even with this choice our error rate with all labels is only 3.05% without augmentation.
Table 2 compares our method to the previous state-of-the-art. With the most commonly used 1000 labels we observe an improvement of 2.7 percentage points, from 8.11% to 5.43% without augmentation, and further to 4.42% with standard augmentations.
We also investigated the behavior with 500 labels, where we obtained an error rate less than half of Salimans et al. (2016) without augmentations, with a significantly lower standard deviation as well. When augmentations were enabled, temporal ensembling further reduced the error rate to 5.12%. In this test the difference between Π-model and temporal ensembling was quite significant at 1.5 percentage points.
In SVHN Sajjadi et al. (2016b) provide results without augmentation, with the caveat that they use fractional max pooling, which is a very augmentation-like technique due to the random, local stretching it introduces inside the network. It leads to a superb error rate of 2.28% in supervisedonly training, while our corresponding baseline is 3.05% (or 2.88% with translations). Given that in a separate experiment our network matched the best published result for non-augmented SVHN when extra data is used (1.69% from Lee et al. (2015)), this gap is quite surprising, and leads us to conclude that fractional max pooling leads to a powerful augmentation of the dataset, well beyond what simple translations can achieve. Our temporal ensembling technique obtains better error rates for both 500 and 1000 labels (5.12% and 4.42%, respectively) compared to the 6.03% reported by Sajjadi et al. for 732 labels.
3.3 CIFAR-100 AND TINY IMAGES
The CIFAR-100 dataset consists of 32× 32 pixel RGB images from a hundred classes. We are not aware of previous semi-supervised results in this dataset, and chose 10000 labels for our experiments. Table 3 shows error rates of 43.43% and 38.65% without and with augmentation, respectively. These correspond to 7.8 and 5.9 percentage point improvements compared to supervised learning with labeled inputs only.
We ran two additional tests using unlabeled extra data from Tiny Images dataset (Torralba et al., 2008): one with randomly selected 500k extra images, most not corresponding to any of the CIFAR100 categories, and another with a restricted set of 237k images from the categories that correspond to those found in the CIFAR-100 dataset (see appendix A for details). The results are shown in Table 4. The addition of randomly selected, unlabeled extra images improved the error rate by 2.7 percentage points (from 26.30% to 23.63%), indicating a desirable ability to learn from random natural images. Temporal ensembling benefited much more from the extra data than the Π-model. Interestingly, restricting the extra data to categories that are present in CIFAR-100 did not improve
the classification accuracy further. This indicates that in order to train a better classifier by adding extra data as unlabeled inputs, it is enough to have the extra data roughly in the same space as the actual inputs—in our case, natural images. We hypothesize that it may even be possible to use properly crafted synthetic data as unlabeled inputs to obtain improved classifiers.
In order to keep the training times tolerable, we limited the number of unlabeled inputs to 50k per epoch in these tests, i.e., on every epoch we trained using all 50k labeled inputs from CIFAR-100 and 50k additional unlabeled inputs from Tiny Images. The 50k unlabeled inputs were chosen randomly on each epoch from the 500k or 237k extra inputs. In temporal ensembling, after each epoch we updated only the rows of Z that corresponded to inputs used on that epoch.
3.4 SUPERVISED LEARNING
When all labels are used for traditional supervised training, our network approximately matches the state-of-the-art error rate for a single model in CIFAR-10 with augmentation (Lee et al., 2015; Mishkin & Matas, 2016) at 6.05%, and without augmentation (Salimans & Kingma, 2016) at 7.33%. The same is probably true for SVHN as well, but there the best published results rely on extra data that we chose not to use.
Given this premise, it is perhaps somewhat surprising that our methods reduce the error rate also when all labels are used (Tables 1 and 2). We believe that this is an indication that the consistency requirement adds a degree of resistance to ambiguous labels that are fairly common in many classification tasks, and that it encourages features to be more invariant to stochastic sampling.
3.5 TOLERANCE TO INCORRECT LABELS
In a further test we studied the hypothesis that our methods add tolerance to incorrect labels by assigning a random label to a certain percentage of the training set before starting to train. Figure 2 shows the classification error graphs for standard supervised training and temporal ensembling.
Clearly our methods provide considerable resistance to wrong labels, and we believe this is because the unsupervised loss term encourages the mapping function implemented by the network to be flat in the vicinity of all input data points, whereas the supervised loss term enforces the mapping function to have a specific value in the vicinity of the labeled input data points. This means that even the wrongly labeled inputs play a role in shaping the mapping function—the unsupervised loss term smooths the mapping function and thus also the decision boundaries, effectively fusing the inputs into coherent clusters, whereas the excess of correct labels in each class is sufficient for locking the clusters to the right output vectors through the supervised loss term. The difference to classical regularizers is that we induce smoothness only on the manifold of likely inputs instead
of over the entire input domain. For further analysis about the importance of the gradient of the mapping function, see Simard et al. (1998).
4 RELATED WORK
There is a large body of previous work on semi-supervised learning (Zhu, 2005). In here we will concentrate on the ones that are most directly connected to our work.
Γ-model is a subset of a ladder network (Rasmus et al., 2015) that introduces lateral connections into an encoder-decoder type network architecture, targeted at semi-supervised learning. In Γ-model, all but the highest lateral connections in the ladder network are removed, and after pruning the unnecessary stages, the remaining network consists of two parallel, identical branches. One of the branches takes the original training inputs, whereas the other branch is given the same input corrupted with noise. The unsupervised loss term is computed as the squared difference between the (pre-activation) output of the clean branch and a denoised (pre-activation) output of the corrupted branch. The denoised estimate is computed from the output of the corrupted branch using a parametric nonlinearity that has 10 auxiliary trainable parameters per unit. Our Π-model differs from the Γ-model in removing the parametric nonlinearity and denoising, having two corrupted paths, and comparing the outputs of the network instead of pre-activation data of the final layer.
Sajjadi et al. (2016b) recently introduced a new loss function for semi-supervised learning, so called transform/stability loss, which is founded on the same principle as our work. During training, they run augmentation and network evaluation n times for each minibatch, and then compute an unsupervised loss term as the sum of all pairwise squared distances between the obtained n network outputs. As such, their technique follows the general pseudo-ensemble agreement (PEA) regularization framework of Bachman et al. (2014). In addition, they employ a mutual exclusivity loss term (Sajjadi et al., 2016a) that we do not use. Our Π-model can be seen as a special case of the transform/stability loss obtained by setting n = 2. The computational cost of training with transform/stability loss increases linearly as a function of n, whereas the efficiency of our temporal ensembling technique remains constant regardless of how large effective ensemble we obtain via the averaging of previous epochs’ predictions.
In bootstrap aggregating, or bagging, multiple networks are trained independently based on subsets of training data (Breiman, 1996). This results in an ensemble that is more stable and accurate than the individual networks. Our approach can be seen as pulling the predictions from an implicit ensemble that is based on a single network, and the variability is a result of evaluating it under different dropout and augmentation conditions instead of training on different subsets of data. In work parallel to ours, Huang et al. (2017) store multiple snapshots of the network during training, hopefully corresponding to different local minima, and use them as an explicit ensemble.
The general technique of inferring new labels from partially labeled data is often referred to as bootstrapping or self-training, and it was first proposed by Yarowsky (1995) in the context of linguistic analysis. Whitney & Sarkar (2012) analyze Yarowsky’s algorithm and propose a novel graph-based label propagation approach. Similarly, label propagation methods (Zhu & Ghahramani, 2002) infer labels for unlabeled training data by comparing the associated inputs to labeled training inputs using a suitable distance metric. Our approach differs from this in two important ways. Firstly, we never compare training inputs against each other, but instead only rely on the unknown labels remaining constant, and secondly, we let the network produce the likely classifications for the unlabeled inputs instead of providing them through an outside process.
In addition to partially labeled data, considerable amount of effort has been put into dealing with densely but inaccurately labeled data. This can be seen as a semi-supervised learning task where part of the training process is to identify the labels that are not to be trusted. For recent work in this area, see, e.g., Sukhbaatar et al. (2014) and Patrini et al. (2016). In this context of noisy labels, Reed et al. (2014) presented a simple bootstrapping method that trains a classifier with the target composed of a convex combination of the previous epoch output and the known but potentially noisy labels. Our temporal ensembling differs from this by taking into account the evaluations over multiple epochs.
Generative Adversarial Networks (GAN) have been recently used for semi-supervised learning with promising results (Maaløe et al., 2016; Springenberg, 2016; Odena, 2016; Salimans et al., 2016). It
could be an interesting avenue for future work to incorporate a generative component to our solution. We also envision that our methods could be applied to regression-type learning tasks.
5 ACKNOWLEDGEMENTS
We thank the anonymous reviewers, Tero Karras, Pekka Jänis, Tim Salimans, Ian Goodfellow, as well as Harri Valpola and his colleagues at Curious AI for valuable suggestions that helped to improve this article.
A NETWORK ARCHITECTURE, TEST SETUP, AND TRAINING PARAMETERS
Table 5 details the network architecture used in all of our tests. It is heavily inspired by ConvPoolCNN-C (Springenberg et al., 2014) and the improvements made by Salimans & Kingma (2016). All data layers were initialized following He et al. (2015), and we applied weight normalization and mean-only batch normalization (Salimans & Kingma, 2016) with momentum 0.999 to all of them. We used leaky ReLU (Maas et al., 2013) with α = 0.1 as the non-linearity, and chose to use max pooling instead of strided convolutions because it gave consistently better results in our experiments.
All networks were trained using Adam (Kingma & Ba, 2014) with a maximum learning rate of λmax = 0.003, except for temporal ensembling in the SVHN case where a maximum learning rate of λmax = 0.001 worked better. Adam momentum parameters were set to β1 = 0.9 and β2 = 0.999 as suggested in the paper. The maximum value for the unsupervised loss component was set to wmax ·M/N , where M is the number of labeled inputs and N is the total number of training inputs. For Π-model runs, we used wmax = 100 in all runs except for CIFAR-100 with Tiny Images where we set wmax = 300. For temporal ensembling we used wmax = 30 in most runs. For the corrupted label test in Section 3.5 we used wmax = 300 for 0% and 20% corruption, and wmax = 3000 for corruption of 50% and higher. For basic CIFAR-100 runs we used wmax = 100, and for CIFAR-100 with Tiny Images we used wmax = 1000. The accumulation decay constant of temporal ensembling was set to α = 0.6 in all runs.
In all runs we ramped up both the learning rate λ and unsupervised loss component weight w during the first 80 epochs using a Gaussian ramp-up curve exp[−5(1 − T )2], where T advances linearly from zero to one during the ramp-up period. In addition to ramp-up, we annealed the learning rate λ to zero and Adam β1 to 0.5 during the last 50 epochs, but otherwise we did not decay them during training. The ramp-down curve was similar to the ramp-up curve but time-reversed and with a scaling constant of 12.5 instead of 5. All networks were trained for 300 epochs with minibatch size of 100.
CIFAR-10 Following previous work in fully supervised learning, we pre-processed the images using ZCA and augmented the dataset using horizontal flips and random translations. The translations were drawn from [−2, 2] pixels, and were independently applied to both branches in the Π-model. SVHN We pre-processed the input images by biasing and scaling each input image to zero mean and unit variance. We used only the 73257 items in the official training set, i.e., did not use the provided 531131 extra items. The training setups were otherwise similar to CIFAR-10 except that horizontal flips were not used.
Implementation Our implementation is written in Python using Theano (Theano Development Team, 2016) and Lasagne (Dieleman et al., 2015), and is available at https://github.com/smlaine2/tempens.
Model convergence As discussed in Section 2.1, a slow ramp-up of the unsupervised cost is very important for getting the models to converge. Furthermore, in our very preliminary tests with 250 labels in SVHN we noticed that optimization tended to explode during the ramp-up period, and we eventually found that using a lower value for Adam β2 parameter (e.g., 0.99 instead of 0.999) seems to help in this regard.
We do not attempt to guarantee that the occurrence of labeled inputs during training would be somehow stratified; with bad luck there might be several consecutive minibatches without any labeled inputs when the label density is very low. Some previous work has identified this as a weakness, and have solved the issue by shuffling the input sequences in such a way that stratification is guaranteed, e.g. Rasmus et al. (2015) (confirmed from the authors). This kind of stratification might further improve the convergence of our methods as well.
Tiny Images, extra data from restricted categories The restricted extra data in Section 3.3 was extracted from Tiny Images by picking all images with labels corresponding to the 100 categories used in CIFAR-100. As the Tiny Images dataset does not contain CIFAR-100 categories aquarium fish and maple tree, we used images with labels fish and maple instead. The result was a total of 237 203 images that were used as unlabeled extra data. Table 6 shows the composition of this extra data set.
It is worth noting that the CIFAR-100 dataset itself is a subset of Tiny Images, and we did not explicitly prevent overlap between this extra set and CIFAR-100. This led to approximately a third of the CIFAR-100 training and test images being present as unlabeled inputs in the extra set. The other test with 500k extra entries picked randomly out of all 79 million images had a negligible overlap with CIFAR-100.

This work explores taking advantage of the stochasticity of neural network outputs under randomized augmentation and regularization techniques to provide targets for unlabeled data in a semi-supervised setting. This is accomplished by either applying stochastic augmentation and regularization on a single image multiple times per epoch and encouraging the outputs to be similar (Π-model) or by keeping a weighted average of past epoch outputs and penalizing deviations of current network outputs from this running mean (temporal ensembling). The core argument is that these approaches produce ensemble predictions which are likely more accurate than the current network and are thus good targets for unlabeled data. Both approaches seem to work quite well on semi-supervised tasks and some results show that they are almost unbelievably robust to label noise.
The paper is clearly written and provides sufficient details to reproduce these results in addition to providing a public code base. The core idea of the paper is quite interesting and seems to result in higher semi-supervised accuracy than prior work. I also found the attention to and discussion of the effect of different choices of data augmentation to be useful.
I am a little surprised that a standard supervised network can achieve 30% accuracy on SVHN given 90% random training labels. This would only give 19% correctly labeled data (9% by chance + 10% unaltered). I suppose the other 81% would not provide a consistent training signal such that it is possible, but it does seem quite unintuitive. I tried to look through the github for this experiment but it does not seem to be included.
As for the resistance of Π-model and temporal ensembling to this label noise, I find that somewhat more believable given the large weights placed on the consistency constraint for this task. The authors should really include discussion of w(t) in the main paper. Especially because the tremendous difference in w_max in the incorrect label tolerance experiment (10x for Π-model and 100x for temporal ensembling from the standard setting).
Could the authors comment towards the scalability for larger problems? For ImageNet, you would need to store around 4.8 gigs for the temporal ensembling method or spend 2x as long training with Π-model.
Can the authors discuss sensitivity of this approach to the amount and location of dropout layers in the architecture?
Preliminary rating:
I think this is a very interesting paper with quality results and clear presentation.
Minor note:
2nd paragraph of page one 'without neither' -> 'without either'


PIXELVAE: A LATENT VARIABLE MODEL FOR NATURAL IMAGES
Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64 × 64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.
1 INTRODUCTION
Building high-quality generative models of natural images has been a long standing challenge. Although recent work has made significant progress (Kingma & Welling, 2014; van den Oord et al., 2016a;b), we are still far from generating convincing, high-resolution natural images.
Many recent approaches to this problem are based on an efficient method for performing amortized, approximate inference in continuous stochastic latent variables: the variational autoencoder (VAE) (Kingma & Welling, 2014) jointly trains a top-down decoder generative neural network with a bottom-up encoder inference network. VAEs for images typically use rigid decoders that model the output pixels as conditionally independent given the latent variables. The resulting model learns a useful latent representation of the data and effectively models global structure in images, but has difficulty capturing small-scale features such as textures and sharp edges due to the conditional independence of the output pixels, which significantly hurts both log-likelihood and quality of generated samples compared to other models.
PixelCNNs (van den Oord et al., 2016a;b) are another state-of-the-art image model. Unlike VAEs, PixelCNNs model image densities autoregressively, pixel-by-pixel. This allows it to capture fine details in images, as features such as edges can be precisely aligned. By leveraging carefully constructed masked convolutions (van den Oord et al., 2016b), PixelCNNs can be trained efficiently in parallel on GPUs. Nonetheless, PixelCNN models are still very computationally expensive. Unlike typical convolutional architectures they do not apply downsampling between layers, which means that each layer is computationally expensive and that the depth of a PixelCNN must grow linearly with the size of the images in order for it to capture dependencies between far-away pixels. PixelCNNs also do not explicitly learn a latent representation of the data, which can be useful for downstream tasks such as semi-supervised learning.
∗Corresponding author; igul222@gmail.com
Our contributions are as follows:
• We present PixelVAE, a latent variable model which combines the largely complementary advantages of VAEs and PixelCNNs by using PixelCNN-based masked convolutions in the conditional output distribution of a VAE.
• We extend PixelVAE to a hierarchical model with multiple stochastic layers and autoregressive decoders at each layer. This lets us autoregressively model not only the output pixels but also higher-level latent feature maps.
• On MNIST, we show that PixelVAE: (1) establishes a new state-of-the-art likelihood, (2) performs comparably to PixelCNN using far fewer computationally expensive autoregressive layers, (3) learns more compressed latent codes than a standard VAE while still accounting for most non-trivial structure, and (4) learns a latent code which separates digits better than a standard VAE.
• We evaluate hierarchical PixelVAE on two challenging natural image datasets (64 × 64 ImageNet and LSUN bedrooms). On 64× 64 ImageNet, we report likelihood competitive with the state of the art at significantly less computational cost. On LSUN bedrooms, we generate high-quality samples and show that hierarchical PixelVAE learns to model different properties of the scene with each of its multiple layers.
2 RELATED WORK
There have been many recent advancements in generative modeling of images. We briefly discuss some of these below, especially those that are related to our approach.
The Variational Autoencoder (VAE) (Kingma & Welling, 2014) is a framework to train neural networks for generation and approximate inference jointly by optimizing a variational bound on the data log-likelihood. The use of normalizing flows (Rezende & Mohamed, 2015) improves the flexibility of the VAE approximate posterior. Based on this, Kingma et al. (2016) develop an efficient formulation of an autoregressive approximate posterior model using MADE (Germain et al., 2015). In our work, we avoid the need for such flexible inference models by using autoregressive priors.
The idea of using autoregressive conditional likelihoods in VAEs has been explored in the context of language modeling in (Bowman et al., 2016), however in that work the use of latent variables fails to improve likelihood over a purely autoregressive model.
Simultaneously to our work, Chen et al. (2016) present a VAE model for images with an an autoregressive output distribution. In constrast to Chen et al. (2016), who focus on models with a single layer of latent variables, we also investigate models with a hierarchy of latent variables (and corresponding autoregressive priors) and show that they enable us to scale our model to challenging natural image datasets.
Another promising recent approach is Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which pit a generator network and a discriminator network against each other. Recent work has improved training stability (Radford et al., 2015; Salimans et al., 2016) and incorporated inference networks into the GAN framework (Dumoulin et al., 2016; Donahue et al., 2016). GANs generate compelling samples compared to our work, but still exhibit unstable training dynamics and are known to underfit by ignoring modes of the data distribution (Dumoulin et al., 2016). Further, it is difficult to accurately estimate the data likelihood in GANs.
3 PIXELVAE MODEL
Like a VAE, our model jointly trains an “encoder” inference network, which maps an image x to a posterior distribution over latent variables z, and a “decoder” generative network, which models a distribution over x conditioned on z. The encoder and decoder networks are composed of a series of convolutional layers, respectively with strided convolutions for downsampling in the encoder and transposed convolutions for upsampling in the decoder.
As opposed to most VAE decoders that model each dimension of the output independently (for example, by modeling the output as a Gaussian with diagonal covariance), we use a conditional PixelCNN in the decoder. Our decoder models x as the product of each dimension xi conditioned on all previous dimensions and the latent variable z:
p(x|z) = ∏ i p(xi|x1, . . . , xi−1, z)
We first transform z through a series of convolutional layers into feature maps with the same spatial resolution as the output image and then concatenate the resulting feature maps with the image. The resulting concatenated feature maps are then further processed by several PixelCNN masked convolutional layers and a final PixelCNN 256-way softmax output.
Unlike typical PixelCNN implementations, we use very few PixelCNN layers in our decoder, relying on the latent variables to model the structure of the input at scales larger than the combined receptive
field of our PixelCNN layers. As a result of this, our architecture captures global structure at a much lower computational cost than a standard PixelCNN implementation.
3.1 HIERARCHICAL ARCHITECTURE
The performance of VAEs can be improved by stacking them to form a hierarchy of stochastic latent variables: in the simplest configuration, the VAE at each level models a distribution over the latent variables at the level below, with generation proceeding downward and inference upward through each level (i.e. as in Fig. 3). In convolutional architectures, the intermediate latent variables are typically organized into feature maps whose spatial resolution decreases toward higher levels.
Our model can be extended in the same way. At each level, the generator is a conditional PixelCNN over the latent features in the level below. This lets us autoregressively model not only the output distribution over pixels but also the prior over each set of latent feature maps. The higher-level PixelCNN decoders use diagonal Gaussian output layers instead of 256-way softmax, and model the dimensions within each spatial location (i.e. across feature maps) independently. This is done for simplicity, but is not a limitation of our model.
The output distributions over the latent variables for the generative and inference networks decompose as follows (see Fig. 3).
p(z1, · · · , zL) = p(zL)p(zL−1|zL) · · · p(z1|z2) q(z1, · · · , zL|x) = q(z1|x) · · · q(zL|x)
We optimize the negative of the evidence lower bound (sum of data negative log-likelihood and KL-divergence of the posterior over latents with the prior).
−L(x, q, p) = −Ez1∼q(z1|x) log p(x|z1) +DKL(q(z1, · · · zL|x)||p(z1, · · · , zL)) = −Ez1∼q(z1|x) log p(x|z1) + ∫
z1,··· ,zL
L∏ j=1 q(zj |x) L∑ i=1 log q(zi|x) p(zi|zi+1) dz1...dzL
= −Ez1∼q(z1|x) log p(x|z1) + L∑
i=1 ∫ z1,··· ,zL L∏ j=1 q(zj |x) log q(zi|x) p(zi|zi+1) dz1...dzL
= −Ez1∼q(z1|x) log p(x|z1) + L∑
i=1 ∫ zi,zi+1 q(zi+1|x)q(zi|x) log q(zi|x) p(zi|zi+1) dzidzi+1
= −Ez1∼q(z1|x) log p(x|z1) + L∑
i=1
Ezi+1∼q(zi+1|x) [ DKL(q(zi|x)||p(zi|zi+1)) ]
Note that when specifying an autoregressive prior over each latent level zi, we can leverage masked convolutions (van den Oord et al., 2016b) and samples drawn independently from the approximate posterior q(zi | x) (i.e. from the inference network) to train efficiently in parallel on GPUs.
4 EXPERIMENTS
4.1 MNIST
We evaluate our model on the binarized MNIST dataset (Salakhutdinov & Murray, 2008; Lecun et al., 1998) and report results in Table 1. We also experiment with a variant of our model in which each PixelCNN layer is directly conditioned on a linear transformation of latent variable, z (rather than transforming z first through several upsampling convolutional layers) (as in (van den Oord et al., 2016b) and find that this further improves performance, achieving an NLL upper bound comparable with the current state of the art. We estimate the marginal likelihood of our MNIST model using the importance sampling technique in Burda et al. (2015), which computes a lower bound on the likelihood whose tightness increases with the number of importance samples per datapoint. We use N = 5000 samples per datapoint (higher values don’t appear to significantly affect the likelihood estimate) and achieve state-of-the-art likelihood.
4.1.1 USING FEW PIXELCNN LAYERS
The masked convolutional layers in PixelCNN are computationally expensive because they operate at the full resolution of the image and in order to cover the full receptive field of the image, PixelCNN typically needs a large number of them. One advantage of our architecture is that we can achieve strong performance with very few PixelCNN layers, which makes training and sampling from our model significantly faster than PixelCNN. To demonstrate this, we compare the performance of our model to PixelCNN as a function of the number of PixelCNN layers (Fig. 4a). We find that with fewer than 10 autoregressive layers, our PixelVAE model performs much better than PixelCNN. This is expected since with few layers, the effective receptive field of the PixelCNN output units is too small to capture long-range dependencies in the data.
We also observe that adding even a single PixelCNN layer has a dramatic impact on the NLL bound of PixelVAE. This is not surprising since the PixelCNN layer helps model local characteristics which
are complementary to the global characteristics which a VAE with a factorized output distribution models.
4.1.2 LATENT VARIABLE INFORMATION CONTENT
Because the autoregressive conditional likelihood function of PixelVAE is expressive enough to model some properties of the image distribution, it isn’t forced to account for those properties through its latent variables as a standard VAE is. As a result, we can expect PixelVAE to learn latent representations which are invariant to textures, precise positions, and other attributes which are more efficiently modeled by the autoregressive decoder. To empirically validate this, we train PixelVAE models with different numbers of autoregressive layers (and hence, different PixelCNN receptive field sizes) and plot the breakdown of the NLL bound for each of these models into the reconstruction term log p(x|z) and the KL divergence term DKL(q(z|x)||p(z)) (Fig. 4b). The KL divergence term can be interpreted as a measure of the information content in the posterior distribution q(z|x) (in the sense that in expectation, samples from q(z|x) require KL(q||p) fewer bits to code under a code optimized for q than under one optimized for p (Burnham & Anderson, 2003)) and hence, models with smaller KL terms encode less information in their latent variables.
We observe a sharp drop in the KL divergence term when we use a single autoregressive layer compared to no autoregressive layers, indicating that the latent variables have been freed from having to encode small-scale details in the images. Since the addition of a single PixelCNN layer allows the decoder to model interactions between pixels which are at most 2 pixels away from each other (since our masked convolution filter size is 5×5), we can also say that most of the non-trivial (long-range) structure in the images is still encoded in the latent variables.
4.1.3 LATENT REPRESENTATIONS
On MNIST, given a sufficiently high-dimensional latent space, VAEs have already been shown to learn representations in which digits are well-separated (Sønderby et al., 2016). However, this task becomes more challenging as the capacity of the latent space is decreased. PixelVAE’s flexible output distribution should allow it to learn a latent representation which is invariant to small details and thus better models global factors of variation given limited capacity.
To test this, we train a PixelVAE with a two-dimensional latent space, and an equivalent VAE. We visualize the distribution of test set images in latent space and observe that PixelVAE’s latent representation separates digits significantly better than VAE (Figure 5). To quantify this difference, we train a K-nearest neighbors classifier in the latent space of each model and find that PixelVAE
significantly outperforms VAE, achieving a test error of 7.2% compared to VAE’s 22.9%. We also note that unlike VAE, PixelVAE learns a representation in which digit identity is largely disentangled from other generative factors.
4.2 LSUN BEDROOMS
To evaluate our model’s performance with more data and complicated image distributions, we perform experiments on the LSUN bedrooms dataset (Yu et al., 2015). We use the same preprocessing as in Radford et al. (2015) to remove duplicate images in the dataset. For quantitative experiments we use a 32× 32 downsampled version of the dataset, and we present samples from a model trained on the 64× 64 version. We train a two-level PixelVAE with latent variables at 1×1 and 8×8 spatial resolutions. We find that this outperforms both a two-level convolutional VAE with diagonal Gaussian output and a singlelevel PixelVAE in terms of log-likelihood and sample quality. We also try replacing the PixelCNN layers at the higher level with a diagonal Gaussian decoder and find that this hurts log-likelihood, which suggests that multi-scale PixelVAE uses those layers effectively to autoregressively model latent features.
4.2.1 FEATURES MODELED AT EACH LAYER
To see which features are modeled by each of the multiple layers, we draw multiple samples while varying the sampling noise at only a specific layer (either at the pixel-wise output or one of the latent layers) and visually inspect the resulting images (Fig. 6). When we vary only the pixellevel sampling (holding z1 and z2 fixed), samples are almost indistinguishable and differ only in precise positioning and shading details, suggesting that the model uses the pixel-level autoregressive distribution to model only these features. Samples where only the noise in the middle-level (8 × 8) latent variables is varied have different objects and colors, but appear to have similar basic room geometry and composition. Finally, samples with varied top-level latent variables have diverse room geometry.
4.3 64× 64 IMAGENET
The 64×64 ImageNet generative modeling task was introduced in (van den Oord et al., 2016a) and involves density estimation of a difficult, highly varied image distribution. We trained a heirarchical PixelVAE model (with a similar architecture to the model in section 4.2) on 64× 64 ImageNet and report validation set likelihood in Table 2. Our model achieves a likelihood competitive with van den Oord et al. (2016a;b), despite being substantially less computationally complex. A visual inspection of ImageNet samples from our model (Fig. 7) also reveals them to be significantly more globally coherent than samples from PixelRNN.
5 CONCLUSIONS
In this paper, we introduced a VAE model for natural images with an autoregressive decoder that achieves strong performance across a number of datasets. We explored properties of our model, showing that it can generate more compressed latent representations than a standard VAE and that it can use fewer autoregressive layers than PixelCNN. We established a new state-of-the-art on binarized MNIST dataset in terms of likelihood on 64× 64 ImageNet and demonstrated that our model generates high-quality samples on LSUN bedrooms.
The ability of PixelVAE to learn compressed representations in its latent variables by ignoring the small-scale structure in images is potentially very useful for downstream tasks. It would be interesting to further explore our model’s capabilities for semi-supervised classification and representation learning in future work.
ACKNOWLEDGMENTS
The authors would like to thank the developers of Theano (Theano Development Team, 2016) and Blocks and Fuel (van Merriënboer et al., 2015). We acknowledge the support of the following agencies for research funding and computing support: Ubisoft, Nuance Foundation, NSERC, Calcul Quebec, Compute Canada, CIFAR, MEC Project TRA2014-57088-C2-1-R, SGR project 2014- SGR-1506 and TECNIOspring-FP7-ACCI grant.
C MNIST RECONSTRUCTIONS
B MNIST SAMPLES
D MORE SAMPLES FOR HIERARCHICAL LATENT SPACE VISUALIZATIONS
E MODEL ARCHITECTURE
E.1 MNIST
For our quantitative MNIST experiments, the architectures of our encoder and decoder are as follows. Unless otherwise specified, all convolutional layers use ReLU nonlinearity. We also make an open-source implementation of this model available at https://github.com/igul222/ PixelVAE.
Encoder x→ (µ, σ) Kernel size Stride Output channels
Convolution 3x3 1 32 Convolution 3x3 2 32 Convolution 3x3 1 32 Convolution 3x3 2 64
Pad 7×7 feature maps to 8×8 Convolution 3x3 1 64 Convolution 3x3 2 64 Convolution 3x3 1 64 Convolution 3x3 1 64 Convolution 3x3 1 64
Flatten
Linear - - 2×latent dimensionality
Decoder z → x Kernel size Stride Output channels
Linear - - 4×4×64 Reshape to (64, 4, 4)
Convolution 3x3 1 64 Convolution 3x3 1 64 Transposed convolution 3x3 2 64 Convolution 3x3 1 64
Crop 8×8 feature maps to 7×7 Transposed convolution 3x3 2 32 Convolution 3x3 1 32 Transposed convolution 3x3 2 32 Convolution 3x3 1 32 PixelCNN gated residual block 7x7 1 32
PixelCNN gated residual block(s) [ 5x5 ] ×N 1 32 PixelCNN gated convolution 1x1 1 32 PixelCNN gated convolution 1x1 1 32 Convolution 1x1 1 1
E.2 LSUN BEDROOMS AND 64×64 IMAGENET
The LSUN and ImageNet models use the same architecture: all encoders and decoders are residual networks; we use pre-activation residual blocks with two 3 × 3 convolutional layers each and ELU nonlinearity. Some residual blocks perform downsampling, using a 2 × 2 stride in the second convolutional layer, or upsampling, using subpixel convolution in the first convolutional layer. Weight normalization is used in masked convolutional layers; in all other layers, batch normalization is used. We optimize using Adam with learning rate 1e-3. Training proceeds for 400K iterations using batch size 48.
For further architectural details, please refer to our open-source implementation at https:// github.com/igul222/PixelVAE.
Bottom-level Encoder x→ h1 Kernel size Resample Output channels
Embedding - - 48 Convolution 1x1 - 192 Residual block [ 3x3 ] × 2 - 192 Residual block [ 3x3 ] × 2 Down ×2 256 Residual block [ 3x3 ] × 2 - 256 Residual block [ 3x3 ] × 2 Down ×2 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512
Bottom-level Decoder z1 → x Kernel size Resample Output channels
Convolution 1x1 1 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 Up ×2 256 Residual block [ 3x3 ] × 2 - 256 Residual block [ 3x3 ] × 2 Up ×2 192 Residual block [ 3x3 ] × 2 - 192 Embedding - - 48 PixelCNN gated residual block [ 3x3 ] × 2 - 384 PixelCNN gated residual block [ 3x3 ] × 2 - 384 PixelCNN gated residual block [ 3x3 ] × 2 - 384
Top-level Encoder h1 → h2 Kernel size Resample Output channels
Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 Down ×2 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 Down ×2 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512
Top-level Decoder z2 → z1 Kernel size Resample Output channels
Linear - - 4×4×512 Reshape to (512, 4, 4)
Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 Up ×2 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 Up ×2 512 Residual block [ 3x3 ] × 2 - 512 Residual block [ 3x3 ] × 2 - 512 PixelCNN convolution 5x5 - 512 PixelCNN gated residual block [ 3x3 ] × 2 - 512 PixelCNN gated residual block [ 3x3 ] × 2 - 512 PixelCNN gated residual block [ 3x3 ] × 2 - 512 Convolution 1x1 - 256

UPDATE: The authors addressed all my concerns in the new version of the paper, so I raised my score and now recommend acceptance.
--------------
This paper combines the recent progress in variational autoencoder and autoregressive density modeling in the proposed PixelVAE model. The paper shows that it can match the NLL performance of a PixelCNN with a PixelVAE that has a much shallower PixelCNN decoder.
I think the idea of capturing the global structure with a VAE and modeling the local structure with a PixelCNN decoder makes a lot of sense and can prevent the blurry reconstruction/samples of VAE. I specially like the hierarchical image generation experiments.
I have the following suggestions/concerns about the paper:
1) Is there any experiment showing that using the PixelCNN as the decoder of VAE will result in better disentangling of high-level factors of variations in the hidden code? For example, the authors can train a PixelVAE and VAE on MNIST with 2D hidden code and visualize the 2D hidden code for test images and color code each hidden code based on the digit and show that the digits have a better separation in the PixelVAE representation. A semi-supervised classification comparison between VAE and the PixelVAE will also significantly improve the quality of the paper.
2) A similar idea is also presented in a concurrent ICLR submission "Variational Lossy Autoencoder". It would be interesting to have a discussion included in the paper and compare these works.
3) The answer to the pre-review questions made the architecture details of the paper much more clear, but I still ask the authors to include the exact architecture details of all the experiments in the paper and/or open source the code. The clarity of the presentation is not satisfying and the experiments are difficult to reproduce.
4) As pointed out in my pre-review question, it would be great to include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.
I will gladly raise the score if the authors address my concerns.