This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
1 INTRODUCTION
Representation learning methods have received a lot of attention by researchers and practitioners because of its successful application to complex problems in areas such as computer vision, speech recognition and text processing (LeCun et al., 2015). Most of these efforts have concentrated on data involving one type of information (images, text, speech, etc.), despite data being naturally multimodal. Multimodality refers to the fact that the same real-world concept can be described by different views or data types. Collaborative encyclopedias (such as Wikipedia) describe a famous person through a mixture of text, images and, in some cases, audio. Users from social networks comment events like concerts or sport games with small phrases and multimedia attachments (images/videos/audios). Medical records are represented by a collection of images, sound, text and signals, among others. The increasing availability of multimodal databases from different sources has motivated the development of automatic analysis techniques to exploit the potential of these data as a source of knowledge in the form of patterns and structures that reveal complex relationships (Bhatt & Kankanhalli, 2011; Atrey et al., 2010). In recent years, multimodal tasks have acquired attention by the representation learning community. Strategies for visual question answering (Antol et al., 2015), or image captioning (Vinyals et al., 2015; Xu et al., 2015; Johnson et al., 2015) have developed interesting ways of combining different representation learning architectures.
Most of these models are focused on mapping from one modality to another or solving an auxiliary task to create a common representation with the information of all modalities. In this work, we design a novel module that combines multiple sources of information, which is optimized with respect to the end goal objective function. Our proposed module is based on the idea of gates for selecting which parts of the input are more likely to contribute for correctly generating the desired output. We
use multiplicative gates that assign importance to various features simultaneously, creating a rich multimodal representation that does not require manual tuning, but instead it learns directly from the training data. Our gated model can be reused in different network architectures for solving different tasks, and can be optimized end-to-end with other modules in the architecture using standard gradient-based optimization algorithms.
As an application use case, we explore the task of identifying a movie genre based on its plot and its poster. Genre classification has several application areas like document categorization (Kanaris & Stamatatos, 2009), recommendation systems (Makita & Lenskiy, 2016a), and information retrieval systems, among others. Figure 1 depicts the challenging task of assigning genres to a particular movie based solely on the usage of one modality. Such predictions were done with MaxoutMLP w2v and VGG transfer approaches (See Section 3), both of them are models based on representation learning. It can be seen that even a human might be confused if both modalities are not available. The main hypothesis of this work is that a model using gating units, in contrast to a hand-coded multimodal fusion architecture, will be able to learn an input-dependent gate-activation pattern that determines how each modality contribute to the output of hidden units.
The rest of the paper is organized as follows: Section 2 presents a literature review and some considerations of the previous work. Section 3 describes the methods used as baseline as well as our representation-leaning-based model proposed. Section 4 presents the experimental evaluation setup along with the details of the MM-IMDb dataset. Section 5 shows and discusses the results for movie genre classification. Finally, Section 6 draws the conclusions and future work.
2 RELATED WORK
2.1 MULTIMODAL FUSION
Different reviews (Atrey et al., 2010; Bhatt & Kankanhalli, 2011; Li Deng, 2014; Deng, 2014) have summarized strategies that addressed multimodal analysis. Most of the collected works claimed the superiority of multimodal over unimodal approaches for automatic analysis tasks. A conventional multimodal analysis system receives as input two or more modalities that describe a particular concept. The most common multimodal sources are video, audio, images and text. In recent years there has been a consensus with respect to the use of representation learning models to characterize the information of this kind of sources (LeCun et al., 2015). However, the way that such extracted features are combined is still in exploration.
Multimodal combination seeks to generate a single representation that makes easier automatic analysis tasks when building classifiers or other predictors. A simple approach is to concatenate features to get a final representation (Kiela & Bottou, 2014; Pei et al., 2013; Suk & Shen, 2013). Although it is a straightforward strategy, it ignores inherent correlations between different modalities.
More complex fusion strategies include Restricted Boltzmann Machines (RBMs) and autoencoders. Ngiam et al. (2011) concatenated higher level representations and train two RBMs to reconstruct the original audio and video representations respectively. Additionally, they trained a model to recon-
struct both modalities given only one of them as input. In an interesting result, Ngiam et al. (2011) were able to mimic a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception known as McGurk effect. A similar approach was proposed by Srivastava & Salakhutdinov (2012). They modified feature learning and reconstruction phases with Deep Boltzmann Machines. Authors claimed that such strategy is able to exploit large amounts of unlabeled data by improving the performance in retrieval and annotation tasks. Other similar strategies propose to fusion modalities using neural network architectures (Andrew et al., 2013; Feng et al., 2013; Kang et al., 2012; Kiros et al., 2014a; Lu et al., 2014; Mao et al., 2014; Tu et al., 2014; Wu et al., 2013) with two input layers separately and including a final supervised layer such as softmax regression classifier.
An alternative approach involves an objective or loss function suited for the target task (Akata et al., 2014; Frome et al., 2013; Kiros et al., 2014b; Mao et al., 2014; Socher et al., 2013; 2014; Zheng et al., 2014). These strategies usually assume that there exists a common latent space where modalities can express the same semantic concept through a set of transformations of the raw data. The semantic embedding representations are such that two concepts are similar if and only if their semantic embeddings are close (Norouzi et al., 2014). In (Socher et al., 2013) a multimodal strategy to perform zero-shot classification was proposed. They trained a word-based neural network model (Huang et al., 2012) to represent textual information, whilst use unsupervised feature learning models proposed in (Coates & Ng, 2011) to get image representation. The fusion was done by learning an image linear mapping to project images into the semantic word space learned in the neural network model. Additionally a Bayesian framework was included to decide whether an image is of a seen or unseen class. Frome et al. (2013) learn the image representation using a CNN trained with the Imagenet dataset and a word-based neural language model (Mikolov et al., 2013b) to represent the textual modality. To perform the fusion they re-train CNN using text representation as targets. This work outperforms scalability with respect to (Socher et al., 2013) from 2 to 20,000 unknown classes in the zero-shot learning task. A modified strategy of Frome et al. (2013) was presented by Norouzi et al. (2014). Instead of re-train the CNN network, they built a convex combination with probabilities estimated by the classifier and semantic embedding vector of the unseen label. This simple strategy outperforms state-of-the-art results. Because the cost function involves both multimodal combination and supervision, these family of models are tied to the task of interest. Thus, if the domain or task conditions changes, adaptations are required.
The proposed model is closely related to the mixture of experts (MoE) approach (Jacobs et al., 1991). However, the common usage of MoE is focused on performing decision fusion, i.e. combining predictors to address a supervised learning problem (Yuksel et al., 2012). Our model is devised as a new component in the representation learning scheme, making it independent from the final task (e.g. classification, regression, unsupervised learning, etc) provided that the defined cost function be differentiable.
2.2 MOVIE GENRE CLASSIFICATION
With respect to movie genre classification, several strategies also have been proposed. These strategies have used different modalities to characterize each movie, such as textual features, image features and multimedia features (audio and/or video). Huang et al. (2007) were one of the first teams exploring this task. They classified movie previews into 3 genres by extracting handcrafted features from the video and training a decision tree classifier. They evaluated the model using 44 films. Using only textual modality, Shah et al. (2013) performed single-label genre classification of movie scripts using clustering algorithms with 260 movies. Later, combining two modalities, Pais et al. (2012) classified movies between drama and non-drama using visual and textual features with 107 samples. Hong & Hwang (2015) explored different PLSA models to combine 3 modalities: audio, image and text to predict genre of movie previews. It was single label classification with 4 genres for 140 movies taken from IMDb.
Recently, Fu et al. (2015) used a set of handcrafted visual features for poster characterization and bag-of-words for synopsis. Then, they trained one SVM per each modality to combine their predictions. The dataset contained 2, 400 movies with one genre (out of 4) each.
The previous mentioned works present this problem in a single label setup. However, a more realistic scenario would be multilabel, since most of the movies belong to more than one genre, (e.g.
Matrix(2000) is a Sci-fi/Action movie). In this setup, Anand (2014) explores the efficiency of using keywords and users’ tags to perform multilabeling using the movies from MovieLens 1M dataset which contains 1, 700 movies. Also Ivasic-Kos et al. (2014; 2015) performed multilabel classification using handcrafted features from posters, with 1, 500 samples for 6 genres. Makita & Lenskiy (2016a;b) use movie ratings matrix and genre correlation matrix to predict the genre. It used a smaller version of the Movielens dataset with 18 movie genres.
Most of the above works have used the publicly available MovieLens datasets. However, there is not a single experimental setup defined so that all methods can be systematically compared. Also, to the best of our knowledge, none of the previous works contain more than 10, 000 samples. With this work we will release a dataset created with the movies of the MovieLens 20M dataset. We include not only genre, poster and plot information used in this work, but also the poster of the movie as well as more than 50 characteristics taken from the IMDb website. We will also release the source code to automatically add more movies and genres.
3 METHODS
This paper presents a neural-network-based strategy for multilabel classification of multimodal data. The key component of the strategy is a novel type of hidden unit, the Gated Multimodal Unit (GMU), which learns to decide how modalities influence the activation of the unit using gates. The details of the GMU are presented in Subsection 3.1.
Statistical properties usually are not shared across modalities (Srivastava & Salakhutdinov, 2012). And thus, they require different representation strategies according to the nature of data. This work explored several strategies to address text and visual representation. For text information we evaluated word2vec models, n-grams models and RNN models. The details are discussed in Subsection 3.2. On the other hand, two different convolutional neural networks were evaluated for processing visual data and are presented in Subsection 3.3.
3.1 GATED MULTIMODAL UNIT FOR MULTIMODAL FUSION
Multimodal learning is closely related to data fusion. Data fusion looks for optimal ways of combining different information sources into an integrated representation that provides more information than the individual sources (Bhatt & Kankanhalli, 2011). This fusion can be performed at different levels, that can be categorized into two broad categories: feature fusion and decision fusion. Feature fusion, also called early fusion, looks for a subset of features from different modalities, or combinations of them, that better represent the information needed to solve a particular problem. On the other hand, decision fusion, or late fusion, combines decisions from different systems, e.g. classifiers, to produce consensus. This consensus may be reached by a simple average, a voting system or a more complex Bayesian framework.
In this work we present a model, based on gated neural networks, for data fusion that combines ideas from both feature and decision fusion. The model, called Gated Multimodal Unit (GMU), is inspired by the flow control in recurrent architectures like GRU or LSTM. A GMU is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. Figure 2.a depicts the structure of a GMU. Each xi corresponds to a feature vector associated with modality i. Each feature vector feeds a neuron with a tanh activation function, which is intended to encode an internal representation feature based on the particular modality. For each input modality, xi, there is a gate neuron (represented by σ nodes in the diagram), which controls the contribution of the feature calculated from xi to the overall output of the unit. When a new sample is fed to the network, a gate neuron associated to modality i receives as input the feature vectors from all the modalities and uses them to decide whether the modality i may contribute, or not, to the internal encoding of the particular input sample.
Figure 2.b shows a simplified version of the GMU for two input modalities, xv (visual modality) and xt (textual modality), that will be used in the remaining of the paper. It should be noted that both models are not completely equivalent, since in the bimodal case the gates are tied. Such weight tying constraints the model, so that the units trade off between both modalities while they use less
parameters than the multimodal case. The equations governing this GMU are as follows:
hv = tanh (Wv · xv) ht = tanh (Wt · xt) z = σ (Wz · [xv, xt]) h = z ∗ hv + (1− z) ∗ ht Θ = {Wv,Wt,Wz}
with Θ the parameters to be learned and [·, ·] the concatenation operator. Since all are differentiable operations, this model can be easily coupled with other neural network architectures and trained with stochastic gradient descent.
3.2 TEXT REPRESENTATION
Text representation is a critical step when classification tasks are addressed using machine learning methods. Traditional approaches are based on counting frequencies of n-gram occurrences such as words or sequences of characters (e.g. bag-of-words models). The main drawback of such approaches is the difficulty to model relationships between words and their context. An alternative approach was initially proposed by Bengio et al. (2003), by building a language model based on a neural network architecture (NNLM). The NNLM was able to learn distributed representations of words that capture contextual information. Later, this model was simplified to deal with large corpora by removing hidden layers in the neural network architecture (word2vec) (Mikolov et al., 2013a). This is a fully unsupervised model that takes advantage of large sets of unlabeled documents. Herein, three text representations were evaluated:
n-gram Following the strategy proposed by Kanaris & Stamatatos (2009), we used the n-gram strategy for representing text. Despite their simplicity, n-gram models have shown to be a competitive baseline.
Word2Vec Word2vec is an unsupervised learning algorithm that finds a vector representation for each word based on its context (Mikolov et al., 2013a). It has been shown that this model is able to find semantic and syntactic relationships using arithmetic operations between the vectors. Based on this property, we represent a movie as the average of the vectors of words in the plot outline. The main motivation to aggregate word2vec vectors is the property of additive compositionality that this representation has exposed over different set of tasks such as word analogies. The usual way to aggregate is to sum vectors. We instead take the average to avoid large input values to the neural network.
Recurrent neural network Here we take the plot outline as a sequence of words and train a supervised recurrent neural network. We evaluated two variants. The first one (RNN w2v) is a transfer learning model that takes as input the word vectors of word2vec as representations. The second one learns the word vectors from scratch (RNN end2end).
3.3 VISUAL REPRESENTATION
In computer vision tasks, Convolutional neural networks have become the de facto standard. It has been shown that CNN models trained with a huge amount of data are able to learn common features shared across different domains. This characteristic is usually exploited by transfer learning approaches. For visual representation we explored 2 strategies: transfer learning and end-to-end training.
VGG Transfer In this approach, the VGG Network (Simonyan & Zisserman, 2014) trained with the ImageNet dataset is used as feature extractor by taking the last hidden activations as the visual representation.
End2End CNN Here, a CNN with 5 convolutional layers and an MLP (see Section 3.4) on top was trained from scratch.
3.4 CLASSIFICATION MODEL
Based on the defined representation, we explored two methods to map from feature vectors to genre classification. In particular we explored a simple Logistic regression and a neural network architecture. This is a multilayer perceptron (MLP) with two fully connected layers and maxout activation function. In particular, the maxout activation function hi : Rn → R is a defined as:
hi (s) = max j∈[1,k] zi,j (1)
where s ∈ Rn is the input vector, zi,j = sTW···ij+bij is the output of the j-th linear transformation of the i-th hidden unit, and W ∈ Rd×m×k and b ∈ Rm×k are learned parameters. It has been shown that maxout models with just 2 hidden units behave as universal approximators, while are less prone to saturate units (Goodfellow et al., 2013).
0 100 200 300 400 500 600 size (pixels)
0
1000
2000
3000
4000
5000
6000
7000
# o
f sa
m p le
s
width height
Figure 4: Size distribution of movie posters. 0 100 200 300 400 500 600 # of words
0
500
1000
1500
2000
2500
3000
# o
f sa
m p le
s
Figure 5: Length distribution of movie plots.
4 EXPERIMENTAL EVALUATION
4.1 MULTIMODAL IMDB DATASET
With this work we will make publicly available the Multimodal IMDb (MM-IMDb)1 dataset. MMIMDb dataset is built with the IMDb id’s provided by the Movielens 20M dataset 2 that contains ratings of 27, 000 movies. Using the IMDbPY 3 library, movies which do not contain their poster image were filtered out. As the final result, the MM-IMDb dataset comprises 25, 959 movies along with their plot, poster, genres and other 50 additional metadata fields such as year, language, writer, director, aspect ratio, etc.
Notice that one movie may belong to more than one genre. Figure 3 shows the co-occurrence matrix, where the color bar indicates the representative co-occurrence per row, while Figure 4 and Figure 5 depict the distribution of the movie poster sizes and length of movie plots respectively. Each plot contains on average 92.5 words, while the longest one contains 1, 431 words and the average of genres per movie is 2.48. In this work, we defined the task of movie genre prediction based on its plot and image poster. Nevertheless, the additional metadata information encourages other interesting tasks such as rating prediction and content-based retrieval, among others.
4.2 EXPERIMENTAL SETUP
The MM-IMDb dataset has been split in three subsets. Train, development and test subsets contain 15552, 2608 and 7799 respectively. The distribution of samples is listed in Table 1. The sample was stratified so that training, dev and test sets comprises 60%, 10%, 30% samples of each genre respectively.
In the multilabel classification the performance evaluation can be more complex than traditional multi-class classification and the differences can be significant among several measures (Madjarov et al., 2012). Herein, four averages of the f-score (f1) are reported: samples computes the f-score per sample and then averages the results, micro computes the f-score using all predictions at once, macro computes the f-score per genre and then averages the results. weighted is the same as macro with a weighted average based on the number of positive samples per genre. Concretely, we calculate them as follows (Madjarov et al., 2012):
fsample1 = 1
N N∑ i=1 2× |ŷi ∩ yi| |ŷi|+ |yi| fmacro1 = 1 Q ∑Q j=1 2×pj×rj pj+rj fweighted1 = 1 Q2 Q∑ j=1 Qj 2× pj × rj pj + rj
1http://lisi1.unal.edu.co/mmimdb/ 2http://grouplens.org/datasets/movielens/ 3http://imdbpy.sourceforge.net/
Table 1: Genre distribution per subset
Genre Train Dev Test Genre Train Dev Test
Drama 8424 1401 4142 Family 978 172 518 Comedy 5108 873 2611 Biography 788 144 411 Romance 3226 548 1590 War 806 128 401 Thriller 3113 512 1567 History 680 118 345 Crime 2293 382 1163 Music 634 100 311 Action 2155 351 1044 Animation 586 105 306 Adventure 1611 278 821 Musical 503 85 253 Horror 1603 275 825 Western 423 72 210 Documentary 1234 219 629 Sport 379 64 191 Mystery 1231 209 617 Short 281 48 142 Sci-Fi 1212 193 586 Film-Noir 202 34 102 Fantasy 1162 186 585
pmicro = ∑Q j=1 tpj∑Q
j=1 tpj + ∑Q j=1 fpj rmicro =
∑Q j=1 tpj∑Q
j=1 tpj+ ∑Q j=1 fnj fmicro1 = 2× pmicro × rmicro pmicro + rmicro
With N the number of examples; Q the number of labels; Qj the number of true instances for the j-th label; p the precision, r the recall; ŷi, yi ∈ (0, 1)Q the prediction and ground truth binary tuples respectively; tpj , fpjandfnj the number of true positives, false positives and false negatives for the j-th label respectively.
TEXTUAL REPRESENTATION
The pretrained Google Word2vec4 embedding space was used. After intersecting the Google word2vec available words with the MM-IMDb plots, the final vocabulary contains 41,612 words. Other than lowercase, no text preprocessing was applied. Since it is our intention to measure how the network’s depth affects the performance of the model, we also evaluate the architecture with a single fully connected layer. In order to compare the performance of this textual representation, we evaluate it using two publicly available datasets: 7genre dataset that comprises 1,400 web pages with 7 disjoint genres and ki-04 dataset that comprises 1,239 samples classified under 8 genres. We compare the model with the state of the art results (Kanaris & Stamatatos, 2009) which used character n-grams with structured information from the HTML tags to predict the genre of web pages.
VISUAL REPRESENTATION
Since the first approach was to use VGG as a feature extractor. This model is referred as VGG Transfer. The second approach takes as input the raw images to a CNN. Since all the images do not have the same size, all images were scaled, and cropped when required, to 160 × 256 pixels keeping the aspect ratio. This CNN comprises 5 CNN layers of 5, 3, 3, 3, 3 squared filters and 2 × 2 pool sizes. Each convolutional layer has 16 hidden units. The convolutional layers are connected with the MaxoutMLP on top.
MULTIMODAL REPRESENTATION
We evaluate 4 different ways to combine both modalities as baselines.
Average probability This can be seen as a late-fusion strategy. The probabilities obtained by the best model of each modality are averaged and thresholded.
concatenation Different works have found that a simple concatenation of representations of different modalities are good for combining the information (Suk & Shen, 2013; Pei et al., 2013; Kiela & Bottou, 2014). Herein, we concatenated both representations to train the MaxoutMLP architecture.
4https://code.google.com/archive/p/word2vec/
linear sum Following the way Vinyals et al. (2015) combine text and images representation into a single space, this model adds a linear transformation for each modality so that both outputs have the same size to be summed up and then followed by the MaxoutMLP architecture.
MoE The mixture of experts (MoE) (Jacobs et al., 1991) model was adapted for multilabel classification. two gating strategies were explored: tied, where a single gate multiplies all the logistics outputs, and untied where every logistic output has its own gate. Logistic regression and MaxoutMLP were evaluated as experts.
NEURAL NETWORK TRAINING
Neural network models were trained using using Batch Normalization scheme (Ioffe & Szegedy, 2015). This strategy applies a normalization step across samples that belong to the same batch, so that each hidden unit in the network receive a zero-mean and unit variance. Stochastic gradient descent with ADAM optimization (Kingma & Ba, 2014) was used to learn the weights of the neural network. Dropout and max-norm regularization were used to control overfitting. Hidden size ({64, 128, 256, 512}), learning rate ( [ 10−3, 10−1 ] ), dropout ([0.3, 0.7]), max-norm ([5, 20]) and
initialization ranges ( [ 10−3, 10−1 ] ) parameters were explored by training 25 models with random (uniform) hyperparameter initializations and the best was chosen according to validation performance. It has been reported that this strategy is preferable over grid search when training deep models (Bergstra & Bengio, 2012). All the implementation was carried on with the Blocks framework (Van Merriënboer et al., 2015)5.
During the training process, we noticed that batch normalization considerably helped in terms of training time and convergence, resulting in less sensitivity to hyperparameters such as initialization ranges or learning rate. Also, dropout and max-norm regularization strategies helped to increase the performance at test time.
5 RESULTS
5.1 EVALUATION OVER SYNTHETIC DATA
In order to evaluate if the model is able to identify which modality is contributing more information to classify a particular sample, we created a synthetic task based on a generative model, which is depicted in Figure 6. In this model we define the random binary variable C as the target and xv, xt ∈ R2 as the input features. M is a random binary variable that decides which modality will contain the relevant information that determines the class. The input features of each modality can
5https://github.com/johnarevalo/gmu-mmimdb
be generated by a random source, ŷv and ŷt, or by an informed source, yv and yt. The generative model is specified as follows:
C ∼ Bernoulli(pC) M ∼ Bernoulli(pM ) yv ∼ N (γCv ) ŷv ∼ N (γ̂v)
xv = Myv + (1−M)ŷv yt ∼ N (γCt ) ŷt ∼ N (γ̂t) xt = Mŷt + (1−M)yt
We trained a model with a single GMU and applied a sigmoid function over h, then the binary cross entropy was used as loss function. Using the generative model, 200 samples per class were generated for each experiment. 1000 synthetic experiments with different random seeds were run and the GMU outperformed a logistic regression classifier in 370 of them, while obtaining equal results in the remainder ones. Our goal in these simulations was to show that the model was able to learn a latent variable that determines which modality carries the useful information for the classification. An interesting result is that between M and the activations of the gate z there is a correlation of 1. This means the model was capable of learning such latent variable by only observing the xv and xt input features.
We also wanted to project back the z activations to the feature space in order to visualize regions depending on the modality. Figure 7 shows the activations in a synthetic experiment generated by the setup of Figure 6 for xv, xt ∈ R1. Each axis represents a modality, red and blue dots are the samples generated for the two classes and black Gaussian curves represent the γ̂v and γ̂t noises. The contour of the left figure (gray) represents the activation of z. Notice that in white regions (z = 1), the model gives more importance to the xv modality while in gray regions (z = 0) the xt modality is more relevant; i.e. the z gate is isolating the noise. The contour of the right figure (blue-red) represents the model prediction. It is noteworthy that the boundary defined by the gates still holds when the model solves the task. This also encourages the inclusion of non-linearities to the z gate so that it is able to discriminate more complex interactions between modalities.
5.2 GENRE CLASSIFICATION RESULTS
Before using our text representation in the multimodal task, we wanted to be sure such representation was good enough to address the genre classification task. Thus, we evaluated it on 2 public datasets. We found MaxoutMLP w2v achieves the state of the art results on the ki-04 dataset and increases the performance in the 7Genre dataset from 0.841 to 0.854 (Kanaris & Stamatatos, 2009). Notice
that the baseline uses additional information from the HTML structure from the web page, while this representation uses only the text data.
Table 2 shows the results in the proposed dataset. For the textual modality, the best performance is obtained by the combination of word2vec representation with an MLP classifier. The behavior of all representation methods are consistent across the performance measures. Learning from scratch the RNN model performed the worst. We hypothesize this has to do with the lack of data to learn meaningful relations among words. It has been shown that millions of words are required to train a model such as word2vec that is able to exploit common regularities between word co-occurrences.
For the visual modality, the usage of pretrained models works better than training the model from scratch. It seems it is still a small dataset to learn all the complexities of the posters. Now, comparing the performance independently per genre, as in Table 3, it is interesting to notice that in Animation the visual modality outperforms the textual one.
In the multimodal scenario, by adding the GMU as building block to learn the fusion we obtained the best performance, improving independent modalities in the averaged measures and in 16 of out 23 genres and outperforming all other evaluated fusion strategies. The concatenation or the linear
combination approaches were not enough to model the correlation between the modalities and MoE models did not perform better than simpler approaches. This is an expected behavior for MoE in a relatively small dataset because the data is fractionated over different experts, and thus it doesn’t make an efficient use of the training samples.
In order to evaluate which modality influences more the model when assigning a particular label, we averaged the activations of a subset of z gates of the test samples to which the model assigned them such label. We counted the number of samples that pays more attention to the textual modality (z <= 0.5) or to the visual modality (z > 0.5). The units were chosen taking into account the mutual information between the predictions and the z activations. The result of this analysis is depicted in Figure 8. As expected, the model is generally more influenced by the textual modality. But, in particular cases such as Animation or Family genres, the visual modality affects more the model. This is also consistent with results of Table 3 which reports better performances for visual modality.
We wanted to qualitative explore test examples in which performance was improved by a relative large margin. Table 4 illustrates cases where the model takes advantage of the most accurate modality, and in some cases removes false positives. It is noteworthy that some of these examples can be confusing for a human if one modality is missing, or additional context information is not given.
6 CONCLUSIONS
This work presented a strategy to learn fusion transformations from multimodal sources. Similarly to the way recurrent models control the information flow, the proposed model is based on multiplicative gates. The Gated Multimodal Unit (GMU) receives two or more input sources and learns to determine how much each input modality affects the unit activation. In synthetic experiments the GMU was able to learn hidden latent variables, and in a real scenario it outperformed the singlemodality approaches. An interesting property of GMU is that, being a differentiable operation, it
is easily coupled in any other neural network architecture and trained with standard gradient-based optimization algorithms. With this work we will also release a new dataset that contains around 27, 000 movie plots, images and other metadata. To the best of our knowledge, this is the biggest dataset used to perform movie genre classification based on multimodal information and the first one to be publicly available. In our future work we expect to explore deep architectures of GMU layers as well as integration with attention mechanism over the input modalities. Also, It will be interesting to explore in more depth the interpretability of the learned features.
ACKNOWLEDGMENTS
Arevalo thanks Colciencias for its support through a doctoral grant in call 617/2013. The authors also thank for K40 Tesla GPU donated by NVIDIA and which was used for some representation learning experiments.

This paper proposed The Gated Multimodal Unit (GMU) model for information fusion. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. The paper collected a large genre dataset from IMDB and showed that GMU gets good performance.
The proposed approach seems quite interesting, and the audience may expect it can be used in general scenarios beyond movie genre prediction. So it is quite straightforward that the paper should test the algorithm in other applications, which was not done yet. That is the biggest shortcoming of this paper in my opinions.
Another concern lies in how to evaluate the performance of information fusion. The abstract claims "The model improves the macro f-score performance of single-modality models by 30% and 4% with respect to visual and textual information respectively", however, such an improvement is off the key. If two modals are complementary to each other, the fusion results will always be higher. The key fact is how much better than baselines the proposed GMU is. There is a long list of techniques for fusions, so it is difficult to conduct an impressive comparison on only one real dataset. I think GMU did a nice work on movie dataset, but I would also expect other techniques, including fine-tuning, dropout, distillation may help too.  It would be nice if the author could compare these techniques.
I also hope this paper could talk in more details the connection with mixture-of-expert (MoE) model. Both models are based on the nonlinear gated functions, while both method may suffer from local minimum for optimization on small datasets. I would like more in-depth discussion in their similarity and difference.
To gain more attention for GMU, I would encourage the author to open-source their code and try more datasets.


Reinforcement learning is concerned with learning to interact with environments that are initially unknown. State-of-the-art reinforcement learning approaches, such as DQN, are model-free and learn to act effectively across a wide range of environments such as Atari games, but require huge amounts of data. Modelbased techniques are more data-efficient, but need to acquire explicit knowledge about the environment dynamics or the reward structure. In this paper we take a step towards using model-based techniques in environments with high-dimensional visual state space when system dynamics and the reward structure are both unknown and need to be learned, by demonstrating that it is possible to learn both jointly. Empirical evaluation on five Atari games demonstrate accurate cumulative reward prediction of up to 200 frames. We consider these positive results as opening up important directions for model-based RL in complex, initially unknown environments.
1 INTRODUCTION
When humans or animals receive reward for taking a particular action in a given situation, the probability is increased that they will act similarly in similar situations in the future. This is described by principles such as the law of effect (Thorndike, 1898), operant conditioning (Skinner, 1938) and trial-and-error learning (Thorpe, 1979) in behaviorist psychology, and has inspired a discipline of artificial intelligence called reinforcement learning (RL, Sutton & Barto (1998)). RL is concerned with finding optimal behavior policies in order to maximize agents’ cumulative future reward.
Approaches to RL can be divided into model-free and model-based approaches. In model-free approaches, agents learn by trial and error but do not aim to explicitly capture the dynamics of the environment or the structure of the reward function underlying the environment. State-of-the-art modelfree approaches, such as DQN (Mnih et al., 2015), effectively approximate so-called Q-values, i.e., the value of taking specific actions in a given state, using deep neural networks. The impressive effectiveness of these approaches comes from their ability to learn complex policies directly from high-dimensional input (e.g., video frames). Despite their effectiveness, model-free approaches require large amounts of training data that have to be collected through direct interactions with the environment, which makes them expensive to apply in settings where interactions are costly (such as most real-world applications). Additionally, model-free RL requires access to reward observations during training, which is problematic in environments with sparse reward structure—unless coupled with an explicit exploration mechanism.
RL approaches that explicitly learn statistics about the environment or the reward are generally referred to as model-based—in a more narrow definition these statistics comprise environment dynamics and the reward function. In recent work, model-based techniques were successfully used to learn statistics about cumulative future reward (Veness et al., 2015) and to improve exploration by favoring actions that are likely to lead to novel states (Bellemare et al., 2016; Oh et al., 2015),
∗Research conducted while interning at Microsoft.
resulting in substantially more data efficient learning compared to model-free approaches. When an accurate model of the true environment dynamics and the true reward function is available, modelbased approaches, such as planning via Monte-Carlo tree search (Browne et al., 2012) outperform model-free state-of-the-art approaches (Guo et al., 2014).
A key open question is whether effective model-based RL is possible in complex settings where the environment dynamics and the reward function are initially unknown, and the agent has to acquire such knowledge through experience. In this paper, we take a step towards addressing this question by extending recent work on video frame prediction (Oh et al., 2015), which has been demonstrated to effectively learn system dynamics, to enable joint prediction of future states and rewards using a single latent representation. We propose a network architecture and training procedure for joint state and reward prediction, and evaluate our approach in the Arcade Learning Environment (ALE, Bellemare et al. (2013)).
Our empirical results on five Atari games demonstrate that our approach can successfully predict cumulative reward up to roughly 200 frames. We complement our quantitative results with a detailed error analysis by visualizing example predictions. Our results are the first to demonstrate the feasibility of using a learned dynamics and reward model for accurate planning. We see this as a significant step towards data efficient RL in high-dimensional environments without prior knowledge.
2 RELATED WORK AND MOTIVATION
Two lines of research are related to the work presented in this paper: model-based RL and optimal control theory. Model-based RL utilizes a given or learned model of some aspect of a task to, e.g., reduce data or exploration requirements (Bellemare et al., 2016; Oh et al., 2015; Veness et al., 2015). Optimal control theory describes mathematical principles for deriving control policies in continuous action spaces that maximize cumulative future reward in scenarios with known system dynamics and known reward structure (Bertsekas, 2007; 2005).
There has been recent interest in combining principles from optimal control theory and model-based learning in settings where no information on system dynamics is available a priori and instead has to be acquired from visual data (Finn et al., 2016; Wahlström et al., 2015; Watter et al., 2015). The general idea behind these approaches is to learn a compressed latent representation of the visual state space from raw images through autoencoder networks (Bengio, 2009) and to utilize the acquired latent representation to infer system dynamics. System dynamics are then used to specify a planning problem which can be solved by optimization techniques to derive optimal policies. Watter et al. (2015) introduce an approach for learning system dynamics from raw visual data by jointly training a variational autoencoder (Kingma & Welling, 2014; Rezende et al., 2014) and a state prediction model that operates in the autoencoder’s compressed latent state representation. A similar approach for jointly learning a compressed state representation and a predictive model is pursued by Wahlström et al. (2015).Finn et al. (2016) devise a sequential approach that first learns a latent state representation from visual data and that subsequently exploits this latent representation to augment a robot’s initial state space describing joint angles and end-effector positions. The augmented state space is then used to improve estimates of local system dynamics for planning.
The approaches presented above assume knowledge of the functional form of the true reward signal and are hence not directly applicable in settings like ALE (and many real-world settings) where the reward function is initially unknown. Planning in such settings therefore necessitates learning both system dynamics and reward function in order to infer optimal behavioral policies. Recent work by Oh et al. (2015) introduced an approach for learning environment dynamics from pixel images and demonstrated that this enabled successful video frame prediction over up to 400 frames. In our current paper, we extend this recent work to enable reward prediction as well by modifying the network’s architecture and training objective accordingly. The modification of the training objective bears a positive side effect: since our network must optimize a compound loss consisting of the video frame reconstruction loss and the reward loss, reward-relevant aspects in the video frames to which the reconstruction loss alone might be insensitive are explicitly captured by the optimization objective. In the subsequent section, we elucidate the approach from Oh et al. (2015) as well as our extensions for reward prediction in more detail.
3 NETWORK ARCHITECTURE AND TRAINING
The deep network proposed by Oh et al. (2015) for video frame prediction in Atari games aims at learning a function that predicts the video frame st+1 at the next time step t + 1, given the current history of frames st−h+1:t with time horizon h and the current action at taken by the agent—see Section 3.1. Here, we extend this work to enable joint video frame and reward prediction such that the network anticipates the current reward rt as well—see Sections 3.2 and 3.3.
3.1 VIDEO FRAME PREDICTION
The video-frame-predictive architecture from Oh et al. (2015) comprises three informationprocessing stages: an encoding stage that maps input frames to some compressed latent representation, a transformation stage that integrates the current action into the compressed latent representation, and a decoding stage that maps the compressed latent representation to the predicted next frame—see Figure 1. The initial encoding stage is a sequence of convolutional and forward operations that map the current frame history st−h+1:t—a three-dimensional tensor—to a compressed feature vector henct . The transformation stage converts this compressed feature vector h enc t into an action-conditional representation hdect in vectorized form by integrating the current action at. The current action at is represented as a one-hot vector with length varying from game to game since there are at least 3 and at most 18 actions in ALE. The integration of the current action into the compressed feature vector includes an element-wise vector multiplication—depicted as ’×’ in Figure 1—with the particularity that the two neuron layers involved in this element-wise multiplication are the only layers in the entire network without bias parameters, see Section 3.2 in Oh et al. (2015). Finally, the decoding stage performs a series of forward and deconvolutional operations (Dosovitskiy et al., 2015; Zeiler et al., 2010) by mapping the action-conditional representation hdect of the current frame history st−h+1:t and the current action at to the predicted video frame st+1 of the next time step t+1. Note that this necessitates a reshape operation at the beginning of the decoding cascade in order to transform the vectorized hidden representation into a three-dimensional tensor. The whole network uses linear and rectified linear units (Glorot et al., 2011) only. In all our experiments, following DQN (Mnih et al., 2015), the video frames processed by the network are 84 × 84 grey-scale images down-sampled from the full-resolution 210× 160 Atari RGB images from ALE. Following Mnih et al. (2015) and Oh et al. (2015), the history frame time horizon h is set to 4.
3.2 REWARD PREDICTION
In this section we detail our proposed network architecture for joint state and reward prediction. Our model assumes ternary rewards which result from reward clipping in line with Mnih et al. (2015). Original game scores in ALE are integers that can vary significantly between different Atari games and the corresponding original rewards are clipped to assume one of three values: −1 for negative rewards, 0 for no reward and 1 for positive rewards. Because of reward clipping, rewards can be represented as vectors rt in one-hot encoding of size 3.
In Figure 1, our extension of the video-frame-predictive architecture from Oh et al. (2015) to enable reward prediction is highlighted in red. We add an additional softmax layer to predict the current reward rt with information contained in the action-conditional encoding hdect . The motivation behind this extension is twofold. First, our extension makes it possible to jointly train the network with a compound objective that emphasizes both video frame reconstruction and reward prediction, and thus encourages the network to not abstract away reward-relevant features to which the reconstruction loss alone might be insensitive. Second, this formulation facilitates the future use of the model for reward prediction through virtual roll-outs in the compressed latent space, without the computational expensive necessity of reconstructing video frames explicitly—note that this requires another ”shortcut” predictive model to map from hdect to h enc t+1.
Following previous work (Oh et al., 2015; Mnih et al., 2015), actions are chosen by the agent on every fourth frame and are repeated on frames that were skipped. Skipped frames and repeated actions are hence not part of the data sets used to train and test the predictive network on, and original reward values are accumulated over four frames before clipping.
3.3 TRAINING
Training the model for joint video frame and reward prediction requires trajectory samples{( s (i) n ,a (i) n , r (i) n )N n=1 }I i=1 collected by some agent playing the Atari game, where i is an index over trajectories and n is a time index over samples within one trajectory i. The parameter I denotes the number of trajectories in the training set or the minibatch respectively and the parameter N denotes the length of an individual trajectory. In our case, we use agents trained according to Mnih et al. (2015) in order to collect trajectory samples.
The original training objective in Oh et al. (2015) consists of a video frame reconstruction loss in terms of a squared loss function aimed at minimizing the quadratic l2-norm of the difference vector between the ground truth image and its action-conditional reconstruction. We extend this training objective to enable joint reward prediction. This results in a compound training loss consisting of the original video frame reconstruction loss and a reward prediction loss given by the cross entropy (Simard et al., 2003) between the ground truth reward and the corresponding prediction:
LK(θ) = 1
2 · I · T ·K I∑ i=1 T−1∑ t=0 K∑ k=1  ∣∣∣∣∣∣s(i)t+k − ŝ(i)t+k∣∣∣∣∣∣2
2︸ ︷︷ ︸ video frame reconstruction loss
+λ · (−1) 3∑
l=1
r (i) t+k[l] · lnp (i) t+k[l]︸ ︷︷ ︸
reward prediction loss  , where ŝ(i)t+k denotes the k-step look ahead frame prediction with target video frame s (i) t+k and p (i) t+k denotes the k-step look ahead probability values of the reward-predicting softmax layer—depicted in red in Figure1—with target reward vector r(i)t+k. The parameter λ > 0 controls the trade-off between video frame reconstruction and reward loss. The parameter T is a time horizon parameter that determines how often a single trajectory sample i is unrolled into the future, and K determines the look ahead prediction horizon dictating how far the network predicts into the future by using its own video frame predicted output as input for the next time step. Following Oh et al. (2015) and Michalski et al. (2014), we apply a curriculum learning (Bengio et al., 2009) scheme by successively increasing K in the course of training such that the network initially learns to predict over a short time horizon and becomes fine-tuned on longer-term predictions as training advances (see Section A.1 for details). The network parameters θ are updated by stochastic gradient descent, derivatives of the training objective w.r.t. θ are computed with backpropagation through time (Werbos, 1988).
4 RESULTS
In our evaluations, we investigate cumulative reward predictions quantitatively and qualitatively on five different Atari games (Q*bert, Seaquest, Freeway, Ms Pacman and Space Invaders). The quantitative analysis comprises evaluating the cumulative reward prediction error—see Section 4.1. The qualitative analysis comprises visualizations of example predictions in Seaquest—see Section 4.2.
4.1 QUANTITATIVE REWARD PREDICTION ANALYSIS: CUMULATIVE REWARD ERROR
Our quantitative evaluation examines whether our joint model of system dynamics and reward function results in a shared latent representation that enables accurate cumulative reward prediction. We assess cumulative reward prediction on test sets consisting of approximately 50,000 video frames per game, including actions and rewards. Each network is evaluated on 1,000 trajectories—suitable to analyze up to 100-step ahead prediction—drawn randomly from the test set. Look ahead prediction is measured in terms of the cumulative reward error which is the difference between ground truth cumulative reward and predicted cumulative reward. For each game, this results in 100 empirical distributions over the cumulative reward error—one distribution for each look ahead step—consisting of 1,000 samples each (one for each trajectory). We compare our model predictions to a baseline model that samples rewards from the marginal reward distribution observed on the test set for each game. Note that negative reward values are absent in the games investigated for this study.
Figure 2 illustrates 20 of the 100 empirical cumulative reward error distributions in all games for our network model in blue and for the baseline model in red (histograms, bottom), together with the median and the 5 to 95 percentiles of the cumulative reward error over look ahead steps (top). Across all games, we observe that our joint state and reward prediction model accurately predicts future cumulative rewards at least 20 look ahead steps, and that it predicts future rewards substantially more accurately than the baseline model. This is evidenced by cumulative reward error distributions that maintain a unimodal form with mode zero and do not flatten out as quickly as the distributions for the random-prediction baseline model. Best results are achieved in Freeway and Q*bert where the probability of zero cumulative reward error at 51 look ahead steps is still around 80% and 60% respectively—see Figure 2. Note that 51 look ahead steps correspond to 204 frames because the underlying DQN agent, collecting trajectory samples for training and testing our model, skipped every fourth frame when choosing an action—see Section 3.2. Lowest performance is obtained in Seaquest where the probability of zero cumulative reward error at 26 steps (104 frames) is around 40% and begins to flatten out soon thereafter—see Figure 2. Running the ALE emulator at a frequency of 60fps, 26 steps correspond to more than 1 second real-time game play because of frame skipping. Since our model is capable of predicting 26 steps ahead in less than 1 second, our model enables real-time planning and could be therefore utilized in an online fashion.
We now turn our attention to error analysis. While the look ahead step at which errors become prominent differs substantially from game to game, we find that overall our model underestimates cumulative reward. This can be seen in the asymmetry towards positive cumulative reward error values when inspecting the 5 to 95 percentile intervals in the first plot per each game in Figure 2. We identify a likely cause in (pseudo-)stochastic transitions inherent in these games. Considering Seaquest as our running example, objects such as divers and submarines can enter the scene randomly from the right and from the left and at the same time have an essential impact on which rewards the agent can potentially collect. In the ground truth trajectories, the agent’s actions are reactions to these objects. If the predicted future trajectory deviates from the ground truth, targeted actions such as shooting will miss their target, leading to underestimating true reward. We analyze this effect in more detail in Section 4.2.
All our experiments were conducted in triplicate with different initial random seeds. Different initial random seeds did not have a significant impact on cumulative reward prediction in all games except Freeway—see Section A.5 for a detailed analysis. So far, we discussed results concerning reward prediction only. In the appendix, we also evaluate the joint performance of reward and video frame prediction on the test set in terms of the optimization objective as in Oh et al. (2015), where the authors report successful video frame reconstruction up to approximately 100 steps (400 frames), and observe similar results—see Section A.6.
4.2 QUALITATIVE REWARD PREDICTION ANALYSIS: EXAMPLE PREDICTIONS IN SEAQUEST
In the previous section, we identified stochasticity in state transitions as a likely cause for relatively low performance in long-term cumulative reward prediction in games such as Seaquest. In Seaquest objects may randomly enter a scene in a non-deterministic fashion. Errors in predicting these events result in predicted possible futures that do not match actually observed future states, resulting in inaccurate reward predictions. Here, we support this hypothesis by visualizations in Seaquest illustrating joint video frame and reward prediction for a single network over 20 steps (80 frames)—see Figure 3 where ground truth video frames are compared to predicted video frames in terms of error maps. Error maps emphasize the difference between ground truth and predicted frames through squared error values between pixels in black or white depending on whether objects are absent or present by mistake in the network’s prediction. Actions, ground truth rewards and model-predicted rewards are shown between state transitions. Peculiarities in the prediction process are shown in red.
In step 2, the model predicts reward by mistake because the agent barely misses its target. Steps 4 to 6 report how the model predicts reward correctly but is off by one time step. Steps 7 to 14 depict problems caused by objects randomly entering the scene from the right which the model cannot predict. Steps 26 to 30 show how the model has problems to predict rewards at steps 26 and 28 as these rewards are attached to objects the model failed to notice entering the scene earlier.
5 CONCLUSION AND FUTURE WORK
In this paper, we extended recent work on video frame prediction (Oh et al., 2015) in Atari games to enable reward prediction. Our approach can be used to jointly predict video frames and cumulative rewards up to a horizon of approximately 200 frames in five different games (Q*bert, Seaquest, Freeway, Ms Pacman and Space Invaders). We achieved best results in Freeway and Q*bert where the probability of zero cumulative reward error after 200 frames is still around 80% and 60% respectively, and worst results in Seaquest where the probability of zero cumulative reward error after 100 frames is around 40%. Our study fits into the general line of research using autoencoder networks to learn a latent representation from visual data (Finn et al., 2016; Goroshin et al., 2015; Gregor et al., 2015; Kulkarni et al., 2015; Srivastava et al., 2015; Wahlström et al., 2015; Watter et al., 2015; Kingma & Welling, 2014; Rezende et al., 2014; Lange et al., 2012; Hinton et al., 2011; Ranzato et al., 2007), and extends this line of research by showing that autoencoder networks are capable of learning a combined representation for system dynamics and the reward function in reinforcement learning settings with high-dimensional visual state spaces—a first step towards applying modelbased techniques for planning in environments where the reward function is not initially known.
Our positive results open up intriguing directions for future work. Our long-term goal is the integration of model-based and model-free approaches for effective interactive learning and planning in complex environments. Directions for achieving this long-standing challenge include the Dyna method (Sutton, 1990), which uses a predictive model to artificially augment expensive training data, and has been shown to lead to substantial reductions in data requirements in tabular RL approaches. Alternatively, the model could be could be utilized for planning via Monte-Carlo tree search (Guo et al., 2014; Browne et al., 2012). We hypothesize that such an approach would be particularly beneficial in multi-task or life-long learning scenarios where the reward function changes but the environment dynamics are stationary. Testing this hypothesis requires a flexible learning framework where the reward function and the artificial environment can be changed by the experimenter in an arbitrary fashion, which is not possible in ALE where the environment and the reward function are fixed per game. A learning environment providing such a flexibility is the recently released Malmö platform for Minecraft (Johnson et al., 2016) where researchers can create user-defined environments and tasks in order to evaluate the performance of artificial agents. In the shorter-term, we envision improving the prediction performance of our network by regularization methods such as dropout and max norm regularization (Srivastava et al., 2014)—a state-of-the-art regularizer in supervised learning—and by modifying the optimization objective to enforce similarity between hidden encodings in multi-step ahead prediction and one-step ahead prediction—see Watter et al. (2015). Finally, extensions of our model to non-deterministic state transitions through dropout and variational autoencoder schemes (Kingma & Welling, 2014; Rezende et al., 2014) is a promising direction to alleviate the limitations highlighted in Section 4.2—paving the way for models that adequately predict and reason over alternative possible future trajectories.
Steps
Error mapPredictionGround truth
Steps Error mapPredictionGround truth
A APPENDIX
A.1 TRAINING DETAILS
We performed all our experiments in Python with Chainer and adhered to the instructions in Oh et al. (2015) as close as possible. Trajectory samples for learning the network parameters were obtained from a previously trained DQN agent according to Mnih et al. (2015). The dataset for training comprised around 500, 000 video frames per game in addition to actions chosen by the DQN agent and rewards collected during game play. Video frames used as network input were 84×84 grey-scale images with pixel values between 0 and 255 down-sampled from the full-resolution 210× 160 ALE RGB images. We applied a further preprocessing step by dividing each pixel by 255 and subtracting mean pixel values from each image leading to final pixel values ∈ [−1; 1]. A detailed network architecture is shown in Figure 1 in the main paper. All weights in the network were initialized according to Glorot & Bengio (2010) except for those two layers that participate in the element-wise multiplication in Figure 1: the weights of the action-processing layer were initialized uniformly in the range [−0.1; 0.1] and the weights of the layer receiving the latent encoding of the input video frames were initialized uniformly in the range [−1; 1]. Training was performed for 1, 500, 000 minibatch iterations with a curriculum learning scheme increasing the look ahead parameter K every 500, 000 iterations from 1 to 3 to 5. When increasing the look ahead parameter K for the first time after 500, 000 iterations, the minibatch size I was also altered from 32 to 8 as was the learning rate for parameter updates from 10−4 to 10−5. Throughout the entire curriculum scheme, the time horizon parameter determining the number of times a single trajectory is unrolled into the future was T = 4. The optimizer for updating weights was Adam (Kingma & Ba, 2015) with gradient momentum 0.9, squared gradient momentum 0.95 and epsilon parameter 10−8. In evaluation mode, network outputs were clipped to [−1; 1] so that strong activations could not accumulate over roll-out time in the network.
In our experiments, we modified the reward prediction loss slightly in order to prevent exploding gradient values by replacing the term − ln p with a first-order Taylor approximation for p-values smaller than e−10—a similar technique is used in DQN (Mnih et al., 2015) to improve the stability of the optimization algorithm. To identify optimal values for the reward weight λ, we performed initial experiments on Ms Pacman without applying the aforementioned curriculum learning scheme instead using a fixed look ahead parameter K = 1. We evaluated the effect of different λ-values ∈ {0.1, 1, 10, 100} on the training objective and identified λ = 1 for conducting further experiments—see Section A.2. After identifying an optimal reward weight, we conducted additional initial experiments without curriculum learning with fixed look ahead parameterK = 1 on all of the five different Atari games used in this paper. We observed periodic oscillations in the reward prediction loss of the training objective in Seaquest, which was fixed by adding gradient clipping (Pascanu et al., 2013) with threshold parameter 1 to our optimization procedure—experiments investigating the effect of gradient clipping in Seaquest are reported in Section A.3. The fine-tuning effect of curriculum learning on the training objective in our final experiments is shown in Section A.4 for all of the five analysed Atari games.
A.2 EFFECT OF REWARD WEIGHT IN MS PACMAN
To identify optimal values for the reward weight λ, we conducted initial experiments in Ms Pacman without curriculum learning and a fixed look ahead horizon K = 1. We tested four different λvalues ∈ {0.1, 1, 10, 100} and investigated how the frame reconstruction loss and the reward loss of the training objective evolve over minibatch iterations—see Figure 4. Best results were obtained for λ = 1 and for λ = 10, whereas values of λ = 0.1 and λ = 100 lead to significantly slower convergence and worse overall training performance respectively.
A.3 EFFECT OF GRADIENT CLIPPING IN SEAQUEST
After identifying an optimal value for the reward weight, see Section A.2, we observed oscillations in the reward loss of the training objective in Seaquest—see first column in Figure 5—which was solved by adding gradient clipping to our optimization procedure—see second and third column in Figure 5. We tested two different values for the gradient clipping threshold (5 and 1) both of which worked, but for a value of 1 the oscillation vanished completely.
A.4 EFFECT OF CURRICULUM LEARNING
In our final experiments with curriculum learning, the networks were trained for 1, 500, 000 minibatch iterations in total but the look ahead parameter K was gradually increased every 500, 000 iterations from 1 to 3 to 5. The networks were hence initially trained on one-step ahead prediction only and later on fine-tuned on further-step ahead prediction. Figure 6 shows how the training objective evolves over iterations. The characteristic ”bumps” in the training objective every 500, 000 iterations as training evolves demonstrate improvements in long-term predictions in all games except Freeway where the training objective assumed already very low values within the first 500, 000 iterations and might have been therefore insensitive to further fine-tuning by curriculum learning.
A.5 EFFECT OF RANDOM SEEDS
We conducted three different experiments per game with different initial random seeds. The effect of different initial random seeds on the cumulative reward error is summarized in Figure 7 which reports how the median and the 5 to 95 percentiles of the cumulative reward error evolve over look ahead steps in the different experiments per game. Note that the results of the first column in Figure 7 are shown in Figure 2 from the main paper together with a more detailed analysis depicting empirical cumulative reward error distributions for some look ahead steps. The random initial seed does not seem to have a significant impact on the cumulative reward prediction except for Freeway where the network in the third experiment starts to considerably overestimate cumulative rewards at around 30 to 40 look ahead steps.
In order to investigate this reward overestimation in Freeway further, we analyse visualizations of joint video frame and reward prediction for this particular seed (similar in style to Figure 3 from Section 4.2 in the main paper). The results are shown in Figure 8 where a peculiar situation occurs after 31 predicted look ahead steps. In Freeway, the agent’s job is to cross a busy road from the bottom to the top without bumping into a car in order to receive reward. If the agent bumps into a car, the agent is propelled downwards further away from the reward-yielding top. This propelled downwards movement happens even when the agent tries to move upwards. Exactly that kind of situation is depicted at the beginning of Figure 8 and occurs for this particular prediction after 31 steps. Our predictive model is however not able to correctly predict the aforementioned downwards movement caused by the agent hitting the car, which is highlighted in red throughout steps 31 to 35 documenting an increasing gap between ground truth and predicted agent position as the propelled downwards movement of the ground truth agent continues. In the course of further prediction, the network model assumes the agent to reach the reward-yielding top side of the road way too early which results in a sequence of erroneous positive reward predictions throughout steps 41 to 50, and as a side effect seemingly that the predictive model loses track of other objects in the scene. Concluding, this finding may serve as a possible explanation for cumulative reward overestimation for that particular experiment in Freeway.
Steps
Steps
Ground truth Prediction Error map Ground truth Prediction Error map
A.6 LOSS ON TEST SET
In the main paper, our analysis focuses on evaluating how well our model serves the purpose of cumulative reward prediction. Here, we evaluate network performance in terms of both the video frame reconstruction loss as well as the reward prediction loss on the test set following the analysis conducted in Oh et al. (2015). For each game, we sample 300 minibatches of size I = 50 from the underlying test set and compute the test loss over K = 100 look ahead steps with the formula presented in the main paper in Section 3.3 used for learning network parameters, but without averaging over look ahead steps because we aim to illustrate the test loss as a function of look ahead steps—statistics of this analysis are plotted in Figure 9.
Best overall test loss is achieved in Freeway and for initial look ahead steps (up to roughly between 40 and 60 steps) in Q*bert, which is in accordance with results for cumulative reward prediction from the main paper. Also in line with results from the main paper is the finding that the reward loss on the test set is worse in Seaquest, Ms Pacman and Space Invaders when compared to Q*bert (up to approximately 40 steps) and Freeway. Worst video frame reconstruction loss is observed for Space Invaders in compliance with Oh et al. (2015) where the authors report that there are objects in the scene moving at a period of 9 time steps which is hard to predict by a network only taking the last 4 frames from the last 4 steps as input for future predictions. At first sight, it might seem a bit surprising that the reward prediction loss in Space Invaders is significantly lower than in Seaquest and Ms Pacman for long-term ahead prediction despite the higher frame reconstruction loss in Space Invaders. A possible explanation for this paradox might be the frequency at which rewards are collected—this frequency is significantly higher in Seaquest and Ms Pacman than in Space Invaders. A reward prediction model with bias towards zero rewards—as indicated by the main results in the paper—might therefore err less often in absolute terms when rewards are collected at a lower frequency and may hence achieve lower overall reward reconstruction loss.
Compound loss Reconstruction loss Reward loss

The paper extends a recently proposed video frame prediction method with reward prediction in order to learn the unknown system dynamics and reward structure of an environment. The method is tested on several Atari games and is able to predict the reward quite well within a range of about 50 steps. The paper is very well written, focussed and is quite clear about its contribution to the literature. The experiments and methods are sound. However, the results are not really surprising given that the system state and the reward are linked deterministically in Atari games. In other words, we can always decode the reward from a network that successfully encodes future system states in its latent representation. The contribution of the paper is therefore minor. The paper would be much stronger if the authors could include experiments on the two future work directions they suggest in the conclusions: augmenting training with artificial samples and adding Monte-Carlo tree search. The suggestions might decrease the number of real-world training samples and increase performance, both of which would be very interesting and impactful.