MULTI-TASK LEARNING WITH DEEP MODEL BASED REINFORCEMENT LEARNING
In recent years, model-free methods that use deep learning have achieved great success in many different reinforcement learning environments. Most successful approaches focus on solving a single task, while multi-task reinforcement learning remains an open problem. In this paper, we present a model based approach to deep reinforcement learning which we use to solve different tasks simultaneously. We show that our approach not only does not degrade but actually benefits from learning multiple tasks. For our model, we also present a new kind of recurrent neural network inspired by residual networks that decouples memory from computation allowing to model complex environments that do not require lots of memory. The code will be released before ICLR 2017.
1 INTRODUCTION
Recently, there has been a lot of success in applying neural networks to reinforcement learning, achieving super-human performance in many ATARI games (Mnih et al. (2015); Mnih et al. (2016)). Most of these algorithms are based on Q-learning, which is a model free approach to reinforcement learning. This approaches learn which actions to perform in each situation, but do not learn an explicit model of the environment. Apart from that, learning to play multiple games simultaneously remains an open problem as these approaches heavily degrade when increasing the number of tasks to learn.
In contrast, we present a model based approach that can learn multiple tasks simultaneously. The idea of learning predictive models has been previously proposed (Schmidhuber (2015); Santana & Hotz (2016)), but all of them focus on learning the predictive models in an unsupervised way. We propose using the reward as a means to learn a representation that captures only that which is important for the game. This also allows us to do the training in a fully supervised way. In the experiments, we show that our approach can surpass human performance simultaneously on three different games. In fact, we show that transfer learning occurs and it benefits from learning multiple tasks simultaneously.
In this paper, we first discuss why Q-learning fails to learn multiple tasks and what are its drawbacks. Then, we present our approach, Predictive Reinforcement Learning, as an alternative to overcome those weaknesses. In order to implement our model, we present a recurrent neural network architecture based on residual nets that is specially well suited for our task. Finally, we discuss our experimental results on several ATARI games.
2 PREVIOUS WORK: DEEP Q-LEARNING
In recent years, approaches that use Deep Q-learning have achieved great success, making an important breakthrough when Mnih et al. (2015) presented a neural network architecture that was able to achieve human performance on many different ATARI games, using just the pixels in the screen as input.
As the name indicates, this approach revolves around the Q-function. Given a state s and an action a, Q(s, a) returns the expected future reward we will get if we perform action a in state s. Formally, the Q-function is defined in equation 1.
Q(s, a) = Es′ [ r + γmax
a′ Q(s′, a′)|s, a
] (1)
For the rest of this subsection, we assume the reader is already familiar with Deep Q-learning and we discuss its main problems. Otherwise, we recommend skipping to the next section directly as none of the ideas discussed here are necessary to understand our model.
As the true value of the Q-function is not known, the idea of Deep Q-learning is iteratively approximating this function using a neural network1 which introduces several problems.
First, the Q-values depend on the strategy the network is playing. Thus, the target output for the network given a state-action pair is not constant, since it changes as the network learns. This means that apart from learning an strategy, the network also needs to remember which strategy it is playing. This is one of the main problems when learning multiple tasks, as the networks needs to remember how it is acting on each of the different tasks. Rusu et al. (2015) and Parisotto et al. (2015) have managed to successfully learn multiple tasks using Q-learning. Both approaches follow a similar idea: an expert network learns to play a single game, while a multi-tasking network learns to copy the behavior of an expert for each different game. This means that the multi-tasking network does not iteratively approximate the Q-function, it just learns to copy the function that the single-task expert has approximated. That is why their approach works, they manage to avoid the problem of simultaneously approximating all the Q-functions, as this is done by each single task expert.
Apart from that, the network has to change the strategy very slightly at each update as drastically changing the strategy would change the Q-values a lot and cause the approximation process to diverge/slow-down. This forces the model to interact many times with the environment in order to find good strategies. This is not problematic in simulated environments like ATARI games where the simulation can easily be speed up using more computing power. Still, in real world environments, like for example robotics, this is not the case and data efficiency can be an important issue.
3 PREDICTIVE REINFORCEMENT LEARNING
In order to avoid the drawbacks of Deep Q-learning, we present Predictive Reinforcement Learning (PRL). In our approach, we separate the understanding of the environment from the strategy. This has the advantage of being able to learn from different strategies simultaneously while also being able to play strategies that are completely different to the ones that it learns from. We will also argue that this approach makes generalization easier. But before we present it, we need to define what we want to solve.
3.1 PREDICTION PROBLEM
The problem we want to solve is the following: given the current state of the environment and the actions we will make in the future, how is our score going to change through time?
To formalize this problem we introduce the following notation:
• ai: The observation of the environment at time i. In the case of ATARI games, this corresponds to the pixels of the screen.
• ri: The total accumulated reward at time i. In the case of ATARI games, this corresponds to the in-game score.
• ci: The control that was performed at time i. In the case of ATARI games, this corresponds to the inputs of the ATARI controller: up, right, shoot, etc.
1We do not explain the process, but Mnih et al. (2015) give a good explanation on how this is done.
Then, we want to solve the following problem: For a given time i and a positive integer k, let the input to our model be an observation ai and a set of future controls ci+1, . . . ci+k. Then, we want to predict the change in score for the next k time steps, i.e. (ri+1 − ri), . . . , (ri+k − ri). Figure 1 illustrates this with an example.
Observe that, unlike in Q-learning, our predictions do not depend on the strategy being played. The outputs only depend on the environment we are trying to predict. So, the output for a given state-actions pair is always the same or, in the case of non-deterministic environments, it comes from the same distribution.
3.2 MODEL
We have defined what we want to solve but we still need to specify how to implement a model that will do it. We will use neural networks for this and we will divide it into three different networks as follows:
• Perception: This network reads a state ai and converts it to a lower dimensional vector h0 that is used by the Prediction.
• Prediction: For each j ∈ {1, . . . , k}, this network reads the vector hj−1 and the corresponding control ci+j and generates a vector hj that will be used in the next steps of the Prediction and Valuation. Observe that this is actually a recurrent neural network.
• Valuation: For each j ∈ {1, . . . , k}, this network reads the current vector hj of the Prediction and predicts the difference in score between the initial time and the current one, i.e, ri+j − ri.
Figure 2 illustrates the model. Observe that what we actually want to solve is a supervised learning problem. Thus, the whole model can be jointly trained with simple backpropagation. We will now proceed to explain each of the components in more detail.
3.2.1 PERCEPTION
The Perception has to be tailored for the kind of observations the environment returns. For now, we will focus only on vision based Perception. As we said before, the idea of this network is to convert the high dimensional input to a low dimensional vector that contains only the necessary information for predicting the score. In the case of video games, it is easy to see that such vector exists. The input will consists of thousands of pixels but all we care about is the position of a few key objects, like for example, the main character or the enemies. This information can easily be
encoded using very few neurons. In our experiments, we convert an input consisting of 28K pixels into a vector of just 100 real values.
In order to do this, we use deep convolutional networks. These networks have recently achieved super-human performance in very complex image recognition tasks (He et al., 2015). In fact, it has been observed that the upper layers in these models learn lower dimensional abstract representations of the input (Yosinski et al. (2015), Karpathy & Li (2015)). Given this, it seems reasonable to believe that if we use any of the successful architectures for vision, our model will be able to learn a useful representation that can be used by the Prediction.
3.2.2 PREDICTION
For the Prediction network, we present a new kind of recurrent network based on residual neural networks (He et al., 2015), which is specially well suited for our task and it achieved better results than an LSTM (Hochreiter & Schmidhuber, 1997) with a similar number of parameters in our initial tests.
Residual Recurrent Neural Network (RRNN) We define the RRNN in Figure 3 using the following notation: LN is the layer normalization function (Ba et al., 2016) which normalizes the activations to have a median of 0 and standard deviation of 1. ”·” is the concatenation of two vectors. f can be any parameterizable and differentiable function, e.g., a multilayer perceptron.
As in residual networks, instead of calculating what the new state of the network should be, we calculate how it should change (ri). As shown by He et al. (2015) this prevents vanishing gradients or optimization difficulties. LN outputs a vector with mean 0 and standard deviation 1. As we
proof2 in Observation 1, this prevents internal exploding values that may arise from repeatedly adding r to h. It also avoids the problem of vanishing gradients in saturating functions like sigmoid or hyperbolic tangent.
Observation 1. Let x ∈ Rn be a vector with median 0 and standard deviation 1. Then, for all 1 ≤ i ≤ n, we get that xi ≤ √ n.
Proof. Taking into account that the median is 0 and the standard deviation is 1, simply substituting the values in the formula for the standard deviation shows the observation.
σ = √√√√ 1 n n∑ j=1 (xj − µ)2 (4)
1 = √√√√ 1 n n∑ j=1 x2j (5)
√ n = √√√√ n∑ j=1 x2j (6) √ n ≥ xi (7)
The idea behind this network is mimicking how a video game’s logic works. A game has some variables (like positions or speeds of different objects) that are slightly modified at each step. Our intuition is that the network can learn a representation of these variables (h), while f learns how they are transformed at each frame. Apart from that, this model decouples memory from computation allowing to increase the complexity of f without having to increase the number of neurons in h. This is specially useful as the number of real valued neurons needed to represent the state of a game is quite small. Still, the function to move from one frame to the next can be quite complex, as it has to model all the interactions between the objects such as collisions, movements, etc.
Even if this method looks like it may be just tailored for video games, it should work equally well for real world environments. After all, physics simulations that model the real world work in the same way, with some variables that represent the current state of the system and some equations that define how that system evolves over time.
3.2.3 VALUATION
The Valuation network reads the h vector at time i + j and outputs the change in reward for that time step, i.e. ri+j − rj . Still, it is a key part of our model as it allows to decouple the representation learned by the Prediction from the reward function. For example, consider a robot in a real world environment. If the Perception learns to capture the physical properties of all surrounding objects (shape, mass, speed, etc.) and the Prediction learns to make a physical simulation of the environment, this model can be used for any possible task in that environment, only the Valuation would need to be changed.
3.3 STRATEGY
As we previously said, finding an optimal strategy is a very hard problem and this part is the most complicated. So, in order to test our model in the experiments, we opted for hard-coding a strategy. There, we generate a set of future controls uniformly at random and then we pick the one that would maximize our reward, given that the probability of dying is low enough. Because of this, the games we have tried have been carefully selected such that they do not need very sophisticated and long-term strategies.
2The bound is not tight but it is sufficient for our purposes and straightforward to prove.
Still, our approach learns a predictive model that is independent of any strategy and this can be beneficial in two ways. First, the model can play a strategy that is completely different to the ones it learns from. Apart from that, learning a predictive model is a very hard task to over-fit. Consider a game with 10 possible control inputs and a training set where we consider the next 25 time steps. Then, there are 1025 possible control sequences. This means that every sequence we train on is unique and this forces the model to generalize. Unfortunately, there is also a downside. Our approach is not able to learn from good strategies because we test our model with many different ones in order to pick the best. Some of these strategies will be quite bad and thus, the model needs to learn what makes the difference between a good and a bad set of moves.
4 EXPERIMENTS
4.1 ENVIRONMENT
Our experiments have been performed on a computer with a GeForce GTX 980 GPU and an Intel Xeon E5-2630 CPU. For the neural network, we have used the Torch7 framework and for the ATARI simulations, we have used Alewrap, which is a Lua wrapper for the Arcade Learning Environment (Bellemare et al., 2015).
4.2 MODEL
For the Perception, we used a network inspired in deep residual networks (He et al., 2015). Figure 4 shows the architecture. The reason for this, is that even if the Perception is relatively shallow, when unfolding the Prediction network over time, the depth of the resulting model is over 50 layers deep.
For the Prediction, we use a Residual Recurrent Neural Network. Table 1 describes the network used for the f function. Finally, Table 2 illustrates the Valuation network.
4.3 SETUP
In our experiments, we have trained on three different ATARI games simultaneously: Breakout, Pong and Demon Attack.
We preprocess the images following the same technique of Mnih et al. (2015). We take the maximum from the last 2 frames to get a single 84 × 84 black and white image for the current observation. The input to the Perception is a 4×84×84 tensor containing the last 4 observations. This is necessary to be able to use a feed-forward network for the Perception. If we observed a single frame,
it would not be possible to infer the speed and direction of a moving object. Not doing this would force us to use a recurrent network on the Perception, making the training of the whole model much slower.
In order to train the Prediction, we unfold the network over time (25 time steps) and treat the model as a feed-forward network with shared weights. This corresponds to approximately 1.7 seconds.
For our Valuation, network we output two values. First, the probability that our score is higher than in the initial time step. Second, we output the probability of dying. This is trained using cross entropy loss.
To train the model, we use an off-line learning approach for simplicity. During training we alternate between two steps. First, generate and store data and then, train the model off-line on that data.
4.4 GENERATING DATA
In order to generate the data, we store tuples (ai, C = {ci+1, . . . ci+25}, R = {ri+1 − ri, . . . ri+25 − ri}) as we are playing the game. That is, for each time i, we store the following:
• ai: A 4× 84× 84 tensor, containing 4 consecutive black and white frames of size 84× 84 each.
• C: For j ∈ {i+ 1, . . . , i+ 25}, each cj is a 3 dimensional vector that encodes the control action performed at time j. The first dimension corresponds to the shoot action, the second to horizontal actions and the third to vertical actions. For example, [1,−1, 0] represent pressing shoot and left.
• R: For j ∈ {i+1, . . . , i+25}, we store a 2 dimensional binary vector rj . rj1 is 1 if we die between time i and j. rj2 is 1 if we have not lost a life and we also earn a point between time i and j.
Initially, we have an untrained model, so at each time step, we pick an action uniformly at random and perform it. For the next iterations, we pick a k and do the following to play the game:
1. Run the Perception network on the last 4 frames to obtain the initial vector.
2. Generate k − 1 sequences of 25 actions uniformly at random. Apart from that, take the best sequence from the previous time step and also consider it. This gives a total of k sequences. Then, for each sequence, run the Prediction and Valuation networks with the vector obtained in Step 1.
3. Finally, pick a sequence of actions as follows. Consider only the moves that have a low enough probability of dying. From those, pick the one that has the highest probability of earning a point. If none has a high enough probability, just pick the one with the lowest probability of dying.
We start with k = 25 and increase it every few iterations up to k = 200. For the full details check Appendix A. In order to accelerate training, we run several games in parallel. This allows to run the Perception, Prediction and Valuation networks together with the ATARI simulation in parallel, which heavily speeds up the generation of data without any drawback.
4.5 TRAINING
In the beginning, we generate 400K training cases for each of the games by playing randomly, which gives us a total of 1.2M training cases. Then, for the subsequent iterations, we generate 200K additional training cases per game (600K in total) and train again on the whole dataset. That is, at first we have 1.2M training cases, afterwards 1.8M , then 2.4M and so on.
The training is done in a supervised way as depicted in Figure 2b. ai and C are given as input to the network and R as target. We minimize the cross-entropy loss using mini-batch gradient descent. For the full details on the learning schedule check Appendix A.
In order to accelerate the process, instead of training a new network in each iteration, we keep training the model from the previous iteration. This has the effect that we would train much more on the initial training cases while the most recent ones would have an ever smaller effect as the training set grows. To avoid this, we assign a weight to each iteration and sample according to these weights during training. Every three iterations, we multiply by three the weights we assign to them. By doing this, we manage to focus on recent training cases, while still preserving the whole training set.
Observe that we never tell our network which game it is playing, but it learns to infer it from the observation ai. Also, at each iteration, we add cases that are generated using a different neural network. So our training set contains instances generated using many different strategies.
4.6 RESULTS
We have trained a model on the three games for a total of 19 iterations, which correspond to 4M time steps per game (74 hours of play at 60 Hz). Each iteration takes around two hours on our hardware. We have also trained an individual model for each game for 4M time steps. In the individual models, we reduced the length of the training such that the number of parameter updates per game is the same as in the multi-task case. Unless some kind of transfer learning occurs, one would expect some degradation in performance in the multi-task model. Figure 5 shows that not only there is no degradation in Pong and Demon Attack, but also that there is a considerable improvement in Breakout. This confirms our initial belief that our approach is specially well suited for multi-task learning.
We have also argued that our model can potentially play a very different strategy from the one it has observed. Table 3 shows that this is actually the case. A model that has learned only from random play is able to play at least 7 times better.
Demon Attack’s plot in Figure 5c shows a potential problem we mentioned earlier which also happens in the other two games to a lesser extent. Once the strategy is good enough, the agent dies very rarely. This causes the model to ”forget” which actions lead to a death and makes the score oscillate.
5 DISCUSSION
We have presented a novel model based approach to deep reinforcement learning. Despite not achieving state of the art results, this papers opens new lines of research showing that a model based approach can work in environments as complex as ATARI. We have also shown that it can beat human performance in three different tasks simultaneously and that it can benefit from learning multiple tasks.
Still, the model has two areas that can be addressed in future work: long-term dependencies and the instability during training. The first, can potentially be solved by combining our approach with Q-learning based techniques. For the instability, balancing the training set or oversampling hard training cases could alleviate the problem.
Finally, we have also presented a new kind of recurrent network which can be very useful for problems were little memory and a lot of computation is needed.
ACKNOWLEDGMENTS
I thank Angelika Steger and Florian Meier for their hardware support in the final experiments and comments on previous versions of the paper.

This paper proposes a model-based reinforcement learning approach focusing on predicting future rewards given a current state and future actions. This is achieved with a "residual recurrent neural network", that outputs the expected reward increase at various time steps in the future. To demonstrate the usefulness of this approach, experiments are conducted on Atari games, with a simple playing strategy that consists in evaluating random sequences of moves and picking the one with highest expected reward (and low enough chance of dying). Interestingly, out of the 3 games tested, one of them exhibits better performance when the agent is trained in a multitask setting (i.e. learning all games simultaneously), hinting that transfer learning is occurring.
This submission is easy enough to read, and the reward prediction architecture looks like an original and sound idea. There are however several points that I believe prevent this work from reaching the ICLR bar, as detailed below.
The first issue is the discrepancy between the algorithm proposed in Section 3 vs its actual implementation in Section 4 (experiments): in Section 3 the output is supposed to be the expected accumulated reward in future time steps (as a single scalar), while in experiments it is instead two numbers, one which is the probability of dying and another one which is the probability of having a higher score without dying. This might work better, but it also means the idea as presented in the main body of the paper is not actually evaluated (and I guess it would not work well, as otherwise why implement it differently?)
In addition, the experimental results are quite limited: only on 3 games that were hand-picked to be easy enough, and no comparison to other RL techniques (DQN & friends). I realize that the main focus of the paper is not about exhibiting state-of-the-art results, since the policy being used is only a simple heuristic to show that the model predictions can ne used to drive decisions. That being said, I think experiments should have tried to demonstrate how to use this model to obtain better reinforcement learning algorithms: there is actually no reinforcement learning done here, since the model is a supervised algorithm, used in a manually-defined hardcoded policy. Another question that could have been addressed (but was not) in the experiments is how good these predictions are (e.g. classification error on dying probability, MSE on future rewards, ...), compared to simpler baselines.
Finally, the paper's "previous work" section is too limited, focusing only on DQN and in particular saying very little on the topic of model-based RL. I think a paper like for instance "Action-Conditional Video Prediction using Deep Networks in Atari Games" should have been an obvious "must cite".
Minor comments:
- Notations are unusual, with "a" denoting a state rather than an action, this is potentially confusing and I see no reason to stray away from standard RL notations
- Using a dot for tensor concatenation is not a great choice either, since the dot usually indicates a dot product
- The r_i in 3.2.2 is a residual that has nothing to do with r_i the reward
- c_i is defined as "The control that was performed at time i", but instead it seems to be the control performed at time i-1
- There is a recurrent confusion between mean and median in 3.2.2
- x should not be used in Observation 1 since the x from Fig. 3 does not go through layer normalization
- The inequality in Observation 1 should be about |x_i|, not x_i
- Observation 1 (with its proof) takes too much space for such a simple result
- In 3.2.3 the first r_j should be r_i
- The probability of dying comes out of nowhere in 3.3, since we do not know yet it will be an output of the model
- "Our approach is not able to learn from good strategies" => did you mean "*only* from good strategies"?
- Please say that in Fig. 4 "fc" means "fully connected"
- It would be nice also to say how the architecture of Fig. 4 differs from the classical DQN architecture from Mnih et al (2015)
- Please clarify r_j2 as per your answer in OpenReview comments
- Table 3 says "After one iteration" but has "PRL Iteration 2" in it, which is confusing
- "Figure 5 shows that not only there is no degradation in Pong and Demon Attack"=> to me it seems to be a bit worse, actually
- "A model that has learned only from random play is able to play at least 7 times better." => not clear where this 7 comes from
- "Demon Attack's plot in Figure 5c shows a potential problem we mentioned earlier" => where was it mentioned?


OPENING THE VOCABULARY OF NEURAL LANGUAGE MODELS WITH CHARACTER-LEVEL WORD REPRESEN- TATIONS
This paper introduces an architecture for an open-vocabulary neural language model. Word representations are computed on-the-fly by a convolution network followed by pooling layer. This allows the model to consider any word, in the context or for the prediction. The training objective is derived from the NoiseContrastive Estimation to circumvent the lack of vocabulary. We test the ability of our model to build representations of unknown words on the MT task of IWSLT2016 from English to Czech, in a reranking setting. Experimental results show promising results, with a gain up to 0.7 BLEU point. They also emphasize the difficulty and instability when training such models with character-based representations for the predicted words.
1 INTRODUCTION
Most of neural language models, such as n-gram models Bengio et al. (2003) are word based and rely on the definition of a finite vocabulary V . As a consequence, a Look-up table is associated to V in which each word w ∈ V is mapped to a vector of dE real valued features stored in a matrix L ∈ R|V|∗dE . While this approach has proven successful for a variety of tasks and languages, see for instance Schwenk (2007) in speech recognition and Le et al. (2012); Devlin et al. (2014); Bahdanau et al. (2014) in machine translation, it induces several limitations.
For morphologically-rich languages, like Czech or German, the lexical coverage is still an important issue, since there is a combinatorial explosion of word forms, most of which are hardly observed on training data. On the one hand, growing the Look-up table is not a solution, since it would increase the number of parameters without having enough training example for a proper estimation. On the other hand, rare words can be replaced by a special token. Nevertheless, this acts as a word class merging very different words without any distinction and using different word classes to handle outof-vocabulary words Allauzen & Gauvain (2005) does not really solve this issue, since rare words are difficult to classify.
Moreover, for most inflected or agglutinative forms, as well as for compound words, the word structure is overlooked, wasting parameters for modeling forms that could be more efficiently handled by word decomposition. While the use of subword units Botha & Blunsom (2014); Sennrich et al. (2016) could improve the generalization power of such models, it relies on a proper and efficient method to induce these subword units.
To overcome these issues, we propose to investigate a word based language model with an open vocabulary. Since most of existing models and training criteria rely on the assumption of a finite vocabulary, the definition of an open vocabulary model, along with a training criterion, constitutes a scientific challenge. Our goal is to build word representations every words. Word representations are inferred on-the-fly from its character sequence, using convolution filters which implicitly capture subword patterns, as described in section 2. The architecture is based on a neural ngram model inspired from Bengio et al. (2003), while this idea can be extended to other kind of models. By relaxing the normalized constraint, the objective function borrows from the noise contrastive estimation Gutmann & Hyvärinen (2012) to allow our model to consider a possibly infinite vocabulary. This paper focusses on this challenge and its related training issues. To assess the efficiency of
this approach, the experimental setup described in section 3 uses a large scale translation task in a reranking setting. The experimental results summarized in section 4 show promising results as well as training issues.
2 MODEL DESCRIPTION
Word embeddings are parameters, stored in a Look-up matrix L. The embedding ewordw of a word w is simply the column of L corresponding to its index in the vocabulary:
ewordw = [L]w
2.1 CHARACTER-LEVEL WORD EMBEDDINGS
To infer a word embedding from its character embeddings, we use a convolution layer Waibel et al. (1990); Collobert et al. (2011), similar to layers used in Santos & Zadrozny (2014); Kim et al. (2015). As illustrated in figure 1, a word w is a character sequence {c1, .., c|w|} represented by their embeddings {Cc1 , ..,Cc|w|}, where Cci denotes the vector associated to the character ci. A convolution filter Wconv ∈ Rde × Rdc∗nc is applied over a sliding window of nc characters, producing local features :
xn = W conv(Ccn−nc+1 : .. : Ccn) T + bconv
where xn is a vector of size de obtained for each position n in the word1. The notation (Ccn−1 : Ccn ) denotes the concatenation of two embeddings. The i-th element of the embedding of w is the mean over the i-th elements of the feature vectors, passed by the activation function φ :
[echar]i = φ |w|−nc+1∑ n=1 [xn]i |w| − nc + 1  (1) Using a mean after a sliding convolution window ensures that the embedding combines local features from the whole word, and that the gradient is redistributed at scale for each character n-gram. The parameters of the layer are the matrices C and Wconv and the bias bconv .
2.2 MODELS
Our model follows the classic n-gram feedforward architecture. The input of the network is a nwords context Hi = (wi−1, . . . , wN−i+1), and its output the probability P (w|Hi) for each word w ∈ V . The embeddings of the word in the context are concatenated and fed into a hidden layer:
hHi = φ(Whidden(ei−1 : . . . : eN−i+1) + b hidden)
A second hidden layer my be added. Finally, the output layer computes scores for each word:
sHi = exp (WouthHi + bout)
Whidden, bhidden, Wout and bout are the parameters of the model. As the input Lookup-matrix L, the output weight matrix Wout contains word embeddings, that are output representations of the words in the vocabulary:
eoutw = [W out]w
Then, the output probabilities are expressed as:
P (w|Hi) = exp eoutw h Hi∑ 1<j<|V| exp eoutj h Hi
Later, we will use three different input layer to obtain word representations: 1Two padding character tokens are used to deal with border effects. The first is added at the beginning and the second at the end of the word, as many times as it is necessary to obtain the same number of windows than the length of the word. Their embeddings are added to C.
• A classic NLM using word-level embeddings only, that we will note WE, which uses |V| ∗ de parameters. • A NLM using embeddings constructed from character n-grams by convolution + pooling,
that we will note CE, which uses |Vc| ∗ dc + dc ∗ nc ∗ de parameters. • A NLM using a concatenation of these two types of embeddings as word representation,
that we will note CWE.
2.3 OBJECTIVE FUNCTION FOR OPEN VOCABULARY MODELS
Usually, such a model is trained by maximizing the log-likelihood. For a given word given its context, the model parameters θ are estimated in order to maximize the following function for all the n-grams observed in the training data:
LL(θ) = ∑
1<i<|D|
logPθ(wi|Hi).
This objective function raises two important issues. For conventional word models, it implies a very costly summation imposed by the softmax activation of the output layer. More importantly, this objective requires the definition of a finite vocabulary, while the proposed model may use characterbased word embeddings, especially at the output, making the notion of vocabulary obsolete.
Therefore, the parameters estimation relies on Noise Contrastive Estimation (NCE) introduced in Gutmann & Hyvärinen (2012); Mnih & Teh (2012). This criterion allows us to train both types of models based on conventional word embeddings, along with character-based embeddings. The NCE objective function aims to discriminate between examples sampled from the real data and from a noise distribution. When presented with examples coming from a mixture of one sample from the data distribution Pd and k from the noise distribution Pn, PH(w ∈ D) denotes the posterior probability of a word w given its context H to be sampled from the training data D. This probability can be expressed as follows:
PH(w ∈ D) = P H d (w)
PHd (w) + kPn(w)
As suggested in Mnih & Teh (2012), Pn only depends on w here, since we chose the unigram distribution estimated on the training data. If
sHθ (w) = exp (e outhH + bout) (2)
denotes the non-normalized score given by the model to a specific word w, as a function of the parameters θ and the context H , the final NCE objective function has the following form Gutmann
& Hyvärinen (2012):
JHθ = EsHθ
[ log
sHθ (w)
sH(w) + kPn(w)
] + kEPn [ log
kPn(w)
sHθ (w) + kPn(w)
] ,
where sHθ will tend to P H d without the need for an explicit normalization.
2.4 CHARACTER-BASED OUTPUT WEIGHTS WITH NOISE-CONTRASTIVE ESTIMATION
The output weights representing each word in the vocabulary eout can also be replaced by embeddings computed by a convolution layer on character n-grams. In this case the model can efficiently represent and infer a score to any word, observed during the training process or not, while with conventional word embeddings, out of vocabulary words only share the same representation and distribution. Instead of using a parameter matrix Wout to estimate the score like in equation 2, the output representation of a word w, eoutw can be replaced by a vector e char−out w estimated on the fly based on its character sequence as described in equation 1, using |Vc| ∗dc +dc ∗nc ∗dh parameters. With this extension the model does not rely on a vocabulary anymore, hence motivating our choice of the NCE. This unnormalized objective allows us to handle an open vocabulary, since we only need to compute k+ 1 word representations for each training examples. Models that use character-based embeddings both for input and output words are denoted by CWE-CWE.
Moreover, with this extension, the representations of words sharing character n-grams are tied. This is an important property to let the model generalize to unseen words. However, it can be also an issue: the limited number of updates for output representations (k+ 1 words) has a “rich get richer” effect: the most frequent words are usually short and will get most of the update. They may therefore ”contaminate” the representation of longer words with which they share character n-grams, even if these words are not related. This issue is further addressed in section 4.1.
3 EXPERIMENTAL SET-UP
The impact of the models described in section 2 is evaluated within the machine translation (MT) shared task of IWSLT-20162 from Englih to Czech. This language pair is highly challenging since Czech is a morphologically-rich language. Neural language models are integrated in a two steps approach: the first step uses a conventional MT system to produce an n-best list (the n most likely translations); in the second step, these hypothesis are re-ranked by adding the score of the neural language model. To better benefit from the open vocabulary models introduced in section 2.1, a more complex system is also used: first an MT system is used to translate from English to a simplified form of Czech which is reinflected. With this pipeline we expect n-best lists with more diversity and also words unseen during the training process. The neural language models are then used to re-rank the reinflected n-best lists.
3.1 DATA
The IWSLT16 MT task is focused on the translation of TED talks. The translation systems are trained on parallel data from the TED, QED and europarl. Our Neural language models are trained on the same data, but training examples are sampled from these corpora given weights that are computed to balance between in-domain parallel data (TED), out-of domain parallel data, and additional monolingual data. Finally, we use the concatenation of TED.dev2010, TED.dev2011 and TED.tst2010 as development set, while TED.tst2012 and TED.tst2013 provide the test set.
3.2 CZECH RE-INFLECTION
In Czech, a morphologically rich language, each lemma can take a lot of possible word forms. Most of them won’t appear - or with a very low frequency - in training data. For an important part of the words found in test data and unseen during training, their lemmas however can be observed but with a different morphological derivation.
2http://workshop2016.iwslt.org
A non-observed word form can’t be generated by the translation system, and one seen too rarely won’t be used in a relevant way. To circumvent this limitation, in a similar fashion as the method described in Marie et al. (2015), each noun, pronoun and adjective is replaced in the training corpora by its lemma along with some morphological features. These word forms are considered in factored way, where some of the POS tags are discarded to reduce the vocabulary. After the translation process, a cascade of Conditional Random Fields (CRF) are used to reintroduce the discarded features, such as gender, number and case, and to generate a new word form.
Formally, the MT system translates English into a simplified version of Czech, that is reinflected. Within this process, the MT system can produce a n-best list, that can be extended to a nk-best list, considering for each translation hypothesis the k-best reinflected sentences given by the factorized CRF. Intuitively, this process can introduce word forms potentially not yet seen in training data, but based on known paradigms, which can give an advantage to language models able to build a word representation from character n-grams.
3.3 BASELINE TRANSLATION SYSTEM
Our baseline is built with a Statistical Machine Translation system based on bilingual n-grams, NCODE3, described in Crego et al. (2011). We follow the same setup as in Marie et al. (2015).
3.4 NLM TRAINING AND OPTIMIZATION
First, some comparative experiments on a smaller dataset are carried out to better understand how open vocabulary NLM behave and to set the hyper-parameters. First trained using stochastic gradient descent, we observed a quite unstable training process, restricting a proper hyper-parameters choices. We found that especially the embedding dimensions, and the activation functions used could make the NCE-objective hard to optimize. This was aggravated in Czech, which we found more difficult to work with than other morphologically complex languages, like German and Russian. The use of Adagrad Duchi et al. (2010) clearly helps to solve most of these issues, but adds consequent computation time. Following preliminary results on our work with a similar model on a different task Labeau et al. (2015), we made the choice of not implementing LSTMs to obtain character-level word representations. It gave similar results, at the cost of unstable training and extended computation time. We then train using batches of 128, for various context sizes, WE, CWE, and CWE-CWE models. The ReLu activation function is used, along with an embedding size of de = 128. When relevant, we used a character embedding size of dc = 32 and a convolution on nc = 5-grams of characters for all experiments4. Concerning the NCE training, we sampled k = 25 examples from the unigram distribution obtained from the training data, for each example sampled from the data. The models were implemented using C++5.
3.5 RERANKING
The re-ranking step uses additional features to find a better translation among the n-best generated by the decoder (in our case, n = 300): we use the score (probability) of WE, CWE and CWECWE models given to each sentence by our models as such a feature. Tuning for re-ranking was performed with KB-MIRA Cherry & Foster (2012), and evaluation using BLEU score.
4 EXPERIMENTAL RESULTS
The first set of experiments investigates the impact of the padding design on the character-level representation followed by a study of the learning behavior of our proposed models and training criterion. Then, the proposed models are evaluated within the MT task. The final set of experiments analyzes the issues of the model based on character-level representation for output words, in order to propose remedies.
3http://ncode.limsi.fr 4Results did not differ significantly when increasing these embedding sizes, with an impact on convergence
speed and computation time. 5Implementation will be made available.
4.1 TIES BETWEEN CHARACTER-LEVEL REPRESENTATION OF OUTPUT WORDS
Preliminary results on smaller dataset are quite poor for models using character-level representation, and far worse when used for the output layer. We suspect that groups of characters are updated far more together, yielding a ”contamination” of several character n-grams by very frequent short words.
Indeed, our simple padding scheme, as shown in the left part of table 1, makes words sharing first or last letter(s) systematically share at least one character n-gram: we suppose it gives the models more chance to detect similarities in word forms sharing prefixes and suffixes.
The representations of any of the character n-grams that are included in the frequent words will thus be re-used in a large part of the other words in the corpus. A huge number of word forms are affected: a little more than one third of the training data shares its first character n-gram with one of the ten most frequent words, and a little more than one quarter shares its last.
While considering varying size of character n-grams when building our word representation, as in Kim et al. (2015), would certainly help, it would increase our computation time. We thus choose to alleviate our padding scheme, as shown on the right part of table 1. We add only one character token at the beginning of the word, and one at the end6. While it may inhibit the capacity of the model to build links between words sharing prefixes or suffixes, it improves results drastically, especially when using character-level outputs, as shown in figure 3. This limited padding scheme is used for the following experiments.
4.2 NLM TRAINING
While the perplexity of our language models is not our main focus, it is still related to the quantity that our training seeks to optimize - since the NCE gradient approaches the maximum likelihood gradient Mnih & Teh (2012). On figure 2 are shown perplexity values of each model during training. These values are based on a vocabulary containing the 250K most frequent words on the training data - it is also the vocabulary used in the model when relevant. They are computed on the development set after each epoch. An epoch includes 2,5M N-grams sampled from the training data. On table 2 are shown the best perplexity obtained on the development set by each model, during training.
Figure 4: Model perplexity measured on the development set during training. The context size is 3 words. Figure 3 shows models based on character-level word representations, with and without complete padding. Models are trained on the same data than Figure 2 but on smaller epochs (250K n-grams).
same vocabulary as for other models, and use the ’unknown’ tokens for words and characters-based representations. Hence, the perplexity computed is difficult to interpret. The main downside of Adagrad is that the learning rate determined by accumulating the history of past gradients is usually too aggressive and stops learning rather early. We simply reset this history every five epochs to give the model a chance to improve, which explains the flattening followed by small improvements we see for WE and CWE models. We choose to do that reset 2 times, based on previous experiments. Despite adaptive gradient, training of CWE-CWE models stays unstable.
4.3 RERANKING
The reranking results are shown in table 3. The first line corresponds to experiments with a direct translation from English to Czech, where n-best lists generated by the MT system are simply rescored by our models. The best result is given by the longest-context CWE model, which produces a +0.7 BLEU score improvement. CWE models gives on average +0.1 BLEU point compared to WE models, while CWE-CWE are−0.2 BLEU point under. Doubling the context size consistently improves results of +0.2 BLEU point.
Experimental results on reinflected Czech seems to follow a similar trend: CWE models behave a little better than WE models, while CWE-CWE models are under. While simply reranking n-best lists is not as efficient as doing it directly in Czech, reranking nk-best lists extended by the factorized CRF gives a small improvement, reaching an improvement of +0.7 BLEU point. As a general rule, small context models seem to have difficulties with reinflected Czech. The main advantage given by the CWE model is an ability to better rerank nk-best lists. These results suggest that, while the normalization + reinflection procedure may introduce diversity in the output to be reranked, our models are not able to draw any significant advantage from it.
4.4 ANALYSIS OF CHARACTER-LEVEL OUTPUT REPRESENTATIONS PERFORMANCE
Models using character-level output representations gave sub-par results on re-ranking. It is surprising, especially for re-inflected Czech: such a model is supposed to behave better on unknown words, and thus should benefit from diversity given by generating new words. However, as we can see in table 4, re-inflection doesn’t add that much diversity (About 0.1 % of OOV words, and about 0.001 % of words never seen by the model before). Diversity is also inhibited by our training algorithm: while we train open-vocabulary models, the negative examples used with Noise-contrastive estimation come from a closed vocabulary.
This can related to the nature of the unigram distribution used to sample negative examples. As explained in section 4.1, it makes frequent short words completely outweigh the others in number of updates, and we are forced to reduce the ability of the model to find common morphological attributes between words to avoid ’contamination’ of character n-gram representations.
5 RELATED WORKS
There is a number of different strategies to efficiently train NNLMs with large vocabularies, such as different types of hierarchical softmax Mnih & Hinton (2009); Le et al. (2011), importance sampling Bengio & Sénécal (2003), and Noise contrastive estimation Gutmann & Hyvärinen (2012); Mnih & Teh (2012). Vaswani et al. (2013) has showed the interest of training a NLM with NCE to re-rank k-best lists, while Devlin et al. (2014) uses a self-normalization. Recently, a comparative study Chen et al. (2016) has been made on how to deal with a large vocabulary. However, the purpose of this paper is to explore models with open vocabulary rather large vocabulary.
There is a surge of interest into using character-level information for a wide range of NLP tasks, with improved results in POS Tagging Santos & Zadrozny (2014), Text classification Zhang & LeCun (2015), Parsing Ballesteros et al. (2015), Named entity recognition Lample et al. (2016).
In language modeling, first applications to language modeling were strictly using characters, and performed less than word-level models Mikolov et al. (2012), while showing impressive results for text generation Sutskever et al. (2011); Graves (2013), using bi-directional LSTM Graves et al. (2013). Recently, Ling et al. (2015) has used bi-directional LSTM to build word representations from characters, with improvements in language modeling and POS-tagging.
The recent work of Kim et al. (2015), that uses convolutional networks and pooling to construct a word representation from character n-grams, coupled with highway networks Srivastava et al. (2015), showed on various languages that using characters improves results on the language modeling task (for a small corpus), even more so for languages with complex morphology. A similar architecture was used Józefowicz et al. (2016) on a larger dataset, conjointly with bi-directional LSTMs, and trained with importance sampling, showing great results.
On the study of NNLMs in the context of Machine Translation, we can mention the work of Luong et al. (2015) on the effect of the number of layers on reranking n-best lists. Finally, while not directly related to our work, Luong & Manning (2016) very recently showed great improvements on a translation task by handling rare words with character-level recurrent networks, with a neural translation model.
6 CONCLUSION
In this work, we addressed the challenge of designing an open vocabulary Neural Language Model. For that purpose, word representations are estimated on-the-fly from n-grams of characters. Two kinds of models are introduced: first, NLMs using word and character-level embeddings to represent the input context (CWE); then its extension to an open-vocabulary even for the predicted words (CWE-CWE). These models were used to re-rank outputs of translation systems from English to Czech. We also carried out experiments on translation systems from English to a simplified Czech, which is then re-inflected into Czech before re-ranking.
We obtained a slight improvement in BLEU score using a CWE model, which, given the little variety of the words generated by translation systems, makes us suppose there is room for more. We plan to investigate with more complex translation systems, as well as with other applications, such as morphological re-inflection.
While the performance of our open-vocabulary models are to some extent disappointing, they open questions about the learned representations we will explore. We also plan to investigate on a more fitted noise distribution to use with NCE when training open-vocabulary models.
ACKNOWLEDGMENTS

In this submission, an interesting approach to character-based language modeling is pursued that retains word-level representations both in the context, and optionally also in the output. However, the approach is not new, cf. (Kim et al. 2015) as cited in the submission, as well as (Jozefowicz et al. 2016). Both Kim and Jozefowicz already go beyond this submission by applying the approach using RNNs/LSTMs. Also, Jozefowicz et al. provide a comparative discussion of different approaches to character-level modeling, which I am missing here, at least by discussing this existing work. THe remaining novelty of the approach then would be its application to machine translation, although it remains somewhat unclear, inhowfar reranking of N-best lists can handle the OOV problem - the translation-related part of the OVV problem should be elaborated here. That said, some of the claims of this submission seems somewhat exaggerated, like the statement in Sec. 2.3: "making the notion of vocabulary obsolete", whereas the authors e.g. express doubts concerning the interpretation of perplexity w/o an explicit output vocabulary. For example modeling of especially frequent word forms still can be expected to contribute, as shown in e.g. arXiv:1609.08144
Sec. 2.3: You claim that the objective requires a finite vocabulary. This statement only is correct if the units considered are limited to full word forms. However, using subwords and even individual characters, implicitly larger and even infinite vocabularies can be covered with the log-likelihood criterion. Even though this require a model different from the one proposed here, the corresponding statement should qualified in this respect.
The way character embeddings are used for the output should be clarified. The description in Sec. 2.4 is not explicit enough in my view.
Concerning the configuration of NCE, it would be desirable to get a better idea of how you arrived at your specific configuration and parameterization described in Sec. 3.4.
Sec. 4.1: you might want to mention that (Kim et al. 2015) came to similar conclusions w.r.t. the performance of using character embeddings at the output, and discuss the suggestions for possible improvements given therein.
Sec. 4.2: there are ways to calculate and interpret perplexity for unknown words, cf. (Shaik et al. IWSLT 2013).
Sec. 4.4 and Table 4: the size of the full training vocabulary should be provided here.
Minor comments:
p. 2, bottom: three different input layer -> three different input layers (plural)
Fig. 1: fonts within the figure are way too small
p. 3, first item below Fig. 1: that we will note WE -> that we will denote WE
Sec. 2.3: the parameters estimation -> the parameter estimation (or: the parameters' estimation)
p. 5, first paragraph: in factored way -> in a factored way
p. 5, second paragraph: a n-best list, a nk-best list -> an n-best list, an nk-best list
Sec. 4.2, last sentence: Despite adaptive gradient, -> verb and article missing


ADVERSARIAL IMAGINATION PRIORS
Given an image, humans effortlessly run the image formation process backwards in their minds: they can tell albedo from shading, foreground from background, and imagine the occluded parts of the scene behind foreground objects. In this work, we propose a weakly supervised inversion machine trained to generate similar imaginations that when rendered using differentiable, graphics-like decoders, produce the original visual input. We constrain the imagination spaces by providing exemplar memory repositories in the form of foreground segmented objects, albedo, shading, background scenes and imposing adversarial losses on the imagination spaces. Our model learns to perform such inversion with weak supervision, without ever having seen paired annotated data, that is, without having seen the image paired with the corresponding ground-truth imaginations. We demonstrate our method by applying it to three Computer Vision tasks: image in-painting, intrinsic decomposition and object segmentation, each task having its own differentiable renderer. Data driven adversarial imagination priors effectively guide inversion, minimize the need for hand designed priors of smoothness or good continuation, or the need for paired annotated data. Consider Figure 1. We imagine a missing triangle occluding three small black circles rather than three carefully arranged pacman shapes – which is what the pixels depict. In (b), we do not perceive two parts of the sea separated by a standing person, rather a continuous sea landscape. In (c), we explain the input as a ”masked 8” rather than two semicircles. Consistent explanations of visual observations in terms of familiar concepts and memories we call “imaginations”. Imaginations invert the image formation process and propose 3D shape, camera pose, scene layering, spatial layout, albedo, shading, inpainted, un-occluded perceptions of the world, necessary for the understanding of the visual scene and interaction with it. Gestalt philosophers (Smith (1988)) proposed a set or principles to explain formation of such percepts, such as, closure, center surround pop-out, good continuity, smoothness etc, which many works attempt to hand design principles to incorporate those into computational frameworks of e.g., perceptual grouping (Yu (2003)). In this work, we present a learning-based inversion model that uses data-driven priors instead. We propose a computational model that addresses inverse problems in Computer Vision using adversarial imagination priors. Figure 2 illustrates our model. It is comprised of a generator neural network that given a visual input predicts visual imaginations, such as, in-painted image, un-occluded background scene, object segmentation, albedo and shading etc. Relevant memories, assumed to +
Consider Figure 1. We imagine a missing triangle occluding three small black circles rather than three carefully arranged pacman shapes – which is what the pixels depict. In (b), we do not perceive two parts of the sea separated by a standing person, rather a continuous sea landscape. In (c), we explain the input as a ”masked 8” rather than two semicircles. Consistent explanations of visual observations in terms of familiar concepts and memories we call “imaginations”. Imaginations invert the image formation process and propose 3D shape, camera pose, scene layering, spatial layout, albedo, shading, inpainted, un-occluded perceptions of the world, necessary for the understanding of the visual scene and interaction with it. Gestalt philosophers (Smith (1988)) proposed a set or principles to explain formation of such percepts, such as, closure, center surround pop-out, good continuity, smoothness etc, which many works attempt to hand design principles to incorporate those into computational frameworks of e.g., perceptual grouping (Yu (2003)). In this work, we present a learning-based inversion model that uses data-driven priors instead.
We propose a computational model that addresses inverse problems in Computer Vision using adversarial imagination priors. Figure 2 illustrates our model. It is comprised of a generator neural network that given a visual input predicts visual imaginations, such as, in-painted image, un-occluded background scene, object segmentation, albedo and shading etc. Relevant memories, assumed to
+
How much is it?
Under review as a conference paper at ICLR 2017
Distribution Matching
have been acquired from past experience, are retrieved based on coarse attribute matching. Fullyconvolutional discriminator networks match statistics of the generated imaginations with retrieved relevant memories. A non-parametric graphics-like differentiable renderer projects such imaginations accordingly and reconstructs the original image. Our model is trained using a combination of adversarial and reconstruction losses.
Architectures we explore ensure the original images can be reconstructed from the inferred imaginations using basic, parameter-free differentiable renderers. This particular choice of decoder function further enforces the imagination spaces to take the particular desired forms, along with the adversarial priors. We are inspired by work of capsules (Tieleman (2014)) that first introduced such domain specific, graphics-like decoders for image generation. We empirically validate the choice of such decoders against standard parametric deconvolutional networks employed by previous works, e.g., inverse graphics network of Kulkarni et al. (2015b).
Our model can infers visual imaginations without having seen paired annotations, that is, each input image paired with the corresponding ground-truth. Instead, repositories of relevant memories suffice, in the form of collections of albedo, shading, segmented objects, complete background scenes etc. This distinguishes it from previous works that rely on supervision for decomposing an image into imaginations (Kulkarni et al. (2015b)) or that train image conditioned generator networks using a combination of adversarial and L2 reconstruction loss on imaginations -such as works of Pathak et al. (2016) for in-painting, work of Jiajun Wu (2016) for 3D object reconstruction, work of Dong et al. (2015) for super-resolution. In these works, L2 loss is used to condition the imagination on the input image, e.g., in Jiajun Wu (2016), the adversarial loss ensures the generated voxel grid looks like a 3D object and the L2 loss ensures it is the 3D object corresponding to that particular input image instead of an arbitrary one. Instead, we employ a different method of conditioning: we add a reconstruction loss after our graphics-like decoder, when imaginations are projected (rendered back) to image pixels, reconstructing the original image; our model is like an autoencoder in that sense. In this way, we do not need paired supervision, we can take advantage of unlabelled data and we do not discriminate between training and test phases: adversarial priors useful for inversion can be employed at any time, and the relevant memory repositories may be updated. However, any available annotated pairs can always be used to pretrain the generator network. We did consider such small amount of pretraining in one of the considered tasks and this in this paper to emphasize the power of adversarial priors.
Our model enables feedback from the input image directly to its memory priors: the relevant memory engine retrieves memories based on matching of attribute/ feature descriptors. In this way, priors are tailored to the visual input which alleviates the rare sample problem of traditional training methods, which suffer from the imbalance of training data samples: some examples are way more typical than others, and thus more represented on the neural network weights. Hard negative mining has been used to fight such skewness of training distributions. We empirically show that fixing the prior
distribution instead of adapting it results in undesirable, wrong imaginations, and makes the balance of adversarial and reconstruction losses dependent on example by example basis.
In our model, distribution matching of predicted imaginations and retrieved memories concerns local image statistics. In contrast to most previous use cases of adversarial networks, our work (1) conditions imaginations to the input image -does not generate from random noise- and (2) has a feedback loop by projecting imaginations back to the image through rendering and L2 reconstruction loss. Both (1) and (2) constrain the imagination space and thus our adversarial distribution matching cares mostly about local statistics, rather than global structure, texture matching rather than semantic content. For example, for grass in-painting, we do not care whether the imagination looks similar or exactly the same as a retrieve grass image, we only want to make sure each part in the imagination follows a grass-like texture. We propose fully-convolutional discriminator networks, that predict real versus fake binary tests densely across the feature grid, rather than once for the whole image. This accelerates training, makes our model robust to the size of the network input size and across different field-of-views.
In summary, our contributions are as follows:
• A weakly supervised model for inverse problems given visual input based on adversarial imagination priors and graphics-like decoders.
• Relevant memory retrieval for informative adversarial priors. • Fully-convolutional discriminator networks for matching local image statistic distributions
robust to network input size and image field-of-view.
We demonstrate our model in the tasks of image in-painting, figure-ground layer extraction and intrinsic image decomposition. We show successful imagination prediction without using paired ground-truth annotations. We are working towards updating the draft with inverse problems in videos to convey the generality of the proposed model.
1 RELATED WORK
Vision as an inference problem Both Computer and Human Vision fields have worked towards models that given visual observations attempt to infer hidden properties about the visual scene by inverting the image formation process, ”un”-doing camera projection, occlusions, motion blur, downsampling, image masking. Examples are inferring 3D shape and camera pose in videos or images in Tomasi & Kanade (1992), decoupling 3D shape, lighting and albedo interactions in Barron & Malik (2013); Kong et al. (2014), inferring scene depth segmentation layering in Yang et al. (2012), super-resolving low resolution input in Yang et al. (2010), filling in pixels in masked (”hole”) images (in-painting) Efros & Leung (1999) etc.
Multimodality Inverse problems are ill-posed: There are many imagination solutions whose projection or rendering would result in the same visual image. Multi-modality of the desired hidden representation causes methods that rely on maximum likelihood to suffer from regression-to-themean problem. Despite this fact, direct feed-forward neural networks regressors or classifiers have been trained in a supervised way to achieve such inversion, e.g., depth estimation in Eigen et al. (2014), albedo estimation in Narihira et al. (2015b), volumetric inference in Firman et al. (2016), super-resolution in Dong et al. (2015) etc. The argument is that with large enough receptive fields, ambiguities of inversion are diminished. However, such approaches require human supervision, may not generalize well enough to handle different inputs, adapt to the example at hand effectively, or achieve global consistency of the solution (Narihira et al. (2015a)).
Generative adversarial networks (Goodfellow et al. (2014); Sebastian Nowozin (2016); Radford et al. (2015)) instead have shown to minimize Jensen-Shannon divergence between the matched distributions and exhibit a mode seeking behaviour (Theis et al. (2016)) desirable for inversion. Our adversarial priors can be thought as a surrogate to true perceptual losses, which would involve humans in the loop and would be very expensive to obtain in practise.
Concurrent work of Sønderby et al. (2016) proposes a model for super-resolution that does not required paired supervision, similar to our model. They have an adversarial loss in the high resolution
image and their decoder is a downsampler. Their model is a special case of our model, in which we consider general graphics-like decoders, tailored to each task. Further, they do not consider relevant memory retrieval and do not consider fully-convolutional decoders.
Priors Other research approaches on inverse problems do not employ learning but rather rely on hand designed priors, such as sparsity in Yang et al. (2010), spatial smoothness (for optical flow, depth, albedo etc), temporal smoothness (for shading in Kong et al. (2014)), low-rank 3D shape or trajectory priors in Akhter et al. (2008); Wu et al. (2016), deformable 3D scene models in Kulkarni et al. (2015a). Such hand designed priors, though do not suffer from generalization issues, cannot exploit data available effectively.
Our work proposes data driven priors implemented through adversarial distribution matching between inferred imaginations and retrieval memories. Such priors exploit unlabelled data available in the form of imagination repositories, do not suffer from training and test discrepancies, do not need paired supervision and alleviate the engineering burden of designing good prior models.
Feedback Feedback is visual processing has been incorporated in recent computational models through iterative processing, where each step produces a better estimate of the relevant memory, let is be image reconstruction Raiko et al. (2014), body pose estimation Carreira et al. (2015) etc. Such feedback is incorporated in our adversarial prior model through a memory retrieval mechanism which uses coarse feature extraction and attributes on unoccluded parts of the visual input to retrieve relevant examples, and thus influence the reconstruction in an example by example case, alleviating the problem of data imbalance and finetuning, catastrophic forgetting, hard negative mining of traditional training paradigms.
Domain specific non-parametric decoders Model architectures we explore are based on the fact that the inferred imaginations are such that the original image can be reconstructed using basic, parameter-free operations, such as, camera projection, that project inferred 3D and camera pose to 2D scene Handa et al. (2016), pointwise multiplication for image decomposition, layering that assembles different imaginations based on their depth and segmentation masks. Our work is inspired by work of Tieleman (2014) which proposes capsules, a model for image generation by assembling 2D image pieces and their poses predicted from the encoder into one canvas.
2 MODEL
Our model is illustrated in Figure 2. Given a set of images X = {xi, x2, · · · , xn}, and a memory database M , a generator network inverts each image x into a set of imaginations z1,z2, · · ·zK , which, (1) when rendered back to pixels, the projection should match the corresponding input image; and (2) the imagination statistics should match the distribution of relevant memories retrieved from M through a memory retrieval engine. Our model is trained to minimize the combination of (1) an image reconstruction loss and (2) an adversarial imagination loss that constraints the imagination space(s). The imagination spaces and renderer architecture depend on the inversion task. We consider thress tasks in this work: 1) image in-painting, 2) intrinsic image decomposition, 3) figure-ground layer extraction.
We denote the generator as a mapping function from input to imaginations G(x), the renderer as a mapping function from the imagination to the input image P (z), the image retrieval engine as a mapping function from memories M and input image x to relevant memories R(M,x), and the discriminator for imposing distribution matching between imaginations and retrieved memories as D. In case of multiple imagination spaces (e.g., shading and albedo, in-painted background and foreground object mask etc.) we will use Gi(x) to denote the i-th imagination proposed by the generator. Suppose there are K imagination spaces, the memory retrieval engine will need to retrieved K relevant memory that corresponds to each of the imaginations. Besides, we also need K discriminators to look after each generated imagination. Here we use Ri(M,x) and Di to denote
the corresponding retrieved relevant memory and discriminator for the i-th imagination space. Our loss reads as follows:
min D max G Ex∈X ||P (G(x))− x||2︸ ︷︷ ︸ reconstruction loss +β n∑ i=1 logDi(Ri(M,x)) + log(1−Di(Gi(x)))︸ ︷︷ ︸ adversarial loss , (1)
where β the relative weight of reconstruction and adversarial losses.
2.1 IMAGINATION GENERATOR G
Given an image, the generator outputs one or more imaginations. In the tasks we consider, imaginations have a retinotopic representation, that is, they have the same size as the input image. Our generators are convolutional/deconvolutional neural networks with skip-layer connections from the encoding to the decoding layers. Skip-layer connections much improve the precision of the produced imaginations. We share weights of the first convolutional layers across multiple imagination spaces. Figure 3 shows the generator architectures we used for the different inversion tasks.
2.2 IMAGINATION RENDERER P
Below we present our domain-specific renderers for three Computer Vision tasks: image in-painting, intrinsic image decomposition and figure-ground layer extraction.
Image in-painting The input is a masked image x, an image whose content is covered by a black contiguent mask m. The task is to invert such masking and produce an imagination that corresponds to the complete (in-painted) image before the masking operation, as shown in Figure 3 (a). The rendering function P in this case is defined as P (z) = m z,where denotes pointwise multiplication. Intrinsic image decomposition Given an image x, the generator generates albedo z1 and shading z2, as shown in Figure 3 (b). For Lambertian surfaces that the product of albedo and shading should recover the original image, we thus define our renderer to be: P (z1,z2) =z1 z2. Note that we need two discriminator networks, one that controls the statistics distribution of generated albedos and one that controls the statistics distribution of generated shading imaginations. In practise, instead of pointwise multiplication, we used addition in the log space.
Figure-ground layer extraction In this task, given an image, we want to invert the layering superimposition caused by the objects against their background and produce imaginations of the segmented objects and in-painted background scene. Given an image x, the generator outputs a foreground segmentation mask zm, corresponding image foreground z1 = x zm and an in-painted background z2 such that the in-painted background matches the relevant background memories and the image foreground matches memories of segmented relevant objects with clean (black) background, as shown in Figure 3 (c). Our renderer in this case is defined to be: P (z1,z2,zm) = (1−zm) z2+z1 x, that is, it overlays the object on the in-painted background.
2.3 FULLY-CONVOLUTIONAL DISCRIMINATOR D
We propose fully-convolutional discriminator architectures for matching local image statistics between inferred imaginations and retrieved relevant memories. Fully-convolutional discriminators employ many -instead of one- classifiers centered at grid points of the feature maps, that calculate the confidence scores of being real of fake pattern for each of the local receptive fields, in different layers of the network. Fully-convolutional discriminators allow better generalization from relevant memories to generated imaginations as they match only local statistics and not global patterns. Further, they are much faster and more stable to train as the number of examples fed into the discriminator increases. In our experimental section, we show empirically that fully-convolutional adversarial loss accelerates and stabilizes training.
2.4 MEMORY RETRIEVAL ENGINE R
Given an input image and a memory database of the same imagination types we want to generate, e.g., albedos of natural images, the memory retrieval engine retrieves the most relevant memories. The details of memory retrieval depend on the inversion task.
Image in-painting In this case, we measure the L2 pixel distance between the visible part of the input image and images in our memory database, and retrieve the top nearest neighbors.
Intrinsic image decomposition We retrieve relevant shadings by L2 pixel matching between the grayscale version of the input image and albedo memories. We retrieve relevant albedos by computing pixel matching between the input image and albedo memories.
Figure-ground layer extraction Our foreground object memories are segmented objects, as shown in Figure 3 (c) z1. We retrieve relevant segmented objects according to an object detector output, that makes sure our segmented object imaginations agree on the object category with retrieved memories (here the object category of interest is “chair”). For the in-painted background memories, we use L2 pixel distance between the current image and images from the SUN scene dataset.
In any of the aforementioned inversion tasks, after the model starts to generate reasonable initial imaginations, we can use those to retrieve more relevant memories. Such iterative feedback between memory and visual processing though very reasonable to do, we did not consider it in this work to keep the framework simple.
3 EXPERIMENTS
We show results of our model for (1) image in-painting, (2) intrinsic image decomposition and (3) figure-ground layer extraction. The corresponding model architectures are shown in Figure 3 and further training details are provided in the Appendix.
3.1 IMAGE IN-PAINTING
We used the MNIST dataset and masked parts of its digit images. Specifically, we randomly selected 2500 samples of digits 0, 1, 2, 3 from the dataset and overlayed a squared mask at the center over them to create our input images. Our memory databaseM contains 1000 samples for each of the ten digits. We purposefully designed such distribution mismatch between the input image dataset and memory database to study the usefulness of retrieved memories under a controlled setup. The set of digit images contained in M does not intersect the set of images we used to create our input images. In other words, the groundtruth imaginations for our input images are not contained in our memory database.
Figure 4 shows the results of four in-painting models: (1) a baseline with L2 pixel loss between imaginations and retrieved relevant memories (BmemL2), (2) our model with memories retrieved uniformly at random from M rather than conditioned on input images (we supress our memory retrieval engine R) (Bmemrand), (3) our model with memories retrieved uniformly at random from M and with larger weight on the reconstruction loss (BmemrandHR), (4) our model.
Treating retrieved relevant memories as the golden ground-truth produces blurry images, as shown in Figure 4 Row 2. L2 matching optimizes the wrong objective, aside of the fact that it suffers from regression-to-the-mean error even with perfectly correct paired ground-truth, as noted in previous works (e.g., Sønderby et al. (2016)). Bmemrand produces imaginations that look like reasonable digits but do not match the corresponding input image, as shown in Figure 4 Row 3, rightmost column. Such discrepancy between memories and desired imagination distributions cannot be corrected by increasing the reconstruction loss over the adversarial loss (BmemrandHR), shown in Figure 4 Row 4. Then, the resulting imaginations do not look like correct digits anymore. Our model correctly in-paints the masked digits, as shown in Figure 4 Row 5.
For each input digit image we show in Figure 4 Row 6 the closest retrieved memory from our engine R. By comparing the output of our model with the closest memory, we see that we learn to
Inputs
Bmemrand BmemrandHR our model
ima ination BmemL2
Retrieved Memory With R Ra dom draw
interpolate on the memory space and form an imagination that fits the current input image, without copy pasting, as a nearest neighbor memory engine alone would do.
3.2 INTRINSIC IMAGE DECOMPOSITION
We use the MIT intrinsic image dataset of Grosse et al. (2009). We use ten objects for training and ten objects for testing. During training, our inputs are images of the training objects and our memory database contains albedos and shadings for the training objects. At test time, we just evaluate our generator on images of the test objects, without finetuning our model. We used random image cropping for data augmentation as described in the Appendix.
Figure 5 Left shows results of our model which never uses paired annotations, that is, does not have access to pairing of each RGB image with its ground-truth albedo and shading. The results are comparable to an oracle model that has access to such paired supervision and optimizes a regression loss, similar to previous work of Narihira et al. (2015b), shown in Figure 5 Right. Our model effectively generalizes to unseen objects (Figure 5 Bottom Right). Figure 6 shows how using fullyconvolutional discriminators on albedo and shading stabilize training and converge faster.
A
I
S
R
A
I
S
R
regression with full supervisionOur approach with unpaired groundtruth
Testing on unseen objects
Figure 5: MIT intrinsic decomposition with unpaired shading and albedo. I: Input Image, A: Inferred albedo, S: Inferred shading, R: Reconstructed Image using A and S. Left: inferred albedo and shading using our weakly supervised method. Right: inferred albedo and shading using a fully supervised model that minimizes regression loss. The bottom part shows the results of decompositions on unseen objects.
A
I
S
R
0 5k 10k 15k 20k 25k
0.02 0.04 0.06 0.08 0.10
0.12 0.14 0.16 0.18 0.20
Fully-Conn D Fully-Conv D + Fully-Conn D Fully-Conv D
Figure 6: L2 distance between inferred albedo imagination and the ground truth. Fully-convolutional discriminators (purple and green lines) converge faster than fully connected ones, that employ only one fake/real classifier per image.
3.3 FIGURE-GROUND LAYER EXTRACTION
We use the seeing 3D chairs dataset of Aubry et al. (2014) as object memory database which contains 1200 different chairs and the SUN scene dataset Xiao et al. (2010) as the background memory database which contains 131000 images. Our input images are generated by randomly selecting an SUN image and cropping it to 64 × 64. After that, we overlaid an chair image on top of it as our input image.
We use 200 background images and 200 chair images to generate a small subset of ”labelled data”, where we provide the network with ground-truth of mask and background and train the network using regression loss, as described in the Appendix. Such small scale supervised pretraining suffices for stability of our model in this task; it is very realistic to assume the existence of such strong sparse supervision. Figure 7 shows the inferred mask and background we obtain.
4 CONCLUSION
We have presented a weakly supervised inverse model of images that predicts imaginations of hidden representations which then renders through image formation or layering to reconstruct the original image. It regularizes the inferred hidden representations using convolutional adversarial priors by distribution matching against retrieved relevant memories. It does not assume paired supervision and can handle multimodal imagination spaces. We have empirically validated are design choices of fully-convolutional adversarial discriminator networks and relevant memory retrieval. We believe the proposed learning paradigm better exploits unlabelled data in the form of images, depth maps, albedo, shading or segmentation maps and complements well human paired annotations.
We are working towards updating the paper with two inversion problems in videos, visual odometry and motion object segmentation. Videos allow for much stronger observation module, with imagination projections from frame to frame, as well as temporal constraining of the imaginations in time. Further, we are working towards quantifying generalization of our imaginations from training to test images, specifically measuring how well our model can do with increasing dissimilarity between memories in the database and input images.

The paper describes a network architecture for inverse problems in computer vision. Example inverse problems considered are image inpainting, computing intrinsic image decomposition and foreground/background separation.
The architecture is composed of (i) a generator that produces target (latent) output (such as foreground / background regions),
(ii) renderer that composes that latent output back to the image that can be compared with the input to measure reconstruction error,
and (iii) adversarial prior that ensures the target output (latent) image respects a certain image statistics.
Strong  points.
- The proposed architecture with memory database is interesting and appears to be novel.
Weak points:
- Experimental results are only proof-of-concept in toy set-ups and do not clearly demonstrate benefits of the proposed architecture.
- It is unclear whether the memory retrieval engine that retrieves images based on L2 distance on pixel values is going generalize to other more realistic scenarios.
- Clarity. The clarity of explanation can be also improved (see below).
Detailed evaluation.
Originality:
- The novelty of this work lies in the (iii) adversarial prior that places an adversarial loss between the generated latent output and a single image retrieved from a large unlabelled database of target output examples (called memory). The adversarial prior has a convolutional form matching local image statistics, rather than the entire image.  The particular form of network architecture with the memory-based fully convolutional adversarial loss appears to be novel and potentially interesting.
- Motivation for the Architecture. The weakest point of the proposed architecture is the "Memory retrieval engine" R (section 2.4),
where images are retrieved from the memory by measuring L2 distance on pixel intensities. While this maybe ok for simple problems considered in this work, it is unclear how this can generalize to other more complicated datasets and problems.
This should be better discussed, better justified and ideally results in some more realistic set-up shown (see below).
Quality:
- Experiments. Results are shown for inpainting of MNIST digits, intrinsic image decomposition on the MIT intrinsic image database, and figure/ground layer extraction on the synthesized dataset of 3D chairs rendered onto background from real photographs.
 The experimental validation of the model is not very strong and proof-of-concept only. All the experiments are performed in simplified toy set-ups. The MNIST digit inpainting is far from current state-of-the-art on image inpainting in real photographs (see e.g. Pathak et al., 2016). The foreground background separation is done on  only synthetically generated test data. Even for intrinsic image demposition problem there is now relatively large-scale dataset of (Bell et al., 2014), see the citation below.
While this is probably ok for the ICLR paper, it diminishes the significance of the work. Is this model going to be useful in a real settings? One possibility to address this would be to focus on one of the problems and show results on a challenging state-of-the-art data. It would be great to see the benefits of the memory database.
S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild.
ACM Transactions on Graphics, 33(4):159, 2014.
Clarity:
- The clarity of the writing can be improved. I found some of the terminology of the paper, specially the “imagination” and “memory” confusing. From figure 2, it is not clear how the “memories” for the given input image are obtained, which also took me some time to understand.
- To help understand the proposed architecture, it would be useful to draw an illustration of what is happening in the "feature space”, similar in spirit e.g. to figure 2 in