We present an algorithm for policy search in stochastic dynamical systems using model-based reinforcement learning. The system dynamics are described with Bayesian neural networks (BNNs) that include stochastic input variables. These input variables allow us to capture complex statistical patterns in the transition dynamics (e.g. multi-modality and heteroskedasticity), which are usually missed by alternative modeling approaches. After learning the dynamics, our BNNs are then fed into an algorithm that performs random roll-outs and uses stochastic optimization for policy learning. We train our BNNs by minimizing α-divergences with α = 0.5, which usually produces better results than other techniques such as variational Bayes. We illustrate the performance of our method by solving a challenging problem where model-based approaches usually fail and by obtaining promising results in real-world scenarios including the control of a gas turbine and an industrial benchmark.
1 INTRODUCTION
In model-based reinforcement learning, an agent uses its experience to first learn a model of the environment and then uses that model to reason about what action to take next. We consider the case in which the agent observes the current state st, takes some action a, and then observes the next state st+1. The problem of learning the model corresponds then to learning a stochastic transition function p(st+1|st,a) specifying the conditional distribution of st+1 given st and a. Most classic control theory texts, e.g. Bertsekas (2002), will start with the most general model of dynamical systems:
st+1 = f(st,a, z,W)
where f is some deterministic function parameterized by weightsW that takes as input the current state st, the control signal a, and some stochastic disturbance z.
However, to date, we have not been able to robustly learn dynamical system models to such a level of generality. Popular modes for transition functions include Gaussian processes (Rasmussen et al., 2003; Ko et al., 2007; Deisenroth & Rasmussen, 2011), fixed bases such as Laguerre functions (Wahlberg, 1991), and adaptive basis functions or neural networks (Draeger et al., 1995). All of these methods assume deterministic transition functions, perhaps with some addition of Gaussian observation noise. Thus, they are severely limited in the kinds of stochasticity—or transition noise—they can express. In many real-world scenarios stochasticity may often arise due to some unobserved environmental feature that can affect the dynamics in complex ways (such as unmeasured gusts of wind on a boat).
In this work we use Bayesian neural networks (BNNs) in conjunction with a random input noise source z to express stochastic dynamics. We take advantage of a very recent inference advance based on α-divergence minimization (Hernández-Lobato et al., 2016), with α = 0.5, to learn with
high accuracy BNN transition functions that are both scalable and expressive in terms of stochastic patterns. Previous work achieved one but not both of these two characteristics.
We focus our evaluation on the off-policy batch reinforcement learning scenario, in which we are given an initial batch of data from an already-running system and are asked to find a better (ideally near-optimal) policy. Such scenarios are common in real-world industry settings such as turbine control, where exploration is restricted to avoid possible damage to the system. We propose an algorithm that uses random roll-outs and stochastic optimization for learning an optimal policy from the predictions of BNNs. This method produces (to our knowledge) the first model-based solution of a 20-year-old benchmark problem: the Wet-Chicken (Tresp, 1994). We also obtain very promising results on a real-world application on controlling gas turbines and on an industrial benchmark.
2 BACKGROUND
2.1 MODEL-BASED REINFORCEMENT LEARNING
We consider reinforcement learning problems in which an agent acts in a stochastic environment by sequentially choosing actions over a sequence of time steps, in order to minimize a cumulative cost. We assume that our environment has some true dynamics Ttrue(st+1|s,a), and we are given a cost function c(st). In the model-based reinforcement learning setting, our goal is to learn an approximation Tapprox(st+1|s,a) for the true dynamics based on collected samples (st,a, st+1). The agent then tries to solve the control problem in which Tapprox is assumed to be the true dynamics.
2.2 BAYESIAN NEURAL NETWORKS WITH STOCHASTIC INPUTS
Given dataD = {xn,yn}Nn=1, formed by feature vectors xn ∈ RD and targets yn ∈ R K , we assume that yn = f(xn, zn;W) + n, where f(·, ·;W) is the output of a neural network with weightsW . The network receives as input the feature vector xn and the random disturbance zn ∼ N (0, γ). The activation functions for the hidden layers are rectifiers: ϕ(x) = max(x, 0). The activation functions for the output layers are the identity function: ϕ(x) = x. The network output is corrupted by the additive noise variable n ∼ N (0,Σ) with diagonal covariance matrix Σ. The role of the noise disturbance zn is to capture unobserved stochastic features that can affect the network’s output in complex ways. Without zn, randomness is only given by the additive Gaussian observation noise n, which can only describe limited stochastic patterns. The network has L layers, with Vl hidden units in layer l, andW = {Wl}Ll=1 is the collection of Vl × (Vl−1 + 1) weight matrices. The +1 is introduced here to account for the additional per-layer biases.
One could argue why n is needed at all when we are already using the more flexible stochastic model based on zn. The reason for this is that, in practice, we make predictions with the above model by averaging over a finite number of samples of zn and W . By using n, we obtain a predictive distribution whose density is well defined and given by a mixture of Gaussians. If we eliminate n, the predictive density is degenerate and given by a mixture of delta functions.
Let Y be an N ×K matrix with the targets yn and X be an N ×D matrix of feature vectors xn. We denote by z the N -dimensional vector with the values of the random disturbances z1, . . . , zN that were used to generate the data. The likelihood function is
p(Y |W, z,X) = N∏ n=1 p(yn |W, z,xn) = N∏ n=1 K∏ k=1 N (yn,k | f(xn, zn;W),Σ) . (1)
The prior for each entry in z is N (0, γ). We also specify a Gaussian prior distribution for each entry in each of the weight matrices inW . That is,
p(z) = N∏ n=1 N (zn|0, γ) , p(W) = L∏ l=1 Vl∏ i=1 Vl−1+1∏ j=1 N (wij,l | 0, λ) , (2)
where wij,l is the entry in the i-th row and j-th column of Wl and γ and λ are a prior variances. The posterior distribution for the weightsW and the random disturbances z is given by Bayes’ rule:
p(W, z | D) = p(Y |W, z,X)p(W)p(z) p(Y |X) . (3)
Given a new input vector x?, we can then make predictions for y? using the predictive distribution p(y? |x?,D) = ∫ [∫ N (y? | f(x?, z?;W),Σ)N (z?|0, 1) dz? ] p(W, z | D) dW dz . (4)
Unfortunately, the exact computation of (4) is intractable and we have to use approximations.
2.3 α-DIVERGENCE MINIMIZATION
We approximate the exact posterior distribution p(W, z | D) with the factorized Gaussian distribution
q(W, z) =  L∏ l=1 Vl∏ i=1 Vl−1+1∏ j=1 N (wij,l|mwij,l, vwij,l) [ N∏ n=1 N (zn |mzn, vzn) ] . (5)
The parameters mwij,l, v w ij,l and m z n, v z n are determined by minimizing a divergence between p(W, z | D) and the approximation q. After fitting q, we make predictions by replacing p(W, z | D) with q in (4) and approximating the integrals in (4) with empirical averages over samples ofW ∼ q. We aim to adjust the parameters of (5) by minimizing the α-divergence between p(W, z | D) and q(W, z) (Minka, 2005):
Dα[p(W, z | D)||q(W, z)] = 1
α(α− 1)
( 1− ∫ p(W, z | D)αq(W, z)(1−α) ) dW dz , (6)
which includes a parameter α ∈ R that controls the properties of the optimal q. Figure 1 illustrates these properties for the one-dimensional case. When α ≥ 1, q tends to cover the whole posterior distribution p. When α ≤ 0, q tends to fit a local mode in p. The value α = 0.5 is expected to achieve a balance between these two tendencies. Importantly, when α→ 0, the solution obtained is the same as with variational Bayes (VB) (Wainwright & Jordan, 2008).
The direct minimization of (6) is infeasible in practice for arbitrary α. Instead, we follow HernándezLobato et al. (2016) and optimize an energy function whose minimizer corresponds to a local minimization of α-divergences, with one α-divergence for each of the N likelihood factors in (1). Since q is Gaussian and the priors p(W) and p(z) are also Gaussian, we represent q as
q(W, z) ∝ [ N∏ n=1 f(W)fn(zn) ] p(W)p(z) , (7)
where f(W) is a Gaussian factor that approximates the geometric mean of the N likelihood factors in (1) as a function ofW . Each fn(zn) is also a Gaussian factor that approximates the n-th likelihood factor in (1) as a function of zn. We adjust f(W) and the fn(zn) by minimizing local α-divergences. In particular, we minimize the energy function
Eα(q) = − logZq − 1
α N∑ n=1 log EW,zn∼ q [( p(yn |W,xn, zn,Σ) f(W)fn(zn) )α] , (8)
(Hernández-Lobato et al., 2016), where f(W) and fn(zn) are in exponential Gaussian form and parameterized in terms of the parameters of q and the priors p(W) and p(zn), that is,
f(W) = exp  L∑ l=1 Vl∑ i=1 Vl−1+1∑ j=1 1 N ( λvwi,j,l λ− vwi,j,l w2i,j,l + mwi,j,l vwi,j,l wi,j,l ) ∝ [ q(W) p(W) ] 1 N , (9)
fn(zn) = exp { γvzn γ − vzn z2n + mzn vzn zn } ∝ q(zn) p(zn) , (10)
and logZq is the logarithm of the normalization constant of the exponential Gaussian form of q:
logZq = L∑ l=1 Vl∑ i=1 Vl−1+1∑ j=1 1 2 log ( 2πvwi,j,l ) + ( mwi,j,l )2 vwi,j,l + N∑ n=1 [ 1 2 log (2πvzn) + (mzn) 2 vzn ] . (11)
The scalable optimization of (8) is done in practice by using stochastic gradient descent. For this, we subsample the sums for n = 1, . . . , N in (8) and (11) using mini-batches and approximate the expectations over q in (8) with an average over K samples drawn from q. We can then use the reparametrization trick (Kingma et al., 2015) to obtain gradients from the resulting stochastic approximator to (8). The hyper-parameters Σ, λ and γ can also be tuned by minimizing (8). In practice we only tune Σ and keep λ = 1 and γ = d. The latter means that the prior scale of each zn grows with the data dimensionality. This guarantees that, a priori, the effect of each zn in the neural network’s output does not diminish when more and more features are available.
Minimizing (8) when α→ 0 is equivalent to running the method VB (Hernández-Lobato et al., 2016), which has recently been used to train Bayesian neural networks in reinforcement learning problems (Blundell et al., 2015; Houthooft et al., 2016; Gal et al., 2016). However, we propose to minimize (8) using α = 0.5, which often results in better test log-likelihood values.
We have also observed α = 0.5 to be more robust than VB when q(z) is not fully optimized. In particular, α = 0.5 can still capture complex stochastic patterns even when we do not learn q(z) and instead keep it fixed to the prior p(z). By contrast, VB fails completely in this case (see Appendix A).
3 POLICY SEARCH USING BNNS WITH STOCHASTIC INPUTS
We now describe a gradient-based policy search algorithm that uses the BNNs with stochastic disturbances from the previous section. The motivation for our approach lies in its applicability to industrial systems: we wish to estimate a policy in parametric form, using only an available batch of state transitions obtained from an already-running system. We assume that the true dynamics present stochastic patterns that arise due to some unobserved process affecting the system in complex ways.
Model-based policy search methods include two key parts (Deisenroth et al., 2013). The first part consists in learning a dynamics model from data in the form of state transitions (st,at, st+1), where st denotes the current state, at is the action applied and st+1 is the resulting state. The second part consists in learning the parametersWπ of a deterministic policy function π that returns the optimal action at = π(st;Wπ) as function of the current state st. The policy function can be a neural network with deterministic weights given byWπ . The first part in the aforementioned procedure is a standard regression task, which we solve by using the modeling approach from the previous section. We assume the dynamics to be stochastic with the following true transition model:
st = ftrue(st−1,at−1, zt;Wtrue) , zt ∼ N (0, γ) . (12) where the input disturbances zt ∼ N (0, γ) account for the stochasticity in the dynamics. When the Markov state st is hidden and we are given only observations ot , we can use the time embedding theorem using a suitable window of length n and approximate:
ŝ(t) = [ot−n, · · · ,ot] . (13) The transition model in equation 12 specifies a probability distribution p(st|st−1,at−1) that we approximate using a BNN with stochastic inputs:
p(st|st−1,at−1) ≈ ∫ N (st|f(st−1,at−1, zt;W),Σ)q(W)N (zt|0, γ) dW dzt , (14)
Algorithm 1 Model-based policy search using Bayesian neural networks with stochastic inputs.
1: Input: D = {sn, an,∆n} for n ∈ 1..N 2: Fit q(W) and Σ by optimizing (8). 3: function UNFOLD(s0) 4: sample{W1, ..,WK} from q(W) 5: C ← 0 6: for k = 1 : K do 7: for t = 0 : T do 8: zkt+1 ∼ N (0, γ) 9: ∆t ← f(st, π(st;Wπ), zkt+1;Wk)
10: kt+1 ∼ N (0,Σ) 11: st+1 ← st + ∆t + kt+1 12: C ← C + c(st+1) 13: return C/K 14: FitWπ by optimizing 1N ∑N n=1 UNFOLD(sn)
0 1 2 3 4 5 yt+1
0
0.6
p (y t+
1 )
st = (2.0, 3.0) , at = (0, 0.0) GP
V B α = 0.5 Ground Truth
0 1 2 3 4 5 yt+1
0
0.6
p (y t+
1 )
st = (2.0, 4.0) , at = (−1.0, 0.0) GP
V B α = 0.5 Ground Truth
0 1 2 3 4 5 yt+1
0
0.6
p (y t+
1 )
st = (4.3, 3.0) , at = (0.3, 1.0) GP
V B α = 0.5 Ground Truth
0 1 2 3 4 5 yt+1
0
0.6
p (y t+
1 )
st = (0.0, 1.5) , at = (0.0, 0.0) GP
V B α = 0.5 Ground Truth
Figure 2: Predictive distribution of yt given by different methods in four different scenarios. Ground truth (red) is obtained by sampling from the real dynamics.
where the feature vectors in our BNN are now st−1 and at−1 and the targets are given by st. In this expression, the integration with respect toW accounts for stochasticity arising from lack of knowledge of the model parameters, while the integration with respect to zt accounts for stochasticity arising from unobserved processes that cannot be modeled. In practice, these integrals are approximated by an average over samples of zt ∼ N (0, γ) andW ∼ q. In the second part of our model-based policy search algorithm, we optimize the parametersWπ of a policy that minimizes the sum of expected cost over a finite horizon T with respect to our belief q(W). This expected cost is obtained by averaging over multiple virtual roll-outs. For each roll-out we sampleWi ∼ q and then simulate state trajectories using the model st+1 = f(st,at, zt;Wi) + t+1 with policy at = π(st;Wπ), input noise zt ∼ N (0, γ) and additive noise t+1 ∼ N (0,Σ). This procedure allows us to obtain estimates of the policy’s expected cost for any particular cost function. If model, policy and cost function are differentiable, we are then able to tune Wπ by stochastic gradient descent over the roll-out average.
Given a cost function c(st), the objective to be optimized by our policy search algorithm is J(Wπ) = E [∑T t=1 c(st) ] . (15)
We approximate (15) by using (14), replacing at with π(st;Wπ) and using sampling to approximate the expectations:
J(Wπ) = ∫ [ T∑
t=1
c(st) ][ T∏ t=1 ∫ N (st|f(st−1, π(st−1;Wπ), zt;W),Σ)q(W)N (zt|0, γ) dW dzt ] p(s0)ds0 · · · dsT
= ∫ [ T∑ t=1 c(s W,{z1,...,zt},{ 1,..., t},Wπ t ) ] q(W)dW [ T∏ t=1 N ( t|0,Σ)N (zt|0, γ)d tdzt ] p(s0) ds0
≈ 1 K K∑ k=1 [ T∑ t=1 c(s Wk,{zk1 ,...,z k t },{ k 1 ,..., k t },Wπ t ) ] . (16)
The first line in (16) is obtained by using the assumption that the dynamics are Markovian with respect to the current state and the current action and by replacing p(st|st−1,at−1) with the right-hand side of (14). In the second line, sW,{z1,...,zt},{ 1,..., t},Wπt is the state that is obtained at time t in a roll-out generated by using a policy with parametersWπ , a transition function parameterized byW and input noise z1, . . . , zt, with additive noise values 1, . . . , t. In the last line we have approximated the integration with respect toW, z1, . . . , zT , 1, . . . , T and s0 by averaging over K samples of these variables. To sample s0, we draw this variable uniformly from the available transitions (st,at, st+1).
The expected cost (15) can then be optimized by stochastic gradient descent using the gradients of the Monte Carlo approximation given by the last line of (16). Algorithm 1 computes this Monte
Carlo approximation. The gradients can then be obtained using automatic differentiation tools such as Theano (Theano Development Team). Note that Algorithm 1 uses the BNNs to make predictions for the change in the state ∆t = st+1 − st instead of for the next state st+1 since this approach often performs better in practice (Deisenroth & Rasmussen, 2011).
4 EXPERIMENTS
We now evaluate the performance of our algorithm for policy search in different benchmark problems. These problems are chosen based on two reasons. First, they contain complex stochastic dynamics and second, they represent real-world applications common in industrial settings. A theano implementation of algorithm 1 is available online1. See the appendix B for a short introduction to all methods we compare to and appendix C for the hyper-parameters used.
4.1 WET-CHICKEN BENCHMARK
The Wet-Chicken benchmark (Tresp, 1994) is a challenging problem for model-based policy search that presents both bi-modal and heteroskedastic transition dynamics. We use the two-dimensional version of the problem (Hans & Udluft, 2009) and extend it to the continuous case.
In this problem, a canoeist is paddling on a two-dimensional river. The canoeist’s position at time t is (xt, yt). The river has width w = 5 and length l = 5 with a waterfall at the end, that is, at yt = l. The canoeist wants to move as close to the waterfall as possible because at time t he gets reward rt = −(l − yt). However, going beyond the waterfall boundary makes the canoeist fall down, having to start back again at the origin (0, 0). At time t the canoeist can choose an action (at,x, at,y) ∈ [−1, 1]2 that represents the direction and magnitude of his paddling. The river dynamics have stochastic turbulences st and drift vt that depend on the canoeist’s position on the x axis. The larger xt, the larger the drift and the smaller xt, the larger the turbulences. The underlying dynamics are given by the following system of equations. The drift and the turbulence magnitude are given by vt = 3xtw −1 and st = 3.5− vt, respectively. The new location (xt+1, yt+1) is given by the current
1https://github.com/siemens/policy_search_bb-alpha
location (xt, yt) and current action (at,x, at,y) using
xt+1 =  0 if xt + at,x < 0 0 if ŷt+1 > l w if xt + at,x > w xt + at,x otherwise , yt+1 =  0 if ŷt+1 < 0 0 if ŷt+1 > l ŷt+1 otherwise , (17)
where ŷt+1 = yt + (at,y − 1) + vt + stτt and τt ∼ Unif([−1, 1]) is a random variable that represents the current turbulence. These dynamics result in rich transition distributions depending on the position as illustrated by the plots in Figure 2. As the canoeist moves closer to the waterfall, the distribution for the next state becomes increasingly bi-modal (see Figure 1c) because when he is close to the waterfall, the change in the current location can be large if the canoeist falls down the waterfall and starts again at (0, 0). The distribution may also be truncated uniform for states close to the borders (see Figure 1d). Furthermore the system has heteroskedastic noise, the smaller the value of xt the higher the noise variance (compare Figure 1a with 1b). Because of these properties, the Wet-Chicken problem is especially difficult for model-based reinforcement learning methods. To our knowledge it has only been solved using model-free approaches after a discretization of the state and action sets (Hans & Udluft, 2009). For model training we use a batch 2500 random state transitions.
The predictive distributions of different models for yt+1 are shown in Figure 2 for specific choices of (xt, yt) and (ax,t, ay,t). These plots show that BNNs with α = 0.5 are very close to the ground-truth. While it is expected that Gaussian processes fail to model multi-modalities in Figure 1c, the FTIC approximation allows them to model the heteroskedasticity to an extent. VB captures the stochastic patterns on a global level, but often under or over-estimates the true probability density in specific regions. The test-loglikelihood and test MSE in y-dimension are reported in Table 2 for all methods. (the transitions for x are deterministic given y).
After fitting the models, we train policies using Algorithm 1 with a horizon of size T = 5. Table 1 shows the average reward obtained by each method. BNNs with α = 0.5 perform best and produce policies that are very close to the optimal upper bound, as indicated by the performance of the particle swarm optimization policy (PSO-P). In this problem VB seems to lack robustness and has much larger empirical variance across experiment repetitions than α = 0.5 or α = 1.0.
Figure 3 shows three example policies, πVB ,πα=0.5 and πGP (Figure 3a,3b and 3c, respectively). The policies obtained by BNNs with random inputs (VB and α = 0.5) show a richer selection of actions. The biggest differences are in the middle-right regions of the plots, where the drift towards the waterfall is large and the bi-modal transition for y (missed by the GP) is more important.
4.2 INDUSTRIAL APPLICATIONS
We now present results on two industrial cases. First, we focus on data generated by a real gas turbine and second, we consider a recently introduced simulator called the "industrial benchmark", with code publicly available2 (Hein et al., 2016b). According to the authors: "The "industrial benchmark" aims at being realistic in the sense, that it includes a variety of aspects that we found to be vital in industrial applications."
2http://github.com/siemens/industrialbenchmark
4.2.1 GAS TURBINE DATA
For the experiment with gas turbine data we simulate a task with partial observability. To that end we use 40,000 observations of a 30 dimensional time-series of sensor recordings from a real gas turbine. We are also given a cost function that evaluates the performance of the current state of the turbine. The features in the time-series are grouped into three different sets: a set of environmental variables Et (e.g. temperature and measurements from sensors in the turbine) that cannot be influenced by the agent, a set of variables relevant for the cost function Nt (e.g. the turbines current pollutant emission) and a set of steering variables At that can be manipulated to control the turbine.
We first train a world model as a reflection of the real turbine dynamics. To that end we define the world model’s transitions for Nt to have the functional form Nt = f(Et−5, .., Et, At−5, ..At). The world model assumes constant transitions for the environmental variables: Et+1 = Et. To make fair comparisons, our world model is given by a non-Bayesian neural network with deterministic weights and with additive Gaussian output noise.
We then use the world model to generate an artificial batch of data for training the different methods. The inputs in this batch are still the same as in the original turbine data, but the outputs are now sampled from the world model. After generating the artificial data, we only keep a small subset of the original inputs to the world model. The aim of this experiment is to learn policies that are robust to noise in the dynamics. This noise would originate from latent factors that cannot be controlled, such as the missing features that were originally used to generate the outputs by the world model but which are no longer available. After training the models for the dynamics, we use algorithm 1 for policy optimization. The resulting policies are then finally evaluated in the world model.
Tables 2 and 1 show the respective model and policy performances for each method. The experiment was repeated 5 times and we report average results. We observe that α = 0.5 performs best in this scenario, having the highest test log-likelihood and best policy performance.
4.2.2 INDUSTRIAL BENCHMARK
In this benchmark the hidden Markov state space st consists of 27 variables, whereas the observable state ot is only 5 dimensional. This observable state consists of 3 adjustable steering variables At: the velocity v(t), the gain g(t) and the shift s(t). We also observe the fatigue f(t) and consumption
c(t) that together form the reward signal R(t) = −(3f(t) + c(t)). Also visible is the setpoint S, a constant hyper-parameter of the benchmark that indicates the complexity of the dynamics.
For each setpoint S ∈ {10, 20, · · · , 100} we generate 7 trajectories of length 1000 using random exploration. This batch with 70, 000 state transitions forms the training set. We use 30, 000 state transitions, consisting of 3 trajectories for each setpoint, as test set.
For data preprocessing, in addition to the standard normalization process, we apply a log transformation to the reward variable. Because the reward is bounded in the interval [0, Rmax], we use a logit transformation to map this interval into the real line. We define the functional form for the dynamics as Rt = f(At−15, · · · , At, Rt−15, · · · , Rt−1). The test errors and log-likelihood are given in Table 2. We see that BNNs with α = 0.5 and α = 1.0 perform best here, whereas Gaussian processes or the MLP obtain rather poor results.
Each row in Figure 4 visualizes long term predictions of the MLP and BNNs trained with VB and α = 0.5 in two specific cases. In the top row we see that while all three methods produce wrong predictions in expectation (compare dark blue curve to red curve). However, BNNs trained with V B and with α = 0.5 exhibit a bi-modal distribution of predicted trajectories, with one mode following the ground-truth very closely. By contrast, the MLP misses the upper mode completely. The bottom row shows that the VB and α = 0.5 also produce more tight confident bands in other settings.
Next, we learn policies using the trained models. Here we use a relatively long horizon of T = 75 steps. Table 1 shows average rewards obtained when applying the policies to the real dynamics. Because both benchmark and models have an autoregressive component, we do an initial warm-up phase using random exploration before we apply the policies to the system and start to measure rewards.
We observe that GPs perform very poorly in this benchmark. We believe the reason for this is the long search horizon, which makes the uncertainties in the predictive distributions of the GPs become very large. Tighter confidence bands, as illustrated in Figure 4 seem to be key for learning good policies. Overall, α = 1.0 performs best with α = 0.5 being very close.
5 RELATED WORK
There has been relatively little attention to using Bayesian neural networks for reinforcement learning. In Blundell et al. (2015) a Thompson sampling approach is used for a contextual bandits problem; the focus is tackling the exploration-exploitation trade-off, while the work in Watter et al. (2015) combines variational auto-encoder with stochastic optimal control for visual data. Compared to our approach the first of these contributions focusses on the exploration/exploitation dilemma, while the second one uses a stochastic optimal control approach to solve the learning problem. By contrast, our work seeks to find an optimal parameterized policy.
Policy gradient techniques are a prominent class of policy search algorithms (Peters & Schaal, 2008). While model-based approaches were often used in discrete spaces (Wang & Dietterich, 2003), model-free approaches tended to be more popular in continuous spaces (e.g. Peters & Schaal (2006)).
Our work can be seen as a Monte-Carlo model-based policy gradient technique in continuous stochastic systems. Similar work was done using Gaussian processes (Deisenroth & Rasmussen, 2011) and with recurrent neural networks (Schaefer et al., 2007) . The Gaussian process approach, while restricted to a Gaussian state distribution, allows propagating beliefs over the roll-out procedure. More recently Gu et al. (2016) augment a model-free learning procedure with data generated from model-based roll-outs.
6 CONCLUSION AND FUTURE WORK
We have extended the standard Bayesian neural network (BNN) model with the addition of a random input noise source z. This enables principled Bayesian inference over complex stochastic functions. We have shown that our BNNs with random inputs can be trained with high accuracy by minimizing α-divergences, with α = 0.5, which often produces better results than variational Bayes. We have
also presented an algorithm that uses random roll-outs and stochastic optimization for learning a parameterized policy in a batch scenario. This algorithm particular suited for industry domains.
Our BNNs with random inputs have allowed us to solve a challenging benchmark problem where model-based approaches usually fail. They have also shown promising results on industry benchmarks including real-world data from a gas turbine. In particular, our experiments indicate that a BNN trained with α = 0.5 as divergence measure in conjunction with the presented algorithm for policy optimization is a powerful black-box tool for policy search.
As future work we will consider safety and exploration. For safety, we believe having uncertainty over the underlaying stochastic functions will allows us to optimize policies by focusing on worst case results instead of on average performance. For exploration, having uncertainty on the stochastic functions will be useful for efficient data collection.
ACKNOWLEDGEMENTS
José Miguel Hernández-Lobato acknowledges support from the Rafael del Pino Foundation. The authors would like to thank Ryan P. Adams, Hans-Georg Zimmermann, Matthew J. Johnson, David Duvenaud and Justin Bayer for helpful discussions.
B METHODS
In the experiments we compare to the following methods:
Standard MLP. The standard multi-layer preceptron (MLP) is equivalent to our BNNs, but does not have uncertainty over the weightW and does not include any stochastic inputs. We train this method using early stopping on a subset of the training data. When we perform roll-outs using algorithm 1, the predictions of the MLP are made stochastic by adding Gaussian noise to its output. The noise variance is fixed by maximum likelihood on some validation data after model training.
Variational Bayes (VB). The most prominent approach in training modern BNNs is to optimize the variational lower bound (Blundell et al., 2015; Houthooft et al., 2016; Gal et al., 2016). This is in practice equivalent to α-divergence minimization when α→ 0 (Hernández-Lobato et al., 2016). In our experiments we use α-divergence minimization with α = 10−6 to implement this method.
Gaussian Processes (GPs). Gaussian Processes have recently been used for policy search under the name of PILCO (Deisenroth & Rasmussen, 2011). For each dimension of the target variables, we fit a different sparse GP using the FITC approximation (Snelson & Ghahramani, 2005). In particular, each sparse GP is trained using 150 inducing inputs by using the method stochastic expectation propagation (Bui et al., 2016). After this training process we approximate the sparse GP by using a feature expansion with random basis functions (see supplementary material of Hernández-Lobato et al. 2014). This allows us to draw samples from the GP posterior distribution over functions, enabling the use of Algorithm 1 for policy training. Note that PILCO will instead moment-match at every roll-out step as it works by propagating Gaussian distributions. However, in our experiments we obtained better performance by avoiding the moment matching step with the aforementioned approximation based on random basis functions.
Particle Swarm Optimization Policy(PSO-P). We use this method to estimate an upper bound for reward performance. PSO-P is a model predictive control (MPC) method that uses the true dynamics when applicable (Hein et al., 2016a). For a given state st, the best action is selected using the standard receding horizon approach on the real environment. Note that this is not a benchmark method to compare to, we use it instead as an indicator of what the best possible reward can be achieved for a fixed planning horizon T .
C MODEL PARAMETERS
For all tasks we will use a standard MLP with two hidden layer with 20 hidden units each as policy representation. The activation functions for the hidden units are rectifiers: ϕ(x) = max(x, 0). If present, bounding of the actions is realized using the tanh activation function on the outputs of the policy. All models based on neural network will share the same hyperparameter. We use ADAM as learning algorithm in all tasks.
WetChicken The neural network models are set to 2 hidden layers and 20 hidden units per layer. We use 2500 random state transitions for training. We found that assuming no observation noise by setting Γ to a constant of 10−5 helped the models converge to lower energy values.
For policy training we use a horizon of size T = 5 and optimize the policy network for 100 epochs, averaging over K = 20 samples in each gradient update, with mini-batches of size 10 and learning rate set to 10−5.
Turbine The world model and the BNNs have two hidden layers with 50 hidden units each. For policy training and world-model evaluation we perform a roll-out with horizon T = 20. For learning the policy we use minibaches of size 10 and draw K = 10 samples from q.
Industrial Benchmark For the neural network models we use two hidden layers with 75 hidden units.We use a horizon of T = 75, training for 500 epochs with batches of size 50 and K = 25 samples for each rollout.
D COMPUTATIONAL COMPLEXITY
MODEL TRAINING
All models were trained using theano and a single GPU. Training the standard neural network is fast, the training time for this method was between 5 - 20 minutes, depending on data set size and dimensionality of the benchmark. In theano, the computational graph of the BNNs is similar to that of an ensemble of standard neural networks. The training time for the BNNs varied between 30 minutes to 5 hours depending on data size and dimensionality of benchmark. The sparse Gaussian Process was optimized using an expectation propagation algorithm and after training, it was approximated with a Bayesian linear model with fixed basis functions whose weights are initialized randomly (see Appendix B). We choose the inducing points in the GPs and the number of training epochs for these models so that the resulting training time was comparable to that of the BNNs.
POLICY SEARCH
For policy training we used a single CPU. All methods are of similar complexity as they are all trained using Algorithm 1. Depending on the horizon, data set size and network topology, training took between 20 minutes (Wet-Chicken, T = 5), 3-4 hours (Turbine, T = 20) and 14-16 hours (industrial benchmark, T = 75).

This paper introduces an approach for model-based control of stochastic dynamical systems with policy search, based on (1) learning the stochastic dynamics of the underlying system with a Bayesian deep neural network (BNN) that allows some of its inputs to be stochastic, and (2) a policy optimization method based on simulated rollouts from the learned dynamics. BNN training is carried out using \alpha-divergence minimization, the specific form of which was introduced in previous work by the authors. Validation and comparison of the approach is undertaken on a simulated domain, as well as real-world scenarios.
The paper is tightly written, and easy to follow. Its approach to fitting Bayesian neural networks with \alpha divergence is interesting and appears novel in this context. The resulting application to model-based control appears to have significant practical impact, particularly in light of the explainability that a system model can bring to specific decisions made by the policy. As such, I think that the paper brings a valuable contribution to the literature.
That said, I have a few questions and suggestions:
1) In section 2.2, it should be explained how the random z_n input is used by the neural network: is it just concatenated to the other inputs and used as-is, or is there a special treatment?
2) Moreover, much case is made for the need to have stochastic inputs, but only a scalar input seems to be provided throughout. Is this enough? How computationally difficult would providing stochastic inputs of higher dimensionality be?
3) How important is the normality assumption in z_n? How is the variance \gamma established?
4) It is mentioned that the hidden layers of the neural network are made of rectifiers, but no further utilization of this fact is made in the paper. Is this assumption somehow important in the optimization of the alpha-divergence (beyond what we know about rectifiers to mitigate the vanishing gradient problem) ?
5) Equation (3), denominator \mathbf{y} should be \mathbf{Y} ?
6) Section 2.3: it would be helpful to have an overview or discussion of the computational complexity of training BNNs, to understand whether and when they can practicably be used.
7) Between eq (12) and (13), a citation to the statement of the time embedding theorem would be helpful, as well as an indication of how the embedding dimension should be chosen.
8) Figure 1: the subplots should have the letters by which they are referenced in the text on p. 7.
9) In section 4.2.1, it is not clear if the gas turbine data is publicly available, and if so where. In addition more details should be provided, such as the dimensionality of the variables E_t, N_t and A_t.
10) Perhaps the comparisons with Gaussian processes should include variants that support stochastic inputs, such as Girard et al. (2003), to provide some of the same modelling capabilities as what’s made use of here. At least, this strand of work should be mentioned in Section 5.
References:
Girard, A., Rasmussen, C. E., Quiñonero Candela, J., & Murray Smith, R. (2003). Gaussian process priors with uncertain inputs-application to multiple-step ahead time series forecasting. Advances in Neural Information Processing Systems, 545-552.


THE PREIMAGE OF RECTIFIER NETWORK ACTIVITIES
The preimage of the activities of all the nodes at a certain level of a deep network is the set of inputs that result in the same node activity. For fully connected multi layer rectifier networks we demonstrate how to compute the preimages of activities at arbitrary levels from knowledge of the parameters in a deep rectifying network by disregarding the effects of max-pooling. If the preimage set of a certain activity in the network contains elements from more than one class it means that these classes are irreversibly mixed. This implies that preimage sets which are piecewise linear manifolds are building blocks for describing the input manifolds specific classes, i.e. all preimages should ideally be from the same class. We believe that the knowledge of how to compute preimages will be valuable in understanding the efficiency displayed by deep learning networks and could potentially be used in designing more efficient training algorithms
1 INTRODUCTION
The activity of the nodes at each level of a deep network contains all the information that will be used for classification. Ideally, if the activities are generated by two inputs from the same class, they should be similar and if the classes are distinct the activities should be distinct. The map from the input to any layer of a deep network can however easily be shown to be many to one. This means that classes can potentially get mixed at any level of the network by mapping to the same activity. This mixing cannot be undone at later stages and must therefore be avoided. Given a certain activity it is therefore essential to know the set of inputs to the network that result in this activity. This set should obviously not contain exemplars from more than one class. This means that they are potential building blocks for designing efficient classifiers. In this paper we will demonstrate that the set of inputs resulting in a specific activity at any level of a deep rectifier network can be completely characterised and we will give a procedure for computing them. For a specific activity at any level they are known as the preimage of the function mapping the input to the activity. In this procedure we disregard the effects of pooling the outputs of node activities. This can be seen as complementary to the work in Mahendran & Vedaldi (2015; 2016) where specific preimages are computed numerically by a regularised optimisation procedure that tries to map the image to the natural image manifold.
For multi layer rectifier networks where each layer consists of linear mappings followed by a rectifying linear unit (ReLU), the set of possible functions that map inputs to node activities can be shown to be piecewise linear Glorot et al. (2011); Montufar et al. (2014). We will demonstrate that for a specific activity at any level, the equivalence class of inputs that can generate this activity consists of piecewise linear manifolds in the input space. For efficient classification by the network, these manifolds must only contain a single class. They therefore constitute building blocks for efficient approximation of the distribution of classes in the input space.
Multi layer networks with rectifier linear units (ReLU) as non-linear elements starts with the ndimensional input vector x and produces successive outputs of the form:
y = [ n∑ i=1 aixi + b ]+ (1)
Where [x]+ denotes the ReLU function max(0, x). By augmenting the input vector with a one xT = (x1 . . . xn, 1) we can absorb the bias b into the vector as a weight an+1 and write
y = [ n+1∑ i=1 aixi]+ = [w Tx]+ (2)
where we have collected the weights ai into the vector w We will consider networks that are fully connected at each level l. I.e. we will consider the set of mappings between layers:
x→ x(1) → x(2) . . .→ x(k) (3)
where the j:th output node at the l + 1:th layer is computed as:
x (l+1) j = [w l+1T j x (l)]+ (4)
This is a somewhat generalised model compared to the more standard convolutional networks with multiple kernels at each level. It can however be easily specialised to the convolutional case which we will will do later.
For each point xl+1 in the activity output space of level l+1 we can define the preimage set P (xl+1) of activities xl at level l that maps to this activity. We can illustrate this as in Figure 1 with examples of preimages of the mapping between successive layers for a simple 2-node network. If there was no non linear rectifying element, the mapping would be just the linear plus bias transformation from layer l to layer l + 1 and and it could be read out from the figure by just noting the respective coordinates in the orthogonal l-system and the skewed l + 1-system.
x (l+1) 1 = [w1,1x (l) 1 + w1,2x (l) 2 + w1,3]+ (5)
x (l+1) 2 = [w2,1x (l) 1 + w2,2x (l) 2 + w2,3]+ (6)
For the all-positive quadrant in the output (x(l+1)1 , x (l+1) 2 ) the mapping is unique and the preimage is just the corresponding input (x(l)1 , x (l) 2 ). The dashes lines in the two other quadrants illustrate the preimage for the case of points on the lines x(l+1)1 = 0 and x (l+1) 2 = 0 respectively. The grey area in the last quadrant indicate points that all get mapped to the origin (0, 0) of the (x(l+1)1 , x (l+1) 2 ) coordinate system.  xl+11 xl+12
1
 =  w (l+1) 1,1 w (l+1) 1,2 w (l+1) 1,3 w (l+1) 2,1 w (l+1) 2,2 w (l+1) 2,3
0 0 1
  xl1 xl2
1  By concatenating this these transformations from the first layer we can express the activity at level (l + 1) directly as am affine mapping from the input space (x1, x2) xl+11 xl+12
1
 =  w (l+1) 1,1 w (l+1) 1,2 w (l+1) 1,3 w (l+1) 2,1 w (l+1) 2,2 w (l+1) 2,3
0 0 1
  w (l) 1,1 w (l) 1,2 w (l) 1,3 w (l) 2,1 w (l) 2,2 w (l) 2,3
0 0 1
 . . .  w (1) 1,1 w (1) 1,2 w (1) 1,3 w (1) 2,1 w (1) 2,2 w (1) 2,3
0 0 1
  x1 x2
1  In fig 2 the coordinate systems depicts these mappings from the input space (x1, x2) to the node activities and levels 1, 2 and 3 and how the preimages at the output level can be traced as elements in input space forming piecewise linear manifolds Note that disjoint piecewise linear regions in input space (x1, x2) gets mapped to distinct points on the two output axis (x (3) 1 , x (3) 2 ) while all the points in the grey shaded area get mapped to the same point x(3)1 = 0, x (3) 2 = 0 This illustrates the network’s ability to map non-linear input regions into linear output regions which is essential for successful classification.
2 = 0. In the all positive
quadrant, the mapping is just linear while in the other quadrants, the input gets mapped to 0 output of either x(l+1)1 or x (l+1) 2 or both. This generates preimage sets depending on the quadrant.
2 GENERAL FULLY CONNECTED RECTIFIER NETWORKS
We collect the linear mappings at level l in the matrix W . The number of rows of this matrix is the dimensionality of the output which we can be varying but we will assume that W always has full rank in order to focus on problems induced by the non linear ReLU element. If we denote by [x]+ the output vector with component wise application of the ReLU function on the vector x, we then can write:
x(l+1) = [Wx(l)]+
for the mapping of activities from layer l to layer l+ 1. For each element x(l+1) the preimage set of this mapping will be the set:
P (x(l+1)) = {x : xl+1 = [Wx]+} which can be specified in more detail as:
P (x(l+1)) = {x : wTi x = xl+1i ∀x l+1 i > 0, w T i x ≤ 0 ∀xl+1i = 0}
Let i1, i2, . . . ip be the indices of the components of xl+1 that are = 0 and j1, j2, . . . jq those that are > 0. This means that
wTi1x (l) ≤ 0, wTi2x (l) ≤ 0 . . . wTipx (l) ≤ 0 (7)
wTj1x (l) > 0, wTj2x (l) > 0 . . . wTjpx (l) > 0
(x
(3) 1 , x (3) 2 ) All elements in the grey shaded area eventually get mapped to output activity (0, 0) and
are irreversibly mixed.
For the case p = 0 we have a trivial linear mapping from the previous layer to only positive values of the output. This means that the preimage is just the point x(l). In the general case where p > 0 the preimage will contain elements x such that wTi x < 0 for i1, i2, . . . ip. In order to identify these we will define the null spaces of the linear mappings wi:
Πi = {x : wTj x = 0 j = 1 . . . n}
These null spaces are sets of hyperplanes in input space. Obviously, any input element x that is mapped to the negative side of the hyperplane generated by the mapping wi will get mapped to this hyperplane by the ReLU function. In order to identify this mapping we will define a set of basis vectors for elements of the input space from the one dimensional linear subspaces generated by the intersections:
πi = Π1 ∩Π2 ∩ . . . ∩Πi−1 ∩Πi+1 ∩ . . . ∩Πn Each one dimensional subspace πi is generated by intersecting the hyperplanes associated with the nullspaces of the remaining linear mapping kernels. The fact that these intersections generate one dimensional subspaces can be seen most easily using e.g. Grassmann-Cayley algebra Carlsson (1993) or by just noting that each intersection of two n-dimensional hyperplanes gives rise to a linear manifold with dimension one lower at each intersection For each subspace πi we can now define a basis unit vector ei such that each element of πi can be expressed as x = αiei. We can also define the direction and length of ei by requiring that wTi ei = 1 The assumed full rank of the mapping W guarantees that the system e1, e2 . . . en is complete in the input space. We can therefore express any vector as:
x = n∑ 1 αiei
Since ei is in the nullspace of every remaining kernel except i we have:
wTj ei = 0 i 6= j This means that:
wTj x = n∑ 1 αiw T j ei = αj
The subspace coordinates αi are therefore a convenient tool for identifying the preimage of the mapping between the successive layers in a rectifier network. Since for j = i1, i2, . . . ip we will have αj > 0 and for j = j1, j2, . . . jq we will have αj ≤ 0 We can therefore finally formulate the procedure for identifying the preimage of a mapping between successive layers in a rectifying network as:
Given the mapping where the activity of the j:th node is computed as:
x (l+1) j = [w T j x (l)]+ (8)
we identify indices j = i1, i2, . . . ip where wTj x (l) > 0 and j = j1, j2, . . . jq where wTj x (l) ≤ 0 Using kernels w1 . . . wn to define their corresponding nullspace hyperplanes Π1 . . .Πn we generate one dimensional subspaces πi by intersecting the complementary set of null-space hyperplanes:
πi = Π1 ∩Π2 ∩ . . . ∩Πi−1 ∩Πi+1 ∩ . . . ∩Πn and define basis vectors for these as ei Any element in the input space can now be expressed as a linear combination:
x = αi1ei1 + αi2ei2 + . . . αipeip − αj1ej1 − αj2ej2 − . . . αjqejq where all αi ≥ 0 The preimage set is then generated by assigning arbitrary values > 0 to the coefficients αj1 , αj2 , . . . αjq
Figure 3 illustrates the associated hyperplanes Π1,Π2,Π3 in the case of three nodes and the respective unit vectors e1, e2, e3 with positive directions indicated by arrows. For the all positive octant, i.e. all wTi x > 0 the linear mapping is just full rank and the preimage is just the associated input (x1, x2, x3). For three other octants the preimages for three selected points are illustrated:
1. For wT1 x > 0, w T 2 x > 0, w T 3 x < 0, the preimage of a point on the plane Π3 consist of all
points on the indicated arrow. 2. For wT1 x > 0, w T 2 x < 0, w T 3 x > 0, the preimage of a point on the plane Π2 consist of all
points on the indicated arrow. 3. For wT1 x > 0, w T 2 x < 0, w T 3 x < 0, the preimage of a point on the intersection of planes
Π2 and Π3 consist of all points on the indicated grey shaded area
In general, for points that are not in the all positive wTi x > 0∀i region they will be located on a linear submanifold spanned by the unit vectors ei1 , ei2 , . . . , eip
x = αi1ei1 + αi2ei2 + . . . αipeip
The preimage then consists of all points on the linear manifold:
x − αj1ej1 − αj2ej2 − . . . αjqejq where all αi ≥ 0 For a multi level network , preimages for elements that are mappings between successive levels will therefore consist of pieces of linear manifolds in the input space at that level of dimensions determined by the number of nodes with positive output for that element. By mapping back to the original input space, preimages for specific elements at a certain level will be piecewise linear manifolds, the elements of which all map to that specific element. This is exactly what is illustrated in figure 2 for the case of 2-dimensional inputs and a network with three levels of two nodes at each level.These piecewise linear manifolds can therefore me considered as fundamental building blocks for mapping input distributions to node outputs at any level of the network.
3 CONVOLUTIONAL NETWORKS HAVE “WELL-BEHAVED” PREIMAGES
Convolutional networks where the mappings W consists of convolutional matrices: wT 0 . . . 0 0 wT 0 . . . 0 0 0 wT 0 . . . 0 0 0 0 wT 0 . . . 0
0 ...  (9) are the standard realisations of multilayer networks. Using a heuristic argument, it can be seen that the preimages of these networks are in general “well-behaved” in the sense that under very general assumptions that are typically valid when training these networks the preimages generated for a specific activity associated with an image will be semantically equivalent to the given image. For any layer, the accumulated kernel w(l) mapping from the input space will be the concatenation with the specific kernel w′(1) of that layer and all specific kernels from the previous layers. I.e. they are generated by repeated convolutions:
w(l) = w′(1) ∗ w′(2) ∗ . . . ∗ w′(l−1) ∗ w′(l) (10) of the specific kernels w′(i) from the previous layers. In general the kernels associated with lower levels of the network will be associated with features such as edges in various orientation. They
are therefore in general not responsive to slowly varying signal intensity inputs. Typically they will therefore have the constant vector (1, 1, . . . 1) as nullvector. Any nullvector of a kernel at a certain level will also be a nullvector of the kernels associated with higher levels. If we study the nullvector (x1 . . . xn) associated with the complementary set of kernels when deleting a specific kernel (i) so that no overlap occurs between kernels (i− 1) and (i+ 1)
 wT 0 . . . 0 0 wT 0 . . . 0 0 0 wT 0 . . . 0 0 0 0 wT 0 . . . 0
0 ...
  x1 x2 ... xn  = 0 (11) it can easily be seen that
(x1, x2 . . . xn) = (a, . . . a, b . . . , b) (12)
i.e. a “ step edge “ where the step occurs at the location (i) of the deleted kernels. Sharp step edges in an image will lead to high likelihood of the convolution being negative. The preimages associated with an image with step edges therefore consists of the original image with step edges overlaid on already existing edges. This will in general not change the semantic content of the image. At higher levels, negative convolutions and nullspaces will be associated with more complex image structures and we can expect more complex variations in the preimage set such as the blurrings associated with the results in Mahendran & Vedaldi (2015; 2016) This suggests that the preimages associated with images in standard convolutional networks are not adversarial but rather fall in the same class as that of the image in question. It will be a focus of further work to outline in more detail the structure of the preimage class of a certain input image in order to find out if it represents e.g variations due to external factors and/or if it contains truly adversarial input exemplars. An interesting possibility would be that the set of preimages can be considered as a model set for any class of images and that the ultimate goal of training a network will be to have the set of preimages of the output nodes of a certain class coincide with the image manifold associated with that class.
4 IMPLICATIONS FOR MODELLING IMAGE MANIFOLDS
4.1 THE “GENERAL IMAGE MANIFOLD”
It is generally believed that a valid model for the distribution of images of various identifiable classes is that they lie on relatively low dimensional manifolds in the high dimensional image input space. One can also think about the “general image manifold” as consisting of all possible images that contain identifiable visual structures. The manifolds associated with specific object classes would then be contained in this general manifold. Due to external factors associated with the imaging situation like viewpoint, illumination , shading etc. the elements of a specific class will be distributed in input image space on the specific object manifold.
4.2 CONSTRAINTS ON KERNELS FOR AVOIDING THE MIXING OF PREIMAGES
Given a specific assumption of the distribution of classes on the image manifold we can postulate various properties of the kernels of rectifier networks at various levels that are necessary in order that the preimages associated with elements of different classes should not overlap. If we make the hypothesis that external factors influence different classes in a similar way we can expect the variation of the class manifolds due to external factors to be similar. Fig 4 depicts a simplified hypothetical case of the input image manifold where the variation due to external factors has been reduced to just one-dimension.
Figure 5 illustrates the placement of the hyperplanes associated with kernels at various levels in a network that would be necessary in order for the preimages of the classes not to mix with each other. Due to the assumed similarity of the variation of the individual class manifolds, the hyperplanes have to be orthogonal to the external factor variation. This highly simplified sketch just illustrates the basic idea that the covariation of class manifolds would induce severe constraints on the kernels at various levels in a network
The covariation of different object classes would have the important implication that the training of the kernels at these levels of the network would benefit from exemplars of multiple classes. This would give an explanation for the relative success of deep learning methods compared to previous approaches that in general relied on training of individual classes.
4.3 SEPARATING CLASSES
At the output level of a network, the representation of of various classes have to be disentangled in order for the final fc layer to perform a linear separation of the classes. We have empirically found that classes at the final layer of a network are highly concentrated on a small set sometimes individual nodes. The whole network can be seen as a dimensionality reducing device that starts out in high dimensional image input space and ends up in very low dimensional output layers. This dimensionality reduction is achieved by the ReLU units at various layers as illustrated in fig. 3 where the preimage, being a linear manifold is mapped from a higher to a lower dimension. At various levels of the network, different classes can located on the same manifold or the same class can be distributed on different manifolds. This means that we need procedures for ReLU networks that splits different classes on the same manifold or merges a class that is represented on several manifolds to one and the same. This can be achieved by ReLU networks and fig. 6 illustrates for the simple 2 node network how the preimages of classes can be split or merged by proper selection of kernels at successive levels.
In summary we have discussed how the properties of a network regarding it’s ability to model input manifolds and achieve efficient classification can be described with the concept of the preimage of the activities of the rectifier network at a certain level. For a specific class the set of preimages expressed in input image space resulting from the totality of activities at the output of the network constitutes the network’s model of the manifold of that class. The efficient training of a network can be seen as obtaining the correct model of every class to be discriminated. The preimage concept allows us to describe how the network is constrained by properties of these manifolds and the requirement of obtaining efficient classification at the end.
5 CONCLUSIONS AND FURTHER WORK
We have described a procedure to compute the preimage of the activity at a certain level in a deep network. I.e the set of inputs to the level that result in the same output activity. By concatenating this procedure we can compute the preimage at input level, i.e. the set of input exemplars that will eventually result in the same activity. Since inputs in the same preimage of any level activity are irreversibly mixed, they should ideally correspond to exemplars in classes to be discriminated. They therefore constitute building blocks for capturing the manifolds of classes in input space. The fact that deep networks can be seen as tools for efficient low dimensional piecewise linear approximation has been pointed out in other works recently Basri & Jacobs (2016) and is an important component in understanding how deep networks achieves their unprecedented efficiency Brahma et al. (2016); An et al. (2015) .
It will also be the objective of further work to investigate empirically if the assumed models of image manifolds and their relation to preimages are valid. This will involve the actual computation of preimage manifolds which essentially involves the computatio of nullspaces of weight matrices at various levels in order to define basis vectors for the manifolds. A deeper analysis of the preimage problem will also have to deal with the pooling that takes place in a deep learning network.
It will also be interesting to investigate how knowledge of preimages in deep networks can be used to enhance the efficiency of the training of the network. In order to do this we will have to consider the specialisation to convolutional layers and what it implies. It will also be relevant to investigate the possible nature of adversarial exemplars Szegedy et al. (2013); Goodfellow et al. (2014); Nguyen et al. (2015) in classification and if they are related to the concept of pre image of activities associated with specific classes.
ACKNOWLEDGMENTS
We would like to thank Andrew Zisserman for pointing out the work of Mahendran & Vedaldi (2015; 2016). Stefan Carlsson, Hossein Azizpour and Ali Razavian are supported by the School of Computer Science and Communication, KTH. In Razavian’s case via a grant held by Atsuto Maki

I have not read the revised version in detail yet.
SUMMARY
This paper studies the preimages of outputs of a feedforward neural network with ReLUs.
PROS
The paper presents a neat idea for changes of coordinates at the individual layers.
CONS
Quite unpolished / not enough contributions for a finished paper.
COMMENTS
- In the first version the paper contains many typos and appears to be still quite unpolished.
- The paper contains nice ideas but in my opinion it does not contribute sufficiently many results for a Conference paper.
I would be happy to recommend for the Workshop track.
- Irreversibly mixed and several other notions from the present paper are closely related to the concepts discussed in [Montufar, Pascanu, Cho, Bengio, NIPS 2014]. I feel that that paper should be cited here and the connections should be discussed. In particular, that paper also contains a discussion on the local linear maps of ReLU networks.
- I am curious about the practical considerations when computing the pre-images. The definition should be rather straight forward really, but the implementation / computation could be troublesome.
DETAILED COMMENTS
- On page 1 ``can easily be shown to be many to one'' in general.
- On page 2 ``For each point x^{l+1}'' The parentheses in the superscript are missing.
- After eq. 6 ``the mapping is unique''  is missing `when w1 and w2 are linearly independent'
- Eq. 1 should be a vector.
- Above eq. 3. ``collected the weights a_i into the vector w'' and bias b. Period is missing.
- On page 2 ``... illustrate the preimage for the case of points on the lines ... respectively''
Please indicate which is which.
- In Figure 1. Is this a sketch, or the actual illustration of a network. In the latter case, please state the specific value of x and the weights that are depicted. Also define and explain the arrows precisely.
What are the arrows in the gray part?
- On page 3 `` This means that the preimage is just the point x^{(l)}''  the points that W maps to x^{(l+1)}.
- On page 3 the first display equation. There is an index i on the left but not on the right hand side.
The quantifier in the right hand side is not clear.
- ``generated by the mapping ... w^i '' subscript
- ``get mapped to this hyperplane'' to zero
- ``remaining'' remaining from what?
- ``using e.g. Grassmann-Cayley algebra''
How about using elementary linear algebra?!
- ``gives rise to a linear manifold with dimension one lower at each intersection''
This holds if the hyperplanes are in general position.
- ``is complete in the input space'' forms a basis
- ``remaining kernel'' remaining from what?
- ``kernel'' Here kernel is referring to nullspace or to a matrix of orthonormal basis vectors of the nullspace, or to what specifically?
- Figure 3. Nullspaces of linear maps should pass through the origin.
- `` from pairwise intersections'' \cap
- ``indicated as arrows or the shaded area'' this description is far from clear.
- typos: peieces, diminsions, netork, me,


Although attention-based Neural Machine Translation have achieved great success, attention-mechanism cannot capture the entire meaning of the source sentence because the attention mechanism generates a target word depending heavily on the relevant parts of the source sentence. The report of earlier studies has introduced a latent variable to capture the entire meaning of sentence and achieved improvement on attention-based Neural Machine Translation. We follow this approach and we believe that the capturing meaning of sentence benefits from image information because human beings understand the meaning of language not only from textual information but also from perceptual information such as that gained from vision. As described herein, we propose a neural machine translation model that introduces a continuous latent variable containing an underlying semantic extracted from texts and images. Our model, which can be trained endto-end, requires image information only when training. Experiments conducted with an English–German translation task show that our model outperforms over the baseline.
1 INTRODUCTION
Neural machine translation (NMT) has achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015). In contrast to statistical machine translation, which requires huge phrase and rule tables, NMT requires much less memory. However, the most standard model, NMT with attention (Bahdanau et al., 2015) entails the shortcoming that the attention mechanism cannot capture the entire meaning of a sentence because it generates a target word while depending heavily on the relevant parts of the source sentence (Tu et al., 2016). To overcome this problem, Variational Neural Machine Translation (VNMT), which outperforms NMT with attention introduces a latent variable to capture the underlying semantic from source and target (Zhang et al., 2016). We follow the motivation of VNMT, which is to capture underlying semantic of a source.
Image information is related to language. For example, we human beings understand the meaning of language by linking perceptual information given by the surrounding environment and language (Barsalou, 1999). Although it is natural and easy for humans, it is difficult for computers to understand different domain’s information integrally. Solving this difficult task might, however, bring great improvements in natural language processing. Several researchers have attempted to link language and images such as image captioning by Xu et al. (2015) or image generation from sentences by Reed et al. (2016). They described the possibility of integral understanding of images and text. In machine translation, we can expect an improvement using not only text information but also image information because image information can bridge two languages.
As described herein, we propose the neural machine translation model which introduces a latent variable containing an underlying semantic extracted from texts and images. Our model includes an explicit latent variable z, which has underlying semantics extracted from text and images by introducing a Variational Autoencoder (VAE) (Kingma et al., 2014; Rezende et al., 2014). Our model,
∗First two authors contributed equally.
Green dotted lines denote that π and encoded y are used only when training.
which can be trained end-to-end, requires image information only when training. As described herein, we tackle the task with which one uses a parallel corpus and images in training, while using a source corpus in translating. It is important to define the task in this manner because we rarely have a corresponding image when we want to translate a sentence. During translation, our model generates a semantic variable z from a source, integrates variable z into a decoder of neural machine translation system, and then finally generates the translation. The difference between our model and VNMT is that we use image information in addition to text information.
For experiments, we used Multi30k (Elliott et al., 2016), which includes images and the corresponding parallel corpora of English and German. Our model outperforms the baseline with two evaluation metrics: METEOR (Denkowski & Lavie, 2014) and BLEU (Papineni et al., 2002). Moreover, we obtain some knowledge related to our model and Multi30k. Finally, we present some examples in which our model either improved, or worsened, the result.
Our paper contributes to the neural machine translation research community in three ways.
• We present the first neural machine translation model to introduce a latent variable inferred from image and text information. We also present the first translation task with which one uses a parallel corpus and images in training, while using a source corpus in translating.
• Our translation model can generate more accurate translation by training with images, especially for short sentences.
• We present how the translation of source is changed by adding image information compared to VNMT which does not use image information.
2 BACKGROUND
Our model is the extension of Variational Neural Machine Translation (VNMT) (Zhang et al., 2016). Our model is also viewed as one of the multimodal translation models. In our model, VAE is used to introduce a latent variable. We describe the background of our model in this section.
2.1 VARIATIONAL NEURAL MACHINE TRANSLATION
The VNMT translation model introduces a latent variable. This model’s architecture shown in Figure 1 excludes the arrow from π. This model involves three parts: encoder, inferrer, and decoder. In the encoder, both the source and target are encoded by bidirectional-Recurrent Neural Networks (bidirectional-RNN) and a semantic representation is generated. In the inferrer, a latent variable z is
modeled from a semantic representation by introducing VAE. In the decoder, a latent variable z is integrated in the Gated Recurrent Unit (GRU) decoder; also, a translation is generated.
Our model is followed by architecture, except that the image is also encoded to obtain a latent variable z.
2.2 MULTIMODAL TRANSLATION
Multimodal Translation is the task with which one might one can use a parallel corpus and images. The first papers to study multimodal translation are Elliott et al. (2015) and Hitschler & Riezler (2016). It was selected as a shared task in Workshop of Machine Translation 2016 (WMT161). Although several studies have been conducted (Caglayan et al., 2016; Huang et al., 2016; Calixto et al., 2016; Libovický et al., 2016; Rodrı́guez Guasch & Costa-jussà, 2016; Shah et al., 2016), they do not show great improvement, especially in neural machine translation (Specia et al., 2016). Here, we introduce end-to-end neural network translation models like our model.
Caglayan et al. (2016) integrate an image into an NMT decoder. They simply put source context vectors and image feature vectors extracted from ResNet-50’s ‘res4f relu’ layer (He et al., 2016) into the decoder called multimodal conditional GRU. They demonstrate that their method does not surpass the text-only baseline: NMT with attention.
Huang et al. (2016) integrate an image into a head of source words sequence. They extract prominent objects from the image by Region-based Convolutional Neural Networks (R-CNN) (Girshick, 2015). Objects are then converted to feature vectors by VGG-19 (Simonyan & Zisserman, 2014) and are put into a head of source words sequence. They demonstrate that object extraction by R-CNN contributes greatly to the improvement. This model achieved the highest METEOR score in NMTbased models in WMT16, which we compare to our model in the experiment. We designate this model as CMU.
Caglayan et al. (2016) argue that their proposed model did not achieve improvement because they failed to benefit from both text and images. We assume that they failed to integrate text and images because they simply put images and text into neural machine translation despite huge gap exists between image information and text information. Our model, however, presents the possibility of benefitting from images and text because text and images are projected to their common semantic space so that the gap of images and text would be filled.
2.3 VARIATIONAL AUTO ENCODER
VAE was proposed in an earlier report of the literature Kingma et al. (2014); Rezende et al. (2014). Given an observed variable x, VAE introduces a continuous latent variable z, with the assumption that x is generated from z. VAE incorporates pθ(x|z) and qϕ(z|x) into an end-to-end neural network. The lower bound is shown below.
LVAE = −DKL [qϕ(z|x)||pθ(z)] + Eqϕ(z|x) [log pθ(x|z)] ≤ log pθ(x) (1)
3 NEURAL MACHINE TRANSLATION WITH LATENT SEMANTIC OF IMAGE AND TEXT
We propose a neural machine translation model which explicitly has a latent variable containing an underlying semantic extracted from both text and image. This model can be seen as an extension of VNMT by adding image information.
Our model can be drawn as a graphical model in Figure 3. Its lower bound is
L = −DKL [qϕ(z|x,y,π)||pθ(z|x)] + Eqϕ(z|x,y,π) [log pθ(y|z,x)] , (2)
where x,y,π, z respectively denote the source, target, image and latent variable, and pθ and qϕ respectively denote the prior distribution and the approximate posterior distribution. It is noteworthy in Eq. (2) that we want to model p(z|x,y,π), which is intractable. Therefore we model qϕ(z|x,y,π)
1http://www.statmt.org/wmt16/
z
x y
z
x y
Figure 2: VNMT
z
x y
z
x y
π
Figure 3: Our model
instead, and also model prior pθ(z|x) so that we can generate a translation from the source in testing. Derivation of the formula is presented in the appendix.
We model all distributions in Eq. (2) by neural networks. Our model architecture is divisible into three parts: 1) encoder, 2) inferrer, and 3) decoder.
3.1 ENCODER
In the encoder, the semantic representation he is obtained from the image, source, and target. We propose several methods to encode an image. We show how these methods affect the translation result in the Experiment section. This representation is used in the inferrer. This section links to the green part of Figure 1.
3.1.1 TEXT ENCODING
The source and target are encoded in the same way as Bahdanau et al. (2015). The source is converted to a sequence of 1-of-k vector and is embedded to demb dimensions. We designate it as the source sequence. Then, a source sequence is put into bidirectional RNN. Representation hi is obtained by concatenating h⃗i and ⃗hi : h⃗i = RNN(h⃗i−1, Ewi), ⃗hi = RNN( ⃗hi+1, Ewi),hi = [h⃗i; ⃗hi], where Ewi is the embedded word in a source sentence, hi ∈ Rdh , and h⃗i, ⃗hi ∈ R dh 2 . It is conducted through i = 0 to i = Tf , where Tf is the sequence length. GRU is implemented in bidirectional RNN so that it can attain long-term dependence. Finally, we conduct mean-pooling to hi and obtain the source representation vector as hf = 1Tf ∑Tf i hi. The exact same process is applied to target to obtain target representation hg .
3.1.2 IMAGE ENCODING AND SEMANTIC REPRESENTATION
We use Convolutional Neural Networks (CNN) to extract feature vectors from images. We propose several ways of extracting image features.
Global (G) The image feature vector is extracted from the image using a CNN. With this method, we use a feature vector in the certain layer as π. Then π is encoded to the image representation vector hπ simply by affine transformation as
hπ = Wππ + bπ where Wπ ∈ Rdπ×dfc7 , bπ ∈ Rdπ . (3)
Global and Objects (G+O) First we extract some prominent objects from images in some way. Then, we obtain fc7 image feature vectors π from the original image and extracted objects using a CNN. Therefore π takes a variable length. We handle π in two ways: average and RNN encoder. In average (G+O-AVG), we first obtain intermediate image representation vector h′π by affine transformation in Eq. (3). Then, the average of h′π becomes the image representation
vector: hπ = ∑l i h ′ πi l , where l is the length of h ′ π .
In RNN encoder (G+O-RNN), we first obtain h′π by affine transformation in Eq. (3). Then, we encode h′π in the same way as we encode text in Section 3.1.1 to obtain hπ .
Global and Objects into source and target (G+O-TXT) Thereby, we first obtain h′π by affine transformation in Eq. (3). Then, we put sequential vector h′π into the head of the source sequence and target sequence. In this case, we set dπ to be the same dimension as demb. In fact, the source sequence including h′π is only used to model qϕ(z|x,y,π). Context vector c (Eq. (15)) and pθ(z|x) are computed by a source sequence that does not include h′π . We encode the source sequence including h′π as Section 3.1.1 to obtain hf and hg . In this case, hπ is not obtained. Image information is contained in hf and hg .
All representation vectors hf , hg and hπ are concatenated to obtain a semantic representation vector as he = [hf ;hg;hπ], where he ∈ Rde=2×dh+dπ (in G+O-TXT: he = [hf ;hg], where he ∈ Rde=2×dh ). It is an input of the multimodal variational neural inferrer.
3.2 INFERRER
We model the posterior qϕ(z|x,y,π) using a neural network and also the prior pθ(z|x) by neural network. This section links to the black and grey part of Figure 1.
3.2.1 NEURAL POSTERIOR APPROXIMATOR
Modeling the true posterior pθ(z|x,y,π) is usually intractable. Therefore, we consider modeling of an approximate posterior qϕ(z|x,y,π) by introducing VAE. We assume that the posterior qϕ(z|x,y,π) has the following form:
qϕ(z|x,y,π) = N (z;µ(x,y,π),σ(x,y,π)2I). (4)
The mean µ and standard deviation σ of the approximate posterior are the outputs of neural networks.
Starting from the variational neural encoder, a semantic representation vector he is projected to latent semantic space as
hz = g(W (1) z he + b (1) z ), (5)
where W (1)z ∈ Rdz×(de) b(1)z ∈ Rdz . g(·) is an element-wise activation function, which we set as tanh(·). Gaussian parameters of Eq. (4) are obtained through linear regression as
µ = Wµhz + bµ, logσ 2 = Wσhz + bσ, (6)
where µ, logσ2 ∈ Rdz .
3.2.2 NEURAL PRIOR MODEL
We model the prior distribution pθ(z|x) as follows:
pθ(z|x) = N (z;µ′(x),σ′(x) 2 I).
(7)
µ′ and σ′ are generated in the same way as that presented in Section 3.2.1, except for the absence of y and π as inputs. Because of the absence of representation vectors, the dimensions of weight in equation (5) for prior model are W ′(1) z ∈ Rdz×dh , b ′(1) z ∈ Rdz . We use a reparameterization trick to obtain a representation of latent variable z: h′z = µ+σϵ, ϵ ∼ N (0, I). During translation, h′z is set as the mean of pθ(z|x). Then, h′z is projected onto the target space as
h′e = g(W (2) z h ′ z + b (2) z ) where h ′ e ∈ Rde . (8)
h′e is then integrated into the neural machine translation’s decoder.
3.3 DECODER
This section links to the orange part of Figure 1. Given the source sentence x and the latent variable z, decoder defines the probability over translation y as
p(y|z,x) = T∏
j=1
p(yj |y<j , z,x). (9)
How we define the probability over translation y is fundamentally the same as VNMT, except for using conditional GRU instead of GRU. Conditional GRU involves two GRUs and an attention mechanism. We integrate a latent variable z into the second GRU. We describe it in the appendix.
3.4 MODEL TRAINING
Monte Carlo sampling method is used to approximate the expectation over the posterior Eq. (2), Eqϕ(z|x,y,π) ≈ 1L ∑L l=1 log pθ(y|x,h (l) z ), where L is the number of samplings. The training objective is defined as
L(θ, ϕ) = −DKL [qϕ(z|x,y,π)||pθ(z|x)] + 1
L L∑ l=1 T∑ j=1 log pθ(yj |y<j ,x,h(l)z ), (10)
where hz = µ+ σ · ϵ, ϵ ∼ N (0, I). The first term, KL divergence, can be computed analytically and is differentiable because both distributions are Gaussian. The second term is also differentiable. We set L as 1. Overall, the objective L is differentiable. Therefore, we can optimize the parameter θ and variational parameter ϕ using gradient ascent techniques.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We used Multi30k (Elliott et al., 2016) as the dataset. Multi30k have an English description and a German description for each corresponding image. We handle 29,000 pairs as training data, 1,014 pairs as validation data, and 1,000 pairs as test data.
Before training, punctuation normalization and lowercase are applied to both English and German sentences by Moses (Koehn et al., 2007) scripts2. Compound-word splitting is conducted only to German sentences using Sennrich et al. (2016)3. Then we tokenize sentences2 and use them as training data. We produce vocabulary dictionaries from training data. The vocabulary becomes 10,211 words for English and 13,180 words for German after compound-word splitting.
Image features are extracted using VGG-19 CNN (Simonyan & Zisserman, 2014). We use 4096- dimensional fc7 features. To extract the object’s region, we use Fast R-CNN (Girshick, 2015). Fast R-CNN is trained on ImageNet and MSCOCO dataset 4.
All weights are initialized byN (0, 0.01I). We use the adadelta algorithm as an optimization method. The hyperparameters used in the experiment are presented in the Appendix. All models are trained with early stopping. When training, VNMT is fine-tuned by NMT model and our models are finetuned using VNMT. When translating, we use beam-search. The beam-size is set as 12. Before evaluation, we restore split words to the original state and de-tokenize2 generated sentences.
We implemented proposed models based on dl4mt5. Actually, dl4mt is fundamentally the same model as Bahdanau et al. (2015), except that its decoder employs conditional GRU6. We implemented VNMT also with conditional GRU so small difference exists between our implementation
2https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/{normalize-punctuation, lowercase, tokenizer, detokenizer}.perl
3https://github.com/rsennrich/subword-nmt 4https://github.com/rbgirshick/fast-rcnn/tree/coco 5https://github.com/nyu-dl/dl4mt-tutorial 6The architecture is described at https://github.com/nyu-dl/dl4mt-tutorial/blob/master/docs/cgru.pdf
and originally proposed VNMT which employs normal GRU as a decoder. We evaluated results based on METEOR and BLUE using MultEval 7.
4.2 RESULT
Table 1 presents experiment results. It shows that our models outperforms the baseline in both METEOR and BLEU. Figure 4 shows the plot of METEOR score of baselines and our models models in validation. Figure 5 shows the plot of METEOR score and the source sentence length.
4.3 QUANTITATIVE ANALYSIS
Table 1 shows that G scores the best in proposed models. In G, we simply put the feature of the original image. Actually, proposed model does not benefit from R-CNN, presumably because we can not handle sequences of image features very well. For example, G+O-AVG uses the average of multiple image features, but it only makes the original image information unnecessarily confusing.
Figure 4 shows that G and G+O-AVG outperforms VNMT almost every time, but all model scores increase suddenly in the 17,000 iteration validation. We have no explanation for this behavior. Figure 4 also shows that G and G+O-AVG scores fluctuate more moderately than others. We state that G and G+O-AVG gain stability by adding image information. When one observes the difference between the test score and the validation score for each model, baseline scores decrease more than proposed model scores. Especially, the G score increases in the test, simply because proposed models produce a better METEOR score on average, as shown in Figure 4.
Figure 5 shows that G and G+O-AVG make more improvements on baselines in short sentences than in long sentences, presumably because qϕ(z|x,y,π) can model z well when a sentence is short. Image features always have the same dimension, but underlying semantics of the image and text differ. We infer that when the sentence is short, image feature representation can afford to approximate the underlying semantic, but when a sentence is long, image feature representation can not approximate the underlying semantic.
Multi30k easily becomes overfitted, as shown in Figure 8 and 9 in the appendix. This is presumably because 1) Multi30k is the descriptions of image, making the sentences short and simple, and 2) Multi30k has 29,000 sentences, which could be insufficient. In the appendix, we show how the parameter setting affects the score. One can see that decay-c has a strong effect. Huang et al. (2016) states that their proposed model outperforms the baseline (NMT), but we do not have that observation. It can be assumed that their baseline parameters are not well tuned.
4.4 QUALITATIVE ANALYSIS
We presented the top 30 sentences, which make the largest METEOR score difference between G and VNMT, to native speakers of German and get the overall comments. They were not informed of
7https://github.com/jhclark/multeval, we use meteor1.5 instead of meteor1.4, which is the default of MultEval.
our model training with image in addition to text. These comments are summarized into two general remarks. One is that G translates the meaning of the source material more accurately than VNMT. The other is that our model has more grammatical errors as prepositions’ mistakes or missing verbs compared to VNMT. We assume these two remarks are reasonable because G is trained with images which mainly have a representation of noun rather than verb, therefore can capture the meaning of materials in sentence.
Figure 6 presents the translation results and the corresponding image which G translates more accurately than VNMT in METEOR. Figure 7 presents the translation results and the corresponding image which G translates less accurately than VNMT in METEOR. Again, we note that our model does not use image during translating. In Figure 6, G translates ”a white and black dog” correctly while VNMT translates it incorrectly implying ”a white dog and a black dog”. We assume that G correctly translates the source because G captures the meaning of material in the source. In Figure 7, G incorrectly translates the source. Its translation result is missing the preposition meaning ”at”, which is hardly represented in image.We present more translation examples in appendix.
5 CONCLUSION
As described herein, we proposed the neural machine translation model that explicitly has a latent variable that includes underlying semantics extracted from both text and images. Our model outperforms the baseline in both METEOR and BLEU scores. Experiments and analysis present that our model can generate more accurate translation for short sentences. In qualitative analysis, we present that our model can translate nouns accurately while our model make grammatical errors.
A DERIVATION OF LOWER BOUNDS
The lower bound of our model can be derived as follows: p(y|x) = ∫ p(y, z|x)dz
= ∫ p(z|x)p(y|z,x)dz
log p(y|x) = log ∫
q(z|x,y,π)p(z|x)p(y|z,x) q(z|x,y,π) dz
≥ ∫
q(z|x,y,π) log p(z|x)p(y|z,x) q(z|x,y,π) dz
= ∫ q(z|x,y,π) ( log
p(z|x) q(z|x,y)
+ log p(y|z,x) ) dz
= −DKL [q(z|x,y,π)||p(z|x)] + Eq(z|x,y,π) [log p(y|z,x)] = L
B CONDITIONAL GRU
Conditional GRU is implemented in dl4mt. Caglayan et al. (2016) extends Conditional GRU to make it capable of receiving image information as input. The first GRU computes intermediate representation s′j as
s′j = (1− o′j)⊙ s′j + o′j ⊙ sj−1 (11) s′j = tanh(W ′E [yj−1] + r ′ j ⊙ (U ′sj−1)) (12)
r′j = σ(W ′ rE [yj−1] + U ′ rsj−1) (13) o′j = σ(W ′ oE [yj−1] + U ′ osj−1) (14)
where E ∈ Rdemb×dt signifies the target word embedding, s′j ∈ Rdh denotes the hidden state, r′j ∈ Rdh and o′j ∈ Rdh respectively represent the reset and update gate activations. dt stands for the dimension of target; the unique number of target words. [W ′,W ′r,W ′ o] ∈ Rdh×demb , [U ′, U ′r, U ′o] ∈ Rdh×dh are the parameters to be learned.
Context vector cj is obtained as
cj = tanh  Tf∑ i=1 αijhi  (15) αij =
exp(eij)∑Tf k=1 exp(ekj)
(16)
eij = Uatttanh(Wcatthi +Watts ′ j) (17)
where [Uatt,Wcatt,Watt] ∈ Rdh×dh are the parameters to be learned. The second GRU computes sj from s′j , cj and h ′ e as
sj = (1− o′j)⊙ sj + oj ⊙ s′j (18) sj = tanh(Wcj + rj ⊙ (Us′j) + V h′e) (19)
rj = σ(Wrcj + Urs ′ j + Vrh ′ e) (20) oj = σ(Wocj + Uos ′ j + Voh ′ e) (21)
where sj ∈ Rdh stands for the hidden state, rj ∈ Rdh and oj ∈ Rdh are the reset and update gate activations. [W,Wr,Wo] ∈ Rdh×dh , [U,Ur, Uo] ∈ Rdh×dh , [V, Vr, Vo] ∈ Rdh×dz are the
parameters to be learned. We introduce h′e obtained from a latent variable here so that a latent variable can affect the representation sj through GRU units.
Finally, the probability of y is computed as
uj = Lutanh(E [yj−1] + Lssj + Lxcj) (22) P (yj |yj−1, sj , cj) = Softmax(uj) (23)
where Lu ∈ Rdt×demb , Ls ∈ Rdemb×dh and Lc ∈ Rdemb×dh are the parameters to be learned.
C TRAINING DETAIL
C.1 HYPERPARAMETERS
Table 2 presents parameters that we use in the experiments.
We found that Multi30k dataset is easy to overfit. Figure 8 and Figure 9 present training cost and validation METEOR score graph of the two experimental settings of the NMT model. Table 3 presents the hyperparameters which were used in the experiments. Large decay-c ans small batchsize give the better METEOR scores in the end. Training is stopped if there is no validation cost improvements over the last 10 validations.
0 10000 20000 30000 40000 50000 60000 70000 iteration
0
20
40
60
80
100
co st
Training cost
1 2
Figure 8: NMT Training Cost 0 10 20 30 40 50 60 70 iteration (x 1000)
0
10
20
30
40
50
60
M E T E O R
Validation METEOR
1 2
Figure 9: NMT Validation METEOR score
Figure 10 presents the English word length histogram of the Multi30k test dataset. Most sentences in the Multi30k are less than 20 words. We assume that this is one of the reasons why Multi30k is easy to overfit.
C.2 COST GRAPH
Figure 11 and 12 present the training cost and validation cost graph of each models. Please note that VNMT fine-tuned NMT, and other models fine-tuned VNMT.
C.3 TRANSLATION EXAMPLES
We present some selected translations from VNMT and our proposed model (G). As of translation 3 to 5 our model give the better METEOR scores than VNMT and as of translation 6 to 8 VNMT give the better METEOR scores than our models.

I have problems understanding the motivation of this paper. The authors claimed to have captured a latent representation of text and image during training and can translate better without images at test time, but didn't demonstrate convincingly that images help (not to mention the setup is a bit strange when there are no images at test time). What I see are only speculative comments: "we observed some gains, so these should come from our image models". The qualitative analysis doesn't convince me that the models have learned latent representations; I am guessing the gains are due to less overfitting because of the participation of images during training.
The dataset is too small to experiment with NMT. I'm not sure if it's fair to compare their models with NMT and VNMT given the following description in Section 4.1 "VNMT is fine-tuned by NMT and our models are fine-tuned with VNMT". There should be more explanation on this.
Besides, I have problems with the presentation of this paper.
(a) There are many symbols being used unnecessary. For example: f & g are used for x (source) and y (target) in Section 3.1.
(b) The ' symbol is not being used in a consistent manner, making it sometimes hard to follow the paper. For example, in section 3.1.2, there are references about h'_\pi obtained from Eq. (3) which is about h_\pi (yes, I understand what the authors mean, but there can be better ways to present that).
(c) I'm not sure if it's correct in Section 3.2.2 h'_z is computed from \mu and \sigma. So how \mu' and \sigma' are being used ?
(d) G+O-AVG should be something like G+O_{AVG}. The minus sign makes it looks like there's an ablation test there. Similarly for other symbols.
Other things: no explanations for Figure 2 & 3. There's a missing \pi symbol in Appendix A before the KL derivation.