Most machine learning applications using neural networks seek to approximate some function g(x) by minimizing some cost criterion. In the simplest case, if one has access to pairs of the form (x, y) where y = g(x), the problem can be framed as a regression problem. Beyond this family of problems, we find many cases where the unavailability of data pairs makes this approach unfeasible. However, similar to what we find in the reinforcement learning literature, if we have some known properties of the function we are seeking to approximate, there is still hope to frame the problem as a regression problem. In this context, we present an algorithm that approximates the solution to a partial differential equation known as the Hamilton-Jacobi-Isaacs partial differential equation (HJI PDE) and compare it to current state of the art tools. This PDE, which is found in the fields of control theory and robotics, is of particular importance in safety critical systems where guarantees of performance are a must.
Most machine learning applications using neural networks seek to approximate some function g(x) by minimizing some cost criterion. In the simplest case, if one has access to pairs of the form (x, y) where y = g(x), the problem can be framed as a regression problem. Beyond this family of problems, we find many cases where the unavailability of data pairs makes this approach unfeasible. However, similar to what we find in the reinforcement learning literature, if we have some known properties of the function we are seeking to approximate, there is still hope to frame the problem as a regression problem. In this context, we present an algorithm that approximates the solution to a partial differential equation known as the Hamilton-Jacobi-Isaacs partial differential equation (HJI PDE) and compare it to current state of the art tools. This PDE, which is found in the fields of control theory and robotics, is of particular importance in safety critical systems where guarantees of performance are a must.
1 INTRODUCTION
Artificial neural networks are remarkable function approximators used in a myriad of applications ranging from complex controllers for robotic actuation (Levine et al., 2016) (Schulman et al., 2015) to simple image classifiers for digit recognition (LeCun et al., 1989) . They even find uses in physics to find approximations to solutions of PDEs and systems of coupled ordinary differential equations (ODEs) (Lagaris et al., 1998). Their success is in part achieved by their property of being universal function approximators (Hornik et al., 1989). In order to train a neural network one usually defines a cost function which captures the ”goodness” of the choice of parameters in our model, and uses gradient descent/ascent algorithms to improve them. In supervised learning, for example, input output data pairs are used to define a cost function such as the mean squared error or the mean absolute error; unfortunately, in many cases the function we want to approximate is unkown. For instance, in many reinforcement learning settings one wants to find the optimal policy, a function from state variables to actions1, which maximizes the expected sum of discounted rewards of an agent in some environment. This function is usually unkown a priori, so this problem can’t readily be framed as a regression problem using input-output pairs. This assertion becomes blurred, however, when looking at the work of Mnih et al. (2013), where a deep Q-network learns by generating targets and minimizing a cost of the form
Li(θi) = Es,a∼ρ[(yi −Q(s, a; θi))2]. (1)
Here, the targets yi are generated from the same Q-network that is being used to approximate the Q-function, hence the neural network has two purposes: approximation and data generation. In this work, we show that this same idea can be extended to the domain of approximating solutions to partial differential equations, and in particular the solution to the Hamiton-Jacobi-Isaacs PDE.
1or states to probabilities over actions
2 THE HAMILTON-JACOBI-ISAACS PDE
In control theory and robotics we often want to know how a system evolves in time given some input signal. In particular, one would like to know whether there exists an (optimal) input signal that drives our system to a particular region of interest in our state space and what that input is. For a deterministic system with continuous states and inputs, this problem can be succinctly expressed as a partial differential equation known as the Hamilton-Jacobi-Isaacs (HJI) PDE.
Let V : Rn × R− → R. Then, given a time invariant system of the form dxdt = f(x, a, b) and boundary condition V (x, 0) = l(x), where x ∈ Rn is the state vector and a ∈ A ⊆ Rma and b ∈ B ⊆ Rmb are inputs to the system2, we wish to find the solution to the minimum-payoff HJI PDE, associated to the reachability problem:
∂V (x, t)
∂t = −min{0, H(x,∇xV )}, (2)
where
H(x,∇xV ) := max a∈A min b∈B ∇xV T f(x, a, b) (3)
is known as the Hamiltonian. The boundary condition V (x, 0) = l(x) encodes in its zero sub-level set (i.e. l(x) ≤ 0) the region of interest in our state space known as the target set T . Lastly, the solution V (x, t) to (2) encodes the information about all the starting states whose induced trajectories will enter (and possibly leave) T within |t|, given the dynamics and input signals. More precisely, for some starting state x0 and t ≤ 0, V (x0, t) < 0 if and only if the trajectory starting from x0 enters T within |t|. To give some intuition as to why V (x, t) encodes the starting states whose trajectories enter T within t, let us consider the simpler problem where dxdt = f(x) is an autonomous system without any inputs. Further, let us write (2) as a finite difference in t. With some rearranging, and absorbing the gradient into V (i.e. ∇xV T f(x)∆t+V (x, t) ≈ V (x+f(x)∆t, t)), one can obtain the following approximation
V (x, t−∆t) ≈ min{ V (x, t) , V (x+ f(x)∆t, t) }. (4)
It is straightforward to see from (4) that at time t = 0 all the states outside of T (i.e. V (x, 0) ≥ 0) but near its boundary, whose induced trajectories enter the target (i.e. V (x+f(x)∆t, 0) < 0) within ∆t, will become negative in V (x,−∆t). Thinking of this update recursively one can intuitively see how the zero sub-level set of V grows backward in time to include more and more states.
For the case of one input trying to drive our system into T , the approximation becomes
V (x, t−∆t) ≈ min{ V (x, t) , min b V (x+ f(x, b)∆t, t) }, (5)
and for two competing inputs,
V (x, t−∆t) ≈ min{ V (x, t) , max a min b V (x+ f(x, a, b)∆t, t) }. (6)
Using the previous analogy of the autonomous system, one can see how (5) and (6) are essentially different ways to expand the zero sub-level set backward in time: (5) can be seen as an input trying to expand the set as fast as possible; (6) can be seen as two inputs with competing goals, where one input tries to expand the set and the other seeks to prevent its growth. Moreover, this last setting shows the relevance of the HJI PDE in safety critical systems. By treating input b as a bounded worse case disturbance and T as some unsafe region, one can establish safety guarantees about the system and claim which states won’t be driven into T within some time horizon.
2a is usually taken to be the input and b is taken to be some bounded input disturbance
Lastly, it is important to note that V (x, t) contains useful information in its gradient ∇xV (x, t). In the case where dxdt = f(x, b) has a single input, the argument minimizing the following optimization problem
b∗ = argmin b∈B ∇xV (xo, t)T f(xo, b) (7)
yields the instantaneous optimal input for state x0 at time t to guide the trajectory into T as fast as possible. Using this fact one can generate an optimal control policy based on the gradient of V . This idea can then be easily extended to the case of two competing inputs to obtain competing control policies. Finally, even though (7) need not be a convex problem, in this work we will only deal with simple dynamical systems, making the optimization problem easy to solve.
3 APPROXIMATING SOLUTIONS OF PDES
The problem presented in section 2 (as in many other cases with PDEs) is general not straightforward to solve. For this reason, trying to find a good approximation instead of the actual solution can be a reasonable approach. Many current state-of-the-art tools used to approximate solutions of PDEs, including (2), use gridding techniques (Mitchell, 2007) whereby finite differences are used to iteratively update values on a grid. Another approach (Lagaris et al., 1998) is to train a feedforward neural network by minimizing the following loss
Lθ := N∑ i=1 G(xi, ψθ(xi),∇ψθ(xi),∇2ψθ(xi))2 (8)
where G(x, ψ(x),∇ψ(x),∇2ψ(x)) = 0 is the PDE whose solution ψ(x) we are trying to approximate and xi are points taken from the discretization of our domain. In (8), the function ψθ(x) := A(x) + F (x,Nθ(x)) is a candidate approximation which by construction satisfies the boundary condition, where Nθ(x) is a feedforward neural network. In order to ensure that the conditions at the boundary are satisfied, F (x,Nθ(x)) = 0 at the boundary and A(x) is a fixed function which satisfies them.
Although this approach is well suited for some problems, special care must be taken when computing the gradient of the loss with respect to the parameters. For instance, following the previous procedure, the loss for HJI PDE would be written as
Lθ := N∑ i=1 ( ∂V (xi, ti) ∂t +min{0, H(xi,∇xV )})2, (9)
but themin in the function makes this expression not differentiable everywhere. There exist ways to circumvent this problem (Djeridane and Lygeros, 2006), however, but they require the cumbersome definition of many intermediary functions which can become hard to find for complicated dynamical models.
In this work, we try to tackle the problem of finding an approximate solution to (2) from a different perspective. We show that a poor approximation to our solution is enough to generate “good enough” new data for regression, which can in turn be used to improve our model.
4 SELF-GENERATED DATA
4.1 ALGORITHM
In this section we present a simple method for approximating the solution to (2) by utilizing a feedforward neural network in two ways: as a function approximator and a data generator. We believe that this parametric approach is better suited for finding good approximations by avoiding some of the limitations found in gridding/tabular techniques due to the curse of dimesionality. To
that end, we start by defining our candidate approximation V̂θ(x) to be of the same form as in (Lagaris et al., 1998); that is, a sum of two terms which help satisfy our boundary condition V (x, 0)
V̂θ(x, t) = V (x, 0) + tNθ(x, t), (10)
where Nθ(x, t) is a neural network mapping from our states and time variables to the real numbers. Next, we sampleN points in the state variable x chosen uniformly at random over some set S which includes T (the target set), and similarly, sampleN points in the time variable t uniformly at random over the set [−T, 0], where T ≥ 0 is the desired time horizon. By sampling from these distributions, we seek to find a good approximation to V (x, t) over the set S × [−T, 0]. Once these points have been gathered, we make use of the update (4),(5) or (6) (depending on our problem) and use V̂θ(x, t), the approximation itself, to generate the new regression points. The complete algorithm 4.1 is shown using update equation (6), but it should be clear how to modify it for the other cases.
Algorithm 1 Recursive Regression via SGD with Momentum 1: Input: N , interval, V (x, 0), f(x, a, b), A, B, S, T, K(batch size), γ (momentum decay), η
(learning rate) 2: Random initialization of the weights and biases of the neural network Nθ(x, t) 3: Define V̂θ(x, t) := V (x, 0) + tNθ(x, t) 4: Define Lθ := ∑K k=0 |yk − V̂θ(xk, tk)| 5: i← 0, νi ← 0, ∆t← 10−2 6: while True do (or stopping criterion) 7: if mod(i,interval) == 0 then 8: R← empty array of size N 9: Sample N pairs (x, t) ∼ Uniform(S × [−T, 0])
10: for j = 0 to N do 11: (a∗j , b ∗ j )← argmax
a∈A argmin b∈B
∇xV̂ Tθ f(x, a, b)
12: yj ← min{V̂θ(xj , tj), V̂θ(xj + f(xj , a∗, b∗)∆t, tj) } 13: Rj ← ((xj , tj), yj) 14: b← K elements from R picked at random 15: νi+1 ← γνi + η∇θL(b) 16: θi+1 ← θi − νi+1 17: i← i+ 1 18: Output: V̂θ(x, t)
4.2 COMMENTS
Algorithm 4.1 is a type of bootstrapping method in that lines 12 and 13 make use of V̂θ(x, t) to generate points for regression to train Nθ(x, t) which in turn modify V̂θ(x, t) itself. At first glance, it is unclear whether the generated pairs ((xj , tj), yj) will result in a good approximation to the solution of our PDE after regression; however, given the form of our candidate function (10) we expect that points sampled near t = 0 will in fact be reasonable approximations of V (x, t) for small t. Given this assumption, we hypothesize that despite the presence of misleading data, our network will be able to do a good job at regressing over all points, thus improving our initial model and allowing the generation of improved data. By repeating this procedure, we expect the accuracy of the boundary condition to ”propagate” backward in time (possibly with some minor error) in the form of better and better points for regression.
Another important aspect from line 13 is that we are simulating our dynamics forward in time using the Euler approximation step xj + f(xj , a∗, b∗)∆t. In practice, depending on the variability and complexity of the dynamics, one might use a Runge-Kutta method or a more involved integration procedure. For the experiments in the next sections a Runge-Kutta method with 4 stages (RK4) was used.
5 EXPERIMENTS
In this section we present a few 2-dimensional experiments to demonstrate the validity of our claim and the effectiveness of the algorithm. To measure the performance of the algorithm, we compare the difference between our computed approximation and the true analytical solution. In case it is not straightforward to obtain the solution, a very accurate approximation taken from state-of-the-art tools is used instead. In particular, we make use of the LevelSet Toolbox from Mitchell (2007), a powerful computational tool for obtaining good approximations to Hamilton-Jacobi (HJ) PDEs.
The first error metric to be used will be
E1(V̂θ(x, t)) := 1
M M∑ i=1 |V (xi, ti)− V̂θ(xi, ti)| (11)
where M are the number of points chosen from our domain to compute the average absolute error and V (x, t) can denote either the true solution or an accurate approximation. In the case where the analytical solution is known, the points are taken uniformly at random over S; otherwise, they are taken over some grid in S and [−T, 0]. Lastly, we also use a second error metric
E2(V̂θ(x, t)) := 1
M M∑ i=1 |∂V (xi, ti) ∂t +min{0, H(xi,∇xV )}| (12)
similar to the one defined in (9), which denotes the extent by which (on average) the approximation is violating the PDE equality. For all experiments M = 3000, all chosen uniformly at random over S × [−T, 0]. In section 5.4 we also show a visual representation of the approximations.
5.1 A LINEAR SYSTEM
In this experiment we study the performance of the algorithm on an autonomous system of the form
ẋ = f(x) = [ −1 −2 2 −1 ] x (13)
with V (x, 0) = ||x||2 − 1 and T = 1.0. For this simple system, the solution to the HJI PDE can be found analytically to be V (x, t) = e−t||x||2 − 1. One can easily verify this by checking it satisfies the boundary condition and (2). For this experiment, a feedforward neural network with a single hidden layer of 10 units and sigmoid activation functions was used. The number of points sampled was chosen to be N = 500, uniformly picked over the set S := {(x1, x2)|x1, x2 ∈ [−5, 5]} and over t ∈ [−T, 0]. The batches were picked to be of size K = 10, momentum decay γ = 0.95 and learning rate η = 0.1. The interval to renew the regression points was chosen to be 1000 iterations and the program was halted at 500,000 iterations.
The results shown in Fig. 1 where taken over 10 runs of the algorithm concurrently executed over multiple threads. The overall time to run the 500,000 iterations for all threads was 1521 seconds. The average E1 error at halting time was in the order of 7× 10−2, whereas the E2 error was in the order of 3× 10−1. The sharp jumps appearing in the loss figure in the majority of cases correspond to the error after new points are generated and used for regression.
5.2 PURSUIT-EVASION GAME: SINGLE INPUT
In this experiment we explore a pursuit-evasion game where a pursuer has to intercept an evader. In a first simplified approach, we assume the evader has a fixed heading and speed, whereas the pursuer has the same speed as the evader but has the liberty to change the direction of its heading. Fixing the evader at the origin with its heading aligned with the x-axis we frame the problem in relative coordinates between the evader and pursuer, that is x = [xr yr]T , where xr and yr represent the x and y position of the pursuer relative to the evader. This system’s dynamics are readily encoded in the following equation [
ẋr ẏr
] = f(x, b) = [ vpcos(b)− ve vpsin(b) ] (14)
where vp = ve = 2.0 represent the speed of the pursuer and evader respectively, b ∈ [0, 2π] represents the input available to the pursuer, which is the angle with respect to the x-axis. In this simplified pursuit-evasion game we say the pursuer has captured the evader if they are within 1 unit of distance from each other. Thus, we define our capture condition by defining V (x, 0) = ||x||2−1, which will ensure that our approximation captures all the states from which the pursuer can capture the evader in within T = 1.0. As in the previous example, we choose the same network architecture and the same values for the halting time, renewal interval, N ,K,γ and η.
The results shown in Fig. 2 where also taken over 10 runs of the algorithm like in section 5.2. The overall time to run the 500,000 iterations was 1952 seconds. The average E1 error at halting time was also in the order of 7× 10−2, whereas the E2 error was in the order of 1.5× 10−1. The points used to compute E1 were taken from a 51 × 51 grid at t = −0.5 (half of the time horizon), using a previously computed approximation from the LevelSet Toolbox. The reason why a single time instance was used to compute E1 was purely to reduce the amount of computation of the error at run-time.
5.3 PURSUIT-EVASION GAME: TWO INPUTS
The last experimental example also consists of a pursuit-evasion game, but in this case the evader has access to a range of speeds through an input a ∈ [−2, 2]. The system dynamics thus become[
ẋr ẏr
] = f(x, a, b) = [ vpcos(b)− a vpsin(b) ] (15)
and, similarly, V (x, 0) = ||x||2 − 1 and T = 1.0. As before, vp = 2.0. The interesting behavior we expect to see from this experiment, in comparison to the single input counterpart, is that this new available action to the evader will make it more difficult for the pursuer to intercept. This should then be evident by looking at our approximation V̂θ and its zero sub-level sets at different times. For this experiment we also chose the same architecture for the network as in the previous experiments and the same parameters, except for the halting time which was 300,000 iterations.
The results shown in Fig. 3 where also taken over 10 runs of the algorithm. The overall time to run the 300,000 iterations over the all threads was 1028 seconds. The average E1 error at halting time was in the order of 6 × 10−2, whereas the E2 error was in the order of 1.5 × 10−1. Like in the single input case, the points used to compute E1 were taken from a 51 × 51 grid at t = −0.5 of a pre-computed approximation.
5.4 CONTOUR VISUALIZATION
In this section we briefly display some of the contours for a neural network picked at random from those computed in the experimental section. Each line corresponds to the set of states where V̂θ(x, t) = 0 for t = 0,−0.25,−0.5,−0.75,−1.0. These contours enclose within them the states from which our system can reach the target set T within the absolute value of its associated time.
As expected, the linear system’s contours expand radially in all directions since the origin is a stable equilibrium point3 where all trajectories converge. For the pursuit-evasion game of one input, we also see that the contours grow toward the right, which is a sensible outcome given that the pursuer can’t catch up with the evader if it starts somewhere where xr < −1.0. Finally, the last set of contours associated with the pursuer-evader game of two competing inputs also make sense, since starting states xr < −1.0 or xr > 1.0 should not permit the pursuer to intercept the evader, and so
3with the same negative real part for the eigenvalues
the contours should not expand in those directions. As a last comparison, in Fig. 5 we display the actual contours that would be obtained using the LevelSet Toolbox.
By comparing Fig. 5 and 4 one can qualitatively see that the neural network has learned an accurate approximation of V (x, t).
6 ADVANTAGES AND DISADVANTAGES
The first advantage of using this method over gridding techniques is a dramatic improvement in memory requirements. For instance, using a standard grid with [51, 51, 10] discretization points per axis (i.e. 51 in xr, 51 in yr and 10 in t) each of the three previous experiments requires the storage of 26, 010 numbers, as opposed to 51 weights for our neural network. For the gridding approach this memory requirement must increase exponentially with the number of dimensions, whereas this need not be the case for our method. Furthermore, points that do not fall exactly on the grid have to be interpolated, whereas the neural network is an approximation that assigns values to all points in the domain. To this we can also add that fact that the neural network can yield the gradient at any point directly with backpropagation, whereas the gradient must once again be approximated for gridding techniques.
The main disadvantage of this method, for small dimensional systems in particular, is the time requirement. Computing values over a grid with the LevelSet Toolbox for the previous systems took less than 10 seconds. This advantage of gridding/tabular procedures, however, quickly disappears in higher dimensions (4D, 5D...) due to the curse of dimensionality. Finally, another disadvantage of using this method is the necessity to tune hyper parameters.
7 CONCLUSION AND FUTURE WORK
In this work we focus our attention on the idea that recursive/bootstrapped regression can be used in some problems where the function we wish to approximate has some known characteristics. In particular, we show that accurate approximations to the HJI PDE solution can be found by assigning a neural network two roles, one of them being function approximation, and the other data generation.To validate our hypothesis three different experiments with three distinct dynamical systems were performed with satisfactory results.
In this work we did not focus on the architecture of the neural network, but rather on its ability to perform well on three distinct tasks using the same algorithm. In future work we will try to find whether one can construct wider or deeper neural networks and obtain better results. We also want to investigate how well this method scales with the number of state and input dimensions. Positive results in that front could suppose an important step to further alleviate the effects of the curse of dimensionality, which are pervasive in griding methods.
ACKNOWLEDGMENTS
Special thanks to Carlos Florensa for his implementation tips and to Jaime F. Fisac for helping in the process of writing this work.
8 EXTRA EXPERIMENT
This experiment was designed to test the applicability of the method to problems beyond those presented in the previous sections. In particular, we show that with small changes we can also compute an accurate approximation to a pursuit-evasion problem in 3 dimensions. Similar to the previous examples, we frame the problem in relative coordinates with the x-axis aligned with the evader’s heading, and give the pursuer and evader control over the rate of rotation. This can be written as follows: ẋrẏr
θ̇r
 = f(x, a, b) = [−ve + vpcos(θr) + ayrvpsin(θr)− axr b− a ] (16)
For this problem the capture condition is encoded in the boundary condition V (x, 0) = ||[xr yr]T ||2 − 1 (where we ignore θr since the capture condition only depends on the distance) and we consider a the time horizon T = 1.0s. For this problem we give both pursuer and evader the same speed vp = ve = 1.0 and the same turning rates a, b ∈ [−1, 1]. Unlike the previous experiments, we used a neural network with two hidden layers with 10 and 5 units respectively and sigmoid activations. The number of points sampled was chosen to be N = 2000, uniformly picked over the set S := {(xr, yr, θr)|xr, yr ∈ [−5, 5], θr ∈ [−π, π]} and over t ∈ [−T, 0]. The batches were picked to be of size K = 25, momentum decay γ = 0.999 and learning rate η = 0.001. The interval to renew the regression points was chosen to be 1000 iterations and the program was halted at 500,000 iterations.
As shown in Fig. 6, both error metrics decrease as the algorithm progresses, reaching an average error for E1 in the order of 5.0× 10−2 and an average error for E2 in the order of 1.0× 10−1. The points used to compute E1 were taken from a 51× 51× 50 approximation grid at t = −0.5s. This set of experiments was run in a different machine4 using 8 threads and the total time for all threads to finish was 1000 seconds. Finally, Fig. 7 shows the zero level set contour at t = −0.5, which is now a 3D surface, from side and top perspectives. The first row shows the output of the LevelSet Toolbox from each perspective, and the second row shows a 3D scatter plot of points on the zero level-set obtained from one of the 8 neural networks that were trained.
4due to heavy usage of the first machine we had to switch to a different one
For this experiment, only 111 numbers were needed to store the approximation, as opposed to 51× 51×50×10 = 1300500 numbers (i.e. 51 in xr, 51 in yr, 50 in θr and 10 in t) for a [51×51×50×10] grid approximation.

This paper presents an algorithm for approximating the solution of certain time-evolution PDEs. The paper presents an interesting learning-based approach to solve such PDEs. The idea is to alternate between:
1. sampling points in space-time
2. generating solution to PDE at "those" sampled points
3. regressing a space-time function to satisfy the latter solutions at the sampled points (and hopefully generalize beyond those points).
I actually find the proposed algorithm interesting, and potentially useful in practice. The classic grid-based simulation of PDEs is often too expensive to be practical, due to the curse of dimensionality. Hence, learning the solution of PDEs makes a lot of sense for practical settings. On the other hand, as the authors point out, simply running gradient descent on the regression loss function does not work, because of the non-differentiablity of the "min" that shows up in the studied PDEs.
Therefore, I think the proposed idea is actually very interesting approach to learning the PDE solution in presence of non-differentability, which is indeed a "challenging" setup for numerically solving PDEs.
The paper motivates the problem (time-evolution PDE with "min" operator applied to the spatial derivatives) by applications in control thery, but I think there is more direct interest in such problems for the machine learning community, and even deep learning community. For example


Latent representation learned from multi-layered neural networks via hierarchical feature abstraction enables recent success of deep learning. Under the deep learning framework, generalization performance highly depends on the learned latent representation. In this work, we propose a novel latent space modeling method to learn better latent representation. We designed a neural network model based on the assumption that good base representation for supervised tasks can be attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables. From this base model, we introduce a semantic noise modeling method which enables semantic perturbation on the latent space to enhance the representational power of learned latent feature. During training, latent vector representation can be stochastically perturbed by a modeled additive noise while preserving its original semantics. It implicitly brings the effect of semantic augmentation on the latent space. The proposed model can be easily learned by back-propagation with common gradient-based optimization algorithms. Experimental results show that the proposed method helps to achieve performance benefits against various previous approaches. We also provide the empirical analyses for the proposed latent space modeling method including t-SNE visualization.
1 INTRODUCTION
Enhancing the generalization performance against unseen data given some sample data is the main objective in machine learning. Under that point of view, deep learning has been achieved many breakthroughs in several domains such as computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), natural language processing (Collobert & Weston, 2008; Bahdanau et al., 2015), and speech recognition (Hinton et al., 2012; Graves et al., 2013). Deep learning is basically realized on deep layered neural network architecture, and it learns appropriate taskspecific latent representation based on given training data. Better latent representation learned from training data results in better generalization over the future unseen data. Representation learning or latent space modeling becomes one of the key research topics in deep learning. During the past decade, researchers focused on unsupervised representation learning and achieved several remarkable landmarks on deep learning history (Vincent et al., 2010; Hinton et al., 2006; Salakhutdinov & Hinton, 2009). In terms of utilizing good base features for supervised learning, the base representation learned from unsupervised learning can be a good solution for supervised tasks (Bengio et al., 2007; Masci et al., 2011).
The definition of ‘good’ representation is, however, different according to target tasks. In unsupervised learning, a model is learned from unlabelled examples. Its main objective is to build a model
∗Corresponding author
to estimate true data distribution given examples available for training, so the learned latent representation normally includes broadly-informative components of the raw input data (e.g., mutual information between the input and the latent variable can be maximized for this objective). In supervised learning, however, a model is learned from labelled examples. In the case of classification, a supervised model learns to discriminate input data in terms of the target task using corresponding labels. Latent representation is therefore obtained to maximize the performance on the target supervised tasks.
Since the meaning of good representations vary according to target tasks (unsupervised or supervised), pre-trained features from the unsupervised model are not be guaranteed to be useful for subsequent supervised tasks. Instead of the two stage learning strategy (unsupervised pre-training followed by supervised fine-tuning), several works focused on a joint learning model which optimizes unsupervised and supervised objectives concurrently, resulting in better generalization performance (Goodfellow et al., 2013; Larochelle & Bengio, 2008a; Rasmus et al., 2015; Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014).
In this work, we propose a novel latent space modeling method for supervised learning as an extension of the joint learning approach. We define a good latent representation of standard feed-forward neural networks under the basis of information theory. Then, we introduce a semantic noise modeling method in order to enhance the generalization performance. The proposed method stochastically perturbs the latent representation of a training example by injecting a modeled semantic additive noise. Since the additive noise is randomly sampled from a pre-defined probability distribution every training iteration, different latent vectors from a single training example can be fully utilized during training. The multiple different latent vectors produced from a single training example are semantically similar under the proposed latent space modeling method, so we can expect semantic augmentation effect on the latent space.
Experiments are performed on two datasets; MNIST and CIFAR-10. The proposed model results in better classification performance compared to previous approaches through notable generalization effect (stochastically perturbed training examples well cover the distribution of unseen data).
2 METHODOLOGY
The proposed method starts from the existing joint learning viewpoint. This section first explains the process of obtaining a good base representation for supervised learning which is the basis of the proposed latent space modeling method. And then, we will describe how the proposed semantic noise modeling method perturbs the latent space while maintaining the original semantics.
2.1 BASE JOINT LEARNING MODEL
In a traditional feed-forward neural network model (Figure 1(a)), output Y of input data X is compared with its true label, and the error is propagated backward from top to bottom, which implicitly learns a task-specific latent representation Z of the input X . As an extension of a joint learning approach, an objective to be optimized can be described in general as below (Larochelle & Bengio, 2008b):
min θ λLunsup + Lsup (1)
where Lunsup and Lsup are respectively an unsupervised loss and a supervised loss, and θ and λ are model parameters to be optimized during training and a loss weighting coefficient, respectively. In terms of modeling Lunsup in Eq. (1), we assume that good latent representation Z is attained by maximizing the sum of hierarchical mutual informations between the input, latent, and output variables; i.e. the sum of the mutual information between the input X and the Z and the mutual information between the Z and the output Y . Each mutual information is decomposed into an entropy and a conditional entropy terms, so the sum of hierarchical mutual informations is expressed as follows:
I(X;Z) + I(Z;Y ) = H(X)−H(X|Z) + H(Z)−H(Z|Y ) (2)
where I(·; ·) is the mutual information between random variables, and H(·) and H(·|·) are the entropy and the conditional entropy of random variables, respectively. Note that the sum of those mutual informations becomes equivalent to the total correlation ofX , Z, and Y under the graphical structure of the general feed-forward model described in Figure 1(a); P (X,Z, Y ) = P (Y |Z)P (Z|X)P (X). The total correlation is equal to the sum of all pairwise mutual informations (Watanabe, 1960).
Our objective is to find the model parameters which maximize I(X;Z)+ I(Z;Y ). Since H(X) and H(Z) are non-negative, and H(X) is constant in this case, the lower bound on I(X;Z) + I(Z;Y ) can be reduced to 1:
I(X;Z) + I(Z;Y ) ≥ −H(X|Z)−H(Z|Y ) . (3)
It is known that maximizing −H(X|Z) can be formulated as minimizing the reconstruction error between the input x(i) (i-th example sampled from X) and its reconstruction x(i)R under the general audo-encoder framework (Vincent et al., 2010). Since H(X|Z) + H(Z|Y ) is proportional to the sum of reconstruction errors of x(i) (with its reconstruction x(i)R ) and z
(i) (with its reconstruction z (i) R ), the target objective can be expressed as follows (refer to Appendix (A1) for the details of mathematical derivations):
min θ ∑ i Lrec(x (i), x (i) R ) + Lrec(z (i), z (i) R ) (4)
where Lrec is a reconstruction loss.
Figure 1(b) shows the target model obtained from the assumption that good latent representation Z can be obtained by maximizing the sum of hierarchical mutual informations. Given an input sample x, feed-forward vectors and their reconstructions are attained deterministically by:
z = fθ1(x)
y = fθ2(fθ1(x))
xR = gθ′1(z) = gθ′1(fθ1(x))
zR = gθ′2(y) = gθ′2(fθ2(fθ1(x)) .
(5)
1Although H(Z) is an upper bound of H(Z|Y ), H(Z) is anyway affected by the process of H(Z|Y ) being minimized in Eq. (3). In Section 4, we experimentally show that we can obtain good base model even from the relatively loose lower bound defined in Eq. (3).
Given a set of training pairs (x(i), t(i)) where x(i) and t(i) are the i-th input example and its label, target objective in Eq. (1) under the model described in Figure 1(b) can be organized as below (with real-valued input samples, L2 loss LL2 is a proper choice for the reconstruction loss Lrec):
min θ:{θ1,θ′1,θ2,θ′2} ∑ i λ ( LL2(x (i), x (i) R ) + LL2(z (i), z (i) R ) ) + LNLL(y (i), t(i)) (6)
where LNLL is a negative log-likelihood loss for the target supervised task. Note that Eq. (6) represents the ‘proposed-base’ in our experiment (see Section 4.3).
2.2 SEMANTIC NOISE MODELING
Based on the architecture shown in Figure 1(b) with the target objective in Eq. (6), we conjecture that stochastic perturbation on the latent space during training helps to achieve better generalization performance for supervised tasks. Figure 1(c) shows this strategy which integrates the stochastic perturbation process during training. Suppose that ZP is a perturbed version of Z, and YP is an output which is feed-forwarded from ZP . Given a latent vector z = fθ1(x) from an input sample x,
z′ = z + ze and ŷ = fθ2(z ′) (7)
where z′ and ŷ are a perturbed latent vector and its output respectively, and ze is an additive noise used in the perturbation process of z. Based on the architecture shown in Figure 1(c), target objective can be modified as:
min θ:{θ1,θ′1,θ2,θ′2} ∑ i λ1 ( LL2(x (i), x (i) R ) + LL2(z (i), z (i) R ) ) + λ2LNLL(y (i), t(i)) + LNLL(ŷ (i), t(i)) .
(8)
Using random additive noise directly on ze is the most intuitive approach (‘proposed-perturb (random)’ in Section 4.3). However, preserving the semantics of the original latent representation z cannot be guaranteed under the direct random perturbation on the latent space. While the latent space is not directly interpretable in general, the output logit y of the latent representation z is interpretable, because the output logit is tightly coupled to the prediction of the target label. In order to preserve the semantics of the original latent representation after perturbation, we indirectly model a semantic noise on the latent space by adding small random noise directly on the output space.
Based on the output (pre-softmax) logit y, the semantic-preserving variation of y (i.e. y′) can be modeled by y′ = y + ye, where ye is a random noise vector stochastically sampled from a zeromean Gaussian with small standard deviation σ; N (0, σ2I). Now, the semantic perturbation z′ can be reconstructed from the random perturbation y′ through the decoding path gθ′2 in Figure 1(c). From the original output logit y and the randomly perturbed output logit y′, semantic additive noise ze on the latent space can be approximately modeled as below:
zR = gθ′2(y) z′R = gθ′2(y ′) = gθ′2(y + ye)
ze ' z′R − zR = gθ′2(y + ye)− gθ′2(y) (9)
By using the modeled semantic additive noise ze and the original latent representation z, we can obtain the semantic perturbation z′ as well as its output ŷ via Eq. (7) for our target objective Eq. (8).
From the described semantic noise modeling process (‘proposed-perturb (semantic)’ in Section 4.3), we expect to achieve better representation on the latent space. The effect of the proposed model in terms of learned latent representation will be explained in more detail in Section 4.4.
3 RELATED WORKS
Previous works on deep neural networks for supervised learning can be categorized into two types as shown in Figure 2; (a) a general feed-forward neural network model (LeCun et al., 1998; Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), and (b) a joint learning model which optimizes unsupervised and supervised objectives at the same time (Zhao et al., 2015; Zhang et al., 2016; Cho & Chen, 2014). Here are the corresponding objective functions:
min θ:{θ1,θ2} ∑ i LNLL(y (i), t(i)) (10)
min θ:{θ1,θ′1,θ2} ∑ i λLL2(x (i), x (i) R ) + LNLL(y (i), t(i)) (11)
where λ is a loss weighting coefficient between unsupervised and supervised losses.
Since the feed-forward neural network model is normally implemented with multiple layers in a deep learning framework, the joint learning model can be sub-classified into two types according to the type of reconstruction; reconstruction only with the input data x (Eq. (11)) and reconstruction with all the intermediate features including the input data x as follows:
min θ ∑ i λ0LL2(x(i), x(i)R ) +∑ j λjLL2(h (i) j , h (i) jR ) + LNLL(y (i), t(i))  . (12) where h(i)j and h (i) jR
are the j-th hidden representation of the i-th training example and its reconstruction.
Another type of the joint learning model, a ladder network (Figure 3), was introduced for semisupervised learning (Rasmus et al., 2015). The key concept of the ladder network is to obtain robust features by learning de-noising functions (gθ′) of the representations at every layer of the model via reconstruction losses, and the supervised loss is combined with the reconstruction losses in order to build the semi-supervised model. The ladder network achieved the best performance in semi-supervised tasks, but it is not appropriate for supervised tasks with small-scale training set (experimental analysis for supervised learning on permutation-invariant MNIST is briefly summarized
in Appendix (A2)). The proposed model in this work can be extended to semi-supervised learning, but our main focus is to enhance the representational power on latent space given labelled data for supervised learning. We leave the study for semi-supervised learning scenario based on the proposed methodology as our future research.
4 EXPERIMENTS
For quantitative analysis, we compare the proposed methodology with previous approaches described in Section 3; a traditional feed-forward supervised learning model and a joint learning model with two different types of reconstruction losses (reconstruction only with the first layer or with all the intermediate layers including the first layer). The proposed methodology includes a baseline model in Figure 1(b) as well as a stochastic perturbation model in Figure 1(c). Especially in the stochastic perturbation model, we compare the random and semantic perturbations and present some qualitative analysis on the meaning of the proposed perturbation methodology.
4.1 DATASETS
We experiment with two public datasets; MNIST (including a permutation-invariant MNIST case) and CIFAR-10. MNIST (10 classes) consists of 50k, 10k, and 10k 28×28 gray-scale images for training, validation, and test datasets, respectively. CIFAR-10 (10 classes) consists of 50k and 10k 32×32 3-channel images for training and test sets, respectively. We split the 50k CIFAR-10 training images into 40k and 10k for training and validation. Experiments are performed with different sizes of training set (from 10 examples per class to the entire training set) in order to verify the effectiveness of the proposed model in terms of generalization performance under varying sizes of training set.
4.2 IMPLEMENTATION
Figure 4 shows the architecture of the neural network model used in this experiment. W ’s are convolution or fully-connected weights (biases are excluded for visual brevity). Three convolution (3×3 (2) 32, 3×3 (2) 64, 3×3 (2) 96, where each item means the filter kernel size and (stride) with the number of filters) and two fully-connected (the numbers of output nodes are 128 and 10, respectively) layers are used for MNIST. For the permutation-invariant MNIST setting, 784-512- 256-256-128-10 nodes of fully-connected layers are used. Four convolution (5×5 (1) 64, 3×3 (2) 64, 3×3 (2) 64, and 3×3 (2) 96) and three fully-connected (128, 128, and 10 nodes) layers are used for CIFAR-10. Weights on the decoding (reconstruction) path are tied with corresponding weights on the encoding path as shown in Figure 4 (transposed convolution for the tied convolution layer and transposed matrix multiplication for the tied fully-connected layer).
In Figure 4, z′ is perturbed directly from z by adding Gaussian random noise for random perturbation. For semantic perturbation, z′ is indirectly generated from y′ which is perturbed by adding Gaussian random noise on y based on Eq. (9). For perturbation, base activation vector (z is the base
vector for the random perturbation and y is the base vector for the semantic perturbation) is scaled to [0.0, 1.0], and the zero-mean Gaussian noise with 0.2 of standard deviation is added (via elementwise addition) on the normalized base activation. This perturbed scaled activation is de-scaled with the original min and max activations of the base vector.
Initial learning rates are 0.005 and 0.001 for MNIST and permutation-invariant MNIST, and 0.002 for CIFAR-10, respectively. The learning rates are decayed by a factor of 5 every 40 epochs until the 120-th epoch. For both datasets, the minibatch size is set to 100, and the target objective is optimized using Adam optimizer (Kingma & Ba, 2015) with a momentum 0.9. All the λ’s for reconstruction losses in Eq. (11) and Eq. (12) are 0.03 and 0.01 for MNIST and CIFAR-10, respectively. The same weighting factors for reconstruction losses (0.03 for MNIST and 0.01 for CIFAR-10) are used for λ1 in Eq (8), and 1.0 is used for λ2.
Input data is first scaled to [0.0, 1.0] and then whitened by the average across all the training examples. In CIFAR-10, random cropping (24×24 image is randomly cropped from the original 32×32 image) and random horizontal flipping (mirroring) are used for data augmentation. We selected the network that performed best on the validation dataset for evaluation on the test dataset. All the experiments are performed with TensorFlow (Abadi et al., 2015).
4.3 QUANTITATIVE ANALYSIS
Three previous approaches (a traditional feed-forward model, a joint learning model with the input reconstruction loss, and a joint learning model with reconstruction losses of all the intermediate layers including the input layer) are compared with the proposed methods (the baseline model in Figure 1(b), and the stochastic perturbation model in Figure 1(c) with two different perturbation methods; random and semantic). We measure the classification performance according to varying sizes of training set (examples randomly chosen from the original training dataset). Performance is averaged over three different random trials.
Table 1 summarizes the classification performance for MNIST and CIFAR-10. As we expected, the base model obtained by maximizing the sum of mutual informations (proposed-base) mostly performs better than previous approaches, and the model with the semantic perturbation (proposedperturb (semantic)) performs best among all the comparison targets. Especially in MNIST, the error rate of ‘proposed-perturb (semantic)’ with 2k per-class training examples is less than the error rate of all types of previous works with the entire training set (approximately 5k per-class examples).
We further verify the proposed method on the permutation-invariant MNIST task with a standard feed-forward neural network. Classification performance is measured against three different sizes of training set (1k, 2k, and 5k per-class training examples). ‘Proposed-perturb (semantic)’ achieves the best performance among all the configurations; 2.57%, 1.82%, and 1.28% error rates for 1k, 2k, and 5k per-class training examples, respectively. The joint learning model with the input reconstruction loss performs best among three previous approaches; 2.72%, 1.97%, and 1.38% error rates for 1k, 2k, and 5k per-class training examples, respectively.
4.4 QUALITATIVE ANALYSIS
As mentioned before, random perturbation by adding unstructured noise directly to the latent representation cannot guarantee preserving the semantics of the original representation. We compared two different perturbation methods (random and semantic) by visualizing the examples reconstructed from the perturbed latent vectors (Figure 5). Top row is the original examples selected from training set (among 2k per-class training examples), and the rest are the reconstructions of their perturbed latent representations. Based on the architecture described in Figure 1(b), we generated five different perturbed latent representations according to the type of perturbation, and reconstructed the perturbed latent vectors through decoding path for reconstruction.
Figure 5(a) and (b) show the examples reconstructed from the random and semantic perturbations, respectively. For both cases, zero-mean Gaussian random noise (0.2 standard deviation) is used for perturbation. As shown in Figure 5(a), random perturbation partially destroys the original semantics; for example, semantics of ‘1’ is mostly destroyed under random perturbation, and some examples of ‘3’ are reconstructed as being similar to ‘8’ rather than its original content ‘3’. Figure 5(b) shows the examples reconstructed from the semantic perturbation. The reconstructed examples show subtle semantic variations while preserving the original semantic contents; for example, thickness difference in ‘3’ (example on the third row) or writing style difference in ‘8’ (openness of the top left corner).
Figure 6 shows the overall effect of the perturbation. In this analysis, 100 per-class MNIST examples are used for training. From the trained model based on the architecture described in Figure 1(b), latent representations z of all the 50k examples (among 50k examples, only 1k examples were used for training) are visualized by using t-SNE (Maaten & Hinton, 2008). Only the training examples of three classes (0, 1, and 9) among ten classes are depicted as black circles for visual discrimination in
Figure 6(a). The rest of the examples which were not used for training (approximately 4.9k examples per class) are depicted as a background with different colors. We treat the colored background examples (not used for training) as a true distribution of unseen data in order to estimate the generalization level of learned representation according to the type of perturbation. Figure 6(b) and (c) show the training examples (100 examples per class with yellow circles) and their perturbed ones (3× sampled from each example with blue crosses) through random and semantic perturbations, respectively.
In Figure 6(b), perturbed samples are distributed near the original training examples, but some samples outside the true distribution cannot be identified easily with appropriate classes. This can be explained with Figure 5(a), since some perturbed samples are ambiguous semantically. In Figure 6(c), however, most of the perturbed samples evenly cover the true distribution. As mentioned before, stochastic perturbation with the semantic additive noise during training implicitly incurs the effect of augmentation on the latent space while resulting in better generalization. Per-class t-SNE results are summarized in Appendix (A4.2).
5 DISCUSSION
We introduced a novel latent space modeling method for supervised tasks based on the standard feed-forward neural network architecture. The presented model simultaneously optimizes both supervised and unsupervised losses based on the assumption that the better latent representation can be obtained by maximizing the sum of hierarchical mutual informations. Especially the stochastic perturbation process which is achieved by modeling the semantic additive noise during training enhances the representational power of the latent space. From the proposed semantic noise modeling process, we can expect improvement of generalization performance in supervised learning with implicit semantic augmentation effect on the latent space.
The presented model architecture can be intuitively extended to semi-supervised learning because it is implemented as the joint optimization of supervised and unsupervised objectives. For semisupervised learning, however, logical link between features learned from labelled and unlabelled data needs to be considered additionally. We leave the extension of the presented approach to semisupervised learning for the future.

This paper introduces a maximum total correlation procedure, adds a target and then adds noise perturbations.
Technical issues:
The move from (1) to (2) is problematic. Yes it is a lower bound, but by igoring H(Z), equation (2) ignores the fact that H(Z) will potentially vary more significantly that H(Z|Y). As a result of removing H(Z), the objective (2) encourages Z that are low entropy as the H(Z) term is ignored, doubly so as low entropy Z results in low entropy Z|Y. Yes the -H(X|Z) mitigates against a complete entropy collapse for H(Z), but it still neglects critical terms. In fact one might wonder if this is the reason that semantic noise addition needs to be done anyway, just to push up the entropy of Z to stop it reducing too much.
In (3) arbitrary balancing paramters lamda_1 and lambda_2 are introduced ex-nihilo - they were not there in (2). This is not ever justified.
Then in (5), a further choice is made by simply adding L_{NLL} to the objective. But in the supervised case, the targets are known and so turn up in H(Z|Y). Hence now H(Z|Y) should be conditioned on the targets. However instead another objective is added again without justification, and the conditional entropy of Z is left disconnected from the data it is to be conditioned on. One might argue the C(X,Y,Z) simply acts as a prior on the networks (and hence implicitly on the weights) that we consider, which is then combined with a likelihood term, but this case is not made. In fact there is no explicit probabilistic or information theoretic motivation for the chosen objective.
Given these issues, it is then not too surprising that some further things need to be done, such as semantic noise addition to actually get things working properly. It may be the form of noise addition is a good idea, but given the troublesome objective being used in the first place, it is very hard to draw conclusions.
In summary, substantially better theoretical justification of the chosen model is needed, before any reasonable conclusion on the semantic noise modelling can be made.


Existing machine translation decoding algorithms generate translations in a strictly monotonic fashion and never revisit previous decisions. As a result, earlier mistakes cannot be corrected at a later stage. In this paper, we present a translation scheme that starts from an initial guess and then makes iterative improvements that may revisit previous decisions. We parameterize our model as a convolutional neural network that predicts discrete substitutions to an existing translation based on an attention mechanism over both the source sentence as well as the current translation output. By making less than one modification per sentence, we improve the output of a phrase-based translation system by up to 0.4 BLEU on WMT15 German-English translation.
1 INTRODUCTION
Existing decoding schemes for translation generate outputs either left-to-right, such as for phrasebased or neural translation models, or bottom-up as in syntactic models (Koehn et al., 2003; Galley et al., 2004; Bahdanau et al., 2015). All decoding algorithms for those models make decisions which cannot be revisited at a later stage, such as when the model discovers that it made an error earlier on.
On the other hand, humans generate all but the simplest translations by conceiving a rough draft of the solution and then iteratively improving it until it is deemed complete. The translator may modify a clause she tackled earlier at any point and make arbitrary modifications to improve the translation.
It can be argued that beam search allows to recover from mistakes, simply by providing alternative translations. However, reasonable beam sizes encode only a small number of binary decisions. A beam of size 50 contains fewer than six binary decisions, all of which frequently share the same prefix (Huang, 2008).1
In this paper, we present models that tackle translation similar to humans. The model iteratively edits the target sentence until it cannot improve it further. As a preliminary study, we address the problem of finding mistakes in an existing translation via a simple classifier that predicts if a word in a translation is correct (§2). Next, we model word substitutions for an existing translation via a convolutional neural network that attends to the source when suggesting substitutions (§3). Finally, we devise a model that attends both to the source as well as to the existing translation (§4). We repeatedly apply the models to their own output by determining the best substitution for each word in the previous translation and then choosing either one or zero substitutions for each sentence. For the latter we consider various heuristics as well as a classifier-based selection method (§5). Our results demonstrate that we can improve the output of a phrase-based translation system on WMT15 German-English data by up to 0.4 BLEU (Papineni et al., 2002) by making on average only 0.6 substitutions per sentence (§6). Our approach differs from automatic post-editing since it does not require post-edited text which is a scarce resource (Simard et al., 2007; Bojar et al., 2016). For our first model (§3) we merely require parallel text and for our second model (§4) the output of a baseline translation system.
∗Roman was interning at Facebook for this work. 125 = 32 < 50 < 26 = 64
2 DETECTING ERRORS
Before correcting errors we consider the task of detecting mistakes in the output of an existing translation system.
In the following, we use lowercase boldface for vectors (e.g. x), uppercase boldface for matrices (e.g. F) and calligraphy for sets (e.g. X ). We use superscripts for indexing or slicing, e.g., xi, Fi,j , Fi = (Fi,1, . . . ,Fi,|F
i|). We further denote x as the source sentence, yg as the guess translation from which we start and which was produced by a phrase-based translation system (§6.1), and yref as the reference translation. Sentences are vectors of indices indicating entries in a source vocabulary X or a target vocabulary Y . For example, x = (x1, . . . ,x|x|) ∈ X |x| with X = {1, . . . , |X |}. We omit biases of linear layers to simplify the notation.
Error detection focuses on word-level accuracy, i.e., we predict for each token in a given translation if it is present in the reference or not. This metric ignores word order, however, we hope that performance on this simple task provides us with a sense of how difficult it will be to modify translations to a positive effect. A token yig in the candidate translation yg is deemed correct iff it is present in the reference translation: yig ∈ yref. We build a neural network f to predict correctness of each token in yg given the source sentence x: f(x,yg) ∈ [0; 1]|yg| , where f(x,yg)i estimates P ( yig ∈ yref ) .
Architecture. We use an architecture similar to the word alignment model of Legrand et al. (2016). The source and the target sequences are embedded via a lookup table that replace each word type with a learned vector. The resulting vector sequences are then processed by alternating convolutions and non-linearities. This results in a vector S (x)i representing each position i in the source x and a vector T (yg)
j representing each position j in the target yg. These vectors are then compared via a dot product. Our prediction estimates the probability of a target word being correct as the largest dot product between any source word and the guess word. We apply the logistic function σ to this score,
f(x,yg) i = σ ( max
16j6|x|
[ S(x)T(yg) T ]j,i) .
Training. At training time we minimize the cross-entropy loss, with the binary supervision 1 for yig ∈ yref, 0 otherwise.
Testing. At test time we threshold the model prediction f(x,yg)i to detect mistakes. We compare the performance of our network to the following baselines:
1. Predicting that all candidate words are always correct fcor ≡ 1, or always incorrect fwrong ≡ 0.
2. The prior probability of a word being correct based on the training data fstat(y) = (P [y ∈ yref | y ∈ yg] > 0.5).
We report word-level accuracy metrics in Table 1. While the model significantly improves over the baselines, the probability of correctly labeling a word as a mistake remains low (62.71%). The task of predicting mistakes is not easy as previously shown in confidence estimation (Blatz et al., 2004; Ueffing & Ney, 2007). Also, one should bear in mind that this task cannot be solved with 100% accuracy since a sentence can be correctly in multiple different ways and we only have a single reference translation. In our case, our final refinement objective might be easier than error detection as we do not need to detect all errors. We need to identify some of the locations where a substitution could improve BLEU. At the same time, our strategy should also suggest these substitutions. This is the objective of the model introduced in the next section.
3 ATTENTION-BASED MODEL
We introduce a model to predict modifications to a translation which can be trained on bilingual text. In §5 we discuss strategies to iteratively apply this model to its own output in order to improve a translation.
Our model F takes as input a source sentence x and a target sentence y, and outputs a distribution over the vocabulary for each target position, F(x,y) ∈ [0, 1]|y|×|Y| . For each position i and any word j ∈ Y , F(x,y)i,j estimates P(yi = j |x,y−i), the probability of word j being at position i given the source and the target context y−i = ( y1, . . . ,yi−1,yi+1, . . . ,y|y| ) surrounding i. In other words, we learn a non-causal language model (Bengio et al., 2003) which is also conditioned on the source x.
Architecture. We rely on a convolutional model with attention. The source sentence is embedded into distributional space via a lookup table, followed by convolutions and non-linearities. The target sentence is also embedded in distributional space via a lookup table, followed by a single convolution and a succession of linear layers and non-linearities. The target convolution weights are zeroed at the center so that the model does not have access to the center word. This means that the model observes a fixed size context of length 2k for any target position i, y−i|k = ( yi−k, . . . ,yi−1,yi+1, . . . ,yi+k
) where 2k + 1 refers to the convolution kernel width. These operations result in a vector Sj representing each position j in the source sentence x and a vector Ti representing each target context y−i|k.
Given a target position i, an attention module then takes as input these representation and outputs a weight for each target position
α(i, j) = exp
( Sj ·Ti )∑|x| j′=1 exp (S j′ ·Ti) .
These weights correspond to dot-product attention scores (Luong et al., 2015; Rush et al., 2015). The attention weights allow to compute a source summary specific to each target context through a weighted sum,
a ( y−i|k,x ) = |x|∑ j=1 α(i, j) Sj
Finally, this summary a(y−i|k,x) is concatenated with the embedding of the target context y−i|k obtained from the target lookup table,
L ( y−i|k ) = { Lj , j ∈ y−i|k } and a multilayer perceptron followed by a softmax computes F(x,y)i from a(y−i|k,x), L(y−i|k). Note that we could alternatively use Ti instead of L(y−i|k) but our preliminary validation experiments showed better result with the lookup table output.
Training. The model is trained to maximize the (log) likelihood of the pairs (x,yref) from the training set.
Testing. At test time the model is given (x,yg), i.e., the source and the guess sentence. Similar to maximum likelihood training for left-to-right translation systems (Bahdanau et al., 2015), the model is therefore not exposed to the same type of context in training (reference contexts from yref) and testing (guess contexts from yg).
Discussion. Our model is similar to the attention-based translation approach of Bahdanau et al. (2015). In addition to using convolutions, the main difference is that we have access to both left and
right target context y−i|k since we start from an initial guess translation. Right target words are of course good predictors of the previous word. For instance, an early validation experiment with the setup from §6.1 showed a perplexity of 5.4 for this model which compares to 13.9 with the same model trained with the left context only.
4 DUAL ATTENTION MODEL
We introduce a dual attention architecture to also make use of the guess at training time. This contrasts with the model introduced in the previous section where the guess is not used during training. Also, we are free to use the entire guess, including the center word, compared to the reference where we have to remove the center word.
At training time, the dual attention model takes 3 inputs, that is, the source, the guess and the reference. At test time, the reference input is replaced by the guess. Specifically, the model Fdual(x,yg,yref) ∈ [0; 1]|yref|×|Y| estimates P ( yiref |x,yg,y −i ref ) for each position i in the reference sentence.
Architecture. The model builds upon the single attention model from the previous section by having two attention functions a with distinct parameters. The first function asource takes the source sentence x and the reference context y−iref to produce the source summary for this context asource ( y−i|k,x ) as in the single attention model. The second function aguess takes the guess sentence yg and the reference context y−iref and produces a guess summary for this context aguess ( y−i|k,yg ) . These two summaries are then concatenated with the lookup representation of
the reference context L ( yref −i|k) and input to a final multilayer perceptron followed by a softmax. The reference lookup table contains the only parameters shared by the two attention functions.
Training. This model is trained similarly to the single attention model, the only difference being the conditioning on the guess yg.
Testing. At test time, the reference is unavailable and we replace yref with yg, i.e., the model is given (x,yg,y −i|k g ) to make a prediction at position i. In this case, the distribution shift when going from training to testing is less drastic than in §3 and the model retains access to the whole yg via attention.
Discussion. Compared to the single attention model (§3), this model reduces perplexity from 5.4 to 4.1 on our validation set. Since the dual attention model can attend to all guess words, it can copy any guess word if necessary. In our dataset, 68% of guess words are in the reference and can therefore be copied. This also means that for the remaining 32% of reference tokens the model should not copy. Instead, the model should propose a substitution by itself (§6.1). During testing, the fact that the guess is input twice (x,yg,y −i|k g ) means that the guess and the prediction context always match. This makes the model more conservative in its predictions, suggesting tokens from yg more often than the single attention model. However, as we show in §6, this turns out beneficial in our setting.
5 ITERATIVE REFINEMENT
The models in §3 and §4 suggest word substitutions for each position in the candidate translation yg given the current surrounding context.
Applying a single substitution changes the context of the surrounding words and requires updating the model predictions. We therefore perform multiple rounds of substitution. At each round, the model computes its predictions, then our refinement strategy selects a substitution and performs it unless the strategy decides that it can no longer improve the target sentence. This means that the refinement procedure should be able to (i) prioritize the suggested substitutions, and (ii) decide to stop the iterative process.
We determine the best edit for each position i in yg by selecting the word with the highest probability estimate: yipred = arg maxj∈Y F (x,yg) i,j . Then we compute a confidence score in this prediction s(yg,ypred) i , possibly considering the prediction for the current guess word at the same position.
These scores are used to select the next position to edit, i? = arg maxi s(yg,ypred)i and to stop the iterative process, i.e., when the confidence falls below a validated threshold t. We also limit the number of substitutions to a maximum of N . We consider different heuristics for s,
• Score positions based on the model confidence in yipred, i.e., sconf(yg,ypred) i = F(x,yg) i,yipred .
• Look for high confidence in the suggested substitution yipred and low confidence in the current word yig: spr(yg,ypred) i = F(x,yg) i,yipred × ( 1− F(x,yg)i,y i g ) .
• Train a simple binary classifier taking as input the score of the best predicted word and the current guess word: scl(yg,ypred)i = nn ( logF(x,yg) i,yipred , logF(x,yg) i,yig ) , where nn
is a 2-layer neural network trained to predict whether a substitution leads to an increase in BLEU or not.
We compare the above strategies, different score thresholds t, and the maximum number of modifications per sentence allowed N in §6.2.
6 EXPERIMENTS & RESULTS
We first describe our experimental setup and then discuss our results.
6.1 EXPERIMENTAL SETUP
Data. We perform our experiments on the German-to-English WMT15 task (Bojar et al., 2015) and benchmark our improvements against the output of a phrase-based translation system (PBMT; Koehn et al. 2007) on this language pair. In principle, our approach may start from any initial guess translation. We chose the output of a phrase-based system because it provides a good starting point that can be computed at high speed. This allows us to quickly generate guess translations for the millions of sentences in our training set.
All data was lowercased and numbers were mapped to a single special “number” token. Infrequent tokens were mapped to an “unknown” token which resulted in dictionaries of 120K and 170K words for English and German respectively.
For training we used 3.5M sentence triples (source, reference, and the guess translation output by the PBMT system). A validation set of 180K triples was used for neural network hyper-parameter selection and learning rate scheduling. Finally, two 3K subsets of the validation set were used to train the classifier discussed in §5 and to select the best model architecture (single vs dual attention) and refinement heuristic.
The initial guess translations were generated with phrase-based systems trained on the same training data as our refinement models. We decoded the training data with ten systems, each trained on 90% of the training data in order to decode the remaining 10%. This procedure avoids the bias of generating guess translation with a system that was trained on the same data.
Implementation. All models were implemented in Torch (Collobert et al., 2011) and trained with stochastic gradient descent to minimize the cross-entropy loss.
For the error detection model in §2 we used two temporal convolutions on top of the lookup table, each followed by a tanh non-linearity to compute S(x) and T(yg). The output dimensions of each convolution was set to 256 and the receptive fields spanned 5 words, resulting in final outputs summarizing a context of 9 words.
For the single attention model we set the shared context embedding dimension dimSj = dimTi = 512 and use a context of size k = 4 words to the left and to the right, resulting in a window of size 9 for the source and 8 for the target. The final multilayer perceptron has 2 layers with a hidden dimension of 512, see §3). For the dual attention model we used 2-layer context embeddings (a convolution followed by a linear with a tanh in between), each having output dimension 512, context of size k = 4. The
final multilayer perceptron has 2 layers with a hidden dimension of 1024, see §4). In this setup, we replaced dot-product attention with MLP attention (Bahdanau et al., 2015) as it performed better on the validation set.
All weights were initialized randomly apart from the word embedding layers, which were precomputed with Hellinger Principal Component Analysis (Lebret & Collobert, 2014) applied to the bilingual co-occurrence matrix constructed on the training set. The word embedding dimension was set to 256 for both languages and all models.
6.2 RESULTS
Table 2 compares BLEU of the single and dual attention models (F vs Fdual) over the validation set. It reports the performance for the best threshold t ∈ {0, 0.1, . . . , 1} and the best maximum number of modifications per sentence N ∈ {0, 1, . . . , 10} for the different refinement heuristics. The best performing configuration is Fdual with the product-based heuristic spr thresholded at t = 0.5 for up to N = 5 substitutions. We report test performance of this configuration in table 3. Tables 4, 5 and 6 show examples of system outputs. Overall the system obtains a small but consistent improvement over all the test sets.
Figure 1 (left) plots accuracy versus the number of allowed substitutions and Figure 1 (right) shows the percentage of actually modified tokens. The dual attention model (§4) outperforms single attention (§3). Both models achieve most of improvement by making only 1-2 substitutions per sentence. Thereafter only very few substitutions are made with little impact on BLEU. Figure 1 (right) shows that the models saturate quickly, indicating convergence of the refinement output to a state where the models have no more suggestions.
To isolate the model contribution from the scoring heuristic, we replace the scoring heuristic with an oracle while keeping the rest of the refinement strategy the same. We consider two types of oracle: The full oracle takes the suggested substitution for each position and then selects which single position should be edited or whether to stop editing altogether. This oracle has the potential to find the largest BLEU improvement. The partial oracle does not select the position, it just takes the heuristic suggestion for the current step and decides whether to edit or stop the process. Notice that both oracles have very limited choice, as they are only able to perform substitutions suggested by our model.
Figure 2 reports the performance of our best single and dual attention models compared to both oracles on the validation set; Figure 3 shows the corresponding number of substitutions. The full and partial oracles result in an improvement of +1.7 and +1.09 BLEU over the baseline in the dual attention setting (compared to +0.35 with spr).
In the single-attention setup the oracles yields a higher improvement (+2.37 and +1.3) and they also perform more substitutions. This supports our earlier conjecture (§4) that Fdual is more conservative and prone to copying words from the guess yg compared to the single attention model. While helpful in validation, the cautious nature of the dual model restricts the options of the oracle.
We make several observations. First, word-prediction models provide high-quality substitutions ypred that can lead to a significant improvements in BLEU (despite that both oracles are limited in their choice of ypred). This is supported by the simple heuristic sconf performing very close to more sophisticated strategies (Table 2).
Second, it is important to have a good confidence estimate on whether a substitution will improve BLEU or not. The full oracle, which yields +1.7 BLEU, acts as an estimate to having a realvalued confidence measure and replaces the scoring heuristic s. The partial oracle, yielding +1.09 BLEU, assesses the benefit of having a binary-valued confidence measure. The latter oracle can only prevent our model from making a BLEU-damaging substitution. However, confidence estimation is a difficult task as we found in §2. Finally, we demonstrate that a significant improvement in BLEU can be achieved through very few substitutions. The full and partial oracle modify only 1.69% and 0.99% of tokens, or 0.4 and 0.24 modifications per sentence, respectively. Of course, oracle substitution assumes access to the reference which is not available at test time. At the same time, our oracle is more likely to generate fluent sentences since it only has access to substitutions deemed likely by the model as opposed to an unrestricted oracle that is more likely to suggest improvements leading to unreasonable sentences. Note that our oracles only allow substitutions (no deletions or insertions), and only those that raise BLEU in a monotonic fashion, with each single refinement improving the score of the previous translation.
7 CONCLUSION AND FUTURE WORK
We present a simple iterative decoding scheme for machine translation which is motivated by the inability of existing models to revisit incorrect decoding decisions made in the past. Our models improve an initial guess translation via simple word substitutions over several rounds. At each round, the model has access to the source as well as the output of the previous round, which is an entire translation of the source. This is different to existing decoding algorithms which make predictions based on a limited partial translation and are unable to revisit previous erroneous decoding decisions.
Our results increase translation accuracy by up to 0.4 BLEU on WMT15 German-English translation and modify only 0.6 words per sentence. In our experimental setup we start with the output of a phrase-based translation system but our model is general enough to deal with arbitrary guess translations.
We see several future work avenues from here. Experimenting with different initial guess translations such as the output of a neural translation system, or even the result of a simple dictionary-based word-by-word translation scheme. Also one can envision editing a number of guess translations simultaneously by expanding the dual attention mechanism to attend over multiple guesses.
So far we only experimented with word substitution, one may add deletion, insertion or even swaps of single or multi-word units. Finally, the dual-attention model in §4 may present a good starting point for neural multi-source translation (Schroeder et al., 2009).
ACKNOWLEDGMENTS
The authors wish to thank Marc’Aurelio Ranzato and Sumit Chopra for their advice and comments.
A EXAMPLES

This paper proposes a model for iteratively refining translation hypotheses. This has several benefits, including enabling the translation model to condition not only on “left context”, but also on “right context”, and potentially enabling more rapid and/or accurate decoding. The motivation given is that often translators (and text generators generally) use a process of refinement in generating outputs.
This is an important idea that is not currently playing much of a role in neural net models, so this paper is a welcome contribution. However, while I think this is an important first step, I do feel that the lack of in depth analysis suggests this paper is not quite ready for a final publication version. For example, there are many possible connections to prior work in NLP, MT, and other parts of ML that could better contextualize this work (see specifics below). More substantively, the model in Section 3 could be interpreted as a globally normalized, undirected (~CRF) translation model trained using a pseudo-likelihood objective. In this analysis, the model squarely back in the context of traditional discriminative translation models which used “undirected” features, and the decoding algorithm then looks more like a standard greedy hill-climbing algorithm (albeit with an extra heuristic model for selecting which variable to update), which is also nothing unfamiliar.
My second criticism the limitations of the model are not well discussed. For example, the proposed editing procedure cannot obviously remove or insert a word from a translation. While I think this is a reasonable assumption than can be made for the sake of tractability, it is very unfortunate since missing or extra words (esp. function words) are a common problem in the baseline models that are being used. Second, the standard objections to absolute positional models (vs. relative positional models) seem particularly crucial to bring up in this work, especially since they might make some of the design decisions a bit more justifiable.
Overall, this is an initial step in an interesting direction, but it needs more thorough analysis to demonstrate its value. A more thorough analysis will also likely suggest some important model variants (for example: is a global translation model really the goal? or is a post-editing model that fixes outputs with more complex operations more ideal?)
Related work:
I think that more could be done to put this work in the context of what has come before and what is currently going on in other parts of ML. The idea of iterative refinement has been proposed in other problems that have complex output spaces, for example the DRAW model of Gregor et al. and the conditional adversarial network models used to refine images proposed recently by Isola et al. In NLP, there have been several (stochastic) hill climbing approaches that have been proposed, such as the work on parsing by Zhang and Lei et al. (2014) who use random initial guesses and then do greedy hill climbing using a series of local refinements, the structured prediction cascades of Weiss and Taskar (2009) (not to mention general coarse-to-fine modeling strategies). Finally, in MT, Arun et al. (2009) who use a Gibbs sampler to refine an initial guess to do decoding with a more complex model. The use of an explicit error model is rather novel in the context of correction, but I would point out that although the proposed architecture is different, the discriminative word lexicon models of Mauser et al. (2009) and the neural version of the same by Ha et al. (2014) are similar in spirit. There have also been a number of papers on “automatic post editing”, including the shared task at WMT2016, and there are not only standard test sets and baselines, but also datasets that could actually be used to train a post editing model with human-generated data. Minimally using the techniques they described could be a useful foil for the models presented in this paper.
“the target sentence is also embedded in distributional space via a lookup table” I think “distributional space” is a bit unclear. Maybe “the target sentence is represented in terms of distributed word representations via a lookup table” or something like that. “distributional” suggests that the representations are derived from how the words are distributed in the corpus, whereas you are learning these representations on this task which isn’t modeling their distribution except only very indirectly.
Section 3 Model: In Section 3, the model computes the distribution over target word types at an absolute position i in the output sentence, given the target language context and the source language context. It is introduced as the model that is used to refine an existing hypothesis, but it is not immediately clear that the training data for this model (at least in this section) are the gold standard translations- “training set” could be interpreted in variety of ways. This becomes clearer when reading later in the paper, but it’s a bit less clear when reading from the beginning for the first time.
The use of a fixed sized window for representing the target word in context also seems to make something like a model 1 assumption since only the lexical features (and not any “alignment” or “positional” features) determine the attention. This should be clarified since it will make the assumptions of the model more transparent (and also suggest possible refinements to the model, e.g., including (representations) of i and j as components of S^j and T^i, which would allow model 2/3-like responses to be learned- although by leaving them out, the model might behave a bit more like a relative positional model than an absolute positional model, which is probably attractive).
Finally, some discussion for why a fixed window is used to represent the target sentence is worth including (since a global context is apparently used to represent the source sentence).
The relationship between this training objective and pseudo likelihood (PL; Besag, 1975) might be worth mentioning. Since I believe this is just a PL objective for a certain global model, this suggests alternative decoding algorithms, or certainly a different analysis of the proposed decoding objective.
The section 4 model conditions on the true context of a position in the true target, the current target guess, and the source. I don’t completely understand the rationale for this model since at test time only two of these variables are available, and the replacement of y_ref with y_g seems hard to justify.