The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say 32–512 data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions—and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say 32–512 data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions—and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.
1 INTRODUCTION
Deep Learning has emerged as one of the cornerstones of large-scale machine learning. Deep Learning models are used for achieving state-of-the-art results on a wide variety of tasks including computer vision, natural language processing and reinforcement learning; see (Bengio et al., 2016) and the references therein. The problem of training these networks is one of non-convex optimization. Mathematically, this can be represented as:
min x∈Rn
f(x) := 1
M M∑ i=1 fi(x), (1)
where fi is a loss function for data point i ∈ {1, 2, · · · ,M} which captures the deviation of the model prediction from the data, and x is the vector of weights being optimized. The process of optimizing this function is also called training of the network. Stochastic Gradient Descent (SGD) (Bottou, 1998; Sutskever et al., 2013) and its variants are often used for training deep networks.
∗Work was performed when author was an intern at Intel Corporation
These methods minimize the objective function f by iteratively taking steps of the form:
xk+1 = xk − αk
( 1
|Bk| ∑ i∈Bk ∇fi(xk)
) , (2)
whereBk ⊂ {1, 2, · · · ,M} is the batch sampled from the data set and αk is the step size at iteration k. These methods can be interpreted as gradient descent using noisy gradients, which and are often referred to as mini-batch gradients with batch size |Bk|. SGD and its variants are employed in a small-batch regime, where |Bk| M and typically |Bk| ∈ {32, 64, · · · , 512}. These configurations have been successfully used in practice for a large number of applications; see e.g. (Simonyan & Zisserman, 2014; Graves et al., 2013; Mnih et al., 2013). Many theoretical properties of these methods are known. These include guarantees of: (a) convergence to minimizers of strongly-convex functions and to stationary points for non-convex functions (Bottou et al., 2016), (b) saddle-point avoidance (Ge et al., 2015; Lee et al., 2016), and (c) robustness to input data (Hardt et al., 2015).
Stochastic gradient methods have, however, a major drawback: owing to the sequential nature of the iteration and small batch sizes, there is limited avenue for parallelization. While some efforts have been made to parallelize SGD for Deep Learning (Dean et al., 2012; Das et al., 2016; Zhang et al., 2015), the speed-ups and scalability obtained are often limited by the small batch sizes. One natural avenue for improving parallelism is to increase the batch size |Bk|. This increases the amount of computation per iteration, which can be effectively distributed. However, practitioners have observed that this leads to a loss in generalization performance; see e.g. (LeCun et al., 2012). In other words, the performance of the model on testing data sets is often worse when trained with largebatch methods as compared to small-batch methods. In our experiments, we have found the drop in generalization (also called generalization gap) to be as high as 5% even for smaller networks.
In this paper, we present numerical results that shed light into this drawback of large-batch methods. We observe that the generalization gap is correlated with a marked sharpness of the minimizers obtained by large-batch methods. This motivates efforts at remedying the generalization problem, as a training algorithm that employs large batches without sacrificing generalization performance would have the ability to scale to a much larger number of nodes than is possible today. This could potentially reduce the training time by orders-of-magnitude; we present an idealized performance model in the Appendix C to support this claim.
The paper is organized as follows. In the remainder of this section, we define the notation used in this paper, and in Section 2 we present our main findings and their supporting numerical evidence. In Section 3 we explore the performance of small-batch methods, and in Section 4 we briefly discuss the relationship between our results and recent theoretical work. We conclude with open questions concerning the generalization gap, sharp minima, and possible modifications to make large-batch training viable. In Appendix E, we present some attempts to overcome the problems of large-batch training.
1.1 NOTATION
We use the notation fi to denote the composition of loss function and a prediction function corresponding to the ith data point. The vector of weights is denoted by x and is subscripted by k to denote an iteration. We use the term small-batch (SB) method to denote SGD, or one of its variants like ADAM (Kingma & Ba, 2015) and ADAGRAD (Duchi et al., 2011), with the proviso that the gradient approximation is based on a small mini-batch. In our setup, the batch Bk is randomly sampled and its size is kept fixed for every iteration. We use the term large-batch (LB) method to denote any training algorithm that uses a large mini-batch. In our experiments, ADAM is used to explore the behavior of both a small or a large batch method.
2 DRAWBACKS OF LARGE-BATCH METHODS
2.1 OUR MAIN OBSERVATION
As mentioned in Section 1, practitioners have observed a generalization gap when using large-batch methods for training deep learning models. Interestingly, this is despite the fact that large-batch methods usually yield a similar value of the training function as small-batch methods. One may put
forth the following as possible causes for this phenomenon: (i) LB methods over-fit the model; (ii) LB methods are attracted to saddle points; (iii) LB methods lack the explorative properties of SB methods and tend to zoom-in on the minimizer closest to the initial point; (iv) SB and LB methods converge to qualitatively different minimizers with differing generalization properties. The data presented in this paper supports the last two conjectures.
The main observation of this paper is as follows:
The lack of generalization ability is due to the fact that large-batch methods tend to converge to sharp minimizers of the training function. These minimizers are characterized by a significant number of large positive eigenvalues in ∇2f(x), and tend to generalize less well. In contrast, small-batch methods converge to flat minimizers characterized by having numerous small eigenvalues of∇2f(x). We have observed that the loss function landscape of deep neural networks is such that large-batch methods are attracted to regions with sharp minimizers and that, unlike small-batch methods, are unable to escape basins of attraction of these minimizers.
The concept of sharp and flat minimizers have been discussed in the statistics and machine learning literature. (Hochreiter & Schmidhuber, 1997) (informally) define a flat minimizer x̄ as one for which the function varies slowly in a relatively large neighborhood of x̄. In contrast, a sharp minimizer x̂ is such that the function increases rapidly in a small neighborhood of x̂. A flat minimum can be described with low precision, whereas a sharp minimum requires high precision. The large sensitivity of the training function at a sharp minimizer negatively impacts the ability of the trained model to generalize on new data; see Figure 1 for a hypothetical illustration. This can be explained through the lens of the minimum description length (MDL) theory, which states that statistical models that require fewer bits to describe (i.e., are of low complexity) generalize better (Rissanen, 1983). Since flat minimizers can be specified with lower precision than to sharp minimizers, they tend to have better generalization performance. Alternative explanations are proffered through the Bayesian view of learning (MacKay, 1992), and through the lens of free Gibbs energy; see e.g. Chaudhari et al. (2016).
2.2 NUMERICAL EXPERIMENTS
In this section, we present numerical results to support the observations made above. To this end, we make use of the visualization technique employed by (Goodfellow et al., 2014b) and a proposed heuristic metric of sharpness (Equation (4)). We consider 6 multi-class classification network configurations for our experiments; they are described in Table 1. The details about the data sets and network configurations are presented in Appendices A and B respectively. As is common for such problems, we use the mean cross entropy loss as the objective function f .
The networks were chosen to exemplify popular configurations used in practice like AlexNet (Krizhevsky et al., 2012) and VGGNet (Simonyan & Zisserman, 2014). Results on other networks
and using other initialization strategies, activation functions, and data sets showed similar behavior. Since the goal of our work is not to achieve state-of-the-art accuracy or time-to-solution on these tasks but rather to characterize the nature of the minima for LB and SB methods, we only describe the final testing accuracy in the main paper and ignore convergence trends.
For all experiments, we used 10% of the training data as batch size for the large-batch experiments and 256 data points for small-batch experiments. We used the ADAM optimizer for both regimes. Experiments with other optimizers for the large-batch experiments, including ADAGRAD (Duchi et al., 2011), SGD (Sutskever et al., 2013) and adaQN (Keskar & Berahas, 2016), led to similar results. All experiments were conducted 5 times from different (uniformly distributed random) starting points and we report both mean and standard-deviation of measured quantities. The baseline performance for our setup is presented Table 2. From this, we can observe that on all networks, both approaches led to high training accuracy but there is a significant difference in the generalization performance. The networks were trained, without any budget or limits, until the loss function ceased to improve.
We emphasize that the generalization gap is not due to over-fitting or over-training as commonly observed in statistics. This phenomenon manifest themselves in the form of a testing accuracy curve that, at a certain iterate peaks, and then decays due to the model learning idiosyncrasies of the training data. This is not what we observe in our experiments; see Figure 2 for the training–testing curve of the F2 and C1 networks, which are representative of the rest. As such, early-stopping heuristics aimed at preventing models from over-fitting would not help reduce the generalization gap. The difference between the training and testing accuracies for the networks is due to the specific choice of the network (e.g. AlexNet, VGGNet etc.) and is not the focus of this study. Rather, our goal is to study the source of the testing performance disparity of the two regimes, SB and LB, on a given network model.
2.2.1 PARAMETRIC PLOTS
We first present parametric 1-D plots of the function as described in (Goodfellow et al., 2014b). Let x?s and x ? ` indicate the solutions obtained by running ADAM using small and large batch sizes respectively. We plot the loss function, on both training and testing data sets, along a line-segment containing the two points. Specifically, for α ∈ [−1, 2], we plot the function f(αx?` + (1 − α)x?s) and also superimpose the classification accuracy at the intermediate points; see Figure 31. For this
1The code to reproduce the parametric plot on exemplary networks can be found in our GitHub repository: https://github.com/keskarnitish/large-batch-training.
experiment, we randomly chose a pair of SB and LB minimizers from the 5 trials used to generate the data in Table 2. The plots show that the LB minima are strikingly sharper than the SB minima in this one-dimensional manifold. The plots in Figure 3 only explore a linear slice of the function, but in Figure 7 in Appendix D, we plot f(sin(απ2 )x ? ` + cos( απ 2 )x ? s) to monitor the function along a curved path between the two minimizers . There too, the relative sharpness of the minima is evident.
2.2.2 SHARPNESS OF MINIMA
So far, we have used the term sharp minimizer loosely, but we noted that this concept has received attention in the literature (Hochreiter & Schmidhuber, 1997). Sharpness of a minimizer can be characterized by the magnitude of the eigenvalues of ∇2f(x), but given the prohibitive cost of this computation in deep learning applications, we employ a sensitivity measure that, although imperfect, is computationally feasible, even for large networks. It is based on exploring a small neighborhood of a solution and computing the largest value that the function f can attain in that neighborhood. We use that value to measure the sensitivity of the training function at the given local minimizer. Now, since the maximization process is not accurate, and to avoid being mislead by the case when a large value of f is attained only in a tiny subspace of Rn, we perform the maximization both in the entire space Rn as well as in random manifolds. For that purpose, we introduce an n× p matrix A, whose columns are randomly generated. Here p determines the dimension of the manifold, which in our experiments is chosen as p = 100.
Specifically, let C denote a box around the solution over which the maximization of f is performed, and letA ∈ Rn×p be the matrix defined above. In order to ensure invariance of sharpness to problem dimension and sparsity, we define the constraint set C as:
C = {z ∈ Rp : − (|(A+x)i|+ 1) ≤ zi ≤ (|(A+x)i|+ 1) ∀i ∈ {1, 2, · · · , p}}, (3)
where A+ denotes the pseudo-inverse of A. Thus controls the size of the box. We can now define our measure of sharpness (or sensitivity). Metric 2.1. Given x ∈ Rn, > 0 and A ∈ Rn×p, we define the (C , A)-sharpness of f at x as:
φx,f ( , A) := (maxy∈C f(x+Ay))− f(x)
1 + f(x) × 100. (4)
Unless specified otherwise, we use this metric for sharpness for the rest of the paper; ifA is not specified, it is assumed to be the identity matrix, In. (We note in passing that, in the convex optimization literature, the term sharp minimum has a different definition (Ferris, 1988), but that concept is not useful for our purposes.)
In Tables 3 and 4, we present the values of the sharpness metric (4) for the minimizers of the various problems. Table 3 explores the full-space (i.e., A = In) whereas Table 4 uses a randomly sampled n × 100 dimensional matrix A. We report results with two values of , (10−3, 5 · 10−4). In all experiments, we solve the maximization problem in Equation (4) inexactly by applying 10 iterations of L-BFGS-B (Byrd et al., 1995). This limit on the number of iterations was necessitated by the
large cost of evaluating the true objective f . Both tables show a 1–2 order-of-magnitude difference between the values of our metric for the SB and LB regimes. These results reinforce the view that the solutions obtained by a large-batch method defines points of larger sensitivity of the training function. In Appedix E, we describe approaches to attempt to remedy this generalization problem of LB methods. These approaches include data augmentation, conservative training and adversarial training. Our preliminary findings show that these approaches help reduce the generalization gap but still lead to relatively sharp minimizers and as such, do not completely remedy the problem.
Note that Metric 2.1 is closely related to the spectrum of ∇2f(x). Assuming to be small enough, when A = In, the value (4) relates to the largest eigenvalue of ∇2f(x) and when A is randomly sampled it approximates the Ritz value of∇2f(x) projected onto the column-space of A.
We conclude this section by noting that the sharp minimizers identified in our experiments do not resemble a cone, i.e., the function does not increase rapidly along all (or even most) directions. By sampling the loss function in a neighborhood of LB solutions, we observe that it rises steeply only along a small dimensional subspace (e.g. 5% of the whole space); on most other directions, the function is relatively flat.
3 SUCCESS OF SMALL-BATCH METHODS
It is often reported that when increasing the batch size for a problem, there exists a threshold after which there is a deterioration in the quality of the model. This behavior can be observed for the F2 and C1 networks in Figure 4. In both of these experiments, there is a batch size (≈ 15000 for F2 and ≈ 500 for C1) after which there is a large drop in testing accuracy. Notice also that the upward drift in value of the sharpness is considerably reduced around this threshold. Similar thresholds exist for the other networks in Table 1.
Let us now consider the behavior of SB methods, which use noisy gradients in the step computation. From the results reported in the previous section, it appears that noise in the gradient pushes the iterates out of the basin of attraction of sharp minimizers and encourages movement towards a flatter minimizer where noise will not cause exit from that basin. When the batch size is greater than the threshold mentioned above, the noise in the stochastic gradient is not sufficient to cause ejection from the initial basin leading to convergence to sharper a minimizer.
To explore that in more detail, consider the following experiment. We train the network for 100 epochs using ADAM with a batch size of 256, and retain the iterate after each epoch in memory. Using these 100 iterates as starting points we train the network using a LB method for 100 epochs and receive a 100 piggybacked (or warm-started) large-batch solutions. We plot in Figure 5 the testing accuracy and sharpness of these large-batch solutions, along with the testing accuracy of the small-batch iterates. Note that when warm-started with only a few initial epochs, the LB method does not yield a generalization improvement. The concomitant sharpness of the iterates also stays high. On the other hand, after certain number of epochs of warm-starting, the accuracy improves and sharpness of the large-batch iterates drop. This happens, apparently, when the SB method has ended its exploration phase and discovered a flat minimizer; the LB method is then able to converge towards it, leading to good testing accuracy.
It has been speculated that LB methods tend to be attracted to minimizers close to the starting point x0, whereas SB methods move away and locate minimizers that are farther away. Our numerical
experiments support this view: we observed that the ratio of ‖x?s − x0‖2 and ‖x?` − x0‖2 was in the range of 3–10.
In order to further illustrate the qualitative difference between the solutions obtained by SB and LB methods, we plot in Figure 6 our sharpness measure (4) against the loss function (cross entropy) for one random trial of the F2 and C1 networks. For larger values of the loss function, i.e., near the initial point, SB and LB method yield similar values of sharpness. As the loss function reduces, the sharpness of the iterates corresponding to the LB method rapidly increases, whereas for the SB method the sharpness stays relatively constant initially and then reduces, suggesting an exploration phase followed by convergence to a flat minimizer.
4 DISCUSSION AND CONCLUSION
In this paper, we present numerical experiments that support the view that convergence to sharp minimizers gives rise to the poor generalization of large-batch methods for deep learning. To this end, we provide one-dimensional parametric plots and perturbation (sharpness) measures for a variety of deep learning architectures. In Appendix E, we describe our attempts to remedy the problem, including data augmentation, conservative training and robust optimization. Our preliminary investigation suggests that these strategies do not correct the problem; they improve the generalization of large-batch methods but still lead to relatively sharp minima. Another prospective remedy includes the use of dynamic sampling where the batch size is increased gradually as the iteration progresses (Byrd et al., 2012; Friedlander & Schmidt, 2012). The potential viability of this approach is suggested by our warm-starting experiments (see Figure 5) wherein high testing accuracy is achieved using a large-batch method that is warm-start with a small-batch method.
Recently, a number of researchers have described interesting theoretical properties of the loss surface of deep neural networks; see e.g. (Choromanska et al., 2015; Soudry & Carmon, 2016; Lee et al., 2016). Their work shows that, under certain regularity assumptions, the loss function of deep learning models is fraught with many local minimizers and that many of these minimizers correspond to a similar loss function value. Our results are in alignment these observations since, in our experiments, both sharp and flat minimizers have very similar loss function values. We do not know, however, if the theoretical models mentioned above provide information about the existence and density of sharp minimizers of the loss surface.
Our results suggest some questions: (a) can one prove that large-batch (LB) methods typically converge to sharp minimizers of deep learning training functions? (In this paper, we only provided some numerical evidence.); (b) what is the relative density of the two kinds of minima?; (c) can one design neural network architectures for various tasks that are suitable to the properties of LB methods?; (d) can the networks be initialized in a way that enables LB methods to succeed?; (e) is it possible, through algorithmic or regulatory means to steer LB methods away from sharp minimizers?
A DETAILS ABOUT DATA SETS
We summarize the data sets used in our experiments in Table 5. TIMIT is a speech recognition data set which is pre-processed using Kaldi (Povey et al., 2011) and trained using a fully-connected network. The rest of the data sets are used without any pre-processing.
B ARCHITECTURE OF NETWORKS
B.1 NETWORK F1
For this network, we use a 784-dimensional input layer followed by 5 batch-normalized (Ioffe & Szegedy, 2015) layers of 512 neurons each with ReLU activations. The output layer consists of 10 neurons with the softmax activation.
B.2 NETWORK F2
The network architecture for F2 is similar to F1. We use a 360-dimensional input layer followed by 7 batch-normalized layers of 512 neurons with ReLU activation. The output layer consists of 1973 neurons with the softmax activation.
B.3 NETWORKS C1 AND C3
The C1 network is a modified version of the popular AlexNet configuration (Krizhevsky et al., 2012). For simplicity, denote a stack of n convolution layers of a filters and a Kernel size of b × c with stride length of d as n×[a, b, c, d]. TheC1 configuration uses 2 sets of [64, 5, 5, 2]–MaxPool(3) followed by 2 dense layers of sizes (384, 192) and finally, an output layer of size 10. We use batchnormalization for all layers and ReLU activations. We also use Dropout (Srivastava et al., 2014) of 0.5 retention probability for the two dense layers. The configuration C3 is identical to C1 except it uses 100 softmax outputs instead of 10.
B.4 NETWORKS C2 AND C4
The C2 network is a modified version of the popular VGG configuration (Simonyan & Zisserman, 2014). TheC3 network uses the configuration: 2×[64, 3, 3, 1], 2×[128, 3, 3, 1], 3×[256, 3, 3, 1], 3× [512, 3, 3, 1], 3× [512, 3, 3, 1] which a MaxPool(2) after each stack. This stack is followed by a 512- dimensional dense layer and finally, a 10-dimensional output layer. The activation and properties of each layer is as in B.3. As is the case with C3 and C1, the configuration C4 is identical to C2 except that it uses 100 softmax outputs instead of 10.
C PERFORMANCE MODEL
As mentioned in Section 1, a training algorithm that operates in the large-batch regime without suffering from a generalization gap would have the ability to scale to much larger number of nodes than is currently possible. Such and algorithm might also improve training time through faster convergence. We present an idealized performance model that demonstrates our goal.
For LB method to be competitive with SB method, the LB method must (i) converge to minimizers that generalize well, and (ii) do it in a reasonably number of iterations, which we analyze here. Let Is and I` be number of iterations required by SB and LB methods to reach the point of comparable test accuracy, respectively. Let Bs and B` be corresponding batch sizes and P be number of processors being used for training. Assume that P < B`, and let fs(P ) be the parallel efficiency of the SB method. For simplicity, we assume that f`(P ), the parallel efficiency of the LB method, is 1.0. In other words, we assume that the LB method is perfectly scalable due to use of a large batch size.
For LB to be faster than SB, we must have
I` B` P < Is Bs Pfs(P ) .
In other words, the ratio of iterations of LB to the iterations of SB should be I` Is < Bs fs(P )B` .
For example, if fs(P ) = 0.2 and Bs/B` = 0.1, the LB method must converge in at most half as many iterations as the SB method to see performance benefits. We refer the reader to (Das et al., 2016) for a more detailed model and a commentary on the effect of batch-size on the performance.
D CURVILINEAR PARAMETRIC PLOTS
The parametric plots for the curvilinear path from x?s to x ? ` , i.e., f(sin( απ 2 )x ? ` + cos( απ 2 )x ? s) can be found in Figure 7.
E ATTEMPTS TO IMPROVE LB METHODS
In this section, we discuss a few strategies that aim to remedy the problem of poor generalization for large-batch methods. As in Section 2, we use 10% as the percentage batch-size for large-batch experiments and 256 for small-batch methods. For all experiments, we use ADAM as the optimizer irrespective of batch-size.
E.1 DATA AUGMENTATION
Given that large-batch methods appear to be attracted to sharp minimizers, one can ask whether it is possible to modify the geometry of the loss function so that it is more benign to large-batch methods. The loss function depends both on the geometry of the objective function and to the size and properties of the training set. One approach we consider is data augmentation; see e.g. (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014). The application of this technique is domain specific but generally involves augmenting the data set through controlled modifications on the training data. For instance, in the case of image recognition, the training set can be augmented through translations, rotations, shearing and flipping of the training data. This technique leads to regularization of the network and has been employed for improving testing accuracy on several data sets.
In our experiments, we train the 4 image-based (convolutional) networks using aggressive data augmentation and present the results in Table 6. For the augmentation, we use horizontal reflections, random rotations up to 10◦ and random translation of up to 0.2 times the size of the image. It is evident from the table that, while the LB method achieves accuracy comparable to the SB method (also with training data augmented), the sharpness of the minima still exists, suggesting sensitivity to images contained in neither training or testing set. In this section, we exclude parametric plots and sharpness values for the SB method owing to space constraints and the similarity to those presented in Section 2.2.
E.2 CONSERVATIVE TRAINING
In (Li et al., 2014), the authors argue that the convergence rate of SGD for the large-batch setting can be improved by obtaining iterates through the following proximal sub-problem.
xk+1 = arg min x
1 |Bk| ∑ i∈Bk fi(x) + λ 2 ‖x− xk‖22 (5)
The motivation for this strategy is, in the context of large-batch methods, to better utilize a batch before moving onto the next one. The minimization problem is solved inexactly using 3–5 iterations of gradient descent, co-ordinate descent or L-BFGS. (Li et al., 2014) report that this not only improves the convergence rate of SGD but also leads to improved empirical performance on convex machine learning problems. The underlying idea of utilizing a batch is not specific to convex problems and we can apply the same framework for deep learning, however, without theoretical guarantees. Indeed, similar algorithms were proposed in (Zhang et al., 2015) and (Mobahi, 2016) for Deep Learning. The former placed emphasis on parallelization of small-batch SGD and asynchrony while the latter on a diffusion-continuation mechanism for training. The results using the conservative training approach are presented in Figure 7. In all experiments, we solve the problem (5) using 3 iterations of ADAM and set the regularization parameter λ to be 10−3. Again, there is a statistically significant improvement in the testing accuracy of the large-batch method but it does not solve the problem of sensitivity.
E.3 ROBUST TRAINING
A natural way of avoiding sharp minima is through robust optimization techniques. These methods attempt to optimize a worst-case cost as opposed to the nominal (or true) cost. Mathematically, given an > 0, these techniques solve the problem
min x φ(x) := max ‖∆x‖≤ f(x+ ∆x) (6)
Geometrically, classical (nominal) optimization attempts to locate the lowest point of a valley, while robust optimization attempts to lower an –disc down the loss surface. We refer an interested reader to (Bertsimas et al., 2010), and the references therein, for a review of non-convex robust optimization. A direct application of this technique is, however, not feasible in our context since each iteration is prohibitively expensive because it involves solving a large-scale second-order conic program (SOCP).
In the context of Deep Learning, there are two inter-dependent forms of robustness: robustness to the data and robustness to the solution. The former exploits the fact that the function f is inherently a statistical model, while the latter treats f as a black-box function. In (Shaham et al., 2015), the authors prove the equivalence between robustness of the solution (with respect to the data) and adversarial training (Goodfellow et al., 2014a).
Given the partial success of the data augmentation strategy, it is natural to question the efficacy of adversarial training. As described in (Goodfellow et al., 2014a), adversarial training also aims to artificially increase the training set but, unlike randomized data augmentation, uses the model’s sensitivity to construct new examples. Despite its intuitive appeal, in our experiments, we found that this strategy did not improve generalization. Similarly, we observed no generalization benefit from the stability training proposed by (Zheng et al., 2016). In both cases, the testing accuracy, sharpness values and the parametric plots were similar to the unmodified (baseline) case discussed in Section 2. It remains to be seen whether adversarial training (or any other form of robust training) can increase the viability of large-batch training.

The paper is an empirical study to justify that: 1. SGD with smaller batch sizes converges to flatter minima, 2. flatter minima have better generalization ability.
Pros and Cons:
Although there is little novelty in the paper, I think the work is of great value in shedding light into some interesting questions around generalization of deep networks.
Significance:
I think such results may have impact on both theory and practice, respectively by suggesting what assumptions are legitimate for real scenarios for building new theories, or be used heuristically to develop new algorithms with generalization by smart manipulation of mini-batch sizes.
Comments:
Earlier I had some concern about the correctness of a claim made by the authors, which is resolved now. They had claimed their proposed sharpness criterion is scale invariance. They took care of it by removing this claim in the revised version.


We present an approach to sensorimotor control in immersive environments. Our approach utilizes a high-dimensional sensory stream and a lower-dimensional measurement stream. The cotemporal structure of these streams provides a rich supervisory signal, which enables training a sensorimotor control model by interacting with the environment. The model is trained using supervised learning techniques, but without extraneous supervision. It learns to act based on raw sensory input from a complex three-dimensional environment. The presented formulation enables learning without a fixed goal at training time, and pursuing dynamically changing goals at test time. We conduct extensive experiments in threedimensional simulations based on the classical first-person game Doom. The results demonstrate that the presented approach outperforms sophisticated prior formulations, particularly on challenging tasks. The results also show that trained models successfully generalize across environments and goals. A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.
1 INTRODUCTION
Machine learning problems are commonly divided into three classes: supervised, unsupervised, and reinforcement learning. In this view, supervised learning is concerned with learning input-output mappings, unsupervised learning aims to find hidden structure in data, and reinforcement learning deals with goal-directed behavior (Murphy, 2012). Reinforcement learning is compelling because it considers the natural setting of an organism acting in its environment. It is generally taken to comprise a class of problems (learning to act), the mathematical formalization of these problems (maximizing the expected discounted return), and a family of algorithmic approaches (optimizing an objective derived from the Bellman equation) (Kaelbling et al., 1996; Sutton & Barto, 2017).
While reinforcement learning (RL) has achieved significant progress (Mnih et al., 2015), key challenges remain. One is sensorimotor control from raw sensory input in complex and dynamic threedimensional environments, learned directly from experience. Another is the acquisition of general skills that can be flexibly deployed to accomplish a multitude of dynamically specified goals (Lake et al., 2016).
In this work, we propose an approach to sensorimotor control that aims to assist progress towards overcoming these challenges. Our approach departs from the reward-based formalization commonly used in RL. Instead of a monolithic state and a scalar reward, we consider a stream of sensory input {st} and a stream of measurements {mt}. The sensory stream is typically high-dimensional and may include the raw visual, auditory, and tactile input. The measurement stream has lower dimensionality and constitutes a set of data that pertain to the agent’s current state. In a physical system, measurements can include attitude, supply levels, and structural integrity. In a three-dimensional computer game, they can include health, ammunition levels, and the number of adversaries overcome.
Our guiding observation is that the interlocked temporal structure of the sensory and measurement streams provides a rich supervisory signal. Given present sensory input, measurements, and goal, the agent can be trained to predict the effect of different actions on future measurements. Assuming that the goal can be expressed in terms of future measurements, predicting these provides all the information necessary to support action. This reduces sensorimotor control to supervised learning, while supporting learning from raw experience and without extraneous data. Supervision is pro-
vided by experience itself: by acting and observing the effects of different actions in the context of changing sensory inputs and goals.
This approach has two significant benefits. First, in contrast to an occasional scalar reward assumed in traditional RL, the measurement stream provides rich and temporally dense supervision that can stabilize and accelerate training. While a sparse scalar reward may be the only feedback available in a board game (Tesauro, 1994; Silver et al., 2016), a multidimensional stream of sensations is a more appropriate model for an organism that is learning to function in an immersive environment (Adolph & Berger, 2006).
The second advantage of the presented formulation is that it supports training without a fixed goal and pursuing dynamically specified goals at test time. Assuming that the goal can be expressed in terms of future measurements, the model can be trained to take the goal into account in its prediction of the future. At test time, the agent can predict future measurements given its current sensory input, measurements, and goal, and then simply select the action that best suits its present goal.
We evaluate the presented approach in immersive three-dimensional simulations that require visually navigating a complex three-dimensional environment, recognizing objects, and interacting with dynamic adversaries. We use the classical first-person game Doom, which introduced immersive three-dimensional games to popular culture (Kushner, 2003). The presented approach is given only raw visual input and the statistics shown to the player in the game, such as health and ammunition levels. No human gameplay is used, the model trains on raw experience.
Experimental results demonstrate that the presented approach outperforms state-of-the-art deep RL models, particularly on complex tasks. Experiments further demonstrate that models learned by the presented approach generalize across environments and goals, and that the use of vectorial measurements instead of a scalar reward is beneficial. A model trained with the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which took place in previously unseen environments. The presented approach outperformed the second best submission, which employed a substantially more complex model and additional supervision during training, by more than 50%.
2 BACKGROUND
The supervised learning (SL) perspective on learning to act by interacting with the environment dates back decades. Jordan & Rumelhart (1992) analyze this approach, review early work, and argue that the choice of SL versus RL should be guided by the characteristics of the environment. Their analysis suggests that RL may be more efficient when the environment provides only a sparse scalar reward signal, whereas SL can be advantageous when temporally dense multidimensional feedback is available.
Sutton (1988) analyzed temporal-difference (TD) learning and argued that it is preferable to SL for prediction problems in which the correctness of the prediction is revealed many steps after the prediction is made. Sutton’s influential analysis assumes a sparse scalar reward. TD and policy gradient methods have since come to dominate the study of sensorimotor learning (Kober et al., 2013; Mnih et al., 2015; Sutton & Barto, 2017). While the use of SL is natural in imitation learning (LeCun et al., 2005; Ross et al., 2013) or in conjunction with model-based RL (Levine & Koltun, 2013), the formulation of sensorimotor learning from raw experience as supervised learning is rare (Levine et al., 2016). Our work suggests that when the learner is exposed to dense multidimensional sensory feedback, direct future prediction can support effective sensorimotor coordination in complex dynamic environments.
Our approach has similarities to Monte Carlo methods. The convergence of such methods was analyzed early on and they were seen as theoretically advantageous, particularly when function approximators are used (Bertsekas, 1995; Sutton, 1995; Singh & Sutton, 1996). The choice of TD learning over Monte Carlo methods was argued on practical grounds, based on empirical performance on canonical examples (Sutton, 1995). While the understanding of the convergence of both types of methods has since improved (Szepesvári & Littman, 1999; Tsitsiklis, 2002; Even-Dar & Mansour, 2003), the argument for TD versus Monte Carlo is to this day empirical (Sutton & Barto, 2017). Sharp negative examples exist (Bertsekas, 2010). Our work deals with the more general setting of vectorial feedback and parameterized goals, and shows that a simple Monte-Carlo-type method performs extremely well in a compelling instantiation of this setting.
Vector-valued feedback has been considered in the context of multi-objective decision-making (Gábor et al., 1998; Roijers et al., 2013). Transfer across related tasks has been analyzed by Konidaris et al. (2012). Parameterized goals have been studied in the context of continuous motor skills such as throwing darts at a target (da Silva et al., 2012; Kober et al., 2012; Deisenroth et al., 2014). A general framework for sharing value function approximators across both states and goals has been described by Schaul et al. (2015). Our work is most closely related to the framework of Schaul et al. (2015), but presents a specific formulation in which goals are defined in terms of intrinsic measurements and control is based on direct future prediction. We provide an architecture that handles realistic sensory and measurement streams and achieves state-of-the-art performance in complex and dynamic three-dimensional environments.
Learning to act in simulated environments has been the focus of significant attention following the successful application of deep RL to Atari games by Mnih et al. (2015). A number of recent efforts applied related ideas to three-dimensional environments. Lillicrap et al. (2016) considered continuous and high-dimensional action spaces and learned control policies in the TORCS simulator. Mnih et al. (2016) described asynchronous variants of deep RL methods and demonstrated navigation in a three-dimensional labyrinth. Oh et al. (2016) augmented deep Q-networks with external memory and evaluated their performance on a set of tasks in Minecraft. In a recent technical report, Kulkarni et al. (2016b) proposed end-to-end training of successor representations and demonstrated navigation in a Doom-based environment. In another recent report, Blundell et al. (2016) considered a nonparametric approach to control and conducted experiments in a three-dimensional labyrinth. Experiments reported in Section 4 demonstrate that our approach significantly outperforms state-ofthe-art deep RL methods.
Prediction of future states in dynamical systems was considered by Littman et al. (2001) and Singh et al. (2003). Predictive representations in the form of generalized value functions were advocated by Sutton et al. (2011). More recently, Oh et al. (2015) learned to predict future frames in Atari games. Prediction of full sensory input in realistic three-dimensional environments remains an open challenge, although significant progress is being made (Mathieu et al., 2016; Finn et al., 2016; Kalchbrenner et al., 2016). Our work considers prediction of future values of meaningful measurements from rich sensory input and shows that such prediction supports effective sensorimotor control.
3 MODEL
Consider an agent that interacts with the environment over discrete time steps. At each time step t, the agent receives an observation ot and executes an action at based on this observation. We assume that the observations have the following structure: ot = 〈st,mt〉, where st is raw sensory input and mt is a set of measurements. In our experiments, st is an image: the agent’s view of its threedimensional environment. More generally, st can include input from multiple sensory modalities. The measurements mt can indicate the attitude, supply levels, and structural integrity in a physical system, or health, ammunition, and score in a computer game.
The distinction between sensory input st and measurements mt is somewhat artificial: both st and mt constitute sensory input in different forms. In our model, the measurement vector mt is distinguished from other sensations in two ways. First, the measurement vector is the part of the observation that the agent will aim to predict. At present, predicting full sensory streams is beyond our capabilities (although see the work of Kalchbrenner et al. (2016) and van den Oord et al. (2016) for impressive recent progress). We therefore designate a subset of sensations as measurements that will be predicted. Second, we assume that the agent’s goals can be defined in terms of future measurements. Specifically, let τ1, . . . , τn be a set of temporal offsets and let f = 〈mt+τ1 −mt, . . . ,mt+τn −mt〉 be the corresponding differences of future and present measurements. We assume that any goal that the agent will pursue can be defined as maximization of a function u(f ;g). Any parametric function can be used. Our experiments use goals that are expressed as linear combinations of future measurements:
u(f ;g) = g>f , (1)
where the vector g parameterizes the goal and has the same dimensionality as f . This model generalizes the standard reinforcement learning formulation: the scalar reward signal can be viewed as a measurement, and exponential decay is one possible configuration of the goal vector.
To predict future measurements, we use a parameterized function approximator, denoted by F :
pat = F (ot, a,g;θ). (2)
Here a ∈ A is an action, θ are the learned parameters of F , and pat is the resulting prediction. The dimensionality of pat matches the dimensionality of f and g. Note that the prediction is a function of the current observation, the considered action, and the goal. At test time, given learned parameters θ, the agent can choose the action that yields the best predicted outcome:
at = argmax a∈A
g>F (ot, a,g;θ). (3)
The goal vector used at test time need not be identical to any goal seen during training.
3.1 TRAINING
The predictor F is trained on experiences collected by the agent. Starting with a random policy, the agent begins to interact with its environment. This interaction takes place over episodes that last for a fixed number of time steps or until a terminal event occurs.
Consider a set of experiences collected by the agent, yielding a set D of training examples: D = {〈oi, ai,gi, fi〉}Ni=1. Here 〈oi, ai,gi〉 is the input and fi is the output of example i. The predictor is trained using a regression loss:
L(θ) = N∑ i=1 ‖F (oi, ai,gi;θ)− fi‖2 . (4)
A classification loss can be used for predicting categorical measurements, but this was not necessary in our experiments.
As the agent collects new experiences, the training setD and the predictor used by the agent change. We maintain an experience memory of the M most recent experiences out of which a mini-batch of N examples is randomly sampled for every iteration of the solver. The parameters of the predictor used by the agent are updated after every k new experiences. This setup departs from pure onpolicy training and we have not observed any adverse effect of using a small experience memory. Additional details are provided in Appendix A.
We have evaluated two training regimes:
1. Single goal: the goal vector is fixed throughout the training process. 2. Randomized goals: the goal vector for each episode is generated at random.
In both regimes, the agent follows an ε-greedy policy: it acts greedily according to the current goal with probability 1− ε, and selects a random action with probability ε. The value of ε is initially set to 1 and is decreased during training according to a fixed schedule.
3.2 ARCHITECTURE
The predictor F is a deep network parameterized by θ. The network architecture we use is shown in Figure 1. The network has three input modules: a perception module S(s), a measurement module M(m) and a goal module G(g). In our experiments, s is an image and the perception module S is implemented as a convolutional network. The measurement and goal modules are fully-connected networks. The outputs of the three input modules are concatenated, forming the joint input representation used for subsequent processing:
j = J(s,m,g) = 〈S(s),M(m), G(g)〉. (5)
Future measurements are predicted based on this input representation. The network emits predictions of future measurements for all actions at once. This could be done by a fully-connected module that absorbs the input representation and outputs predictions. However, we found that introducing additional structure into the prediction module enhances its ability to learn the fine differences between the outcomes of different actions. To this end, we build on the ideas of Wang et al. (2016) and
split the prediction module into two streams: an expectation stream E(j) and an action stream A(j). The expectation stream predicts the average of the future measurements over all potential actions. The action stream concentrates on the fine differences between actions: A(j) = 〈 A1(j), . . . , Aw(j) 〉 , where w = |A| is the number of actions. We add a normalization layer at the end of the action stream that ensures that the average of the predictions of the action stream is zero for each future measurement:
Ai(j) = Ai(j)− 1 w w∑ k=1 Ak(j) (6)
for all i. The normalization layer subtracts the average over all actions from each prediction, forcing the expectation stream E to compensate by predicting these average values. The output of the expectation stream has dimensionality dim(f), where f is the vector of future measurements. The output of the action stream has dimensionality w · dim(f). The output of the network is a prediction of future measurements for each action, composed by summing the output of the expectation stream and the normalized action-conditional output of the action stream:
p = 〈pa1 , . . . ,paw〉 = 〈 A1(j) + E(j), . . . , Aw(j) + E(j) 〉 . (7)
The output of the network has the same dimensionality as the output of the action stream.
4 EXPERIMENTS
We evaluate the presented approach in immersive three-dimensional simulations based on the classical game Doom. In these simulations, the agent has a first-person view of the environment and must act based on the same visual information that is shown to human players in the game. To interface with the game engine, we use the ViZDoom platform developed by Kempka et al. (2016). One of the advantages of this platform is that it allows running the simulation at thousands of frames per second on a single CPU core, which enables training models on tens of millions of simulation steps in a single day.
We compare the presented approach to state-of-the-art deep RL methods in four scenarios of increasing difficulty, study generalization across environments and goals, and evaluate the importance of different aspects of the model.
4.1 SETUP
Scenarios. We use four scenarios of increasing difficulty:
D1 Gathering health kits in a square room. (“Basic”)
D2 Gathering health kits and avoiding poison vials in a maze. (“Navigation”)
D3 Defending against adversaries while gathering health and ammunition in a maze. (“Battle”)
D4 Defending against adversaries while gathering health and ammunition in a more complicated maze. (“Battle 2”)
These scenarios are illustrated in Figure 2 and in the supplementary video (http://bit.ly/ 2f9tacZ).
The first two scenarios are provided with the ViZDoom platform. In D1, the agent is in a square room and its health is declining at a constant rate. To survive, it must move around and collect health kits, which are distributed abundantly in the room. This task is easy: as long as the agent learns to avoid walls and keep traversing the room, performance is good. In D2, the agent is in a maze and its health is again declining at a constant rate. Here it must again collect health kits that increase its health, but it must also avoid blue poison vials that decrease health. This task is harder: the agent must learn to traverse irregularly shaped passageways, and to distinguish health kits from poison vials. In both tasks, the agent has access to three binary sub-actions: move forward, turn left, and turn right. Any combination of these three can be used at any given time, resulting in 8 possible actions. The only measurement provided to the agent in these scenarios is health.
The last two scenarios, D3 and D4, are more challenging and were designed by us using elements of the ViZDoom platform. Here the agent is armed and is under attack by alien monsters. The monsters spawn abundantly, move around in the environment, and shoot fireballs at the agent. Health kits and ammunition are sporadically distributed throughout the environment and can be collected by the agent. The environment is a simple maze in D3 and a more complex one in D4. In both scenarios, the agent has access to eight sub-actions: move forward, move backward, turn left, turn right, strafe left, strafe right, run, and shoot. Any combination of these sub-actions can be used, resulting in
256 possible actions. The agent is provided with three measurements: health, ammunition, and frag count (number of monsters killed).
Model. The future predictor network used in our experiments was configured to be as close as possible to the DQN model of Mnih et al. (2015), to ensure a fair comparison. Additional details on the architecture are provided in Appendix A.
Training and testing. The agent is trained and tested over episodes. Each episode terminates after 525 steps (equivalent to 1 minute of real time) or when the agent’s health drops to zero. Statistics reported in figures and tables summarize the final values of respective measurements at the end of episodes.
We set the temporal offsets τ1, . . . , τn of predicted future measurements to 1, 2, 4, 8, 16, and 32 steps in all experiments. Only the latest three time steps contribute to the objective function, with coefficients (0.5, 0.5, 1). More details are provided in Appendix A.
4.2 RESULTS
Comparison to prior work. We have compared the presented approach to three deep RL methods: DQN (Mnih et al., 2015), A3C (Mnih et al., 2016), and DSR (Kulkarni et al., 2016b). DQN is a standard baseline for visuomotor control due to its impressive performance on Atari games. A3C is more recent and is commonly regarded as the state of the art in this area. DSR is described in a recent technical report and we included it because the authors also used the ViZDoom platform in experiments, albeit with a simple task. Further details on the setup of the prior approaches are provided in Appendix B.
The performance of the different approaches during training is shown in Figure 3. In reporting the results of these experiments, we refer to our approach as DFP (direct future prediction). For the first two scenarios, all approaches were trained to maximize health. For these scenarios, Figure 3 reports average health at the end of an episode over the course of training. For the last two scenarios, all approaches were trained to maximize a linear combination of the three normalized measurements (ammo, health, and frags) with coefficients (0.5, 0.5, 1). For these scenarios, Figure 3 reports average frags at the end of an episode. Each presented curve averages information from three independent training runs, and each data point is computed from 3× 50,000 steps of testing. DQN, A3C, and DFP were trained for 50 million steps. The training procedure for DSR is much slower and can only process roughly 1 million simulation steps per day. For this reason, we were only able to evaluate DSR on the Basic scenario and were not able to perform extensive hyperparameter tuning. We report results for this technique after 10 days of training. (This time was sufficient to significantly exceed the number of training steps reported in the experiments of Kulkarni et al. (2016b), but not sufficient to approach the number of steps afforded by the other approaches.)
Table 1 reports the performance of the models after training. Each fully trained model was tested over 1 million simulation steps. The table reports average health at the end of an episode for scenarios D1 and D2, and average frags at the end of an episode for D3 and D4. We also report the average training speed for each approach, in millions of simulation steps per day of training. The performance of the different models is additionally illustrated in the supplementary video (http://bit.ly/2f9tacZ).
In the Basic scenario, DQN, A3C, and DFP all perform well. As reported in Table 1, the performance of A3C and DFP is virtually identical at 97.5%, while DQN reaches 89%. In the more complex Navigation scenario, a significant gap opens up between DQN and A3C; this is consistent with the experiments of Mnih et al. (2016). DFP achieves the best performance in this scenario, with a 25 percentage point advantage during testing. Note that in these first two scenarios, DFP was only given a single measurement per time step (health).
In the more complex Battle and Battle 2 scenarios (D3 and D4), DFP dominates the other approaches. It outperforms A3C at test time by a factor of 6 in D3 and by a factor of 2.5 in D4. Note that the advantage of DFP is particularly significant in the scenarios that provide richer measurements: three measurements per time step in D3 and D4. The effect of multiple measurements is further evaluated in controlled experiments reported below.
Generalization across environments. We now evaluate how the behaviors learned by the presented approach generalize across different environments. To this end, we have created 100 randomly textured versions of the mazes from scenarios D3 and D4. We used 90 of these for training and 10 for testing, with disjoint sets of textures in the training and testing environments. We call these scenarios D3-tx and D4-tx.
Table 2 shows the performance of the approach for different combinations of training and testing regimes. For example, the entry in the D4-tx row of the D3 column shows the performance (in average number of frags at the end of an episode) of a model trained in D3 and tested in D4-tx. Not surprisingly, a model trained in the simple D3 environment does not learn sufficient invariance to surface appearance to generalize well to other environments. Training in the more complex multitexture environment in D4 yields better generalization: the trained model performs well in D3 and exhibits non-trivial performance in D3-tx and D4-tx. Finally, exposing the model to significant variation in surface appearance in D3-tx or D4-tx during training yields very good generalization.
The last column of Table 2 additionally reports the performance of a higher-capacity model trained in D4-tx. This combination is referred to as D4-tx-L. As shown in the table, this model performs even better. The architecture is detailed in Appendix A.
Visual Doom AI Competition. To further evaluate the presented approach, we participated in the Visual Doom AI Competition, held during September 2016. The competition evaluated sensorimotor control models that act based on raw visual input. The competition had the form of a tournament: the submitted agents play multiple games against each other, their performance measured by aggregate frags. The competition included two tracks. The Limited Deathmatch track was held in a known environment that was given to the participants in advance at training time. The Full Deathmatch track evaluated generalization to previously unseen environments and took place in multiple new environments that were not available to the participating teams at training time. We only enrolled in the Full Deathmatch track. Our model was trained using a variant of the D4-tx-L regime.
Our model won, outperforming the second best submission by more than 50%. That submission, described by Lample & Chaplot (2016), constitutes a strong baseline. It is a deep recurrent Q-network that incorporates an LSTM and was trained using reward shaping and extra supervision from the game engine. Specifically, the authors took advantage of the ability provided by the ViZDoom platform to use the internal configuration of the game, including ground-truth knowledge of the presence of enemies in the field of view, during training. The authors’ report shows that this additional supervision improved performance significantly. Our model, which is simpler, achieved even higher performance without such additional supervision.
Goal-agnostic training. We now evaluate the ability of the presented approach to learn without a fixed goal at training time, and adapt to varying goals at test time. These experiments are performed in the Battle scenario. We use three training regimes: (a) fixed goal vector during training, (b) random goal vector with each value sampled uniformly from [0, 1] for every episode, and (c) random goal vector with each value sampled uniformly from [−1, 1] for every episode. More details are provided in Appendix A. Intuitively, in the second regime the agent is instructed to maximize the different measurements, but has no knowledge of their relative importance. The third regime makes no assumptions as to whether the measured quantities are desirable or not.
The results are shown in Table 3. Each group of columns corresponds to a training regime and each row to a different test-time goal. Goals are given by the weights of the three measurements (ammo, health, and frags) in the objective function. The first test-time goal in Table 3 is the goal vector used in the battle scenarios in the prior experiments, the second seeks to maximize the frag count, the third is a pacifist (maximize ammo and health, minimize frags), the fourth seeks to aimlessly drain ammunition, and the fifth aims to maximize health. For each row, each group of columns reports the average value of each of the three measurements at the end of an episode. Note that health level at the end of an episode can be negative if the agent suffered major damage in the pre-terminal step.
We draw two main conclusions. First, on the main task (first row), models trained without knowing the goal in advance (b,c) perform nearly as well as a dedicated model trained specifically for the eventual goal (a). Without knowing the eventual goal during training, the agent performs the task almost as well as when it was specifically trained for it. Second, all models generalize to new goals but not equally well. Models trained with a variety of goals (b,c) generalize much better than a model trained with a fixed goal.
Ablation study. We now perform an ablation study using the D3-tx scenario. Specifically, we evaluate the importance of vectorial feedback versus a scalar reward, and the effect of predicting measurements at multiple temporal offsets. The results are summarized in Table 4. The table reports the performance (in average frags at the end of an episode) of our full model (predicting three measurements at six temporal offsets) and of ablated variants that only predict frags (a scalar reward) and/or only predict at the farthest temporal offset. As the results demonstrate, predicting multiple measurements significantly improves the performance of the learned model, even when it is evaluated by only one of those
measurements. Predicting measurements at multiple future times is also beneficial. This supports the intuition that a dense flow of multivariate measurements is a better training signal than a scalar reward.
5 DISCUSSION
We presented an approach to sensorimotor control in immersive environments. Our approach is simple and demonstrates that supervised learning techniques can be adapted to learning to act in complex and dynamic three-dimensional environments given raw sensory input and intrinsic measurements. The model trains on raw experience, by interacting with the environment without extraneous supervision. Natural supervision is provided by the cotemporal structure of the sensory and measurement streams. Our experiments have demonstrated that this simple approach outperforms sophisticated deep reinforcement learning formulations on challenging tasks in immersive environments. Experiments have further demonstrated that the use of multivariate measurements provides a significant advantage over conventional scalar rewards and that the trained model can effectively pursue new goals not specified during training.
The presented work can be extended in multiple ways that are important for broadening the range of behaviors that can be learned. First, the presented model is purely reactive: it acts based on the current frame only, with no explicit facilities for memory and no test-time retention of internal representations. Recent work has explored memory-based models (Oh et al., 2016) and integrating such ideas with the presented approach may yield substantial advances. Second, significant progress in behavioral sophistication will likely require temporal abstraction and hierarchical organization of learned skills (Barto & Mahadevan, 2003; Kulkarni et al., 2016a). Third, the presented model was developed for discrete action spaces; applying the presented ideas to continuous actions would be interesting (Lillicrap et al., 2016). Finally, predicting features learned directly from rich sensory input can blur the distinction between sensory and measurement streams (Mathieu et al., 2016; Finn et al., 2016; Kalchbrenner et al., 2016).
B BASELINES
We compared our approach to three prior methods: DQN (Mnih et al., 2015), DSR (Kulkarni et al., 2016b), and A3C (Mnih et al., 2016). We used the authors’ implementations of DQN (https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner) and DSR (https://github.com/Ardavans/DSR), and an independent implementation of A3C (https://github.com/muupan/async-rl). For scenarios D1 and D2 we used the change in health as reward. For D3 and D4 we used a linear combination of changes of the three normalized measurements with the same coefficients as for the presented approach: (0.5, 0.5, 1). For DQN and DSR we tested three learning rates: the default one (0.00025) and two alternatives (0.00005 and 0.00002). Other hyperparameters were left at their default values. For A3C, which trains faster, we performed a search over a set of learning rates ({2, 4, 8, 16, 32} · 10−4) for the first two tasks; for the last two tasks we trained 20 models with random learning rates sampled log-uniformly between 10−4 and 10−2 and random β (entropy regularization) sampled log-uniformly between 10−4 and 10−1. For all baselines we report the best results we were able to obtain.

Deep RL (using deep neural networks for function approximators in RL algorithms) have had a number of successes solving RL in large state spaces. This empirically driven work builds on these approaches. It introduces a new algorithm which performs better in novel 3D environments from raw sensory data and allows better generalization across goals and environments. Notably, this algorithm was the winner of the Visual Doom AI competition.
The key idea of their algorithm is to use additional low-dimensional observations (such as ammo or health which is provided by the game engine) as a supervised target for prediction. Importantly, this prediction is conditioned on a goal vector (which is given, not learned) and the current action. Once trained the optimal action for the current state can be chosen as the action that maximises the predicted outcome according the goal. Unlike in successor feature representations, learning is supervised and there is no TD relationship between the predictions of the current state and the next state.
There have been a number of prior works both in predicting future states as part of RL and goal driven function approximators which the authors review in section 2. The key contributions of this work are the focus on Monte Carlo estimation (rather than TD), the use of low-dimensional ‘measurements’ for prediction, the parametrized goals and, perhaps most importantly, the empirical comparison to relevant prior work.
In addition to the comparison with Visual Doom AI, the authors show that their algorithm is able to learn generalizable policies which can respond, without further training, to limited changes in the goal.
The paper is well-communicated and the empirical results compelling and will be of significant interest.
Some minor potential improvements:
There is an approximation in the supervised training as it is making an on-policy assumption but it learns from a replay buffer (with the Monte Carlo regression the expectation of the remainder of the trajectory is assumed to follow the current policy, but is being sampled from episodes generated by prior versions of the policy). This should be discussed.
The algorithm uses additional metadata (the information about which parts of the sensory input are worth predicting) that the compared algorithms do not. I think this, and the limitations of this approach (e.g. it may not work well in a sensory environment if such measurements are not provided) should be mentioned more clearly.