Different neural network architectures, hyperparameters and training protocols lead to different performances as a function of time. Human experts routinely inspect the resulting learning curves to quickly terminate runs with poor hyperparameter settings and thereby considerably speed up manual hyperparameter optimization. The same information can be exploited in automatic hyperparameter optimization by means of a probabilistic model of learning curves across hyperparameter settings. Here, we study the use of Bayesian neural networks for this purpose and improve their performance by a specialized learning curve layer.
Different neural network architectures, hyperparameters and training protocols lead to different performances as a function of time. Human experts routinely inspect the resulting learning curves to quickly terminate runs with poor hyperparameter settings and thereby considerably speed up manual hyperparameter optimization. The same information can be exploited in automatic hyperparameter optimization by means of a probabilistic model of learning curves across hyperparameter settings. Here, we study the use of Bayesian neural networks for this purpose and improve their performance by a specialized learning curve layer.
1 INTRODUCTION
Deep learning has celebrated many successes, but its performance relies crucially on good hyperparameter settings. Bayesian optimization (e.g, Brochu et al. (2010); Snoek et al. (2012); Shahriari et al. (2016)) is a powerful method for optimizing the hyperparameters of deep neural networks (DNNs). However, its traditional treatment of DNN performance as a black box poses fundamental limitations for large and computationally expensive data sets, for which training a single model can take weeks. Human experts go beyond this blackbox notion in their manual tuning and exploit cheaper signals about which hyperparameter settings work well: they estimate overall performance based on runs using subsets of the data or initial short runs to weed out bad parameter settings; armed with these tricks, human experts can often outperform Bayesian optimization.
Recent extensions of Bayesian optimization and multi-armed bandits therefore also drop the limiting blackbox assumption and exploit the performance of short runs (Swersky et al., 2014; Domhan et al., 2015; Li et al., 2017), performance on small subsets of the data (Klein et al., 2017), and performance on other, related data sets (Swersky et al., 2013; Feurer et al., 2015).
While traditional solutions for scalable Bayesian optimization include approximate Gaussian process models (e.g., Hutter et al.; Swersky et al. (2014)) and random forests (Hutter et al., 2011), a recent trend is to exploit the flexible model class of neural networks for this purpose (Snoek et al., 2015; Springenberg et al., 2016). In this paper, we study this model class for the prediction of learning curves. Our contributions in this paper are:
1. We study how well Bayesian neural networks can fit learning curves for various architectures and hyperparameter settings, and how reliable their uncertainty estimates are.
2. Building on the parametric learning curve models of Domhan et al. (2015), we develop a specialized neural network architecture with a learning curve layer that improves learning curve predictions.
3. We compare different ways to generate Bayesian neural networks: probabilistic back propagation (Hernández-Lobato and Adams, 2015) and two different stochastic gradient based Markov Chain Monte Carlo (MCMC) methods – stochastic gradient Langevin dynamics (SGLD (Welling and Teh, 2011)) and stochastic gradient Hamiltonian MCMC (SGHMC (Chen et al., 2014)) – for standard Bayesian neural networks and our specialized architecture and show that SGHMC yields better uncertainty estimates.
4. We evaluate the predictive quality for both completely new learning curves and for extrapolating partially-observed curves, showing better performance than the parametric function approach by Domhan et al. (2015) at stages were learning curves have not yet converged.
5. We extend the recent multi-armed bandit strategy Hyperband (Li et al., 2017) by sampling using our model rather than uniformly at random, thereby enabling it to approach nearoptimal configurations faster than traditional Bayesian optimization.
2 PROBABILISTIC PREDICTION OF LEARNING CURVES
In this section, we describe a general framework to model learning curves of iterative machine learning methods. We first describe the approach by Domhan et al. (2015) which we will dub LCExtrapolation from here on. Afterwards, we discuss a more general joint model across time steps and hyperparameter values that can exploit similarities between hyperparameter configurations and predict for unobserved learning curves. We also study the observation noise of different hyperparameter configurations and show how we can adapt our model to capture this noise.
2.1 LEARNING CURVE PREDICTION WITH BASIS FUNCTION
An intuitive model for learning curves proposed by Domhan et al. (2015) uses a set of k different parametric functions φi(θi, t) ∈ {φ1(θ1, t), ..., φk(θk, t)} to extrapolate learning curves (y1, . . . , yn) from the first n time steps. Each parametric function φi depends on a time step t ∈ [1, T ] and on a parameter vector θi. The individual functions are combined into a single model by a weighted linear combination
f̂(t|Θ,w) = k∑
i=1
wiφi(t,θi) , (1)
where Θ = (θ1, . . . ,θk) denotes the combined vector of all parameters θ1, . . . ,θk, and w = (w1, . . . , wk) is the concatenated vector of the respective weights of each function. Assuming observation noise around the true but unknown value f(t), i.e., assuming yt ∼ N (f̂(t|Θ,w), σ2), Domhan et al. (2015) define a prior for all parameters P (Θ,w, σ2) and use a gradient-free MCMC method (Foreman-Mackey et al., 2013) to obtain S samples, (Θ1,w1, σ21), . . . , (ΘS ,wS , σ 2 S), from the posterior
P (Θ,w, σ2|y1 . . . , yn) ∝ P (y1, . . . , yn|Θ,w, σ2)P (Θ,w, σ2) (2) using the likelihood
P (y1, . . . , yn|Θ,w, σ2) = n∏
t=1
N (yt; f̂(t|Θ,w), σ2) . (3)
These samples then yield probabilistic extrapolations of the learning curve for future time steps m, with mean and variance predictions
ŷm = E[ym|y1, . . . yn] ≈ 1
S
S∑
s=1
f̂(m|Θs,ws) , and
var(ŷm) ≈ 1
S
S∑
s=1
(f̂(m|Θs,ws)− ŷm)2 + S∑
s=1
σ2s .
(4)
For our experiments, we use the original implementation by Domhan et al. (2015) with one modification: the original code included a term in the likelihood that enforced the prediction at t = T to be strictly greater than the last value of that particular curve. This biases the estimation to never underestimate the accuracy at the asymptote. We found that in some of our benchmarks, this led to instabilities, especially with very noisy learning curves. Removing it cured that problem, and we did not observe any performance degradation on any of the other benchmarks.
The ability to include arbitrary parametric functions makes this model very flexible, and Domhan et al. (2015) used it successfully to terminate evaluations of poorly-performing hyperparameters early for various different architectures of neural networks (thereby speeding up Bayesian optimization by a factor of two). However, the model’s major disadvantage is that it does not use previously evaluated hyperparameters at all and therefore can only make useful predictions after observing a substantial initial fraction of the learning curve.
2.2 LEARNING CURVE PREDICTION WITH BAYESIAN NEURAL NETWORKS
In practice, similar hyperparameter configurations often lead to similar learning curves, and modelling this dependence would allow predicting learning curves for new configurations without the need to observe their initial performance. Swersky et al. (2014) followed this approach based on an approximate Gaussian process model. Their Freeze-Thaw method showed promising results for finding good hyperparameters of iterative machine learning algorithms using learning curve prediction to allocate most resources for well-performing configurations during the optimization. The method introduces a special covariance function corresponding to exponentially decaying functions to model the learning curves. This results in an analytically tractable model, but using different functions to account for cases where the learning curves do not converge exponentially is not trivial.
Here, we formulate the problem using Bayesian neural networks. We aim to model the validation accuracy g(x, t) of a configuration x ∈ X ⊂ Rd at time step t ∈ (0, 1] based on noisy observations y(x, t) ∼ N (g(x, t), σ2). For each configuration x trained for Tx time steps, we obtain Tx data points for our model; denoting the combined data by D = {(x1, t1, y11)), (x1, t2, y12), . . . , (xn, Txn , ynTxn )} we can then write the joint probability of the data D and the network weights W as
P (D,W ) = P (W )P (σ2) |D|∏
i=1
N (yi; ĝ(xi, ti|W ), σ2) . (5)
where ĝ(xi, ti|W ) is the prediction of a neural network. It is intractable to compute the posterior weight distribution p(W |D), but we can use MCMC to sample it, in particular stochastic gradient
MCMC methods, such as SGLD (Welling and Teh, 2011) or SGHMC (Chen et al., 2014). Given M samples W 1, . . . ,WM , we can then obtain the mean and variance of the predictive distribution p(g|x, t,D) as
µ̂(x, t|D) = 1 M
M∑
i=1
ĝ(x, t|W i) , and
σ̂2(x, t|D) = 1 M
M∑
i=1
( ĝ(x, t|W i)− µ̂(x, t|D) )2 ,
(6)
respectively. We will write these shorthand as µ̂(x, t) and σ̂2(x, t). This is similar to Eqs. 4 and exactly the model that Springenberg et al. (2016) used for (blackbox) Bayesian optimization with Bayesian neural networks; the only difference is in the input to the model: here, there is a data point for every time step of the curve, whereas Springenberg et al. (2016) only used a single data point per curve (for its final time step).
2.3 HETEROSCEDASTIC NOISE OF HYPERPARAMETER CONFIGURATION
In the model described above, we assume homoscedastic noise across hyperparameter configurations. To evaluate how realistic this assumption is, we sampled 40 configurations of a fully connected network (see Section 3.1 for a more detailed description of how the data was generated and Table 2 for the list of hyperparameters) and evaluated each configuration R = 10 times with different pseudorandom number seeds (see the right panel of Figure 3 for some examples). Figure 3 (left) shows on the vertical axis the noise σ̄2(x, t) = 1R ∑R r=1(y(x, t)− µ̄(x, t))2 and on the horizontal axis the rank of each configuration based on their asymptotic sample mean performance µ̄(x, t = 1) = 1R ∑R r=1 y(x, t = 1).
Maybe not surprisingly, the noise seems to correlate with the asymptotic performance of a configuration. The fact that the noise between different configurations varies on different orders of magnitudes
suggests a heteroscedastic noise model to best describe this behavior. We incorporate this observation by making the noise dependent on the input data to allow to predict different noise levels for different hyperparameters. In principle, one could also model a t dependent noise, but we could not find the same trend across different datasets.
2.4 NEW BASIS FUNCTION LAYER FOR LEARNING CURVE PREDICTION WITH BAYESIAN
NEURAL NETWORKS
weights w, and the noise σ̂2.
We now combine Bayesian neural networks with the parametric functions to incorporate more knowledge about learning curves into the network itself. Instead of obtaining the parameters Θ and w by sampling from the posterior, we use a Bayesian neural network to learn several mappings simultaneously:
1. µ̂∞: X → R, the asymptotic value of the learning curve
2. Θ: X → RK , the parameters of a parametric function model (see Figure 2 for some example curves from our basis functions)
3. w: X → Rk, the corresponding weights for each function in the model
4. σ̂2: X → R+, the observational noise for this hyperparameter configuration
With these quantities, we can compute the likelihood in (3) which allows training the network.
Phrased differently, we use a neural network to predict the model parameters Θ and weightsw of our parametric functions, yielding the following form for our network’s mean predictions:
ĝ(xi, ti|W ) = f̂(ti|Θ(xi,W ),w(xi,W )). (7)
A schematic of this is shown in Fig. 4. For training, we will use the same MCMC methods, namely SGLD and SGHMC mentioned above.
3 EXPERIMENTS
We now empirically evaluate the predictive performance of Bayesian neural networks, with and without our special learning curve layer. For both networks, we used a 3-layer architecture with tanh activations and 64 units per layer. We also evaluate two different sampling methods for both types of networks: stochastic gradient Langevin dynamics (SGLD) and stochastic gradient Hamiltonian MCMC (SGHMC), following the approach of Springenberg et al. (2016) to automatically adapt the noise estimate and the preconditioning of the gradients.
As baselines, we compare to other approaches suitable for this task. Besides the aforementioned work by Domhan et al. (2015), we also compare against random forests Breimann (2001) with empirical variance estimates (Hutter et al., 2014), a Gaussian process (GP) using the learning curve kernel from Swersky et al. (2014), another Bayesian neural network technique called probabilistic back propagation (PBP) (Hernández-Lobato and Adams, 2015), and the simple heuristic of using the last seen value (LastSeenValue) of each learning curve for extrapolation. The last model has been successfully used by Li et al. (2017) despite its simplicity.
3.1 DATASETS
For our empirical evaluation, we generated the following four datasets of learning curves, in each case sampling hyperparameter configurations at random from the hyperparameter spaces detailed in Table 2 in the appendix (see also Section D for some characteristic of these datasets):
• CNN: We sampled 256 configurations of 5 different hyperparameters of a 3-layer convolutional neural network (CNN) and trained each of them for 40 epochs on the CIFAR10 (Krizhevsky, 2009) benchmark. • FCNet: We sampled 4096 configurations of 10 hyperparameters of a 2-layer feed forward
neural network (FCNet) on MNIST (LeCun et al., 2001), with batch normalization, dropout and ReLU activation functions, annealing the learning rate over time according to a power function. We trained the neural network for 100 epochs. • LR: We sampled 1024 configurations of the 4 hyperparameters of logistic regression (LR)
and also trained it for 100 epochs on MNIST. • VAE: We sampled 1024 configuration of the 4 hyperparameters of a variational auto-
encoder (VAE) (Kingma and Welling, 2014). We trained the VAE on MNIST, optimizing the approximation of the lower bound for 300 epochs.
3.2 PREDICTING ASYMPTOTIC VALUES OF PARTIALLY OBSERVED CURVES
We first study the problem of predicting the asymptotic values of partially-observed learning curves tackled by Domhan et al. (2015). The LC-Extrapolation method by Domhan et al. (2015), the GP, and the last seen value work on individual learning curves and do not allow to model performance across hyperparameter configurations. Thus, we trained them separately on individual partial learning curves. The other models, including our Bayesian neural networks, on the other hand, can use training data from different hyperparameter configurations. Here, we used training data with the same number of epochs for every partial learning curve.1
The left panel of Figure 5 visualizes the extrapolation task, showing a learning curve from the CNN dataset and the prediction of the various models trained only using the first 12 of 40 epochs of the learning curve. The right panel shows the corresponding predictive distributions obtained with these models. LastSeenValue does not yield a distribution and uncertainties are not defined.
For a more quantitative evaluation, we used all models to predict the asymptotic value of all learning curves, evaluating predictions based on observing between 10% and 90% of the learning curves.
1We note that when used inside Bayesian optimization, we would have access to a mix of fully-converged and partially-converged learning curves as training data, and could therefore expect better extrapolation performance.
Figure 6 shows the mean squared error between the true asymptotic value and the models’ predictions (left) and the average log-likelihood of the true value given each model as a function of how much of the learning curves has been observed. Note that we removed some methods from the plots to avoid clutter; the same plots with all methods can be found in the supplementary material. We notice several patterns:
1. Throughout, our specialized network architecture performs better than the standard Bayesian neural networks, and SGHMC outperformed SGLD (see Appendix C).
2. PBP shows mixed results for the extrapolated mean, and does not provide reliable uncertainties (also see Appendix C). We hypothesize that this may be due to its linear activation functions.
3. If no uncertainties are required, LastSeenValue is hard to beat. This is because many configurations approach their final performance quite quickly.
4. The GP mean predictions are very competitive, but the average log-likelihood indicates overconfidence, especially for short learning curves. We assume that the prior assumption of an exponential function is not flexible enough in practice and that after observing some data points the Gaussian process becomes too certain of its predictions.
5. The random forest’s mean predictions almost match the quality of the GP. The uncertainty estimates are better, but still too aggressive when only a small fraction of the learning curve was observed. Random forests do not extrapolate and similar to LastSeenValue also achieve a very good mean prediction if the learning curve has almost converged.
6. Local models for single curves clearly outperform global models for almost complete learning curves. Global models, like our BNN approach, on the other hand, are trained on many configurations and need to generalize across these, yielding somewhat worse performance for this (arguably easy) task.2
3.3 PREDICTING UNOBSERVED LEARNING CURVES
As mentioned before, training a joint model across hyperparameters and time steps allows us to make predictions for completely unobserved learning curves of new configurations. To estimate how well Bayesian neural networks perform in this task, we used the datasets from Section 3.1 and split all of them into 16 folds, allowing us to perform cross-validation of the predictive performance. For each fold, we trained all models on the full learning curves in the training set and let them predict the held-out learning curves.
Table 1 shows the mean squared error and the average log-likelihood (both computed for all points in each learning curve) across the 16 folds. We make two observations: firstly, both neural network architectures lead to a reasonable mean squared error and average log-likelihood, for both SGLD and SGHMC. Except on the VAE dataset our learning curve layer seems to improve mean squared error and average log likelihood. Secondly, SGHMC performed better than SGLD, with the latter resulting in predictions with too small variances. Figure 7 visualizes the results for our CNN dataset in more detail, showing that true and predicted accuracies correlate quite strongly.
2In the future, we aim to improve our BNN architecture for this case of partially-observed learning curves by also giving the network access to the partial learning curve it should extrapolate.
3.4 MODEL-BASED HYPERBAND
In this section, we show how our model can be used to improve hyperparameter optimization of iterative machine learning algorithms. For this, we extended the multi-armed bandit strategy Hyperband (Li et al., 2017), which in each iteration i first samples Ni hyperparameter configurations C = {x1, . . . ,xNi} and then uses successive halving (Jamieson and Talwalkar, 2016) to iteratively discard poorly-performing configurations from C. While the original Hyperband method samples configurations C from a uniform distribution U over hyperparameter configurations, our extension instead samples them based on our model, with all other parts remaining unchanged. More precisely, we sample Mi >> Ni configurations uniformly, Ĉ = {x1, . . . ,xMi} ∼ U , and pick the Ni configurations with the highest predicted asymptotic mean a(x) = µ̂(x, t = 1) or upper confidence value a(x) = µ̂(x, t = 1) + σ̂(x, t = 1).
For a thorough empirical evaluation and to reduce computational requirements we reused the data from Section 3.1 to construct a surrogate benchmark based on a random forest (see Appendix E for more details). After each iteration we report the final performance of the best observed configuration so far, along with the wall clock time that would have been needed for optimizing the true objective function. Reporting the (predicted) wall clock time (rather than the number of iterations) also takes our optimizer’s overhead into account. This is important since one may worry that this overhead may be substantial due to our use of a model; however, as we show it does not harm performance.
Figure 8 shows the immediate regret on the CNN benchmark as a function of wallclock time, for three optimizers: Hyperband, our model-based extension of Hyperband, as well as standard Bayesian optimization with a Gaussian process (a comparison to additional baselines can be found in Appendix E). Standard Bayesian optimization does not make use of learning curves and thus needs to evaluate each configuration for the full amount of epochs. Nevertheless, since training one configuration to the end takes less time than one round of successive halving, Bayesian optimization produces its first results earlier than Hyperband. In accordance with results by Li et al. (2017), in this experiment Hyperband found a configuration with good performance faster than standard Bayesian optimization, but its random sampling did not suffice to quickly approach the best configuration; given enough time Bayesian optimization performed better. However, extended by our model, Hyperband both found a good configuration fast and approached the global optimum fastest.
We would like to emphasize that our model-based extension of Hyperband is not limited to our particular Bayesian neural networks as the underlying model. Since Hyperband stops the majority of the configurations very early, the setting is quite similar to that of very short partially observed learning curves in Section 3.2, with the differences that some configurations have been evaluated for longer and that the model now has to generalize to unseen configurations. For example, for the CNN and LR benchmarks, the experiments in Section 3.2 showed that random forests achieve strong average log-likelihoods for very short partial learning curves, and we would therefore also expect them to work well when combined with Hyperband. However, given our model’s more robust likelihoods and mean squared error values we believe it to be less sensitive to the underlying data. Additionally, we can incorporate more prior knowledge about the general shape of the curves into our model, something that is not easily done for many other model classes.
4 CONCLUSION
We studied Bayesian neural networks for modelling the learning curves of iterative machine learning methods, such as stochastic gradient descent for convolutional neural networks. Based on the parametric learning curve models of Domhan et al. (2015), we also developed a specialized neural network architecture with a learning curve layer that improves learning curve predictions. In future work, we aim to study recurrent neural networks for predicting learning curves and will extend Bayesian optimization methods with Bayesian neural networks (Springenberg et al., 2016) based on our learning curve models.
ACKNOWLEDGMENT
This work has partly been supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme under grant no. 716721, by the European Commission under grant no. H2020-ICT-645403-ROBDREAM, and by the German Research Foundation (DFG) under Priority Programme Autonomous Learning (SPP 1527, grant HU 1900/3-1).
A EXPERIMENTAL SETUP – DETAILS
Table 2 shows the hyperparameters of our 4 benchmarks described in Section 3.1 and their ranges.
B DESCRIPTION OF THE BASIS FUNCTIONS
To reduce complexity, we used a subset of the basis function from Domhan et al. (2015) which we found to be sufficient for learning curve prediction. We adapted these functions to model the residual between the asymptotic value y∞ and we scaled the parameters Θ to be in [0, 1]. Table 3 shows the exact equations we used.
C PREDICTING ASYMPTOTIC VALUES OF PARTIALLY OBSERVED CURVES
Figure 9 shows the mean squared error and the average log-likelihood of predicting the asymptotic value after observing different amounts of the learning curve for all methods. See Section 3.2 in the main text for more details.
D DATASET CHARACTERISTICS
Figure 10 shows the distributions over runtimes for all random configurations of different benchmarks. As it can be seen there is a high variance of the runtime between different configurations for all benchmarks and some configurations need order of magnitudes longer than others.
Figure 11 shows the empirical cumulative distribution of the asymptotic performance of random configurations. These plots give an intuition about the difficulty of a benchmark as they show how many random configurations one has to sample in order to achieve a good performance.
E OPTIMIZATION ON SURROGATE BENCHMARKS
We follow the approach by Eggensperger et al. (2015) and used the generated datasets from Section 3.1 to build surrogates of the objective functions. This allows us to compare different optimizers on this benchmark very efficiently while (approximately) preserving the characteristics of the underlying benchmark. Using surrogates instead of optimizing the real benchmarks allows us to carry out a more thorough empirical evaluation (since single function evaluations are cheap and we therefore can afford more and longer optimization runs). We used random forests as surrogates following Eggensperger et al. (2015) since they do not introduce a bias for our approach (in contrast to, e.g., surrogates based on standard feed forward neural networks trained with stochastic gradient descent ). For each benchmark, we used all configurations to train a random forest predicting the validation accuracy of a configuration at a certain time step as well as the wall clock time for its evaluation.
After each round of successive halving, we return the current best observed configuration and its asymptotic performance. For each function evaluation, we predict the accuracy, and the runtime. By adding the optimization overhead, we can predict the wallclock time necessary to optimize the real objective function. Reporting the wallclock time rather than the number of iterations also takes the additional overhead from running our method into account. Furthermore, we argue that the wallclock time is more interesting in practice.
As baselines, we compare to Gaussian process based Bayesian optimization with the upper confidence bound acquisition function (GP-BO-UCB) and the (log) expected improvement acquisition function (GP-BO-Log-EI), as well as Bayesian optimization with Bayesian neural networks (BOHAMIANN) (Springenberg et al., 2016). As an additional baseline, we use standard Bayesian neural networks (SGHMC-BNN) inside Hyperband instead of our specialized neural network architecture (SGHMCLCNet). We compare our model-based version of Hyperband against all these baselines with sampling from the asymptotic mean (SGHMC-LCNet-mean-sampling) and the asymptotic upper confidence bound (SGHMC-LCNet-ucb-sampling). Figure 12 shows that our specialized neural network architecture helps to speed up the convergence of Hyperband on the CNN and the FCNet benchmark. For the LR benchmarks our model still improves over Hyperband but does not perform better than Bayesian optimization. Hyperband as well as our method find the optimum already after the first round of successive halving for the VAE benchmark, which leads to an regret of 0.

This paper is about using Bayesian neural networks to model learning curves (that arise from training ML algorithms). The application is hyper-parameter optimization: if we can model the learning curve, we can terminate bad runs early and save time. The paper builds on existing work that used parametric learning curves. Here, the parameters of these learning curves form the last layer of a Bayesian neural network. This seems like a totally sensible idea.
I think the main strength of this paper is that it addresses an actual need. Based on my personal experience, there is high demand for a working system to do early termination in hyperparameter optimization. What I'd like to know, which I wish I'd asked during pre-review questions, is whether the authors plan to release their code. Do you? I sincerely hope so, because I think the code would be a significant part of the paper's contribution, since the nature of the paper is more practical than conceptual.
The experiments in the paper seem thorough but the results are a bit underwhelming. I'm less interested in the part about whether the learning curves are actually modeled well, and more interested in the impact on hyperparameter optimization. I was hoping to see BIG speedups as a result of using this method, but I am left feeling unsure how big the speedup really is. Instead of "objective function vs. iterations" I would be more interested in the inverse plot: number of iterations needed to get to a fixed objective function value. Since what I'm really interested in is how much time I can save. Ideally there would also be some mention of real time as sometimes these hyperparameter optimization methods are themselves so slow that they end up being unusable.
Finally, one figure that I feel is missing is a histogram of termination times over different runs. This would provide me with more intuition than all the other figures. Because it would tell me, what fraction of runs are being terminated early. And, how early? Right now I have no sense of this, except that at least *some* runs are clearly being terminated early, since this is neccessary for the proposed method to outperform other methods.
Overall, I think this paper merits acceptance because it is a solid effort on an interesting problem. The progress is fairly incremental but I can live with that.


GENERALIZING SKILLS WITH SEMI-SUPERVISED REINFORCEMENT LEARNING
Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while it is deployed and performing useful tasks. However, this learning requires access to a reward function, to tell the agent whether it is succeeding or failing at its task. Such reward functions are often hard to measure in the real world, especially in domains such as robotics and dialog systems, where the reward could depend on the unknown positions of objects or the emotional state of the user. On the other hand, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present, or in a controlled laboratory setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect in the unstructured real world? In this paper, we formalize this problem setting as semisupervised reinforcement learning (SSRL), where the reward function can only be evaluated in a set of “labeled” MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of “unlabeled” MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent’s own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging, continuous control tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.
1 INTRODUCTION
Reinforcement learning (RL) provides a powerful framework for learning behavior from highlevel goals. RL has been combined with deep networks to learn policies for problems such as Atari games (Mnih et al., 2015), simple Minecraft tasks (Oh et al., 2016), and simulated locomotion (Schulman et al., 2015). To apply reinforcement learning (RL) to real-world scenarios, however, the learned policy must be able to handle the variability of the real-world and generalize to scenarios that it has not seen previously. In many such domains, such as robotics and dialog systems, the variability of the real-world poses a significant challenge. Methods for training deep, flexible models combined with massive amounts of labeled data are known to enable wide generalization for supervised learning tasks (Russakovsky et al., 2015). Lifelong learning aims to address this data challenge in the context of RL by enabling the agent to continuously learn as it collects new experiences “on the job,” directly in the real world (Thrun & Mitchell, 1995). However, this learning requires access to a reward function, to tell the agent whether it is succeeding or failing at its task. Although the reward is a high-level supervision signal that is in principle easier to provide than detailed labels, in practice it often depends on information that is extrinsic to the agent and is therefore difficult to measure in the real world. For example, in robotics, the reward may depend on the poses of all
of the objects in the environment, and in dialog systems, the reward may depend on the happiness of the user. This reward supervision is practical to measure in a small set of instrumented training scenarios, in laboratory settings, or under the guidance of a human teacher, but quickly becomes impractical to provide continuously to a lifelong learning system, when the agent is deployed in varied and diverse real-world settings.
Conceptually, we might imagine that this challenge should not exist, since reinforcement learning should, at least in principle, be able to handle high-level delayed rewards that can always be measured. For example, a human or animal might have their reward encode some higher-level intrinsic goals such as survival, reproduction, or the absence of pain and hunger. However, most RL methods do not operate at the level of such extremely sparse and high-level rewards, and most of the successes of RL have been in domains with natural sources of detailed external feedback, such as the score in a video game. In most real-world scenarios, such a natural and convenient score typically does not exist. It therefore seems that intelligent agents in the real world should be able to cope with only partial reward supervision, and that algorithms that enable this are of both of practical and conceptual value, since they bring us closer to real-world lifelong reinforcement learning, and can help us understand adaptive intelligent systems that can learn even under limited supervisory feedback. So how can an agent continue to learn in the real world without access to a reward function?
In this work, we formalize this as the problem of semi-supervised reinforcement learning, where the agent must perform RL when the reward function is known in some settings, but cannot be evaluated in others. As illustrated in Figure 1, we assume that the agent can first learn in a small range of “labeled” scenarios, where the reward is available, and then experiences a wider range of “unlabeled” scenarios where it must learn to act successfully, akin to lifelong learning in the real world. This problem statement can be viewed as being analogous to the problem of semi-supervised learning, but with the additional complexity of sequential decision making. Standard approaches to RL simply learn a policy in the scenarios where a reward function is available, and hope that it generalizes to new unseen conditions. However, it should be possible to leverage unlabeled experiences to find a more general policy, and to achieve continuous improvement from lifelong real-world experience.
Our main contribution is to propose and evaluate the first algorithm for performing semi-supervised reinforcement learning, which we call semi-supervised skill generalization (S3G). Our approach can leverage unlabeled experience to learn a policy that can succeed in a wider variety of scenarios than a policy trained only with labeled experiences. In our method, we train an RL policy in settings where a reward function is available, and then run an algorithm that resembles inverse reinforcement learning, to simultaneously learn a reward and a more general policy in the wider range of unlabeled settings. Unlike traditional applications of inverse RL algorithms, we use roll-outs from the RL policy in the labeled conditions as demonstrations, rather than a human expert, making our method completely autonomous. Although our approach is compatible with any choice of reinforcement learning and inverse reinforcement learning algorithm, we use the guided cost learning method in our experimental evaluation, which allows us to evaluate on high-dimensional, continuous robotic manipulation tasks with unknown dynamics while using a relatively modest number of samples (Finn et al., 2016). We compare our method to two baselines: (a) a policy trained with RL in settings where reward labels are available (as is standard), and (b) a policy trained in the unlabeled
settings using a reward function trained to regress to available reward labels. We find that S3G recovers a policy that is substantially more effective than the prior, standard approach in a wide variety of settings, without using any additional labeled information. We also find that, by using an inverse RL objective, our method achieves superior generalization to the reward regression approach.
2 RELATED WORK
Utilizing both labeled and unlabeled data is a well-known technique that can improve learning performance when data is limited (Zhu & Goldberg, 2009). These techniques are especially important in domains where large, supervised datasets are difficult to acquire, but unlabeled data is plentiful. This problem is generally known as semi-supervised learning. Methods for solving this problem often include propagating known labels to the unlabeled examples (Zhu & Ghahramani, 2002) and using regularizing side information (Szummer & Jaakkola, 2002) such as the structure of the data. Semi-supervised learning has been performed with deep models, either by blending unsupervised and supervised objectives (Rasmus et al., 2016; Zhang et al., 2016) or by using generative models, with the labels treated as missing data (Kingma et al., 2014). Semi-supervised learning is particularly relevant in robotics and control, where collecting labeled experience on real hardware is expensive. However, while semi-supervised learning has been successful in domains such as object tracking and detection (Teichman & Thrun, 2007), applications to action and control have not been applied to the objective of the task itself.
The generalization capabilities of policies learned through RL (and deep RL) has been limited, as pointed out by Oh et al. Oh et al. (2016). That is, typically the settings under which the agent is tested do not vary from those under which it was trained. We develop a method for generalizing skills to a wider range of settings using unlabeled experience. A related but orthogonal problem is transfer learning (Taylor & Stone, 2009; Barrett et al., 2010), which attempts to use prior experience in one domain to improve training performance in another. Transfer learning has been applied to RL domains for transferring information across environments (Mordatch et al., 2016; Tzeng et al., 2016), robots (Devin et al., 2016), and tasks (Konidaris & Barto, 2006; Stolle & Atkeson, 2007; Dragan et al., 2011; Parisotto et al., 2016; Rusu et al., 2016). The goal of these approaches is typically to utilize experience in a source domain to learn faster or better in the target domain. Unlike most transfer learning scenarios, we assume that supervision cannot be obtained in many scenarios. We are also not concerned with large, systematic domain shift: we assume that the labeled and unlabeled settings come from the same underlying distribution. Note, however, that the method that we develop could be used for transfer learning problems where the state and reward are consistent across domains.
To the best of our knowledge, this paper is the first to provide a practical and tractable algorithm for semi-supervised RL with large, expressive function approximators, and illustrate that such learning actually improves the generalization of the learned policy. However, the idea of semi-supervised reinforcement learning procedures has been previously discussed as a compelling research direction by Christiano (2016) and Amodei et al. (2016).
To accomplish semi-supervised reinforcement learning, we propose a method that resembles an inverse reinforcement learning (IRL) algorithm, in that it imputes the reward function in the unlabeled settings by learning from the successful trials in the labeled settings. IRL was first introduced by Ng et al. (2000) as the problem of learning reward functions from expert, human demonstrations, typically with the end goal of learning a policy that can succeed from states that are not in the set of demonstrations (Abbeel & Ng, 2004). We use IRL to infer the reward function underlying a policy previously learned in a small set of labeled scenarios, rather than using expert demonstrations. We build upon prior methods, including guided cost learning, which propose to learn a cost and a policy simultaneously (Finn et al., 2016; Ho et al., 2016). Note that the problem that we are considering is distinct from semi-supervised inverse reinforcement learning Audiffren et al. (2015), which makes use of expert and non-expert trajectories for learning. We require a reward function in some instances, rather than expert demonstrations.
3 SEMI-SUPERVISED REINFORCEMENT LEARNING
We first define semi-supervised reinforcement learning. We would like the problem definition to be able to capture situations where supervision, via the reward function, is only available in a small set
of labeled Markov decision processes (MDPs), but where we want our agent to be able to continue to learn to perform successfully in a much larger set of unlabeled MDPs, where reward labels are unavailable. For example, if the task corresponds to an autonomous car learning to drive, the labeled MDPs might correspond to a range of closed courses, while the unlabeled MDPs might involve driving on real-world highways and city streets. We use the terms labeled and unlabeled in analogy to semi-supervised learning, but note a reward observation is not as directly informative as a label.
Formally, we consider a distribution p(M) over undiscounted finite-horizon MDPs, each defined as a 4-tupleMi = (S,A,T,R) over states, actions, transition dynamics (which are generally unknown), and reward. The states and actions may be continuous or discrete, and the reward function R is assumed to the same across MDPs in the distribution p(M). Let L and U denote two sets of MDPs sampled from the distribution p(M). Experience may be collected in both sets of MDPs, but the reward can only be evaluated in the set of labeled MDPs L. The objective is to find a policy π∗ that maximizes expected reward in the distribution over MDPs:
π∗ = argmax π Eπ,p(M)
[ H
∑ t=0 R(st ,at)
] ,
where H denotes the horizon. Note that the notion of finding a policy that succeeds on a distribution of MDPs is very natural in many real-world reinforcement learning problems. For example, in the earlier autonomous driving example, our goal is not to find a policy that succeeds on one particular road or in one particular city, but on all roads that the car might encounter. Note that the problem can also be formalized in terms of a single large MDP with a large diversity of initial states, but viewing the expectation as being over a distribution of MDPs provides a more natural analogue with semi-supervised learning, as we discuss below.
In standard semi-supervised learning, it is assumed that the data distribution is the same across both labeled and unlabeled examples, and the amount of labeled data is limited. Similarly, semisupervised reinforcement learning assumes that the labeled and unlabeled MDPs are sampled from the same distribution. In SSRL, however, it is the set of labeled MDPs that is limited, whereas acquiring large amounts of experience within the set of labeled MDPs is permissible, though unlimited experience in the labeled MDPs is not sufficient on its own for good performance on the entire MDP distribution. This is motivated by real-world lifelong learning, where an agent (e.g. a robot) may be initially trained with detailed reward information in a small set of scenarios (e.g. with a human teacher), and is then deployed into a much larger set of scenarios, without reward labels. One natural question is how much variation can exist in the distribution over MDPs. We empirically answer this question in our experimental evaluation in Section 5.
The standard paradigm in reinforcement learning is to learn a policy in the labeled MDPs and apply it directly to new MDPs from the same distribution, hoping that the original policy will generalize (Oh et al., 2016). An alternative approach is to train a reward function with supervised learning to regress from the agent’s observations to the reward labels, and then use this reward function for learning in the unlabeled settings. In our experiments, we find that this approach is often more effective because, unlike the policy, the reward function is decoupled from the rest of the MDP, and can thus generalize more readily. The agent can then continue to learn from unlabeled experiences using the learned reward function. However, because the state distributions in the two sets of MDPs may be different, a function approximator trained on the reward function in the labeled MDPs may not necessarily generalize well to the unlabeled one, due to the domain shift. A more effective solution would be to incorporate the unlabeled experience sampled from U when learning the reward. Unlike typical semi-supervised learning, the goal is not to learn the reward labels per se, but to learn a policy that optimizes the reward. By incorporating both labeled and unlabeled experience, we can develop an algorithm that alternates between inferring the reward function and updating the policy, which effectively provides a shaping, or curriculum, for learning to perform well in the unlabeled settings. In the following section, we discuss our proposed algorithm in detail.
4 SEMI-SUPERVISED SKILL GENERALIZATION
We now present our approach for performing semi-supervised reinforcement learning for generalizing previously learned skills. As discussed previously, our goal is to learn a policy that maximizes expected reward inM∈U , using both unlabeled experience in U and labeled experience in L. We
will use the formalism adopted in the previous section; however, note that performing RL in a set of MDPs can be equivalently be viewed as a single MDP with a large diversity of initial conditions.
In order to perform semi-supervised reinforcement learning, we use the framework of maximum entropy control (Ziebart, 2010; Kappen et al., 2012), also called linear-solvable MDPs (Dvijotham & Todorov, 2010). This framework is a generalization of the standard reinforcement learning formulation, where instead of optimizing the expected reward, we optimize an entropy-regularized objective of the form
πRL = argmax π Eπ,M∈L
[ H
∑ t=0 R(st ,at)
] −H(π). (1)
To see that this is a generalization of the standard RL setting, observe that, as the magnitude of the reward increases, the relative weight on the entropy regularizer decreases, so the classic RL objective can be recovered by putting a temperature β on the reward, and taking the limit as β →∞. For finite rewards, this objective encourages policies to take random actions when all options have roughly equal value. Under the optimal policy πRL, samples with the highest reward R have the highest likelihood, and the likelihood decreases exponentially with decrease in reward. In our work, this framework helps to produce policies in the labeled MDP that are diverse, and therefore better suited for inferring reward functions that transfer effectively to the unlabeled MDP.
After training πRL, we generate a set of samples from πRL in L, which we denote as DπRL . The objective of S3G is to use DπRL to find a policy that maximizes expected reward in U ,
max θ Eπθ ,M∈U
[ T
∑ t=0 R(st ,at)
] −H(πθ ),
where the reward R is not available. By using the agent’s prior experienceDπRL , as well as unlabeled experience in U , we aim to learn a well-shaped reward function to facilitate learning in U . To do so, S3G simultaneously learns a reward function R̃φ with parameters φ and optimizes a policy πθ with parameters θ in the unlabeled MDP U . This consists of iteratively taking samples Dπθ from the current policy πθ in U , updating the reward R̃φ , and updating the policy π using reward values imputed using R̃φ . At the end of the procedure, we end up with a policy πθ optimized in U . As shown in prior work, this procedure corresponds to an inverse reinforcement learning algorithm that converges to a policy that matches the performance observed in DπRL (Finn et al., 2016). We next go over the objectives used for updating the reward and the policy.
Reward update: Because of the entropy regularized objective in Equation 1, it follows that the samples DπRL are generated from the following maximum entropy distribution (Ziebart, 2010):
p(τ) = 1 Z exp(R(τ)), (2)
where τ denotes a single trajectory sample {s0,a0,s1,a1, ...,sT} and R(τ) = ∑t R(st ,at). Thus, the objective of the reward optimization phase is to maximize the log likelihood of the agent’s prior experience DπRL under this exponential model. The computational challenge here is to estimate the partition function Z which is intractable to compute in high-dimensional spaces. We thus use importance sampling, using samples to estimate the partition function Z as follows:
L(φ) = ∑ τ∼DπRL R̃φ (τ)− logZ ≈ ∑ τ∼DπRL R̃φ (τ)− log ∑ τ∼Dsamp
exp(R̃φ (τ)) q(τ) , (3)
where Dsamp is the set of samples used for estimating the partition function Z and q(τ) is the probability of sampling τ under the policy it was generated from. Note that the distribution of this set of samples is crucial for effectively estimating Z. The optimal distribution for importance sampling is the one that is proportional to q(τ) ∝ |exp(R̃φ (τ))| = exp(R̃φ (τ)). Conveniently, this is also the optimal behavior when the reward function is fully optimized such that R̃φ ≈ R. Thus, we adaptively update the policy to minimize the KL-divergence between its own distribution and the distribution induced by the current reward, R̃φ (τ), and use samples from the policy to estimate the partition function. Since the importance sampling estimate of Z will be high variance at the beginning of training when fewer policy samples have been collected, we also use the samples from the RL policy πRL. Thus we set Dsamp to be {Dπθ ⋃ DπRL}.
Algorithm 1 Semi-Supervised Skill Generalization 0: inputs: Set of unlabeled MDPs U ; reward R for labeled MDPs M ∈ L 1: Optimize πRL to maximize R in M ∈ L 2: Generate samples DπRL from πRL in M ∈ L 3: Initialize Dsamp←DπRL 4: for iteration i = 1 to I do 5: Run πθ in M ∈U to generate samples Dπθ 6: Append samples Dsamp←Dsamp∪Dπθ 7: Update reward R̃φ according to Equation 3 using DπRL and Dsamp 8: Update policy πθ according to Equation 4, using R̃φ and Dπθ 9: end for 10: return generalized policy πθ
We parameterize the reward using a neural network, and update it using mini-batch stochastic gradient descent, by backpropagating the gradient of the Equation 3 to the parameters of the reward.
Policy update: Our goal with the policy is two-fold. First, we of course need a policy that succeeds in MDPs M∈U . But since the reward in these MDPs is unavailable, the policy must also serve to generate samples for more accurately estimating the partition function in Equation 2, so that the reward update step can improve the accuracy of the estimated reward function. The policy optimization objective to achieve both of these is to maximize the expected reward R̃φ , augmented with an entropy term as before:
L(θ) = Eπθ ,M∈U
[ T
∑ t=0 R̃φ (st ,at)
] −H(πθ ) (4)
While we could in principle use any policy optimization method in this step, our prototype uses mirror descent guided policy search (MDGPS), a sample-efficient policy optimization method suitable for training complex neural network policies that has been validated on real-world physical robots (Montgomery & Levine, 2016; Montgomery et al., 2016). We interleave reward function updates using the objective in Equation 3 within the policy optimization method. We describe the policy optimization procedure in detail in Appendix A.
The full algorithm is presented in Algorithm 1. Note that this iterative procedure of comparing the current policy to the optimal behavior provides a form of shaping or curriculum to learning. Our method is structured similarly to the recently proposed guided cost learning method (Finn et al., 2016), and inherits its convergence properties and theoretical foundations. Guided cost learning is an inverse RL algorithm that interleaves policy learning and reward learning directly in the target domain, which in our case is the unlabeled MDPs. Unlike guided cost learning, however, the cost (or reward) is not inferred from expert human-provided demonstrations, but from the agent’s own prior experience in the labeled MDPs.
5 EXPERIMENTAL EVALUATION
Since the aim of S3G is to improve the generalization performance of a learned policy by leveraging data from the unlabeled MDPs, our experiments focus on domains where generalization is critical for success. Despite the focus on generalization in many machine learning problems, the generalization capabilities of policies trained with RL have frequently been overlooked. For example, in recent RL benchmarks such as the Arcade Learning Environment (Bellemare et al., 2012) and OpenAI Gym (Brockman et al., 2016), the training conditions perfectly match the testing conditions. Thus, we define our own set of simulated control tasks for this paper, explicitly considering the types of variation that a robot might encounter in the real world. Through our evaluation, we seek to measure how well semi-supervised methods can leverage unlabeled experiences to improve the generalization of a deep neural network policy learned only in only labeled scenarios.
Code for reproducing the simulated experiments is available online1. Videos of the learned policies can be viewed at sites.google.com/site/semisupervisedrl.
1The code is available at github.com/cbfinn/gps/tree/ssrl
5.1 TASKS
Each of the tasks are modeled using the MuJoCo simulator, and involve continuous state and action spaces with unknown dynamics. The task difficulty ranges from simple, low-dimensional problems to tasks with complex dynamics and high-dimensional observations. In each experiment, the reward function is available in some settings but not others, and the unlabeled MDPs generally involve a wider variety of conditions. We visualize the tasks in Figure 2 and describe them in detail below:
obstacle navigation / obstacle height: The goal of this task is to navigate a point robot around an obstacle to a goal position in 2D. The observation is the robot’s position and velocity, and does not include the height of the obstacle. The height of the obstacle is 0.2 in the labeled MDP, and 0.5 in the unlabeled MDP.
2-link reacher / mass: This task involves moving the end-effector of a two-link reacher to a specified goal position. The observation is the robot’s joint angles, end-effector pose, and their timederivatives. In the labeled MDPs, the mass of the arm varies between 7×10−9 and 7×101, whereas the unlabeled MDPs involve a range of 7×10−9 to 7×103.
2-link reacher with vision / target position: The task objective is the same as the 2-link reacher, except, in this task, the MDPs involve a wide 2D range of target positions, shown in Figure 2. Instead of passing in the coordinate of the target position, the policy and the reward function receive a raw 64×80 RGB image of the environment at the first time step.
half-cheetah jump / wall height: In this task, the goal is for a simulated 6-DOF cheetah-like robot with to jump over a wall, with 10% gravity. The observation is the robot’s joint angles, global pose, and their velocities, for a total dimension of 20. The unlabeled MDP involves jumping over a 0.5 meter wall, compared to the labeled MDP with a 0.2 meter wall. Success is measured based on whether or not the cheetah fully clears the wall. Policies for reward regression, S3G, and oracle were initialized from the RL policy.
In all tasks, the continuous action vector corresponds to the torques or forces applied to each of the robot’s joints. For the first three tasks, reaching the goal position within 5 cm is considered a success. For the non-visual tasks, the policy was represented using a neural network with 2 hidden layers of 40 units each. The vision task used 3 convolutional layers with 15 filters of size 5×5 each, followed by the spatial feature point transformation proposed by Levine et al. (2016), and lastly 3 fully-connected layers of 20 units each. The reward function architecture mirrored the architecture as the policy, but using a quadratic norm on the output, as done by Finn et al. (2016).
5.2 EVALUATION
In our evaluation, we compare the performance of S3G to that of (i) the RL policy πRL, trained only in the labeled MDPs, (ii) a policy learned using a reward function fitted with supervised learning, and (iii) an oracle policy which can access the true reward function in all scenarios. The architecture of the reward function fitted with supervised learning is the same as that used in S3G.
To extensively test the generalization capabilities of the policies learned with each method, we measure performance on a wide range of settings that is a superset of the unlabeled and labeled MDPs, as indicated in Figure 3. We report the success rate of policies learned with each method in Table 1,
and visualize the generalization performance in the 2-link reacher, cheetah, and obstacle tasks in Figure 3. The sample complexity of each method is reported in Appendix B.
In all four tasks, the RL policy πRL generalizes worse than S3G, which demonstrates that, by using unlabeled experience, we can indeed improve generalization to different masses, target positions, and obstacle sizes. In the obstacle and both reacher tasks, S3G also outperforms reward regression, suggesting that it is also useful to use unlabeled experience to learn the reward.
In the obstacle task, the results demonstrate that the reward functions learned using S3G actually produce better generalization in some cases than learning on both the labeled and unlabeled MDPs with full knowledge of the true reward function. While this may at first seem counterintuitive, this agrees with the observation in prior work Guo et al. (2013) that the true reward function is not always the best one when learning with limited samples, computational power, or representational capacity (i.e. because it is not sufficiently shaped). S3G also outperforms the oracle and reward regression in the 2-link reacher task, indicating that the learned reward shaping is also beneficial in that task.
For the vision task, the visual features learned via RL in the labeled MDPs were used to initialize the vision layers of the reward and policy. We trained the vision-based reacher with S3G with both end-to-end finetuning of the visual features and with the visual features frozen and only the fully-connected layers trained on the unlabeled MDPs. We found performance to be similar in both cases, suggesting that the visual features learned with RL were good enough, though fine-tuning the features end-to-end with the inverse RL objective did not hurt the performance.
6 CONCLUSION & FUTURE WORK
We presented the first method for semi-supervised reinforcement learning, motivated by real-world lifelong learning. By inferring the reward in settings where one is not available, S3G can improve the generalization of a learned neural network policy trained only in the “labeled” settings. Additionally, we find that, compared to using supervised regression to reward labels, we can achieve higher performance using an inverse RL objective for inferring the reward underlying the agent’s prior experience. Interestingly, this does not directly make use of the reward labels when inferring the reward of states in the unlabeled MDPs, and our results on the obstacle navigation task in fact suggest that the rewards learned with S3G exhibit better shaping.
As we discuss previously, the reward and policy optimization methods that we build on in this work are efficient enough to learn complex tasks with hundreds of trials, making them well suited for
learning on physical systems such as robots. Indeed, previous work has evaluated similar methods on real physical systems, in the context of inverse RL (Finn et al., 2016) and vision-based policy learning (Levine et al., 2016). Thus, it is likely feasible to apply this method for semi-supervised reinforcement learning on a real robotic system. Applying S3G on physical systems has the potential to enable real-world lifelong learning, where an agent is initialized using a moderate amount of labeled experience in a constrained setting, such as a robot learning a skill for the first time in the lab, and then allowed to explore the real world while continuous improving its capabilities without additional supervision. This type of continuous semi-supervised reinforcement learning has the potential to remove the traditional distinction between a training and test phase for reinforcement learning agents, providing us with autonomous systems that continue to get better with use.
ACKNOWLEDGMENTS
The authors would like to thank Anca Dragan for insightful discussions, and Aviv Tamar and Roberto Calandra for helpful feedback on the paper. Funding was provided by the NSF GRFP, the DARPA Simplex program, and Berkeley DeepDrive.
A MIRROR DESCENT GUIDED POLICY SEARCH
To optimize policies with S3G, we chose to use mirror-descent guided policy search (MDGPS), for its superior sample efficiency over other policy optimization methods. MDGPS belongs to a class of guided policy search methods, which simplify policy search by decomposing the problem into two phases: a) a trajectory-centric RL phase (C-phase) and b) a supervised learning phase (S-phase). During the C-phase, a trajectory-centric RL method is used to train ”local” controllers for each of M initial positions. In the S-phase, a global policy πθ (a|s) is trained using supervised learning to match the output of each of the local policies.
MDGPS can be interpreted as an approximate variant of mirror-descent on the expected cost J(θ) = ∑Tt=1Eπθ (st ,at )[−R(st ,at)] under policy’s trajectory distribution, where πθ (st ,at) denotes the marginal of πθ (τ) = p(s1)∏Tt=1 p(st+1|st ,at)π(at |st) and τ = {s1,a1, . . . ,sT ,aT} denotes the trajectory. In the C-phase, we learn new local policies for each initial position, and in the S-phase we project the local policies down to a single global policy πθ , using KL divergence as the distance metric.
To produce local policies, we make use of the iterative linear quadratic regulator (iLQR) algorithm to train time-varying linear-Gaussian controllers. iLQR makes up for its weak representational power by being sample efficient under regimes where it is capable of learning. Usage of iLQR requires a twice-differentiable cost function and linearized dynamics.
In order to fit a dynamics model, we use the recent samples to fit a gaussian mixture model (GMM) on (st ,at ,st+1) tuples. We then use linear regression to fit time-varying linear dynamics of the form st+1 = Ftst + ft on local policy samples from the most recent iteration, using the clusters from the GMM as a normal-inverse Wishart prior.
During the C-step, for each initial condition m, we optimize the entropy-augmented of the form, objective constrained against the global policy:
qm = argmax q Eq,pm(s0)
[ T
∑ t=0 R(st ,at)
] −H(q) s.t. DKL(q||πθ )≤ ε
Where R(st ,at) is a twice-differentiable objective such as L2-distance from a target state.
This optimization results in a local time-varying linear-Gaussian controller qm(st |at)∼N (Km,tst + km,t ,Cm,t) which is executed to obtain supervised learning examples for the S-step.
B SAMPLE COMPLEXITY OF EXPERIMENTS
Because we use guided policy search to optimize the policy, we inherit its sample efficiency. In Table 2, we report the number of samples used in both labeled and unlabeled scenarios for all tasks and all methods. Note that the labeled samples used by the oracle are in from the “unlabeled” MDPs U , where we generally assume that reward labels are not available.

This paper formalizes the problem setting of having only a subset of available MDPs for which one has access to a reward. The authors name this setting "semi-supervised reinforcement learning" (SSRL), as a reference to semi-supervised learning (where one only has access to labels for a subset of the dataset). They provide an approach for solving SSRL named semi-supervised skill generalization (S3G), which builds on the framework of maximum entropy control. The whole approach is straightforward and amounts to an EM algorithm with partial labels (: they alternate iteratively between estimating a reward function (parametrized) and fitting a control policy using this reward function. They provide experiments on 4 tasks (obstacle, 2-link reacher, 2-link reacher with vision, half-cheetah) in MuJoCo.
The paper is well-written, and is overall clear. The appendix provides some more context, I think a few implementation details are missing to be able to fully reproduce the experiments from the paper, but they will provide the code.
The link to inverse reinforcement learning seems to be done correctly. However, there is no reference to off-policy policy learning, and, for instance, it seems to me that the \tau \in D_{samp} term of equation (3) could benefit from variance reduction as in e.g. TB(\lambda) [Precup et al. 2000] or Retrace(\lambda) [Munos et al. 2016].
The experimental section is convincing, but I would appreciate a precision (and small discussion) of this sentence "To extensively test the generalization capabilities of the policies learned with each method, we measure performance on a wide range of settings that is a superset of the unlabeled and labeled MDPs" with numbers for the different scenarios (or the replacement of superset by "union" if this is the case). It may explain better the poor results of "oracle" on "obstacle" and "2-link reacher", and reinforce* the further sentences "In the obstacle task, the true reward function is not sufficiently shaped for learning in the unlabeled MDPs. Hence, the reward regression and oracle methods perform poorly".
Correction on page 4: "5-tuple M_i = (S, A, T, R)" is a 4-tuple.
Overall, I think that this is a good and sound paper. I am personally unsure as to if all the parallels and/or references to previous work are complete, thus my confidence score of 3.
(* pun intended)


Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting.
1 INTRODUCTION
Adversarial examples are examples that are created by making small perturbations to the input designed to significantly increase the loss incurred by a machine learning model (Szegedy et al., 2014; Goodfellow et al., 2015). Several models, including state of the art convolutional neural networks, lack the ability to classify adversarial examples correctly, sometimes even when the adversarial perturbation is constrained to be so small that a human observer cannot perceive it. Adversarial training is the process of training a model to correctly classify both unmodified examples and adversarial examples. It improves not only robustness to adversarial examples, but also generalization performance for original examples. Adversarial training requires the use of labels when training models that use a supervised cost, because the label appears in the cost function that the adversarial perturbation is designed to maximize. Virtual adversarial training (Miyato et al., 2016) extends the idea of adversarial training to the semi-supervised regime and unlabeled examples. This is done by regularizing the model so that given an example, the model will produce the same output distribution as it produces on an adversarial perturbation of that example. Virtual adversarial training achieves good generalization performance for both supervised and semi-supervised learning tasks.
Previous work has primarily applied adversarial and virtual adversarial training to image classification tasks. In this work, we extend these techniques to text classification tasks and sequence models. Adversarial perturbations typically consist of making small modifications to very many real-valued inputs. For text classification, the input is discrete, and usually represented as a series of highdimensional one-hot vectors. Because the set of high-dimensional one-hot vectors does not admit infinitesimal perturbation, we define the perturbation on continuous word embeddings instead of discrete word inputs. Traditional adversarial and virtual adversarial training can be interpreted both as a regularization strategy (Szegedy et al., 2014; Goodfellow et al., 2015; Miyato et al., 2016) and as defense against an adversary who can supply malicious inputs (Szegedy et al., 2014; Goodfellow et al., 2015). Since the perturbed embedding does not map to any word and the adversary presumably does not have access to the word embedding layer, our proposed training strategy is no longer intended as a defense against an adversary. We thus propose this approach exclusively as a means of regularizing a text classifier by stabilizing the classification function.
∗This work was done when the author was at Google Brain.
We show that our approach with neural language model unsupervised pretraining as proposed by Dai & Le (2015) achieves state of the art performance for multiple semi-supervised text classification tasks, including sentiment classification and topic classification. We emphasize that optimization of only one additional hyperparameter , the norm constraint limiting the size of the adversarial perturbations, achieved such state of the art performance. These results strongly encourage the use of our proposed method for other text classification tasks. We believe that text classification is an ideal setting for semi-supervised learning because there are abundant unlabeled corpora for semi-supervised learning algorithms to leverage. This work is the first work we know of to use adversarial and virtual adversarial training to improve a text or RNN model.
We also analyzed the trained models to qualitatively characterize the effect of adversarial and virtual adversarial training. We found that adversarial and virtual adversarial training improved word embeddings over the baseline methods.
2 MODEL
We denote a sequence of T words as {w(t)|t = 1, . . . , T}, and a corresponding target as y. To transform a discrete word input to a continuous vector, we define the word embedding matrix V ∈ R(K+1)×D where K is the number of words in the vocabulary and each row vk corresponds to the word embedding of the i-th word. Note that the (K + 1)-th word embedding is used as an embedding of an ‘end of sequence (eos)’ token, veos. As a text classification model, we used a simple LSTM-based neural network model, shown in Figure 1a. At time step t, the input is the discrete word w(t), and the corresponding word embedding is v(t). We additionally tried the bidirectional
LSTM architecture (Graves & Schmidhuber, 2005) since this is used by the current state of the art method (Johnson & Zhang, 2016b). For constructing the bidirectional LSTM model for text classification, we add an additional LSTM on the reversed sequence to the unidirectional LSTM model described in Figure 1. The model then predicts the label on the concatenated LSTM outputs of both ends of the sequence.
In adversarial and virtual adversarial training, we train the classifier to be robust to perturbations of the embeddings, shown in Figure 1b. These perturbations are described in detail in Section 3. At present, it is sufficient to understand that the perturbations are of bounded norm. The model could trivially learn to make the perturbations insignificant by learning embeddings with very large norm. To prevent this pathological solution, when we apply adversarial and virtual adversarial training to the model we defined above, we replace the embeddings vk with normalized embeddings v̄k, defined as:
v̄k = vk − E(v)√
Var(v) where E(v) = K∑ j=1 fjvj ,Var(v) = K∑ j=1 fj (vj − E(v))2 , (1)
where fi is the frequency of the i-th word, calculated within all training examples.
3 ADVERSARIAL AND VIRTUAL ADVERSARIAL TRAINING
Adversarial training (Goodfellow et al., 2015) is a novel regularization method for classifiers to improve robustness to small, approximately worst case perturbations. Let us denote x as the input
and θ as the parameters of a classifier. When applied to a classifier, adversarial training adds the following term to the cost function:
− log p(y | x+ radv;θ) where radv = arg min r,‖r‖≤ log p(y | x+ r; θ̂) (2)
where r is a perturbation on the input and θ̂ is a constant set to the current parameters of a classifier. The use of the constant copy θ̂ rather than θ indicates that the backpropagation algorithm should not be used to propagate gradients through the adversarial example construction process. At each step of training, we identify the worst case perturbations radv against the current model p(y|x; θ̂) in Eq. (2), and train the model to be robust to such perturbations through minimizing Eq. (2) with respect to θ. However, we cannot calculate this value exactly in general, because exact minimization with respect to r is intractable for many interesting models such as neural networks. Goodfellow et al. (2015) proposed to approximate this value by linearizing log p(y | x; θ̂) around x. With a linear approximation and a L2 norm constraint in Eq.(2), the resulting adversarial perturbation is
radv = − g/‖g‖2 where g = ∇x log p(y | x; θ̂). This perturbation can be easily computed using backpropagation in neural networks.
Virtual adversarial training (Miyato et al., 2016) is a regularization method closely related to adversarial training. The additional cost introduced by virtual adversarial training is the following:
KL[p(· | x; θ̂)||p(· | x+ rv-adv;θ)] (3) where rv-adv = arg max
r,‖r‖≤ KL[p(· | x; θ̂)||p(· | x+ r; θ̂)] (4)
where KL[p||q] denotes the KL divergence between distributions p and q. By minimizing Eq.(3), a classifier is trained to be smooth. This can be considered as making the classifier resistant to perturbations in directions to which it is most sensitive on the current model p(y|x; θ̂). Virtual adversarial loss Eq.(3) requires only the input x and does not require the actual label y while adversarial loss defined in Eq.(2) requires the label y. This makes it possible to apply virtual adversarial training to semi-supervised learning. Although we also in general cannot analytically calculate the virtual adversarial loss, Miyato et al. (2016) proposed to calculate the approximated Eq.(3) efficiently with backpropagation.
As described in Sec. 2, in our work, we apply the adversarial perturbation to word embeddings, rather than directly to the input. To define adversarial perturbation on the word embeddings, let us denote a concatenation of a sequence of (normalized) word embedding vectors [v̄(1), v̄(2), . . . , v̄(T )] as s, and the model conditional probability of y given s as p(y|s;θ) where θ are model parameters. Then we define the adversarial perturbation radv on s as:
radv = − g/‖g‖2 where g = ∇s log p(y | s; θ̂). (5) To be robust to the adversarial perturbation defined in Eq.(5), we define the adversarial loss by
Ladv(θ) = − 1
N N∑ n=1 log p(yn | sn + radv,n;θ) (6)
where N is the number of labeled examples. In our experiments, adversarial training refers to minimizing the negative log-likelihood plus Ladv with stochastic gradient descent.
In virtual adversarial training on our text classification model, at each training step, we calculate the below approximated virtual adversarial perturbation:
rv-adv = g/‖g‖2 where g = ∇s+dKL [ p(· | s; θ̂)||p(· | s+ d; θ̂) ] (7)
where d is a TD-dimensional small random vector. This approximation corresponds to a 2nd-order Taylor expansion and a single iteration of the power method on Eq.(3) as in previous work (Miyato et al., 2016). Then the virtual adversarial loss is defined as:
Lv-adv(θ) = 1
N ′ N ′∑ n′=1 KL [ p(· | sn′ ; θ̂)||p(· | sn′ + rv-adv,n′ ;θ) ] (8)
where N ′ is the number of both labeled and unlabeled examples.
See Warde-Farley & Goodfellow (2016) for a recent review of adversarial training methods.
4 EXPERIMENTAL SETTINGS
All experiments used TensorFlow (Abadi et al., 2016) on GPUs. Code will be available at https: //github.com/tensorflow/models/tree/master/adversarial_text.
To compare our method with other text classification methods, we tested on 5 different text datasets. We summarize information about each dataset in Table 1.
IMDB (Maas et al., 2011)1 is a standard benchmark movie review dataset for sentiment classification. Elec (Johnson & Zhang, 2015b)2 3 is an Amazon electronic product review dataset. Rotten Tomatoes (Pang & Lee, 2005) consists of short snippets of movie reviews, for sentiment classification. The Rotten Tomatoes dataset does not come with separate test sets, thus we divided all examples randomly into 90% for the training set, and 10% for the test set. We repeated training and evaluation five times with different random seeds for the division. For the Rotten Tomatoes dataset, we also collected unlabeled examples using movie reviews from the Amazon Reviews dataset (McAuley & Leskovec, 2013) 4. DBpedia (Lehmann et al., 2015; Zhang et al., 2015) is a dataset of Wikipedia pages for category classification. Because the DBpedia dataset has no additional unlabeled examples, the results on DBpedia are for the supervised learning task only. RCV1 (Lewis et al., 2004) consists of news articles from the Reuters Corpus. For the RCV1 dataset, we followed previous works (Johnson & Zhang, 2015b) and we conducted a single topic classification task on the second level topics. We used the same division into training, test and unlabeled sets as Johnson & Zhang (2015b). Regarding pre-processing, we treated any punctuation as spaces. We converted all words to lower-case on the Rotten Tomatoes, DBpedia, and RCV1 datasets. We removed words which appear in only one document on all datasets. On RCV1, we also removed words in the English stop-words list provided by Lewis et al. (2004)5.
4.1 RECURRENT LANGUAGE MODEL PRE-TRAINING
Following Dai & Le (2015), we initialized the word embedding matrix and LSTM weights with a pre-trained recurrent language model (Bengio et al., 2006; Mikolov et al., 2010) that was trained on both labeled and unlabeled examples. We used a unidirectional single-layer LSTM with 1024 hidden units. The word embedding dimension D was 256 on IMDB and 512 on the other datasets. We used a sampled softmax loss with 1024 candidate samples for training. For the optimization, we used the Adam optimizer (Kingma & Ba, 2015), with batch size 256, an initial learning rate of 0.001, and a 0.9999 learning rate exponential decay factor at each training step. We trained for 100,000 steps. We applied gradient clipping with norm set to 1.0 on all the parameters except word embeddings. To reduce runtime on GPU, we used truncated backpropagation up to 400 words from each end of the sequence. For regularization of the recurrent language model, we applied dropout (Srivastava et al., 2014) on the word embedding layer with 0.5 dropout rate.
1http://ai.stanford.edu/~amaas/data/sentiment/ 2http://riejohnson.com/cnn_data.html 3There are some duplicated reviews in the original Elec dataset, and we used the dataset with removal of the duplicated reviews, provided by Johnson & Zhang (2015b), thus there are slightly fewer examples shown in Table 1 than the ones in previous works(Johnson & Zhang, 2015b; 2016b).
4http://snap.stanford.edu/data/web-Amazon.html 5http://www.ai.mit.edu/projects/jmlr/papers/volume5/lewis04a/lyrl2004_rcv1v2_README.htm
For the bidirectional LSTM model, we used 512 hidden units LSTM for both the standard order and reversed order sequences, and we used 256 dimensional word embeddings which are shared with both of the LSTMs. The other hyperparameters are the same as for the unidirectional LSTM. We tested the bidirectional LSTM model on IMDB, Elec and RCV because there are relatively long sentences in the datasets.
Pretraining with a recurrent language model was very effective on classification performance on all the datasets we tested on and so our results in Section 5 are with this pretraining.
4.2 TRAINING CLASSIFICATION MODELS
After pre-training, we trained the text classification model shown in Figure 1a with adversarial and virtual adversarial training as described in Section 3. Between the softmax layer for the target y and the final output of the LSTM, we added a hidden layer, which has dimension 30 on IMDB, Elec and Rotten Tomatoes, and 128 on DBpedia and RCV1. The activation function on the hidden layer was ReLU(Jarrett et al., 2009; Nair & Hinton, 2010; Glorot et al., 2011). For optimization, we again used the Adam optimizer, with 0.0005 initial learning rate 0.9998 exponential decay. Batch sizes are 64 on IMDB, Elec, RCV1, and 128 on DBpedia. For the Rotten Tomatoes dataset, for each step, we take a batch of size 64 for calculating the loss of the negative log-likelihood and adversarial training, and 512 for calculating the loss of virtual adversarial training. Also for Rotten Tomatoes, we used texts with lengths T less than 25 in the unlabeled dataset. We iterated 10,000 training steps on all datasets except IMDB and DBpedia, for which we used 15,000 and 20,000 training steps respectively. We again applied gradient clipping with the norm as 1.0 on all the parameters except the word embedding. We also used truncated backpropagation up to 400 words, and also generated the adversarial and virtual adversarial perturbation up to 400 words from each end of the sequence.
We found the bidirectional LSTM to converge more slowly, so we iterated for 15,000 training steps when training the bidirectional LSTM classification model.
For each dataset, we divided the original training set into training set and validation set, and we roughly optimized some hyperparameters shared with all of the methods; (model architecture, batchsize, training steps) with the validation performance of the base model with embedding dropout. For each method, we optimized two scalar hyperparameters with the validation set. These were the dropout rate on the embeddings and the norm constraint of adversarial and virtual adversarial training. Note that for adversarial and virtual adversarial training, we generate the perturbation after applying embedding dropout, which we found performed the best. We did not do early stopping with these methods. The method with only pretraining and embedding dropout is used as the baseline (referred to as Baseline in each table).
5 RESULTS
5.1 TEST PERFORMANCE ON IMDB DATASET AND MODEL ANALYSIS
Figure 2 shows the learning curves on the IMDB test set with the baseline method (only embedding dropout and pretraining), adversarial training, and virtual adversarial training. We can see in Figure 2a that adversarial and virtual adversarial training achieved lower negative log likelihood than the baseline. Furthermore, virtual adversarial training, which can utilize unlabeled data, maintained this low negative log-likelihood while the other methods began to overfit later in training. Regarding adversarial and virtual adversarial loss in Figure 2b and 2c, we can see the same tendency as for negative log likelihood; virtual adversarial training was able to keep these values lower than other methods. Because adversarial training operates only on the labeled subset of the training data, it eventually overfits even the task of resisting adversarial perturbations.
Table 2 shows the test performance on IMDB with each training method. ‘Adversarial + Virtual Adversarial’ means the method with both adversarial and virtual adversarial loss with the shared norm constraint . With only embedding dropout, our model achieved a 7.39% error rate. Adversarial and virtual adversarial training improved the performance relative to our baseline, and virtual adversarial training achieved performance on par with the state of the art, 5.91% error rate. This is despite the fact that the state of the art model requires training a bidirectional LSTM whereas our model only
uses a unidirectional LSTM. We also show results with a bidirectional LSTM. Our bidirectional LSTM model has the same performance as a unidirectional LSTM with virtual adversarial training.
A common misconception is that adversarial training is equivalent to training on noisy examples. Noise is actually a far weaker regularizer than adversarial perturbations because, in high dimensional input spaces, an average noise vector is approximately orthogonal to the cost gradient. Adversarial perturbations are explicitly chosen to consistently increase the cost. To demonstrate the superiority of adversarial training over the addition of noise, we include control experiments which replaced adversarial perturbations with random perturbations from a multivariate Gaussian with scaled norm, on each embedding in the sequence. In Table 2, ‘Random perturbation with labeled examples’ is the method in which we replace radv with random perturbations, and ‘Random perturbation with labeled and unlabeled examples’ is the method in which we replace rv-adv with random perturbations. Every adversarial training method outperformed every random perturbation method.
‘good’ ‘bad’
Baseline Random Adversarial Virtual Adversarial
Baseline Random Adversarial Virtual Adversarial
1 great great decent decent terrible terrible terrible terrible 2 decent decent great great awful awful awful awful 3 ×bad excellent nice nice horrible horrible horrible horrible 4 excellent nice fine fine ×good ×good poor poor 5 Good Good entertaining entertaining Bad poor BAD BAD 6 fine ×bad interesting interesting BAD BAD stupid stupid 7 nice fine Good Good poor Bad Bad Bad 8 interesting interesting excellent cool stupid stupid laughable laughable 9 solid entertaining solid enjoyable Horrible Horrible lame lame 10 entertaining solid cool excellent horrendous horrendous Horrible Horrible
We also investigated the 15 nearest neighbors to ‘great’ and its cosine distances with the trained embeddings. We saw that cosine distance on adversarial and virtual adversarial training (0.159–0.331) were much smaller than ones on the baseline and random perturbation method (0.244–0.399). The much weaker positive word ‘good’ also moved from the 3rd nearest neighbor to the 15th after virtual adversarial training.
5.2 TEST PERFORMANCE ON ELEC, RCV1 AND ROTTEN TOMATOES DATASET
Table 4 shows the test performance on the Elec and RCV1 datasets. We can see our proposed method improved test performance on the baseline method and achieved state of the art performance on both datasets, even though the state of the art method uses a combination of CNN and bidirectional LSTM models. Our unidirectional LSTM model improves on the state of the art method and our method with a bidirectional LSTM further improves results on RCV1. The reason why the bidirectional models have better performance on the RCV1 dataset would be that, on the RCV1 dataset, there are some very long sentences compared with the other datasets, and the bidirectional model could better handle such long sentences with the shorter dependencies from the reverse order sentences.
Table 5 shows test performance on the Rotten Tomatoes dataset. Adversarial training was able to improve over the baseline method, and with both adversarial and virtual adversarial cost, achieved almost the same performance as the current state of the art method. However the test performance of only virtual adversarial training was worse than the baseline. We speculate that this is because the Rotten Tomatoes dataset has very few labeled sentences and the labeled sentences are very short. In this case, the virtual adversarial loss on unlabeled examples overwhelmed the supervised loss, so the model prioritized being robust to perturbation rather than obtaining the correct answer.
5.3 PERFORMANCE ON THE DBPEDIA PURELY SUPERVISED CLASSIFICATION TASK
Table 6 shows the test performance of each method on DBpedia. The ‘Random perturbation’ is the same method as the ‘Random perturbation with labeled examples’ explained in Section 5.1. Note that
DBpedia has only labeled examples, as we explained in Section 4, so this task is purely supervised learning. We can see that the baseline method has already achieved nearly the current state of the art performance, and our proposed method improves from the baseline method.
6 RELATED WORKS
Dropout (Srivastava et al., 2014) is a regularization method widely used for many domains including text. There are some previous works adding random noise to the input and hidden layer during training, to prevent overfitting (e.g. (Sietsma & Dow, 1991; Poole et al., 2013)). However, in our experiments and in previous works (Miyato et al., 2016), training with adversarial and virtual adversarial perturbations outperformed the method with random perturbations.
For semi-supervised learning with neural networks, a common approach, especially in the image domain, is to train a generative model whose latent features may be used as features for classification (e.g. (Hinton et al., 2006; Maaløe et al., 2016)). These models now achieve state of the art performance on the image domain. However, these methods require numerous additional hyperparameters with generative models, and the conditions under which the generative model will provide good supervised learning performance are poorly understood. By comparison, adversarial and virtual adversarial training requires only one hyperparameter, and has a straightforward interpretation as robust optimization.
Adversarial and virtual adversarial training resemble some semi-supervised or transductive SVM approaches (Joachims, 1999; Chapelle & Zien, 2005; Collobert et al., 2006; Belkin et al., 2006) in that both families of methods push the decision boundary far from training examples (or in the case of transductive SVMs, test examples). However, adversarial training methods insist on margins on the input space , while SVMs insist on margins on the feature space defined by the kernel function. This property allows adversarial training methods to achieve the models with a more flexible function on the space where the margins are imposed. In our experiments (Table 2, 4) and Miyato et al. (2016), adversarial and virtual adversarial training achieve better performance than SVM based methods.
There has also been semi-supervised approaches applied to text classification with both CNNs and RNNs. These approaches utilize ‘view-embeddings’(Johnson & Zhang, 2015b; 2016b) which use the window around a word to generate its embedding. When these are used as a pretrained model for the classification model, they are found to improve generalization performance. These methods and our method are complementary as we showed that our method improved from a recurrent pretrained language model.
7 CONCLUSION
In our experiments, we found that adversarial and virtual adversarial training have good regularization performance in sequence models on text classification tasks. On all datasets, our proposed method exceeded or was on par with the state of the art performance. We also found that adversarial and virtual adversarial training improved not only classification performance but also the quality of word embeddings. These results suggest that our proposed method is promising for other text domain tasks, such as machine translation(Sutskever et al., 2014), learning distributed representations of words or paragraphs(Mikolov et al., 2013; Le & Mikolov, 2014) and question answering tasks. Our approach could also be used for other general sequential tasks, such as for video or speech.
ACKNOWLEDGMENTS
We thank the developers of Tensorflow. We thank the members of Google Brain team for their warm support and valuable comments. This work is partly supported by NEDO.

*** Paper Summary ***
This paper applies adversarial and virtual adversarial training to LSTM for text classification. Since text inputs are discrete adversarial perturbation are applied to the (normalized) word embeddings. Extensive experiments are reported and demonstrate the advantage of these methods.
*** Review Summary ***
The paper reads well and has sufficent references. The application of adversarial training to text data is a simple but not trivial extension. The experimental section presents extensive experiments with comparison to alternative strategies. The proposed method is simple and effective and can be easily be applied after reading the paper.
*** Detailed Review ***
The paper reads well. I have only a few comments regarding experiments and link to prior resarch:
Experiments:
- In Table 2 (and for other datasets as well), could you include an SVM baseline? e.g. S Wang and C Manning 2012?
- As another baseline, did you consider dropping words, i.e. masking noise? It is generally better than dropout/gaussian noise for text application (e.g. denoising autoencoders)?
- I am not sure I understand why virtual adversarial is worse than the baseline in Table 5. If you tune epsilon, in the worse case you would get the same performance as the baseline? Was it that validation was unreliable?
Related Work:
I think it would be interesting to point at SVM, transductive SVM who achieve something similar to adversarial training. When maximizing the margin in a (transductive) SVM, it is equivalent to move the example toward the decision boundary, i.e. moving them in the direction of increase of the loss gradient.
Also it would be interesting to draw a parallel between adversarial training and contrastive divergence. The adversarial samples are very close in nature to the one step Markov Chain samples from CD. See Bengio 2009. Related to this technique are also approaches that try to explicitely cancel the Jacobian at data points, e.g. Rifai et al 2011.
*** References ***
Marginalized Denoising Autoencoders for Domain Adaptation. Minmin Chen, K Weinberger.
Stacked Denoising Autoencoders. Pascal Vincent. JMLR 2011.
Learning invariant features through local space contraction, Salah Rifai, Xavier Muller, Xavier Glorot, Gregoire Mesnil, Yoshua Bengio and Pascal Vincent, 2011.
Learning Deep Architectures for AI, Yoshua Bengio 2009
Large Scale Transductive SVMs. Ronan Collobert et al 2006
Optimization for Transductive SVM.  O Chapelle, V Sindhwani, SS Keerthi JMLR 2008