NEURAL GRAPH MACHINES: LEARNING NEURAL NETWORKS USING GRAPHS
Label propagation is a powerful and flexible semi-supervised learning technique on graphs. Neural network architectures, on the other hand, have proven track records in many supervised learning tasks. In this work, we propose a training objective for neural networks, Neural Graph Machines, for combining the power of neural networks and label propagation. The new objective allows the neural networks to harness both labeled and unlabeled data by: (a) allowing the network to train using labeled data as in the supervised setting, (b) biasing the network to learn similar hidden representations for neighboring nodes on a graph, in the same vein as label propagation. Such architectures with the proposed objective can be trained efficiently using stochastic gradient descent and scaled to large graphs. The proposed method is experimentally validated on a wide range of tasks (multilabel classification on social graphs, news categorization and semantic intent classification) using different architectures (NNs, CNNs, and LSTM RNNs).
Label propagation is a powerful and flexible semi-supervised learning technique on graphs. Neural network architectures, on the other hand, have proven track records in many supervised learning tasks. In this work, we propose a training objective for neural networks, Neural Graph Machines, for combining the power of neural networks and label propagation. The new objective allows the neural networks to harness both labeled and unlabeled data by: (a) allowing the network to train using labeled data as in the supervised setting, (b) biasing the network to learn similar hidden representations for neighboring nodes on a graph, in the same vein as label propagation. Such architectures with the proposed objective can be trained efficiently using stochastic gradient descent and scaled to large graphs. The proposed method is experimentally validated on a wide range of tasks (multilabel classification on social graphs, news categorization and semantic intent classification) using different architectures (NNs, CNNs, and LSTM RNNs).
1 INTRODUCTION
Semi-supervised learning is a powerful machine learning paradigm that can improve the prediction performance compared to techniques that use only labeled data, by leveraging a large amount of unlabeled data. The need of semi-supervised learning arises in many problems in computer vision, natural language processing or social networks, in which getting labeled datapoints is expensive or unlabeled data is abundant and readily available.
There exist a plethora of semi-supervised learning methods. The simplest one uses bootstrapping techniques to generate pseudo-labels for unlabeled data generated from a system trained on labeled data. However, this suffers from label error feedbacks (Lee, 2013). In a similar vein, autoencoder based methods often need to rely on a two-stage approach: train an autoencoder using unlabeled data to generate an embedding mapping, and use the learnt embeddings for prediction. In practice, this procedure is often costly and inaccurate in practice. Another example is transductive SVMs (Joachims, 1999), which is too computationally expensive to be used for large datasets. Methods that are based on generative models and amortized variational inference (Kingma et al., 2014) can work well for images and videos, but it is not immediately clear on how to extend such techniques to handle sparse and multi-modal inputs or graphs over the inputs. In contrast to the methods above, graph-based techniques such as label propagation (Zhu & Ghahramani; Bengio et al., 2006) often provide a versatile, scalable, and yet effective solution to a wide range of problems. These methods construct a smooth graph over the unlabeled and labeled data. Graphs are also often a natural way to describe the relationships between nodes, such as similarities between embeddings, phrases or images, or connections between entities on the web or relations in a social network. Edges in the graph connect semantically similar nodes or datapoints, and if present, edge weights reflect how strong such similarities are. By providing a set of labeled nodes, such techniques iteratively refine the node labels by aggregating information from neighbours and propagate these labels to the nodes’ neighbours. In practice, these methods often converge quickly and can be scaled to large datasets with a large label space (Ravi & Diao, 2016). We build upon the principle behind label propagation for our method.
∗Work done during an internship at Google
Another key motivation of our work is the recent advances in neural networks and their performance on a wide variety of supervised learning tasks such as image and speech recognition or sequence-tosequence learning (Krizhevsky et al., 2012; Hinton et al., 2012; Sutskever et al., 2014). Such results are however conditioned on training very large networks on large datasets, which may need millions of labeled training input-output pairs. This begs the question: can we harness previous state-of-theart semi-supervised learning techniques, to jointly train neural networks using limited labeled data and unlabeled data to improve its performance?
Contributions: We propose a discriminative training objective for neural networks with graph augmentation, that can be trained with gradient descent and efficiently scaled to large graphs. In particular, we introduce a regularization term for generic neural network architectures that enforces similarity between nodes in the graphs. This is inspired by the objective function of label propagation. The resulting cost is amenable to stochastic training and can be applied to various model classes. We also investigate using graphs as direct inputs to train neural network classifiers and experimentally demonstrate that this procedure is more efficient and accurate than previous two-stage approaches such as finding embeddings and using them for classification.
The closet approach to our work is the framework proposed by Weston et al. (2012), we extend their work in several ways: (a) our proposed training scheme is flexible, for example multiple graphs from multiple domains can be combined, (b) we provide extensive experiments on different types of neural networks and on properly constructed graphs (in contrast to nearest neighbor graphs in Weston et al. (2012), (c) we propose using graphs as inputs to the neural networks if there are no input features. Our work is also different from recent works on using neural networks on graphs (e.g. see Niepert et al. (2016)). Instead, we advocate a training objective that uses graphs to augment neural network learning.
2 BACKGROUND
In this section, we will lay out the groundwork for our proposed training objective in section 3.
2.1 LABEL PROPAGATION
We first provide a concise introduction to label propagation and its training objective. Suppose we are given a graph G = (V,E,W ) where V is the set of nodes, E the set of nodes and W the edge weight matrix. Let Vl, Vu be the labeled and unlabeled nodes in the graph. The goal is to predict a soft assignment of labels for each node in the graph, Ŷ , given the training label distribution for the seed nodes, Y . Mathematically, label propagation performs minimization of the following convex objective function, for L labels,
CLP(Ŷ ) = µ1 ∑ v∈Vl ∥∥∥Ŷv − Yv∥∥∥2 2 + µ2 ∑ v∈V,u∈N (v) wu,v ∥∥∥Ŷv − Ŷu∥∥∥2 2 + µ3 ∑ v∈V ∥∥∥Ŷv − U∥∥∥2 2 , (1)
subject to ∑L l=1 Ŷvl = 1, where N (v) is the neighbour node set of the node v, and U is the prior distribution over all labels, wu,v is the edge weight between nodes u and v, and µ1, µ2, and µ3 are hyperparameters that balance the contribution of individual terms in the objective. The terms in the objective function above encourage that: (a) the label distribution of seed nodes should be close to the ground truth, (b) the label distribution of neighbouring nodes should be similar, and, (c) if relevant, the label distribution should stay close to our prior belief. This objective function can be solved efficiently using iterative methods such as the Jacobi procedure. That is, in each step, each node aggregates the label distributions from its neighbours and adjusts its own distribution, which is then repeated until convergence. In practice, the iterative updates can be done in parallel or in a distributed fashion which then allows large graphs with a large number of nodes and labels to be trained efficiently. Bengio et al. (2006) and Ravi & Diao (2016) are good surveys on the topic for interested readers.
2.2 NEURAL NETWORK LEARNING
Neural networks are a class of non-linear mapping from inputs to outputs and comprised of multiple layers that can potentially learn useful representations for predicting the outputs. We will view
various models such as feedforward neural networks, recurrent neural networks and convolutional networks in the same umbrella. Given a set of N training input-output pairs {xn, yn}Nn=1, such neural networks are often trained by performing maximum likelihood learning, that is, tuning their parameters so that the networks’ outputs are close to the ground truth under some criterion,
CNN(θ) = ∑ n c(gθ(xn), yn), (2)
where gθ(·) denotes the overall mapping, parameterized by θ, and c(·) denotes a loss function such as l-2 for regression or cross entropy for classification. The cost function c and the mapping g are typically differentiable w.r.t θ, which facilitates optimisation via gradient descent. Importantly, this can be scaled to a large number of training instances by employing stochastic training using minibatches of data. However, it is not clear how unlabeled data, if available, can be treated using this objective, or if extra information about the training set, such as relational structures can be used.
3 NEURAL GRAPH MACHINES
In this section, we devise a discriminative training objective for neural networks, that is inspired by the label propagation objective and uses both labeled and unlabeled data, and can be trained by stochastic gradient descent.
First, we take a close look at the two objective functions discussed in section 2. The label propagation objective equation 1 makes sure the predicted label distributions of neighbouring nodes to be similar, while those of labeled nodes to be close to the ground truth. For example: if a cat image and a dog image are strongly connected in a graph, and if the cat node is labeled as animal, the predicted probability of the dog node being animal is also high. In contrast, the neural network training objective equation 2 only takes into account the labeled instances, and ensure correct predictions on the training set. As a consequence, a neural network trained on the cat image alone will not make an accurate prediction on the dog image.
Such shortcoming of neural network training can be rectified by biasing the network using prior knowledge about the relationship between instances in the dataset. In particular, for the domains we are interested in, training instances (either labeled or unlabeled) that are connected in a graph, for example, dog and cat in the above example, should have similar predictions. This can be done by encouraging neighboring data points to have a similar hidden representation learnt by a neural network, resulting in a modified objective function for training neural network architectures using both labeled and unlabeled datapoints:
CNGM(θ) = Vl∑ n=1 c(gθ(xn), yn) + α1 ∑
(u,v)∈ELL
wuvd(hθ(xu), hθ(xv))
+ α2 ∑
(u,v)∈ELU
wuvd(hθ(xu), hθ(xv)) + α3 ∑
(u,v)∈EUU
wuvd(hθ(xu), hθ(xv), (3)
where ELL, ELU , and EUU are sets of labeled-labeled, labeled-unlabeled and unlabeled-unlabeled edges correspondingly, h(·) represents the hidden representations of the inputs produced by the neural network, and d(·) is a distance metric, and {α1, α2, α3} are hyperparameters. We call architectures to be trained using this objective Neural Graph Machines, and schematically illustrate the concept in figure 1. In practice, we choose an l-1 or l-2 distance metric for d(·), and h(x) to be the last layer of the neural network. However, these choices can be changed, to a customized metric, or to using an intermediate hidden layer instead.
3.1 CONNECTIONS TO PREVIOUS METHODS
Note that we have separated the terms based on the edge types, as these can affect the training differently. The graph-dependent α hyperparameters control the balance of these terms. When αi = 0, the proposed objective ignores the similarity constraint and becomes a supervised-only objective as in equation 2. When gθ(x) = hθ(x) = ŷ, where ŷ is the label distribution, the individual cost functions (c and d) are squared l-2 norm, and the objective is trained using ŷ directly instead of θ, we arrive at the label propagation objective in equation 1. Therefore, the proposed objective could
be thought of as a non-linear version of the label propagation objective, and a graph-regularized version of the neural network training objective.
3.2 NETWORK INPUTS AND GRAPH CONSTRUCTION
Similar to graph-based label propagation, the choice of the input graphs is critical, to correctly bias the neural network’s prediction. Depending on the type of the graphs and nodes on the graphs, they can be readily available to use such as social networks or protein linking networks, or they can be constructed (a) using generic graphs such as Knowledge Bases, that consists of links between vertices on the graph, (b) using embeddings learnt by an unsupervised learning technique, or, (c) using sparse feature representations for each vertex. Additionally, the proposed training objective can be easily modified for directed graphs.
We have discussed using node-level features as inputs to the neural network. In the absences of such inputs, our training scheme can still be deployed using input features derived from the graph itself. We show in figure 2 and in the experiment that the neighbourhood information such as rows in the adjacency matrix are simple to construct, yet powerful inputs to the network. These features can also be combined with existing features.
3.3 OPTIMIZATION
The proposed objective function in equation 3 has several summations over the labeled points and edges, and can be equivalently written as follows,
CNGM(θ) = ∑
(u,v)∈ELL
α1wuvd(hθ(xu), hθ(xv)) + c(gθ(xu), yu) + c(gθ(xv), yv)
+ ∑
(u,v)∈ELU
α2wuvd(hθ(xu), hθ(xv)) + c(gθ(xu), yu)
+ ∑
(u,v)∈EUU
α3wuvd(hθ(xu), hθ(xv). (4)
The objective in its new form enables stochastic training to be deployed. In particular, in each training iteration, we use a minibatch of edges and obtain the stochastic gradients of the objective. To further reduce noise, we can select a labeled node and sample from the set of edges that are incident to that node. The number of edges per node to be sampled can be controlled.
3.4 COMPLEXITY
The complexity of each training epoch using equation 4 is O(M) where M = |E| is the number of edges in the graph. In practice, unlabeled-unlabeled edges do not seem to help learning and could be ignored. which further reduces the above complexity.
4 EXPERIMENTS
In this section, we provide several experiments showing the efficacy of the proposed training objective on a wide range of tasks, datasets and network architectures. All the experiments are done using TensorFlow (Abadi et al., 2015).
4.1 MULTI-LABEL CLASSIFICATION OF NODES ON GRAPHS
We first consider a multi-label classification on nodes of a graph. We use the BlogCatalog dataset (Agarwal et al., 2009), which has 10,312 nodes and 333,983 edges, and there are 39 labels. This graph represent a network of social relationships given by bloggers and the labels are the bloggers’ interests. We train a feedforward neural network with one hidden layer of 50 units and train each class as a one-vs-rest binary classification task. Since there are no features for each node, we use the rows of the adjacency matrix as inputs to the network, as discussed in section 3.2. Since we use the test set to construct the graph and augment the training objective, the learning in this experiment is transductive. Since the training set is extremely unbalanced, we employ weighted sampling during training, i.e. making sure each minibatch has both positive and negative examples. In this experiment, we fix αi to be equal, and experiment with α = 0 and 0.1 (0 means no edge information during training); we use the l-2 metric to compute the distance between the hidden representations. We compare our method against a two-stage approach: use node2vec (Grover & Leskovec, 2016) to generate node embeddings and use a linear one-vs-rest classifier for classification. The methods are evaluated using two metrics Macro F1 and Micro F1. The results for different train/test splits and different α values, together with the baseline are included in table 1. The results demonstrate that 1. using the graph itself as direct inputs to the neural network and letting the network learning a non-linear mapping is more effective than the two-stage approach considered, 2. using the graph information improves the performance in the small data regime (for example: when training set is only 20% of the dataset). We observe the same improvement over Node2vec on the Micro F1 metric and α = 0.1 is comparable to α = 0 but performs better on the recall metric.
*These results are different compared to Grover & Leskovec (2016), since we treat the classifiers (one per label) independently. This setting is the same as for our NGM-NN classifiers.
4.2 TEXT CLASSIFICATION USING CHARACTER-LEVEL CNNS
We evaluate the proposed objective function on a multi-class text classification task using a character-level convolutional neural network (CNN). We use the AG news dataset from Zhang et al. (2015), where the task is to classify a news article into one of 4 categories. Each category has 30,000 examples for training and 1,900 examples for testing. In addition to the train and test sets, there are 111,469 examples that are treated as unlabeled examples.
We restrict the graph construction to only the train set and the unlabeled examples and keep the test set only for evaluation. We use the Google News word2vec corpus to calculate the average embedding for each news article and use the cosine similarity of document embeddings as a similarity metric. Each node is restricted to 5 neighbors.
We construct the CNN in the same way as Zhang et al. (2015), but with significantly smaller layers, as shown in table 2:
The network is trained with the same parameters as Zhang et al. (2015) but only for 20 epochs. We compare the final outputs using the cross entropy loss, that is d = cross entropy(g(xu), g(xv)). Using the proposed objective function, the NGM-CNN provides a 1.8% absolute and 2.1% relative improvement in accuracy, despite using a smaller network. We show the results in table 3.
4.3 SEMANTIC INTENT CLASSIFICATION USING LSTM RNNS
Finally, we compare the performance of our approach for training RNN sequence models (LSTM) for a semantic intent classification task as described in the recent work on SmartReply (Kannan et al., 2016) for automatically generating short email responses. One of the underlying tasks in SmartReply is to discover and map short response messages to semantic intent clusters.1 We choose 20 intent classes and created a dataset comprised of 5,483 samples (3,832 for training, 560 for validation and 1,091 for testing). Each sample instance corresponds to a short response message text paired with a semantic intent category that was manually verified by human annotators. For example, “That
1For details regarding SmartReply and how the semantic intent clusters are generated, refer Kannan et al. (2016).
sounds awesome!” and “Sounds fabulous” belong to the sounds good intent cluster. We construct a sparse graph in a similar manner as the news categorization task using word2vec embeddings over the message text and computing similarity to generate a response message graph with fixed node degree (k=10). We use l-2 for the distance metric d(·) and choose α based on the development set. We run the experiments for a fixed number of time steps and pick the best results on the development set. A multilayer LSTM architecture (2 layers, 100 dimensions) is used for the RNN sequence model. The LSTM model and its NGM variant are also compared against other baseline systems—Random baseline ranks the intent categories randomly and Frequency baseline ranks them in order of their frequency in the training corpus. To evaluate the intent prediction quality of different approaches, for each test instance, we compute the rank of the actual intent category ranki with respect to the ranking produced by the method and use this to calculate the Mean Reciprocal Rank:
MRR = 1
N N∑ i=1 1 ranki
We show in table 4 that LSTM RNNs with our proposed graph-augmented training objective function outperform standard baselines by offering a better MRR.
5 CONCLUSIONS
We have proposed a training objective for neural network architectures that can leverage both labeled and unlabeled data. Inspired by the label propagation objective function, the proposed objective biases the neural networks to learn similar hidden representations for nodes connected by an edge on the graph. Importantly, this objective can be trained by stochastic gradient descent, as in supervised neural network training. We validate the efficacy of the graph-augmented objective on various state-of-the-art neural network architectures on bloggers’ interest, text category and semantic intent classification problems. Additionally, the node-level input features can be combine with graph features as inputs to the neural network. We showed that a neural network that simply takes the adjacency matrix of a graph and produces node labels, can perform better than a recently proposed two-stage approach using sophisticated graph embeddings and a linear classifier.
While our objective can be applied to multiple graphs which come from different domains, we have not fully explored this aspect and leave this as future work. We expect the domain-specific networks can interact with the graphs to determine the importance of each domain/graph source in prediction. Another possible future work is to use our objective on directed graphs, that is to control the direction of influence between nodes during training.
ACKNOWLEDGMENTS
We would like to thank the Google Expander team for insightful feedbacks.

This paper proposes the Neural Graph Machine that adds in graph regularization on neural network hidden representations to improve network learning and take the graph structure into account.  The proposed model, however, is almost identical to that of Weston et al. 2012.
As the authors have clarified in the answers to the questions, there are a few new things that previous work did not do:
1. they showed that graph augmented training for a range of different types of networks, including FF, CNN, RNNs etc. and works on a range of problems.
2. graphs help to train better networks, e.g. 3 layer CNN with graphs does as well as than 9 layer CNNs
3. graph augmented training works on a variety of different kinds of graphs.
However, all these points mentioned above seems to simply be different applications of the graph augmented training idea, and observations made during the applications.  I think it is therefore not proper to call the proposed model a novel model with a new name Neural Graph Machine, but rather making it clear in the paper that this is an empirical study of the model proposed by Weston et al. 2012 to different problems would be more acceptable.


In this paper we propose a simple and efficient method for improving stochastic gradient descent methods by using feedback from the objective function. The method tracks the relative changes in the objective function with a running average, and uses it to adaptively tune the learning rate in stochastic gradient descent. We specifically apply this idea to modify Adam, a popular algorithm for training deep neural networks. We conduct experiments to compare the resulting algorithm, which we call Eve, with state of the art methods used for training deep learning models. We train CNNs for image classification, and RNNs for language modeling and question answering. Our experiments show that Eve outperforms all other algorithms on these benchmark tasks. We also analyze the behavior of the feedback mechanism during the training process.
1 INTRODUCTION
Despite several breakthrough results in the last few years, the training of deep learning models remains a challenging problem. This training is a complex, high-dimensional, non-convex, stochastic optimization problem which is not amenable to many standard methods. Currently, the most common approach is to use some variant of stochastic gradient descent. Many extensions have been proposed to the basic gradient descent algorithm - designed to handle specific issues in the training of deep learning models. We review some of these methods in the next section.
Although variants of simple stochastic gradient descent work quite well in practice, there is still room for improvement. This is easily evidenced by the existence of numerous methods to simplify the optimization problem itself like weight initialization techniques and normalization methods.
In this work, we seek to improve stochastic gradient descent with a simple method that incorporates feedback from the objective function. The relative changes in the objective function indicate progress of the optimization algorithm. Our main hypothesis is that incorporating information about this change into the optimization algorithm can lead to improved performance - quantified in terms of the progress rate. We keep a running average of the relative changes in the objective function and use it to divide the learning rate. When the average relative change is high, the learning rate is reduced. This can improve the progress if, for example, the algorithm is bouncing around the walls of the objective function. Conversely, when the relative change is low, the learning rate is increased. This can help the algorithm accelerate through flat areas in the loss surface. As we discuss in the next section, such “plateaus” pose a significant challenge for first order methods and can create the illusion of local minima.
While our method is general i.e. independent of any particular optimization algorithm, in this work we specifically apply the method to modify Adam (Kingma & Ba, 2014), considered to be the state of the art for training deep learning models. We call the resulting algorithm Eve and design experiments to compare it with Adam, as well as other popular methods from the literature.
The paper is organized as follows. In Section 2, we review recent results related to the optimization of deep neural networks. We also discuss some popular algorithms and their motivations. Our general
∗Equal contribution.
method, and the specific algorithm Eve are discussed in Section 3. Then in Section 4, we show that Eve consistently outperforms other methods in training convolutional neural networks (CNNs), and recurrent neural networks (RNNs). We also look in some detail, at the behavior of our method in the simple case of convex non-stochastic optimization. Finally we conclude in Section 5.
2 RELATED WORK
There has been considerable effort to understand the challenges in deep learning optimization. Intuitively, it seems that the non-convex optimization is made difficult by the presence of several poor local optima. However, this geometric intuition proves to be inadequate in reasoning about the high-dimensional case that arises with training deep learning models. Various empirical and theoretical results (Bray & Dean, 2007; Dauphin et al., 2014) have indicated that the problem in high dimensions arises not from local minima, but rather from saddle points. Moreover, a recent paper (Kawaguchi, 2016) proved (for deep linear networks, and under reasonable assumptions, also for deep non-linear networks) that all local minima in fact achieve the same value, and are optimal. The work also showed that all critical points which are not global minima are saddle points. Saddle points can seriously hamper the progress of both first and second order methods. Second order methods like Newton’s method are actually attracted to saddle points and are not suitable for high dimensional non-convex optimization. First order methods can escape from saddle points by following directions of negative curvature. However, such saddle points are usually surrounded by regions of small curvature - plateaus. This makes first order methods very slow near saddle points and can create the illusion of a local minimum.
To tackle the saddle point problem, Dauphin et al. (2014) propose a second order method that fixes the issue with Newton’s method. Their algorithm builds on considering the behavior of Newton’s method near saddle points. Newton’s method rescales gradients in each eigen-direction with the corresponding inverse eigenvalue. However, near a saddle point, negative eigenvalues can cause the method to move towards the saddle point. Based on this observation, the authors propose using the absolute values of the eigenvalues to rescale the gradients. This saddle-free Newton method is backed by theoretical justifications and empirical results; however due to the computational requirements, second order methods are not very suitable for training large scale models. So we do not compare with such approaches in this work.
We instead focus on first order methods which only rely on the gradient information. A key issue in training deep learning models is that of sparse gradients. To handle this, Adagrad (Duchi et al., 2011) adaptively changes the learning rate for each parameter, performing larger updates for infrequently updated parameters. However its update rule causes the learning rate to monotonically decrease, which eventually stalls the algorithm. Adadelta (Zeiler, 2012) and RMSProp (Tieleman & Hinton, 2012) are two extensions that try to fix this issue. Finally, a closely related method, and the base for our algorithm Eve (introduced in the next section), is Adam (Kingma & Ba, 2014). Adam incorporates the advantages of both Adagrad and RMSProp - and it has been found to work quite well in practice. Adam uses a running average of the gradient to determine the direction of descent, and scales the learning rate with a running average of the gradient squared. The authors of Adam also propose an extension based on the infinity norm, called Adamax. In our experiments, we compare Eve with both Adam and Adamax.
3 METHOD
3.1 ASSUMPTION
We do need to make an assumption about the objective function f . We assume that the minimum value of f over its domain is known. While this is true for loss functions encountered in machine learning (like mean squared error or cross entropy), it does not hold if the objective function also includes regularization terms (L1, L2 etc.). In all our experiments, we used dropout for regularization which is not affected by this assumption. Finally, to simplify notation we assume that the minimum has been subtracted from the objective function i.e. the minimum has been made 0.
Algorithm 1 Eve: Adam with feedback. Parameters carried over from Adam have the same default values: α = 10−3, β1 = 0.9, β2 = 0.999, = 10−8. For parameters specific to our method, we recommend default values β3 = 0.999, k = 0.1, K = 10. Wherever applicable, products are elementwise. Require: α: learning rate Require: β1, β2 ∈ [0, 1): exponential decay rates for moment estimation in Adam Require: β3 ∈ [0, 1): exponential decay rate for computing relative change Require: k,K: lower and upper threshold for relative change Require: : fuzz factor Require: f(θ): objective function Require: θ0: initial value for parameters m0 = v0 = 0 d0 = 1 f̂−1 = t = 0 while stopping condition is not reached do
t← t+ 1 gt ← ∇θf(θt−1) mt ← β1mt−1 + (1− β1)gt m̂t ← mt(1−βt1) vt ← β2vt−1 + (1− β2)g2t v̂t ← vt(1−βt2) if t > 1 then
if f(θt−1) ≥ f̂t−2 then δt ← k + 1 ∆t ← K + 1 else δt ← 1K+1 ∆t ← 1k+1 end if ct ← min { max { δt, f(θt−1)
f̂t−2
} ,∆t } f̂t−1 ← ctf̂t−2 rt ← |f̂t−1−f̂t−2| min{f̂t−1,f̂t−2} dt ← β3dt−1 + (1− β3)rt
else f̂t−1 ← f(θt−1) dt ← 1 end if θt ← θt−1 − α m̂tdt√v̂t+
end while return θt
3.2 ALGORITHM
The main component of our proposed method is a feedback term that captures the relative change in the objective value. Let ft−2 and ft−1 denote the values of the objective function at time steps t− 2 and t−1 respectively. Then this change is computed as rt = ft−2−ft−1ft−1 = ft−2−ft−1 ft−1
if ft−2 ≥ ft−1, and ft−1−ft−2ft−2 otherwise. Note that this value is always non-negative but it can be less than or greater than 1 i.e. it captures both relative increase and decrease. We compute a running average using these relative changes to get a smoother estimate. Specifically, we take d1 = 1, and for t > 1 define dt as dt = βdt−1 + (1 − β)rt. Here β ∈ [0, 1) is a decay rate - large values correspond to a slowly changing dt, and vice versa. This simple expression can, however, blow up and lead to instability. To handle this issue, we use a thresholding scheme. A simple thing to do would be to clip dt as min{max{k, dt},K} for some suitable 0 < k < K. But we found this to not work very well in practice due to the abrupt nature of the clipping. Instead we indirectly clip dt by smoothly tracking
the objective function. Let f̂t−2 be the value of the smoothly tracked objective function at time t− 2 with f̂0 = f0. For now, assume ft−1 ≤ f̂t−2. We would like to have k ≤ rt ≤ K which in this case requires ft−1
f̂t−2 ∈
[ 1
K+1 , 1 k+1 ] . So we compute ct = min { max { 1 K+1 , ft f̂t−1 } , 1k+1 } and set
f̂t−1 = ctf̂t−2. Finally rt is f̂t−2−f̂t−1
f̂t−1 . Analogous expressions can also be derived for the case when
ft−1 > f̂t−2. This smooth tracking has the additional advantage of making dt less susceptible to the high variability that comes with training using minibatches.
Once dt has been computed, it can be used to modify any gradient descent algorithm by modifying the learning rate α as αt = α/dt. Large values of dt, caused by large changes in the objective function will lead to a smaller effective learning rate. Similarly, small values of dt will lead to a larger effective learning rate. Since we start with d0 = 1, the initial updates will closely follow that of the base algorithm. In the next section, we will look at how dt evolves during the course of an experiment to get a better understanding of how it affects the training.
We note again that our method is independent of any particular gradient descent algorithm. However, for this current work, we specifically focus on applying the method to Adam (Kingma & Ba, 2014). This modified algorithm, which we call Eve, is shown in Algorithm 1. We modify the final Adam update by multiplying the denominator √ vt with dt. In addition to the hyperparameters in Adam, we introduce 3 new hyperparameters β3, k, and K. In all our experiments we use the values β3 = 0.999, k = 0.1, and K = 10, which we found to work well in practice.
4 EXPERIMENTS
Now we evaluate our proposed method by comparing Eve with several state of the art algorithms for optimizing deep learning models.1. In all experiments, we used ReLU activations, and initialized weights according to the scheme proposed by Glorot & Bengio (2010). We used minibatches of size 128, and linear decay for the learning rate: αt = α/(1 + γt) (γ is the decay rate, picked by searching over a range of values).
In the figures, SGD refers to vanilla stochastic gradient descent, and SGD Nesterov refers to stochastic gradient descent with Nesterov momentum (Nesterov, 1983) where we set the momentum to 0.9 in all experiments.
4.1 CONVOLUTIONAL NEURAL NETWORKS
We first trained a 5 layer convolutional neural network for 10-way classification of images from the CIFAR10 dataset (Krizhevsky & Hinton, 2009). The model consisted of 2 blocks of 3x3 convolutional layers each followed by 2x2 max pooling and 0.25 dropout (Srivastava et al., 2014). The first block
1Full technical details of the experiments and additional results are available at https://github.com/ jayanthkoushik/sgd-feedback.
contained 2 layers with 32 filters each, and the second block contained 2 layers with 64 filters each. The convolutional layers were followed by a fully connected layer with 512 units and a 10-way softmax layer. We trained this model for 500 epochs on the training split using various popular methods for training deep learning models, as well as Eve. For each algorithm, we tried learning rates {10−2, 10−3, 10−4} (for algorithms with suggested default learning rates, we also included them in the search), learning rate decays {0, 10−2, 10−3, 10−4}, and picked the pair of values that led to the smallest final training loss. The loss curves are shown in Figure 1a. Eve quickly surpasses all other methods and achieves the lowest final training loss. In the next section we will look at the behavior of the adaptive coefficient dt to gain some intuition behind this improved performance.
We also trained a larger CNN model using the top-performing algorithms from the previous experiment. This model consisted of 3 blocks of 3x3 convolutional layers (3 layers per block, and 64, 128, 256 filters per layer in the first, second, and third block respectively) each followed by 2x2 max pooling and 0.5 dropout. Then we had 2 fully connected layers with 512 units, each followed by 0.5 dropout, and finally a 100-way softmax. We trained this model on the CIFAR100 (Krizhevsky & Hinton, 2009) dataset for 100 epochs. We again performed a grid search over the same learning rate and decay values as the last experiment. The results are shown in Figure 1b, and once again show that our proposed method improves over state of the art methods for training convolutional neural networks.
4.2 ANALYSIS OF TUNING COEFFICIENT
Before we consider the next set of experiments on recurrent neural networks, we will first look more closely at the behavior of the tuning coefficient dt in our algorithm. We will specifically consider the results from the CNN experiment on CIFAR10. Figure 2 shows the progress of dt throughout the training, and also in two smaller windows. A few things are worth noting here. First is that of the overall trend. There is an initial acceleration followed by a decay. This initial acceleration allows Eve to rapidly overtake other methods, and makes it proceed at a faster pace for about 100 epochs. This acceleration is not equivalent to simply starting with a larger learning rate - in all our experiments we search over a range of learning rate values. The overall trend for dt can be explained by looking at the minibatch losses at each iteration (as opposed to the loss computed over the entire dataset after each epoch) in Figure 3. Initially, different minibatches achieve similar loss values which leads to dt decreasing. But as training proceeds, the variance in the minibatch losses increases and dt eventually increases. However, this overall trend does not capture the complete picture - for example, as shown in the bottom right plot of Figure 2, dt can actually be decreasing in some regions of the training, adjusting to local structures in the error surface.
To further study the observed acceleration, and to also motivate the need for clipping, we consider a simpler experiment. We trained a logistic regression model on 1000 images from the MNIST dataset. We used batch gradient descent for training i.e. all 1000 samples were used for computing the gradient at each step. We trained this model using Eve, Adam, Adamax, and SGD Nesterov
for 10000 iterations, searching over a large range of values for the learning rate and decay: α ∈ {10−1, 10−2, 10−3, 10−4, 10−5}, γ ∈ {10−2, 10−3, 10−4, 10−5, 10−6, 0}. The results are shown in Figure 4. Eve again outperforms all other methods and achieves the lowest training loss. Also, since this is a smooth non-stochastic problem, the tuning coefficient dt continuously decreases - this makes having a thresholding mechanism important since the learning rate would blow up otherwise.
Although in the previous experiment the effect of our method is to increase the learning rate, it is not equivalent to simply starting with a larger learning rate. We will establish this with a couple simple experiments. First we note that in the previous experiment, the optimal decay rates for both Adam and Eve were 0 - no decay. The optimal learning rate for Eve was 10−2. Since the tuning coefficient converges to 0.1, we trained Adam using no decay, and learning rates i× 10−2 where i varies from 1 to 10. The training loss curves are shown in the left plot of Figure 5. While increasing the learning rate with Adam does seem to close the gap with Eve, Eve does remain marginally faster. Moreover, and more importantly, this first plot represents the best-case situation for Adam. With larger learning rates, training becomes increasingly unstable and sensitive to the initial values of the parameters. This is illustrated in the right plot of Figure 5 where we used Eve (with learning rate 10−2) and Adam (with learning rate 10−1) 10 times with different random initializations. In some cases, Adam fails to converge whereas Eve always converges - even though Eve eventually reaches a learning rate of 10−1. This is because very early in the training, the model is quite sensitive at higher learning rates due to larger gradients. Depending on the initial values, the algorithm may or may not converge. So it is advantageous to slowly accelerate as the learning stabilizes rather than start with a larger learning rate.
4.3 RECURRENT NEURAL NETWORKS
Finally, we evaluated our method on recurrent neural networks (RNNs). We first trained a RNN for character-level language modeling on the Penn Treebank dataset (Marcus et al., 1993). Specifically, the model consisted of a 2-layer character-level Gated Recurrent Unit (Chung et al., 2014) with hidden layers of size 256, 0.5 dropout between layers, and sequences of 100 characters. We adopted 10−3 as the initial learning rate for Adam, Eve, and RMSProp. For Adamax, we used 2× 10−3 as the learning rate since it is the suggested value. We used 3× 10−4 for the learning rate decay. We
trained this model for 100 epochs using each of the algorithms. The results, plotted in Figure 6c, clearly show that our method achieves the best results. Eve optimizes the model to a lower final loss than the other models.
We trained another RNN-based model for the question & answering task. Specifically, we chose two question types among 20 types from the bAbI dataset (Weston et al., 2015), Q19 and Q14. The dataset consists of pairs of supporting story sentences and a question. Different types of pairs are said to require different reasoning schemes. For our test case, Q19 and Q14 correspond to Path Finding and Time Reasoning respectively. We picked Q19 since it is reported to have the lowest baseline score, and we picked Q14 randomly from the remaining questions. The model consisted of two parts, one for encoding story sentences and another for query. Both included an embedding layer with 256 hidden units, and 0.3 dropout. Next query word embeddings were fed into a GRU one token at a time, to compute a sentence representation. Both story and query sequences were truncated to the maximum sequence length of 100. Finally, the sequence of word embeddings from story sentences and the repeated encoded representation of a query were combined together to serve as input for each time step in another GRU, with 0.3 dropout. We searched for the learning rate and decay from a range of values, α ∈ {10−2, 10−3, 10−4} and γ ∈ {10−2, 10−3, 10−4, 0}. The results, shown in Figures 6a, and 6b show that Eve again improves over all other methods.
5 CONCLUSION
We proposed a simple and efficient method for incorporating feedback in to stochastic gradient descent algorithms. We used this method to create Eve, a modified version of the Adam algorithm. Experiments with a variety of models showed that the proposed method can help improve the optimization of deep learning models.
For future work, we would look to theoretically analyze our method and its effects. While we have tried to evaluate our algorithm Eve on a variety of tasks, additional experiments on larger scale problems would further highlight the strength of our approach. We are making code for our method and the experiments publicly available to encourage more research on this method.

The paper introduced an extension of Adam optimizer that automatically adjust learning rate by comparing the subsequent values of the cost function during training. The authors empirically demonstrated the benefit of the Eve optimizer on CIFAR convnets, logistic regression and RNN problems.
I have the following concerns about the paper
- The proposed method is VARIANT to arbitrary shifts and scaling to the cost function.
- A more fair comparison with other baseline methods would be using additional exponential decay learning scheduling between the lower and upper threshold of d_t. I suspect 1/d_t just shrinks as an exponential decay from Figure 2.
- Three additional hyper-parameters: k, K, \beta_3.
Overall, I think the method has its fundamental flew and the paper offers very limited novelty. There is no theoretical justification on the modification, and it would be good for the authors to discuss the potential failure mode of the proposed method. Furthermore, it is hard for me to follow Section 3.2. The writing quality and clarity of the method section can be further improved.


EPOPT: LEARNING ROBUST NEURAL NETWORK POLICIES USING MODEL ENSEMBLES
Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.
1 INTRODUCTION
Reinforcement learning with powerful function approximators like deep neural networks (deep RL) has recently demonstrated remarkable success in a wide range of tasks like games (Mnih et al., 2015; Silver et al., 2016), simulated control problems (Lillicrap et al., 2015; Mordatch et al., 2015b), and graphics (Peng et al., 2016). However, high sample complexity is a major barrier for directly applying model-free deep RL methods for physical control tasks. Model-free algorithms like Q-learning, actor-critic, and policy gradients are known to suffer from long learning times (Kakade, 2003), which is compounded when used in conjunction with expressive function approximators like deep neural networks (DNNs). The challenge of gathering samples from the real world is further exacerbated by issues of safety for the agent and environment, since sampling with partially learned policies could be unstable (Garcı́a & Fernández, 2015). Thus, model-free deep RL methods often require a prohibitively large numbers of potentially dangerous samples for physical control tasks.
Model-based methods, where the real-world target domain is approximated with a simulated source domain, provide an avenue to tackle the above challenges by learning policies using simulated data. The principal challenge with simulated training is the systematic discrepancy between source and target domains, and therefore, methods that compensate for systematic discrepancies (modeling errors) are needed to transfer results from simulations to real world using RL. We show that the impact of such discrepancies can be mitigated through two key ideas: (1) training on an ensemble of models in an adversarial fashion to learn policies that are robust to parametric model errors, as well as to unmodeled effects; and (2) adaptation of the source domain ensemble using data from the target domain to progressively make it a better approximation. This can be viewed either as an instance of model-based Bayesian RL (Ghavamzadeh et al., 2015); or as transfer learning from a collection of simulated source domains to a real-world target domain (Taylor & Stone, 2009). While a number of model-free RL algorithms have been proposed (see, e.g., Duan et al. (2016) for a survey), their high sample complexity demands use of a simulator, effectively making them model-based. We
show in our experiments that such methods learn policies which are highly optimized for the specific models used in the simulator, but are brittle under model mismatch. This is not surprising, since deep networks are remarkably proficient at exploiting any systematic regularities in a simulator. Addressing robustness of DNN-policies is particularly important to transfer their success from simulated tasks to physical systems.
In this paper, we propose the Ensemble Policy Optimization (EPOpt− ) algorithm for finding policies that are robust to model mismatch. In line with model-based Bayesian RL, we learn a policy for the target domain by alternating between two phases: (i) given a source (model) distribution (i.e. ensemble of models), find a robust policy that is competent for the whole distribution; (ii) gather data from the target domain using said robust policy, and adapt the source distribution. EPOpt uses an ensemble of models sampled from the source distribution, and a form of adversarial training to learn robust policies that generalize to a broad range of models. By robust, we mean insensitivity to parametric model errors and broadly competent performance for direct-transfer (also referred to as jumpstart like in Taylor & Stone (2009)). Direct-transfer performance refers to the average initial performance (return) in the target domain, without any direct training on the target domain. By adversarial training, we mean that model instances on which the policy performs poorly in the source distribution are sampled more often in order to encourage learning of policies that perform well for a wide range of model instances. This is in contrast to methods which learn highly optimized policies for specific model instances, but brittle under model perturbations. In our experiments, we did not observe significant loss in performance by requiring the policy to work on multiple models (for example, through adopting a more conservative strategy). Further, we show that policies learned using EPOpt are robust even to effects not modeled in the source domain. Such unmodeled effects are a major issue when transferring from simulation to the real world. For the model adaptation step (ii), we present a simple method using approximate Bayesian updates, which progressively makes the source distribution a better approximation of the target domain. We evaluate the proposed methods on the hopper (12 dimensional state space; 3 dimensional action space) and half-cheetah (18 dimensional state space; 6 dimensional action space) benchmarks in MuJoCo. Our experimental results suggest that adversarial training on model ensembles produces robust policies which generalize better than policies trained on a single, maximum-likelihood model (of source distribution) alone.
2 PROBLEM FORMULATION
We consider parametrized Markov Decision Processes (MDPs), which are tuples of the form: M(p) ≡< S,A, Tp,Rp, γ, S0,p > where S, A are (continuous) states and actions respectively; Tp Rp, and S0,p are the state transition, reward function, and initial state distribution respectively, all parametrized by p; and γ is the discount factor. Thus, we consider a set of MDPs with the same state and action spaces. Each MDP in this set could potentially have different transition functions, rewards, and initial state distributions. We use transition functions of the form St+1 ≡ Tp(st, at) where Tp is a random process and St+1 is a random variable.
We distinguish between source and target MDPs usingM andW respectively. We also refer toM andW as source and target domains respectively, as is common in the transfer learning set-up. Our objective is to learn the optimal policy forW; and to do so, we have access toM(p). We assume that we have a distribution (D) over the source domains (MDPs) generated by a distribution over the parameters P ≡ P(p) that capture our subjective belief about the parameters ofW . Let P be parametrized by ψ (e.g. mean, standard deviation). For example,M could be a hopping task with reward proportional to hopping velocity and falling down corresponds to a terminal state. For this task, p could correspond to parameters like torso mass, ground friction, and damping in joints, all of which affect the dynamics. Ideally, we would like the target domain to be in the model class, i.e. {∃p | M(p) =W}. However, in practice, there are likely to be unmodeled effects, and we analyze this setting in our experiments. We wish to learn a policy π∗θ(s) that performs well for allM∼ D. Note that this robust policy does not have an explicit dependence on p, and we require it to perform well without knowledge of p.
3 LEARNING PROTOCOL AND EPOPT ALGORITHM
We follow the round-based learning protocol of Bayesian model-based RL. We use the term rounds when interacting with the target domain, and episode when performing rollouts with the simulator. In each round, we interact with the target domain after computing the robust policy on the current (i.e.
posterior) simulated source distribution. Following this, we update the source distribution using data from the target domain collected by executing the robust policy. Thus, in round i, we update two sets of parameters: θi, the parameters of the robust policy (neural network); and ψi, the parameters of the source distribution. The two key steps in this procedure are finding a robust policy given a source distribution; and updating the source distribution using data from the target domain. In this section, we present our approach for both of these steps.
3.1 ROBUST POLICY SEARCH
We introduce the EPOpt algorithm for finding a robust policy using the source distribution. EPOpt is a policy gradient based meta-algorithm which uses batch policy optimization methods as a subroutine. Batch policy optimization algorithms (Williams, 1992; Kakade, 2001; Schulman et al., 2015) collect a batch of trajectories by rolling out the current policy, and use the trajectories to make a policy update. The basic structure of EPOpt is to sample a collection of models from the source distribution, sample trajectories from each of these models, and make a gradient update based on a subset of sampled trajectories. We first define evaluation metrics for the parametrized policy, πθ:
ηM(θ, p) = Eτ̃ [ T−1∑ t=0 γtrt(st, at) ∣∣∣∣ p ] , (1)
ηD(θ) = Ep∼P [ηM(θ, p)] = Ep∼P [ Eτ̂ [ T−1∑ t=0 γtrt(st, at) ∣∣∣∣ p ]] = Eτ [ T−1∑ t=0 γtrt(st, at) ] .
In (1), ηM(θ, p) is the evaluation of πθ on the model M(p), with τ̃ being trajectories generated by M(p) and πθ: τ̃ = {st, at, rt}Tt=0 where st+1 ∼ Tp(st, at), s0 ∼ S0,p, rt ∼ Rp(st, at), and at ∼ πθ(st). Similarly, ηD(θ) is the evaluation of πθ over the source domain distribution. The corresponding expectation is over trajectories τ generated by D and πθ: τ = {st, at, rt}Tt=0, where st+1 ∼ Tpt(st, at), pt+1 = pt, s0 ∼ S0,p0 , rt ∼ Rpt(st, at), at ∼ πθ(st), and p0 ∼ P . With this modified notation of trajectories, batch policy optimization can be invoked for policy search.
Optimizing ηD allows us to learn a policy that performs best in expectation over models in the source domain distribution. However, this does not necessarily lead to a robust policy, since there could be high variability in performance for different models in the distribution. To explicitly seek a robust policy, we use a softer version of max-min objective suggested in robust control, and optimize for the conditional value at risk (CVaR) (Tamar et al., 2015):
max θ,y ∫ F(θ) ηM(θ, p)P(p)dp s.t. P (ηM(θ, P ) ≤ y) = , (2)
where F(θ) = {p | ηM(θ, p) ≤ y} is the set of parameters corresponding to models that produce the worst percentile of returns, and provides the limit for the integral; ηM(θ, P ) is the random variable of returns, which is induced by the distribution over model parameters; and is a hyperparameter which governs the level of relaxation from max-min objective. The interpretation is that (2) maximizes the expected return for the worst -percentile of MDPs in the source domain distribution. We adapt the previous policy gradient formulation to approximately optimize the objective in (2). The resulting algorithm, which we call EPOpt- , generalizes learning a policy using an ensemble of source MDPs which are sampled from a source domain distribution.
In Algorithm 1, R(τk) ≡ ∑T−1 t=0 γ
trt,k denotes the discounted return obtained in trajectory sample τk. In line 7, we compute the −percentile value of returns from the N trajectories. In line 8, we find the subset of sampled trajectories which have returns lower than Q . Line 9 calls one step of an underlying batch policy optimization subroutine on the subset of trajectories from line 8. For the CVaR objective, it is important to use a good baseline for the value function. Tamar et al. (2015) show that without a baseline, the resulting policy gradient is biased and not consistent. We use a linear function as the baseline with a time varying feature vector to approximate the value function, similar to Duan et al. (2016). The parameters of the baseline are estimated using only the subset of trajectories with return less than Q . We found that this approach led to empirically good results.
For small values of , we observed that using the sub-sampling step from the beginning led to unstable learning. Policy gradient methods adjust parameters of policy to increase probability of trajectories
Algorithm 1: EPOpt– for Robust Policy Search 1 Input: ψ, θ0, niter, N , 2 for iteration i = 0, 1, 2, . . . niter do 3 for k = 1, 2, . . . N do 4 sample model parameters pk ∼ Pψ 5 sample a trajectory τk = {st, at, rt, st+1}T−1t=0 fromM(pk) using policy π(θi) 6 end 7 compute Q = percentile of {R(τk)}Nk=1 8 select sub-set T = {τk : R(τk) ≤ Q } 9 Update policy: θi+1 = BatchPolOpt(θi,T)
10 end
with high returns and reduce probability of poor trajectories. EPOpt− due to the sub-sampling step emphasizes penalizing poor trajectories more. This might constrain the initial exploration needed to find good trajectories. Thus, we initially use a setting of = 1 for few iterations before setting epsilon to the desired value. This corresponds to exploring initially to find promising trajectories and rapidly reducing probability of trajectories that do not generalize.
3.2 ADAPTING THE SOURCE DOMAIN DISTRIBUTION
In line with model-based Bayesian RL, we can adapt the ensemble distribution after observing trajectory data from the target domain. The Bayesian update can be written as:
P(P |τk) = 1
Z × P(τk|P )× P(P ) =
1 Z × T−1∏ t=0 P(St+1 = s(k)t+1|s (k) t , a (k) t , p)× P(P = p), (3)
where 1Z is the partition function (normalization) required to make the probabilities sum to 1, St+1 is the random variable representing the next state, and ( s (k) t , a (k) t , s (k) t+1 )T t=0
are data observed along trajectory τk. We try to explain the target trajectory using the stochasticity in the state-transition function, which also models sensor errors. This provides the following expression for the likelihood:
P(St+1|st, at, p) ≡ Tp(st, at). (4) We follow a sampling based approach to calculate the posterior, by sampling a set of model parameters: pi = [p1, p2, . . . , pM ] from a sampling distribution, PS(pi). Consequently, using Bayes rule and importance sampling, we have:
P(pi|τk) ∝ L(τk|pi)× PP (pi) PS(pi) , (5)
where PP (pi) is the probability of drawing pi from the prior distribution; and L(τk|pi) is the likelihood of generating the observed trajectory with model parameters pi. The weighted samples from the posterior can be used to estimate a parametric model, as we do in this paper. Alternatively, one could approximate the continuous probability distribution using discrete weighted samples like in case of particle filters. In cases where the prior has very low probability density in certain parts of the parameter space, it might be advantageous to choose a sampling distribution different from the prior. The likelihood can be factored using the Markov property as: L(τk|pi) = ∏ t P(St+1 = s (k) t+1|s (k) t , a (k) t , pi). This simple model adaptation rule allows us to illustrate the utility of EPOpt for robust policy search, as well as its integration with model adaptation to learn policies in cases where the target model could be very different from the initially assumed distribution.
4 EXPERIMENTS
We evaluated the proposed EPOpt- algorithm on the 2D hopper (Erez et al., 2011) and halfcheetah (Wawrzynski, 2009) benchmarks using the MuJoCo physics simulator (Todorov et al., 2012).1 Both tasks involve complex second order dynamics and direct torque control. Underactuation,
1Supplementary video: https://youtu.be/w1YJ9vwaoto
high dimensionality, and contact discontinuities make these tasks challenging reinforcement learning benchmarks. These challenges when coupled with systematic parameter discrepancies can quickly degrade the performance of policies and make them unstable, as we show in the experiments. The batch policy optimization sub-routine is implemented using TRPO. We parametrize the stochastic policy using the scheme presented in Schulman et al. (2015). The policy is represented with a Gaussian distribution, the mean of which is parametrized using a neural network with two hidden layers. Each hidden layer has 64 units, with a tanh non-linearity, and the final output layer is made of linear units. Normally distributed independent random variables are added to the output of this neural network, and we also learn the standard deviation of their distributions. Our experiments are aimed at answering the following questions:
1. How does the performance of standard policy search methods (like TRPO) degrade in the presence of systematic physical differences between the training and test domains, as might be the case when training in simulation and testing in the real world?
2. Does training on a distribution of models with EPOpt improve the performance of the policy when tested under various model discrepancies, and how much does ensemble training degrade overall performance (e.g. due to acquiring a more conservative strategy)?
3. How does the robustness of the policy to physical parameter discrepancies change when using the robust EPOpt- variant of our method?
4. Can EPOpt learn policies that are robust to unmodeled effects – that is, discrepancies in physical parameters between source and target domains that do not vary in the source domain ensemble?
5. When the initial model ensemble differs substantially from the target domain, can the ensemble be adapted efficiently, and how much data from the target domain is required for this?
In all the comparisons, performance refers to the average undiscounted return per trajectory or episode (we consider finite horizon episodic problems). In addition to the previously defined performance, we also use the 10th percentile of the return distribution as a proxy for the worst-case return.
4.1 COMPARISON TO STANDARD POLICY SEARCH
In Figure 1, we evaluate the performance of standard TRPO and EPOpt( = 0.1) on the hopper task, in the presence of a simple parametric discrepancy in the physics of the system between the training (source) and test (target) domains. The plots show the performance of various policies on test domains with different torso mass. The first three plots show policies that are each trained on a single torso mass in the source domain, while the last plot illustrates the performance of EPOpt,
which is trained on a Gaussian mass distribution. The results show that no single torso mass value produces a policy that is successful in all target domains. However, the EPOpt policy succeeds almost uniformly for all tested mass values. Furthermore, the results show that there is almost no degradation in the performance of EPOpt for any mass setting, suggesting that the EPOpt policy does not suffer substantially from adopting a more robust strategy.
4.2 ANALYSIS OF ROBUSTNESS
Next, we analyze the robustness of policies trained using EPOpt on the hopper domain. For this analysis, we construct a source distribution which varies four different physical parameters: torso mass, ground friction, foot joint damping, and joint inertia (armature). This distribution is presented in Table 1. Using this source distribution, we compare between three different methods: (1) standard policy search (TRPO) trained on a single model corresponding to the mean parameters in Table 1; (2) EPOpt( = 1) trained on the source distribution; (3) EPOpt( = 0.1) – i.e. the adversarially trained policy, again trained on the previously described source distribution. The aim of the comparison is to study direct-transfer performance, similar to the robustness evaluations common in robust controller design (Zhou et al., 1996). Hence, we learn a policy using each of the methods, and then test policies on different model instances (i.e. different combinations of physical parameters) without any adaptation. The results of this comparison are summarized in Figure 2, where we present the performance of the policy for testing conditions corresponding to different torso mass and friction values, which we found to have the most pronounced impact on performance. The results indicate that EPOpt( = 0.1) produces highly robust policies. A similar analysis for the 10th percentile of the return distribution (softer version of worst-case performance), the half-cheetah task, and different settings are presented in the appendix.
Table 1: Initial source domain distribution
Hopper µ σ low high mass 6.0 1.5 3.0 9.0 ground friction 2.0 0.25 1.5 2.5 joint damping 2.5 1.0 1.0 4.0 armature 1.0 0.25 0.5 1.5
Half-Cheetah µ σ low high mass 6.0 1.5 3.0 9.0 ground friction 0.5 0.1 0.3 0.7 joint damping 1.5 0.5 0.5 2.5 armature 0.125 0.04 0.05 0.2
3 4 5 6 7 8 9 Torso Mass
0
500
1000
1500
2000
2500
3000
3500
4000
P e rf
o rm
a n ce
Ensemble (unmodeled) Maximum-Likelihood
Figure 3: Comparison between policies trained on a fixed maximum-likelihood model with mass (6), and an ensemble where all models have the same mass (6) and other parameters varying as described in Table 1.
4.3 ROBUSTNESS TO UNMODELED EFFECTS
To analyze the robustness to unmodeled effects, our next experiment considers the setting where the source domain distribution is obtained by varying friction, damping, and armature as in Table 1, but does not consider a distribution over torso mass. Specifically, all models in the source domain distribution have the same torso mass (value of 6), but we will evaluate the policy trained on this distribution on target domains where the torso mass is different. Figure 3 indicates that the EPOpt( = 0.1) policy is robust to a broad range of torso masses even when its variation is not considered. However, as expected, this policy is not as robust as the case when mass is also modeled as part of the source domain distribution.
4.4 MODEL ADAPTATION
The preceding experiments show that EPOpt can find robust policies, but the source distribution in these experiments was chosen to be broad enough such that the target domain is not too far from high-density regions of the distribution. However, for real-world problems, we might not have the domain knowledge to identify a good source distribution in advance. In such settings, model (source) adaptation allows us to change the parameters of the source distribution using data gathered from the target domain. Additionally, model adaptation is helpful when the parameters of the target domain could change over time, for example due to wear and tear in a physical system. To illustrate model adaptation, we performed an experiment where the target domain was very far from the high density regions of the initial source distribution, as depicted in Figure 4(a). In this experiment, the source distribution varies the torso mass and ground friction. We observe that progressively, the source distribution becomes a better approximation of the target domain and consequently the performance improves. In this case, since we followed a sampling based approach, we used a uniform sampling distribution, and weighted each sample with the importance weight as described in Section 3.2. Eventually, after 10 iterations, the source domain distribution is able to accurately match the target domain. Figure 4(b) depicts the learning curve, and we see that a robust policy with return more than 2500, which roughly corresponds to a situation where the hopper is able to move forward without falling down for the duration of the episode, can be discovered with just 5 trajectories from the target domain. Subsequently, the policy improves near monotonically, and EPOpt finds a good policy with just 11 episodes worth of data from the target domain. In contrast, to achieve the same level of performance on the target domain, completely model-free methods like TRPO would require more than 2× 104 trajectories when the neural network parameters are initialized randomly.
5 RELATED WORK
Robust control is a branch of control theory which formally studies development of robust policies (Zhou et al., 1996; Nilim & Ghaoui, 2005; Lim et al., 2013). However, typically no distribution over source or target tasks is assumed, and a worst case analysis is performed. Most results from this field have been concentrated around linear systems or finite MDPs, which often cannot adequately model complexities of real-world tasks. The set-up of model-based Bayesian RL maintains a belief over models for decision making under uncertainty (Vlassis et al., 2012; Ghavamzadeh et al., 2015). In Bayesian RL, through interaction with the target domain, the uncertainty is reduced to find the correct or closest model. Application of this idea in its full general form is difficult, and requires either restrictive assumptions like finite MDPs (Poupart et al., 2006), gaussian dynamics (Ross et al., 2008), or task specific innovations. Previous methods have also suggested treating uncertain model parameters as unobserved state variables in a continuous POMDP framework, and solving the POMDP to get optimal exploration-exploitation trade-off (Duff, 2003; Porta et al., 2006). While this approach is general, and allows automatic learning of epistemic actions, extending such methods to large continuous control tasks like those considered in this paper is difficult.
Risk sensitive RL methods (Delage & Mannor, 2010; Tamar et al., 2015) have been proposed to act as a bridge between robust control and Bayesian RL. These approaches allow for using subjective model belief priors, prevent overly conservative policies, and enjoy some strong guarantees typically associated with robust control. However, their application in high dimensional continuous control tasks have not been sufficiently explored. We refer readers to Garcı́a & Fernández (2015) for a survey of related risk sensitive RL methods in the context of robustness and safety.
Standard model-based control methods typically operate by finding a maximum-likelihood estimate of the target model (Ljung, 1998; Ross & Bagnell, 2012; Deisenroth et al., 2013), followed by policy optimization. Use of model ensembles to produce robust controllers was explored recently in robotics. Mordatch et al. (2015a) use a trajectory optimization approach and an ensemble with small finite set of models; whereas we follow a sampling based direct policy search approach over a continuous distribution of uncertain parameters, and also show domain adaptation. Sampling based approaches can be applied to complex models and discrete MDPs which cannot be planned through easily. Similarly, Wang et al. (2010) use an ensemble of models, but their goal is to optimize for average case performance as opposed to transferring to a target MDP. Wang et al. (2010) use a hand engineered policy class whose parameters are optimized with CMA-ES. EPOpt on the other hand can optimize expressive neural network policies directly. In addition, we show model adaptation, effectiveness of the sub-sampling step ( < 1 case), and robustness to unmodeled effects, all of which are important for transfering to a target MDP.
Learning of parametrized skills (da Silva et al., 2012) is also concerned with finding policies for a distribution of parametrized tasks. However, this is primarily geared towards situations where task parameters are revealed during test time. Our work is motivated by situations where target task parameters (e.g. friction) are unknown. A number of methods have also been suggested to reduce sample complexity when provided with either a baseline policy (Thomas et al., 2015; Kakade & Langford, 2002), expert demonstration (Levine & Koltun, 2013; Argall et al., 2009), or approximate simulator (Tamar et al., 2012; Abbeel et al., 2006). These are complimentary to our work, in the sense that our policy, which has good direct-transfer performance, can be used to sample from the target domain and other off-policy methods could be explored for policy improvement.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we presented the EPOpt- algorithm for training robust policies on ensembles of source domains. Our method provides for training of robust policies, and supports an adversarial training regime designed to provide good direct-transfer performance. We also describe how our approach can be combined with Bayesian model adaptation to adapt the source domain ensemble to a target domain using a small amount of target domain experience. Our experimental results demonstrate that the ensemble approach provides for highly robust and generalizable policies in fairly complex simulated robotic tasks. Our experiments also demonstrate that Bayesian model adaptation can produce distributions over models that lead to better policies on the target domain than more standard maximum likelihood estimation, particularly in presence of unmodeled effects.
Although our method exhibits good generalization performance, the adaptation algorithm we use currently relies on sampling the parameter space, which is computationally intensive as the number of variable physical parameters increase. We observed that (adaptive) sampling from the prior leads to fast and reliable adaptation if the true model does not have very low probability in the prior. However, when this assumption breaks, we require a different sampling distribution which could produce samples from all regions of the parameter space. This is a general drawback of Bayesian adaptation methods. In future work, we plan to explore alternative sampling and parameterization schemes, including non-parametric distributions. An eventual end-goal would be to replace the physics simulator entirely with learned Bayesian neural network models, which could be adapted with limited data from the physical system. These models could be pre-trained using physics based simulators like MuJoCo to get a practical initialization of neural network parameters. Such representations are likely useful when dealing with high dimensional inputs like simulated vision from rendered images or tasks with complex dynamics like deformable bodies, which are needed to train highly generalizable policies that can successfully transfer to physical robots acting in the real world.
ACKNOWLEDGMENTS
The authors would like to thank Emo Todorov, Sham Kakade, and students of Emo Todorov’s research group for insightful comments about the work. The authors would also like to thank Emo Todorov for the MuJoCo simulator. Aravind Rajeswaran and Balaraman Ravindran acknowledge financial support from ILDS, IIT Madras.
A APPENDIX
A.1 DESCRIPTION OF SIMULATED ROBOTIC TASKS CONSIDERED IN THIS WORK
Hopper: The hopper task is to make a 2D planar hopper with three joints and 4 body parts hop forward as fast as possible (Erez et al., 2011). This problem has a 12 dimensional state space and a 3 dimensional action space that corresponds to torques at the joints. We construct the source domain by considering a distribution over 4 parameters: torso mass, ground friction, armature (inertia), and damping of foot.
Half Cheetah: The half-cheetah task (Wawrzynski, 2009) requires us to make a 2D cheetah with two legs run forward as fast as possible. The simulated robot has 8 body links with an 18 dimensional state space and a 6 dimensional action space that corresponds to joint torques. Again, we construct the source domain using a distribution over the following parameters: torso and head mass, ground friction, damping, and armature (inertia) of foot joints.
A video demonstration of the trained policies on these tasks can be viewed here: Supplimenrary video ( https://youtu.be/w1YJ9vwaoto )
Reward functions: For both tasks, we used the standard reward functions implemented with OpenAI gym (Brockman et al., 2016), with minor modifications. The reward structure for hopper task is: r(s, a) = vx − 0.001||a||2 + b, where s are the states comprising of joint positions and velocities; a are the actions (controls); and vx is the forward velocity. b is a bonus for being alive (b = 1). The episode terminates when ztorso < 0.7 or when |θy| < 0.2 where θy is the forward pitch of the body. For the cheetah task, we use the reward function:
r(s, a) = vx − 0.1||a||2 + b,
the alive bonus is 1 if head of cheetah is above −0.25 (relative to torso) and similarly episode terminates if the alive condition is violated.
Our implementation of the algorithms and environments are public in this repository to facilitate reproduction of results: https://github.com/aravindr93/robustRL
A.2 HYPERPARAMETERS
1. Neural network architecture: We used a neural network with two hidden layers, each with 64 units and tanh non-linearity. The policy updates are implemented using TRPO.
2. Trust region size in TRPO: The maximum KL divergence between sucessive policy updates are constrained to be 0.01
3. Number and length of trajectory rollouts: In each iteration, we sample N = 240 models from the ensemble, one rollout is performed on each such model. This was implemented in parallel on multiple (6) CPUs. Each trajectory is of length 1000 – same as the standard implimentations of these tasks in gym and rllab.
The results in Fig 1 and Fig 2 were generated after 150 and 200 iterations of TRPO respectively, with each iteration consisting of 240 trajectories as specified in (3) above.
A.3 WORST-CASE ANALYSIS FOR HOPPER TASK
Figure 2 illustrates the performance of the three considered policies: viz. TRPO on mean parameters, EPOpt( = 1), and EPOpt( = 0.1). We similarly analyze the 10th percentile of the return distribution as a proxy for worst-case analysis, which is important for a robust control policy (here, distribution of returns for a given model instance is due to variations in initial conditions). The corresponding results are presented below:
A.4 ROBUSTNESS ANALYSIS FOR HALF-CHEETAH TASK
A.5 DIFFERENT SETTINGS FOR
Here, we analyze how different settings for influences the robustness of learned policies. The policies in this section have been trained for 200 iterations with 240 trajectory samples per iteration. Similar to the description in Section 3.1, the first 100 iterations use = 1, and the final 100 iterations use the desired . The source distribution is described in Table 1. We test the performance on a grid over the model parameters. Our results, summarized in Table 2, indicate that decreasing decreases the variance in performance, along with a small decrease in average performance, and hence enhances robustness.
A.6 IMPORTANCE OF BASELINE FOR BATCHPOLOPT
As described in Section 3.1, it is important to use a good baseline estimate for the value function for the batch policy optimization step. When optimizing for the expected return, we can interpret the baseline as a variance reduction technique. Intuitively, policy gradient methods adjust parameters of the policy to improve probability of trajectories in proportion to their performance. By using a baseline for the value function, we make updates that increase probability of trajectories that perform better than average and vice versa. In practice, this variance reduction is essential for getting policy gradients to work. For the CVaR case, Tamar et al. (2015) showed that without using a baseline, the policy gradient is biased. To study importance of the baseline, we first consider the case where we do not employ the adversarial sub-sampling step, and fix = 1. We use a linear baseline with a time-varying feature vector as described in Section 3.1. Figure 8(a) depicts the learning curve for the source distribution in Table 1. The results indicate that use of a baseline is important to make policy gradients work well in practice.
Next, we turn to the case of < 1. As mentioned in section 3.1, setting a low from the start leads to unstable learning. The adversarial nature encourages penalizing poor trajectories more, which constrains the initial exploration needed to find promising trajectories. Thus we will “pre-train” by using = 1 for some iterations, before switching to the desired setting. From Figure 8(a), it is clear that pre-training without a baseline is unlikely to help, since the performance is poor. Thus, we use the following setup for comparison: for 100 iterations, EPOpt( = 1) is used with the baseline. Subsequently, we switch to EPOpt( = 0.1) and run for another 100 iterations, totaling 200 iterations. The results of this experiment are depicted in Figure 8(b). This result indicates that use of a baseline is crucial for the CVaR case, without which the performance degrades very quickly. We repeated the experiment with 100 iterations of pre-training with = 1 and without baseline, and observed the same effect. These empirical results reinforce the theoretical findings of Tamar et al. (2015).
A.7 ALTERNATE POLICY GRADIENT SUBROUTINES FOR BATCHPOLOPT
As emphasized previously, EPOpt is a generic policy gradient based meta algorithm for finding robust policies. The BatchPolOpt step (line 9, Algorithm 1) calls one gradient step of a policy gradient method, the choice of which is largely orthogonal to the main contributions of this paper. For the
reported results, we have used TRPO as the policy gradient method. Here, we compare the results to the case when using the classic REINFORCE algorithm. For this comparison, we use the same value function baseline parametrization for both TRPO and REINFORCE. Figure 9 depicts the learning curve when using the two policy gradient methods. We observe that performance with TRPO is significantly better. When optimizing over probability distributions, the natural gradient can navigate the warped parameter space better than the “vanilla” gradient. This observation is consistent with the findings of Kakade (2001), Schulman et al. (2015), and Duan et al. (2016).

The paper looks at the problem of transferring a policy learned in a simulator to  a target real-world system.  The proposed approach considers using an ensemble of simulated source domains, along with adversarial training, to learn a robust policy that is able to generalize to several target domains.
Overall, the paper tackles an interesting problem, and provides a reasonable solution.  The notion of adversarial training used here does not seem the same as other recent literature (e.g. on GANs).  It would be useful to add more details on a few components, as discussed in the question/response round.  I also encourage including the results with alternative policy gradient subroutines, even if they don’t perform well (e.g. Reinforce), as well as results with and without the baseline on the value function. Such results are very useful to other researchers.