Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowledge of building highly accurate neural network classifiers. Here, we develop our understanding of GANs with the aim of forming a rich view of this growing area of machine learning—to build connections to the diverse set of statistical thinking on this topic, of which much can be gained by a mutual exchange of ideas. We frame GANs within the wider landscape of algorithms for learning in implicit generative models—models that only specify a stochastic procedure with which to generate data—and relate these ideas to modelling problems in related fields, such as econometrics and approximate Bayesian computation. We develop likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which we are able to derive the objective function used by GANs, and many other related objectives. The testing viewpoint directs our focus to the general problem of density ratio estimation. There are four approaches for density ratio estimation, one of which is a solution using classifiers to distinguish real from generated data. Other approaches such as divergence minimisation and moment matching have also been explored in the GAN literature, and we synthesise these views to form an understanding in terms of the relationships between them and the wider literature, highlighting avenues for future exploration and cross-pollination. 1 IMPLICIT GENERATIVE MODELS It is useful to make a distinction between two types of probabilistic models: prescribed and implicit models (Diggle and Gratton, 1984). Prescribed probabilistic models are those that provide an explicit parametric specification of the distribution of an observed random variable x, specifying a log-likelihood function log qθ(x) with parameters θ. Most models in machine learning and statistics are of this form, whether they be state-of-the-art classifiers for object recognition, complex sequence models for machine translation, or fine-grained spatio-temporal models tracking the spread of disease. Alternatively, we can specify implicit probabilistic models that define a stochastic procedure that directly generates data. Such models are the natural approach for problems in climate and weather, population genetics, and ecology, since the mechanistic understanding of such systems can be used to directly create a data simulator, and hence the model. It is exactly because implicit models are more natural for many problems that they are of interest and importance. Implicit generative models use a latent variable z and transform it using a deterministic function Gθ that maps from R → R using parameters θ. Such models are amongst the most fundamental of models, e.g., many of the basic methods for generating non-uniform random variates are based on simple implicit models and one-line transformations (Devroye, 2006). In general, implicit generative models specify a valid density on the output space that forms an effective likelihood function: x = Gθ(z); z′ ∼ q(z) (1) qθ(x) = ∂ ∂x1 . . . ∂ ∂xd ∫ {Gθ(z)≤x} q(z)dz, (2) where q(z) is a latent variable that provides the external source of randomness and equation (2) is the definition of the transformed density as the derivative of the cumulative distribution function. When the function G is well-defined, such as when the function is invertible, or has dimensions m = d with easily characterised roots, we recover the familiar rule for transformations of probability distributions.
1 IMPLICIT GENERATIVE MODELS
It is useful to make a distinction between two types of probabilistic models: prescribed and implicit models (Diggle and Gratton, 1984). Prescribed probabilistic models are those that provide an explicit parametric specification of the distribution of an observed random variable x, specifying a log-likelihood function log qθ(x) with parameters θ. Most models in machine learning and statistics are of this form, whether they be state-of-the-art classifiers for object recognition, complex sequence models for machine translation, or fine-grained spatio-temporal models tracking the spread of disease. Alternatively, we can specify implicit probabilistic models that define a stochastic procedure that directly generates data. Such models are the natural approach for problems in climate and weather, population genetics, and ecology, since the mechanistic understanding of such systems can be used to directly create a data simulator, and hence the model. It is exactly because implicit models are more natural for many problems that they are of interest and importance.
Implicit generative models use a latent variable z and transform it using a deterministic function Gθ that maps from Rm → Rd using parameters θ. Such models are amongst the most fundamental of models, e.g., many of the basic methods for generating non-uniform random variates are based on simple implicit models and one-line transformations (Devroye, 2006). In general, implicit generative models specify a valid density on the output space that forms an effective likelihood function:
x = Gθ(z′); z′ ∼ q(z) (1)
qθ(x) = ∂
∂x1 . . .
∂
∂xd ∫ {Gθ(z)≤x} q(z)dz, (2)
where q(z) is a latent variable that provides the external source of randomness and equation (2) is the definition of the transformed density as the derivative of the cumulative distribution function. When the function G is well-defined, such as when the function is invertible, or has dimensions m = d with easily characterised roots, we recover the familiar rule for transformations of probability distributions.
We are interested in developing more general and flexible implicit generative models where the function G is a non-linear function with d > m, specified by deep networks. The integral (2) is intractable in this case: we will be unable to determine the set {Gθ(z) ≤ x}, the integral will often be unknown even when the integration regions are known and, the derivative is high-dimensional and difficult to compute. Intractability is also a challenge for prescribed models, but the lack of a likelihood term significantly reduces the tools available for learning. In implicit models, this difficulty motivates the need for methods that side-step the intractability of the likelihood (2), or are likelihood-free.
Generative adversarial networks (GANs) (Goodfellow et al., 2014) provide a solution for exactly this type of problem. This connection makes it possible to export GAN-type solutions to other areas, and to import new approaches from areas addressing the same research problem. GANs specify an algorithmic framework for learning in implicit generative models, also referred to as generator networks or generative neural samplers or differentiable simulators in the literature. The framework relies on a learning principle based on discriminating real from generated data, which we shall show instantiates a core principle of likelihood-free inference, that of hypothesis and two-sample testing.
Note on notation. We denote data by the random variable x, the (unknown) true data density by p∗(x), our (intractable) model density by qθ(x). q(z) is a density over latent variables z. Parameters of the model are θ, and parameters of the ratio and discriminator functions are φ.
2 HYPOTHESIS TESTING AND DENSITY RATIOS
2.1 LIKELIHOOD-FREE INFERENCE
Without a likelihood function, many of the widely-used tools for inference and parameter learning become unavailable. But there are tools that remain, including the method-of-moments (Hall, 2005), the empirical likelihood (Owen, 1988), Monte Carlo sampling (Marin et al., 2012), and mean-shift estimation (Fukunaga and Hostetler, 1975). Since we can easily draw samples from the model, we can use any method that compares two sets of samples—one from the true data distribution and one from the model distribution—to drive learning. This is a process of density estimation-by-comparison that tests the hypothesis that the true data distribution p∗(x) and our model distribution q(x) are equal, using the density ratio function r(x) = p∗(x)/q(x). The density ratio provides information about the departure of our model distribution from the true distribution, and our aim is to see this ratio be one. The density ratio r(x) is the core quantity for hypothesis testing, motivated by either the NeymanPearson lemma or the Bayesian posterior evidence, appearing as the likelihood ratio or the Bayes factor (Kass and Raftery, 1995), respectively. This is the focus of our approach for likelihood-free inference: estimating density ratios and using them as the driving principle for learning in implicit generative models.
The direct approach of computing the density ratio by first computing the individual marginals is not possible with implicit models. By directly estimating the ratio and exploiting knowledge of the probabilities involved, it will turn out that computing the ratio can be a much easier problem than computing the marginal likelihoods, and is what will make this approach appealing. There are four general approaches to consider (Sugiyama et al., 2012a): 1) class-probability estimation, 2) divergence minimisation, 3) ratio matching, and 4) moment matching. These are highly developed research areas in their own right, but their role in providing a learning principle for density estimation is under-appreciated and opens up an exciting range of approaches for learning in implicit generative models. Figure 1 summarises these approaches by showing pathways available for learning, which follow from the choice of inference driven by hypothesis testing and comparison.
2.2 CLASS PROBABILITY ESTIMATION
The density ratio can be computed by building a classifier to distinguish observed data from that generated by the model. This is the most popular approach for density ratio estimation and the first port of call for learning in implicit models. Hastie et al. (2013) call this approach unsupervised-assupervised learning, Qin (1998) explore this for analysis of case-control in observational studies, both Neal (2008) and Cranmer et al. (2015) explore this approach for high-energy physics applications, Gutmann and Hyvärinen (2012) exploit it for learning un-normalised models, Lopez-Paz and Oquab
(2016) for causal discovery, and Goodfellow et al. (2014) for learning in implicit generative models specified by neural networks.
We denote the domain of our data by X ⊂ Rd. The true data distribution has a density p∗(x) and our model has density qθ(x), both defined on X . We also have access to a set of n samples Xp = {x(p)1 , . . . ,x (p) n } from the true data distribution, and a set of n′ samples Xq = {x(q)1 , . . . ,x (q) n′ } from our model. We introduce a random variable y, and assign a label y = 1 to all samples in Xp and y = 0 to all samples in Xq . We can now represent p∗(x) = p(x|y = 1) and qθ(x) = p(x|y = 0). By application of Bayes’ rule, we can compute the ratio r(x) as:
p∗(x) qθ(x) = p(x|y = 1) p(x|y = 0) = p(y = 1|x)p(x) p(y = 1)
/ p(y = 0|x)p(x)
p(y = 0) = p(y = 1|x) p(y = 0|x) · 1− π π , (3)
which indicates that the problem of density ratio estimation is equivalent to that of class probability estimation, since the problem is reduced to computing the probability p(y = 1|x). We assume that the marginal probability over classes is p(y = 1) = π, which allows the relative proportion of data from the two classes to be adjusted if they are imbalanced; in most formulations π = ½ for the balanced case, and in imbalanced cases 1−ππ ≈ n ′/n.
Our task is now to specify a scoring function, or discriminator, D(x;φ) = p(y = 1|x): a function bounded in [0,1] with parameters φ that computes the probability of data belonging to the positive (real data) class. This discriminator is related to the density ratio through the mapping D = r/(r + 1); r = D/(1−D). Conveniently, we can use our knowledge of building classifiers and specify these functions using deep neural networks. Given this scoring function, we must specify a proper scoring rule (Gneiting and Raftery, 2007; Buja et al., 2005) for binary discrimination to allow for parameter learning, such as those in Table 1. A natural choice is to use the Bernoulli (logarithmic) loss:
L(φ,θ) = Ep(x|y)p(y)[−y logD(x;φ)− (1− y) log(1−D(x;φ))] (4) = πEp∗(x)[− logD(x;φ)] + (1− π)Eqθ(x)[− log(1−D(x;φ))]. (5)
Since we know the underlying generative process for qθ(x), using a change of variables, we can express the loss in term of an expectation over the latent variable z and the generative model G(z;θ):
L(φ,θ) = πEp∗(x)[− logD(x;φ)] + (1− π)Eq(z)[− log(1−D(G(z;θ);φ))]. (6)
The final form of this objective (6) is exactly that used in generative adversarial networks (GANs) (Goodfellow et al., 2014). In practice, the expectations are computed by Monte Carlo integration using samples from p∗ and qθ. Equation (6) allows us to specify a bi-level optimisation (Colson et al., 2007) by forming a ratio loss and a generative loss, using which we perform an alternating optimisation. Our convention throughout the paper will be to always form the ratio loss by extracting all terms in L related to the ratio function parameters φ, and minimise the resulting objective. For the generative loss, we will similarly extract all terms related to the model parameters θ, flip the sign, and minimise the resulting objective. For equation (6), the bi-level optimisation is:
Ratio loss: min φ πEp∗(x)[− logD(x;φ)] + (1− π)Eqθ(x)[− log(1−D(x;φ))] (7)
Generative loss: min θ Eq(z)[log(1−D(G(z;θ)))]. (8)
The ratio loss is minimised since it acts as a surrogate negative log-likelihood; the generative loss is minimised since we wish to minimise the probability of the negative (generated-data) class. We explicitly write out these two stages to emphasise that the objectives used are separable. While we can derive the generative loss from the ratio loss as we have done, any generative loss that drives qθ to p∗, such as minimising the widely-used Eq(z)[− logD(G(z;θ))] (Goodfellow et al., 2014; Nowozin et al., 2016) or Eq(z)[− log D(G(z;θ))1−D(G(z;θ)) ] = Eq(z)[− log r(G(z;θ)) (Sønderby et al., 2016), is possible. Any scoring rule from Table 1 can be used to give a loss function for optimisation. These rules are amenable to stochastic approximation and alternating optimisation, as described by Goodfellow et al. (2014), along with many of the insights for optimisation that have been developed since (Salimans et al., 2016; Radford et al., 2015; Zhao et al., 2016; Sønderby et al., 2016). The Bernoulli loss can be criticised in a number of ways and makes other scoring rules interesting to explore. The Brier loss provides a similar decision rule, and its use in calibrated regression makes it appealing; the motivations behind many of these scoring rules are discussed in (Gneiting and Raftery, 2007). Finally, while we have focussed on the two sample hypothesis test, we believe it could be advantageous to extend this reasoning to the case of multiple testing, where we simultaneously test several sets of data (Bickel et al., 2008).
The advantage of using a proper scoring rule is that the global optimum is achieved iff qθ = p∗ (cf. the proof in (Goodfellow et al., 2014) for the Bernoulli loss); however there are no convergence guarantees since the optimisation is non-convex. Goodfellow et al. (2014) discuss the relationship to maximum likelihood estimation, which minimises the divergence KL[p∗‖q], and show that the GAN objective with Bernoulli loss is instead related to the Jensen Shannon divergence JS [p∗||q]. In the objective (6), π denotes the marginal probability of the positive class; however several authors have proposed choosing π depending on the problem. In particular, Huszár (2015) showed that varying π is related to optimizing a generalised Jensen-Shannon divergence JSπ [p∗||q]. Creswell and Bharath (2016) presented results showing that different values of π are desirable, depending on whether we wish to fit one of the modes (a ‘high precision, low recall’ task such as generation) or explain all of the modes (a ‘high recall, low precision’ task such as retrieval).
2.3 DIVERGENCE MINIMISATION
A second approach to testing is to use the divergence between the true density p∗and our model q, and use this as an objective to drive learning of the generative model. A natural class of divergences to use are the f -divergences (or Ali-Silvey (Ali and Silvey, 1966) or Csiszar’s φ-divergence (Csiszàr, 1967)) since they are fundamentally linked to the problem of two-sample hypothesis testing (Liese and Vajda, 2008): f -divergences represent an integrated Bayes risk since they are an expectation of the density ratio. Nowozin et al. (2016) develop f -GANs using this view. The f -divergences contain the KL divergence as a special case and are equipped with an exploitable variational formulation:
Df [p ∗(x)‖qθ(x)]= ∫ qθ(x)f ( p∗(x) qθ(x) ) dx=Eqθ(x)[f(r(x))]≥sup t Ep∗(x)[t(x)]−Eqθ(x)[f†(t(x))] (9)
where f is a convex function with derivative f ′ and Fenchel conjugate f†; this divergence class instantiates many familiar divergences, such as the KL and Jensen-Shannon divergence. The variational formulation introduces the functions t(x) whose optimum is related to the density ratio since t∗(x) = f ′(r(x)). Substituting t∗ in (9), we transform the objective (10) into supremum over rφ (which is attained when rφ = r∗ = p ∗ /qθ). For self-consistency, we flip the sign to make it a minimisation problem in rφ, leading to the bi-level optimisation:
L = Ep∗(x)[−f ′(rφ(x))] + Eqθ(x)[f†(f ′(rφ(x))] (10)
Ratio loss: min φ
Ep∗(x)[−f ′(rφ(x))] + Eqθ(x)[f†(f ′(rφ(x))] (11)
Generative loss: min θ
Eq(z)[−f†(f ′(r(G(z;θ)))], (12)
where we derived equation (12) by extracting all the terms involving qθ(x) in equation (10), used the change of variables to express it in terms of the underlying generative model and flipping the sign to obtain a minimisation. There is no discriminator in this formulation, and this role is taken by the ratio function. We minimise the ratio loss, since we wish to minimise the negative of the variational lower bound; we minimise the generative loss since we wish to drive the ratio to one. By using the function f(u) = u log u, we recover an objective using the KL divergence; when f(u) = u log u − (u + 1) log(u + 1), we recover the objective function in equation (6) from the previous section using the Bernoulli loss, and hence the objective for GANs.
The density ratio implies that p∗(x) ≈ p̃ = rφ(x)qθ(x), since it is the amount by which we must correct our model qθ(x) to match the true distribution. This led us to a divergence formulation that evaluates the divergence between the distributions p∗ and p̃, using the KL divergence:
DKL[p ∗(x)‖p̃(x)] = ∫ p∗(x) log
p∗(x) rφ(x)qθ(x) dx +
∫ (rφ(x)qθ(x)− p∗(x))dx (13)
L = Ep∗(x)[− log rφ(x)] + Eqθ(x)[rφ(x)− 1]− Ep∗(x)[log qθ(x)] + Ep∗(x)[log p∗(x)], (14)
where the first equation is the KL for un-normalised densities (Minka, 2005). This leads to a convenient and valid ratio loss since all terms independent of r can be ignored. But we are unable to derive a useful generative loss from this expression (14) , since the third term with log q cannot be ignored, and is unavailable. Since the generative loss and ratio losses need not be coupled, any other generative loss can be used, e.g., equation (8). But this is not ideal, since we would prefer to derive valid ratio losses from the same principle to make reasoning about correctness and optimality easier. We include this discussion to point out that while the formulation of equation (9) is generally applicable, the formulation (14), while useful for ratio estimation, is not useful for generative learning. The difficulty of specifying stable and correct generative losses is a common theme of work in GANs; we will see a similar difficulty in the next section.
The equivalence between divergence minimisation and class probability estimation is also widely discussed, and most notably developed by Reid and Williamson (2011) using the tools of weighted integral measures, and more recently by Menon and Ong (2016). This divergence minimisation viewpoint (9) was used in f -GANs and explored in depth by Nowozin et al. (2016) who provide a detailed description, and explore many of the objectives that become available and practical guidance.
2.4 RATIO MATCHING
A third approach is to directly minimise the error between the true density ratio and an estimate of it. If we denote the true density ratio as r∗(x) = p∗(x)/qθ(x) and its approximation as rφ(x), we can define a loss using the squared error:
L = 1 2 ∫ qθ(x)(r(x)− r∗(x))2dx = 12Eqθ(x)[rφ(x)2]− Ep∗(x)[rφ(x)] + 12Ep∗(x)[r∗(x)] (15)
= 12Eqθ(x)[rφ(x) 2]− Ep∗(x)[rφ(x)] s.t. rφ(x) ≥ 0, (16)
where the final objective is obtained by ignoring terms independent of rφ(x) and is used to derive ratio and generative losses. When used to learn the ratio function, Sugiyama et al. (2012b) refer to this objective as KL importance estimation procedure (KLIEP). Concurrently with this work, Uehara et al. (2016) recognised the centrality of the density ratio, the approach for learning by ratio matching, and its connections to GANs (Goodfellow et al., 2014) and f -GANs (Nowozin et al., 2016), and provide useful guidance on practical use of ratio matching.
We can generalise (16) to loss functions beyond the squared error using the Bregman divergence for density ratio estimation (Sugiyama et al., 2012a; Uehara et al., 2016), and is the unifying tool exploited in previous work (Reid and Williamson, 2011; Sriperumbudur et al., 2009; Sugiyama et al., 2012a; Menon and Ong, 2016). This leads to a minimisation of the Bregman divergence Bf between ratios:
Bf (r ∗(x)‖rφ(x)) = ∫ ( f(r∗(x))− f(rφ(x))− f ′(rφ(x)) [ r∗(x)− rφ(x) ]) qθ(x)dx (17)
= Eqθ(x) [rφ(x)f ′(rφ(x))− f(rφ(x))]− Ep∗[f ′(rφ(x))] +Df [p∗(x)‖qθ(x)] (18)
= LB(rφ(x)) +Df [p∗(x)‖qθ(x)], (19) where we have used p∗= r∗qθ, and Df is the f -divergence defined in equation (9). We can derive a ratio loss from (19) by extracting all the terms in rφ, leading to the minimisation of LB(rφ). The role of the discriminator in GANs is again taken by the ratio r that provides information about the relationship between the two distributions. This ratio loss, as pointed out by Uehara et al. (2016), is equivalent to the ratio loss we derived using divergence minimisation (10), since:
LB(rφ(x)) = Ep∗[−f ′(rφ(x))] + Eqθ(x)[rφ(x)f ′(rφ(x))− f(rφ(x))] (20) = Ep∗[−f ′(rφ(x))] + Eqθ(x)[f†(f ′(rφ(x)))]. (21)
The equivalence of the second terms in (20) and (21) can be derived by using the definition of the dual function: f†(f ′(x)) = maxr rf ′(x)− f(r). The maximum is attained at x = r, leading to the identity f†(f ′(rφ(x))) = rφ(x)f ′(rφ(x))− f(rφ(x)). If we follow the strategy we used in previous sections to obtain a generative loss, by collecting the terms in equation (19) dependent on qθ, we obtain:
L(qθ) = Eqθ(x)[rφ(x)f ′(rφ(x))]− Eqθ(x)[f(rφ(x))] +Df [p∗(x)||qθ(x)]. (22) But this does not lead to a useful generative loss since there are terms involving qθ whose density is always unknown, similar to the difficulty we encountered with divergence minimisation in equation (14). Since we can use any generative loss that drives the ratio to one, we can employ other losses, like the alternatives described in section 2.2 . One approximation suggested by Uehara et al. (2016) is to assume p∗≈ rφqθ, i.e. assume a near-optimal ratio, which reduces the f -divergence to:
Df [p ∗(x)‖qθ(x)]=Eqθ(x)
[ f ( p∗
qθ(x)
)] ≈ Eqθ(x) [ f ( qθ(x)rφ(x)
qθ(x)
)] = Eqθ(x)[f(rφ(x))] (23)
As a result, the last two terms in equation (22) cancel, leaving a generative loss that can be used for learning. Using ratio matching under the Bregman divergence we obtain the bi-level optimisation:
Ratio loss: min φ
Eqθ(x)[rφ(x)f ′(rφ(x))− f(rφ(x))]− Ep∗[f ′(rφ(x))] (24)
Generative loss: min θ
Eqθ(x)[rφ(x)f ′(rφ(x))] (25)
2.5 MOMENT MATCHING
A final approach for testing is to evaluate whether the moments of the distributions p∗and q are the same, i.e. by moment matching. We compare the moments of the two distributions by minimising their distance, using test statistics s(x) that provides the moments of interest:
L(φ,θ) = (Ep∗(x)[s(x)]− Eqθ(x)[s(x)])2 = (Ep∗(x)[s(x)]− Eq(z)[s(G(z;θ))])2 (26) The choice of test statistics s(x) is critical, since ideally, we wish to match all moments of the two distributions. When the functions s(x) are defined within a reproducing kernel Hilbert space, we obtain kernel-based forms of these objectives, which are highly flexible and allow easy handling of data such as images, strings and graphs. The objective (26) can then be re-expressed in closed-form in terms of kernel functions, leading to the maximum mean discrepancy criterion (MMD). The role of the MMD for learning in implicit generative models defined by neural networks has been explored by both Li et al. (2015) and Dziugaite et al. (2015). While typically expensive to compute, there are now highly efficient approaches for computing the MMD (Chwialkowski et al., 2015). The objective using feature matching by Salimans et al. (2016) is a form of moment matching that defines the test statistics using intermediate layers of the discriminator function. And a further set of objective functions is possible by framing this problem as one of density difference (rather than ratio) estimation (Sugiyama et al., 2013).
The other significant body of research that focusses on learning in implicit generative models using the lens of moment matching is approximate Bayesian computation (ABC) (Marin et al., 2012). The models in ABC are often related to problems in population genetics and ecology, where knowledge of these systems leads to ABC being easily exploited to learn simulator parameters. The ABC literature has widely-explored the use of test statistics, including fixed functions and kernels, and Markov chain Monte Carlo methods to learn the posterior distribution of the parameters. There is a great deal of opportunity for exchange between GANs, ABC and ratio estimation in aspects of scalability, applications, and theoretical understanding. A further insight obtained from a moment-matching approach is that other empirical measures such as the Dudley and Wasserstein distances can easily be considered, establishing even further connections to other growing areas, such as optimal transport (Frogner et al., 2015; Sriperumbudur et al., 2009).
The moment matching approach can be generalised by representing them as an integral probability metric (Sriperumbudur et al., 2009), which is what makes it possible to describe the relationships and ways of transforming from density ratio estimation using using moment-matching, f -divergences, and class-probability estimation. Sriperumbudur et al. (2009) showed that f -divergences and integral probability metrics (MMD, Wasserstein) intersect only at the total variation distance. As we described previously, by cleverly formulating our problem as a Bregman divergence minimisation, we show that all the methods we described for density ratio estimation are very closely related (Sugiyama et al., 2012a; Reid and Williamson, 2011; Sriperumbudur et al., 2009; Sugiyama et al., 2012b; Uehara et al., 2016).
3 DISCUSSION
By using an inferential principle driven by hypothesis testing, we have been able to develop a number of indirect methods for learning the parameters of generative models. These methods do not compute the probability of the data or posterior distributions over latent variables, but instead only involve relative statements of probability by comparing populations of data from the generative model to observed data. This view allows us to better understand how algorithms such as generative adversarial networks, approximate Bayesian computation, noise-contrastive estimation, and density ratio estimation are related. Ultimately, these techniques make it possible for us to make contributions to applications in climate and weather, economics, population genetics, and epidemiology, all areas whose principal tools are implicit generative models.
Distinction between implicit and prescribed models. The distinction between implicit and prescribed models is useful to keep in mind for at least two reasons: the choice of model has direct implications on the types of learning and inferential principles that can be called upon; and it makes explicit that there are many different ways in which to specify a model that captures our beliefs about data generating processes. Any implicit model can be easily turned into a prescribed model by adding a simple likelihood function (noise model) on the generated outputs, so the distinction is
not essential. And models with likelihood functions also regularly face the problem of intractable marginal likelihoods. But the specification of a likelihood function provides knowledge of p∗ that leads to different algorithms by exploiting this knowledge, e.g., NCE resulting from class-probability based testing in un-normalised models (Gutmann and Hyvärinen, 2012), or variational lower bounds for directed graphical models. We strive to maintain a clear distinction between the choice of model, choice of inference, and the resulting algorithm, since it is through such a structured view that we can best recognise the connections between research areas that rely on the same sets of tools.
Model misspecification and non-maximum likelihood methods. Once we have made the choice of an implicit generative model, we cannot use likelihood-based techniques, which then makes testing and estimation-by-comparison appealing. What is striking, is that this leads us to principles for parameter learning that do not require inference of any underlying latent variables, side-stepping one of the major challenges in statistical practice. This piques our interest in more general approaches for non-maximum likelihood and likelihood-free estimation methods, of which there is much work (Lyu, 2011; Gutmann and Hyvärinen, 2012; Hall, 2005; Marin et al., 2012; Frogner et al., 2015). We often deal with misspecified models where qθ cannot represent p∗, and non maximum likelihood methods could be a more robust choice depending on the task (see figure 1 in (Huszár, 2015) for an illustrative example).
Choice of loss function. This density ratio viewpoint has led us to derive multiple different objective functions and an understanding of how they are related. But it has left us unsatisfied since we have not gained the insight needed to choose between them. Sugiyama et al. (2012a) make statements on this choice in the simpler setting where only the density ratio is to be estimated. But when we wish to learn implicit generative models, this choice, at present, remains unclear.
Perceptual losses. Several authors have also proposed using pre-trained discriminative networks to define the test functions since the difference in activations (of say a pre-trained VGG classifier) can better capture perceptual similarity than the reconstruction error in pixel space. This provides a strong motivation for further research into joint models of images and labels. However, it is not completely unsupervised as the pre-trained discriminative network contains information about labels and invariances. This makes evaluation difficult since we lack good metrics and can only fairly compare to other joint models that use both label and image information.
Bayesian inference. We have mainly discussed point estimation methods for parameter learning. It is also desirable to perform Bayesian inference in implicit models, where we learn the posterior distribution over the model parameters p(θ|x), allowing knowledge of parameter uncertainty to be used in risk minimisation and other decision-making tasks. This is the aim of approximate Bayesian computation (ABC) (Marin et al., 2012). The most common approach for thinking about ABC is through moment-matching, but as we explored, there are other approaches available. An approach through class-probability estimation is appealing and leads to classifier ABC (Gutmann et al., 2014). We have highly diverse approaches for Bayesian reasoning in prescribed models, and it is desirable to develop a similar breadth of choice for implicit models.
Evaluation. Our view suggests that value of the density ratio, a quantity we can always compute using the approaches described, can be a useful metric to report. But density ratio estimation has not yet illuminated a method for consistent evaluation of the generative model that is learnt. Without such a measure, it is hard to make fair comparisons; but this is a problem faced by all related fields, and establishing such measures is an important contribution to be made. Furthermore, we have also not yet gained the tools to make theoretical statements that allow us to assess the correctness of the model learning framework, although theoretical developments in the literature on approximate Bayesian computation may help in this regard, e.g., Frazier et al. (2016).
Non-differentiable models. We have restricted our development to implicit models that are differentiable. In many practical applications, the implicit model (or simulator) will be non-differentiable, discrete or defined in other ways, such as through a stochastic differential equation . The stochastic optimisation problem we are generally faced with (for differentiable and non-differentiable models), is to compute ∆ = ∇θEqθ(x)[f(x)], the gradient of the expectation of a function. As our exposition followed, when the implicit model is differentiable, the pathwise derivative estimator can be used, i.e. ∆ = Eq(z)[∇θf(Gθ(z))] by rewriting the expectation in terms of the known and easy to sample distribution q(z). It is commonly assumed that when we encounter non-differentiable functions that the score function estimator (or likelihood ratio or reinforce estimator) can be used; the score-function
estimator is ∆ = Eqθ(x)[f(x)∇θ log qθ(x)]. For implicit models, we do not have knowledge of the density q(x) whose log derivative we require, making this estimator inapplicable. This leads to the first of three tools available for non-differentiable models: the use of the weak derivative and related stochastic finite difference estimators, which require forward-simulation only and compute gradients by perturbation of the parameters (Glasserman, 2003; Fu, 2005).
The two other approaches are: moment matching and ABC-MCMC (Marjoram et al., 2003), which has been successful for many problems with moderate dimension; and the natural choice of gradientfree optimisation methods, which include familiar tools such as Bayesian optimisation (Gutmann and Corander, 2016), evolutionary search like CMA-ES, and the Nelder-Mead method, amongst others (Conn et al., 2009). For all three approaches, new insights will be needed to help scale to high-dimensional data with complex dependency structures.
Ultimately, these concerns serve to highlight the many opportunities that remain for advancing our understanding of inference and parameter learning in implicit generative models.

I just noticed I submitted my review as a pre-review question - sorry about this. Here it is again, with a few more thoughts added...
The authors present a great and - as far as I can tell - accurate and honest overview of the emerging theory about GANs from a likelihood ratio estimation/divergence minimisation perspective. It is well written and a good read, and one I would recommend to people who would like to get involved in GANs.
My main problem with this submission is that it is hard as a reviewer to pin down what precisely the novelty is - beyond perhaps articulating these views better than other papers have done in the past. A sentence from the paper "But it has left us unsatisfied since we have not gained the insight needed to choose between them.” summarises my feeling about this paper: this is a nice 'unifying review’ type paper that - for me - lacks a novel insight.
In summary, my assessment is mixed: I think this is a great paper, I enjoyed reading it. I was left a bit disappointed by the lack of novel insight, or a singular key new idea which you often expect in conference presentations, and this is why I’m not highly confident about this as a conference submission (and hence my low score) I am open to be convinced either way.
Detailed comments:
I think the authors should probably discuss the connection of Eq. (13) to KLIEP: Kullback-Leibler Importance Estimation by Shugiyama and colleagues.
I don’t quite see how the part with equation (13) and (14) fit into the flow of the paper. By this point the authors have established the view that GANs are about estimating likelihood ratios - and then using these likelihood ratios to improve the generator. These paragraphs read like: we also tried to derive another particular formulation for doing this but we failed to do it in a practical way.
There is a typo in spelling Csiszar divergence
Equation (15) is known (to me) as Least Squares Importance Estimation by Kanamori et al (2009). A variant of least-squares likelihood estimation uses the kernel trick, and finds a function from an RKHS that best represents the likelihood ratio between the two distributions in a least squares sense. I think it would be interesting to think about how this function is related to the witness function commonly used in MMD and what the properties of this function are compared to the witness function - perhaps showing the two things for simple distributions.
I have stumbled upon the work of Sugiyama and collaborators on direct density ratio estimation before, and I found that work very insightful. Generally, while some of this work is cited in this paper, I felt that the authors could do more to highlight the great work of this group, who have made highly significant contributions to density ratio estimation, albeit with a different goal in mind.
On likelihood ratio estimation: some methods approximate the likelihood ratio directly (such as least-squares importance estimation), some can be thought of more as approximating the log of this quantity (logistic regression, denoising autoencoders). An unbiased estimate of the ratio will provide a biased estimate of the logarithm and vice versa. To me it feels like estimating the log of the ratio directly is more useful, and in more generality estimating the convex function of the ratio which is used to define the f-divergence seems like a good approach. Could the authors comment on this?
I think the hypothesis testing angle is oversold in the paper.  I’m not sure what additional insight is gained by mixing in some hypothesis testing terminology. Other than using quantities that appear in hypothesis testing as tests statistics, his work does not really talk about hypothesis testing, nor does it use any tools from the hypothesis testing literature. In this sense, this paper is in contrast with Sutherland et al (in review for ICLR) who do borrow concepts from two-sample testing to optimise hyperparameters of the divergence used.


EXPLORING UNDER-APPRECIATED REWARDS
This paper presents a novel form of policy gradient for model-free reinforcement learning (RL) with improved exploration properties. Current policy-based methods use entropy regularization to encourage undirected exploration of the reward landscape, which is ineffective in high dimensional spaces with sparse rewards. We propose a more directed exploration strategy that promotes exploration of under-appreciated reward regions. An action sequence is considered under-appreciated if its log-probability under the current policy under-estimates its resulting reward. The proposed exploration strategy is easy to implement, requiring small modifications to the REINFORCE algorithm. We evaluate the approach on a set of algorithmic tasks that have long challenged RL methods. Our approach reduces hyper-parameter sensitivity and demonstrates significant improvements over baseline methods. The proposed algorithm successfully solves a benchmark multi-digit addition task and generalizes to long sequences, which, to our knowledge, is the first time that a pure RL method has solved addition using only reward feedback.
1 INTRODUCTION
Humans can reason about symbolic objects and solve algorithmic problems. After learning to count and then manipulate numbers via simple arithmetic, people eventually learn to invent new algorithms and even reason about their correctness and efficiency. The ability to invent new algorithms is fundamental to artificial intelligence (AI). Although symbolic reasoning has a long history in AI (Russell et al., 2003), only recently have statistical machine learning and neural network approaches begun to make headway in automated algorithm discovery (Reed & de Freitas, 2016; Kaiser & Sutskever, 2016; Neelakantan et al., 2016), which would constitute an important milestone on the path to AI. Nevertheless, most of the recent successes depend on the use of strong supervision to learn a mapping from a set of training inputs to outputs by maximizing a conditional log-likelihood, very much like neural machine translation systems (Sutskever et al., 2014; Bahdanau et al., 2015). Such a dependence on strong supervision is a significant limitation that does not match the ability of people to invent new algorithmic procedures based solely on trial and error.
By contrast, reinforcement learning (RL) methods (Sutton & Barto, 1998) hold the promise of searching over discrete objects such as symbolic representations of algorithms by considering much weaker feedback in the form of a simple verifier that tests the correctness of a program execution on a given problem instance. Despite the recent excitement around the use of RL to tackle Atari games (Mnih et al., 2015) and Go (Silver et al., 2016), standard RL methods are not yet able to consistently and reliably solve algorithmic tasks in all but the simplest cases (Zaremba & Sutskever, 2014). A key property of algorithmic problems that makes them challenging for RL is reward sparsity, i.e., a policy usually has to get a long action sequence exactly right to obtain a non-zero reward.
∗Work done as a member of the Google Brain Residency program (g.co/brainresidency) †Also at the Department of Computing Science, University of Alberta, daes@ualberta.ca
We believe one of the key factors limiting the effectiveness of current RL methods in a sparse reward setting is the use of undirected exploration strategies (Thrun, 1992), such as -greedy and entropy regularization (Williams & Peng, 1991). For long action sequences with delayed sparse reward, it is hopeless to explore the space uniformly and blindly. Instead, we propose a formulation to encourage exploration of action sequences that are under-appreciated by the current policy. Our formulation considers an action sequence to be under-appreciated if the model’s log-probability assigned to an action sequence under-estimates the resulting reward from the action sequence. Exploring underappreciated states and actions encourages the policy to have a better calibration between its logprobabilities and observed reward values, even for action sequences with negligible rewards. This effectively increases exploration around neglected action sequences.
We term our proposed technique under-appreciated reward exploration (UREX). We show that the objective given by UREX is a combination of a mode seeking objective (standard REINFORCE) and a mean seeking term, which provides a well motivated trade-off between exploitation and exploration. To empirically evaluate our method, we take a set of algorithmic tasks such as sequence reversal, multi-digit addition, and binary search. We choose to focus on these tasks because, although simple, they present a difficult sparse reward setting which has limited the success of standard RL approaches. The experiments demonstrate that UREX significantly outperforms baseline RL methods, such as entropy regularized REINFORCE and one-step Q-learning, especially on the more difficult tasks, such as multi-digit addition. Moreover, UREX is shown to be more robust to changes of hyper-parameters, which makes hyper-parameter tuning less tedious in practice. In addition to introducing a new variant of policy gradient with improved performance, our paper is the first to demonstrate strong results for an RL method on algorithmic tasks. To our knowledge, the addition task has not been solved by any model-free reinforcement learning approach. We observe that some of the policies learned by UREX can successfully generalize to long sequences; e.g., in 2 out of 5 random restarts, the policy learned by UREX for the addition task correctly generalizes to addition of numbers with 2000 digits with no mistakes, even though training sequences are at most 33 digits long.
2 NEURAL NETWORKS FOR LEARNING ALGORITHMS
Although research on using neural networks to learn algorithms has witnessed a surge of recent interest, the problem of program induction from examples has a long history in many fields, including program induction, inductive logic programming (Lavrac & Dzeroski, 1994), relational learning (Kemp et al., 2007) and regular language learning (Angulin, 1987). Rather than presenting a comprehensive survey of program induction here, we focus on neural network approaches to algorithmic tasks and highlight the relative simplicity of our neural network architecture.
Most successful applications of neural networks to algorithmic tasks rely on strong supervision, where the inputs and target outputs are completely known a priori. Given a dataset of examples, one learns the network parameters by maximizing the conditional likelihood of the outputs via backpropagation (e.g., Reed & de Freitas (2016); Kaiser & Sutskever (2016); Vinyals et al. (2015)). However, target outputs may not be available for novel tasks, for which no prior algorithm is known to be available. A more desirable approach to inducing algorithms, followed in this paper, advocates using self-driven learning strategies that only receive reinforcement based on the outputs produced. Hence, just by having access to a verifier for an algorithmic problem, one can aim to learn an algorithm. For example, if one does not know how to sort an array, but can check the extent to which an array is sorted, then one can provide the reward signal necessary for learning sorting algorithms.
We formulate learning algorithms as an RL problem and make use of model-free policy gradient methods to optimize a set parameters associated with the algorithm. In this setting, the goal is to learn a policy πθ that given an observed state st at step t, estimates a distribution over the next action at, denoted πθ(at | st). Actions represent the commands within the algorithm and states represent the joint state of the algorithm and the environment. Previous work in this area has focused on augmenting a neural network with additional structure and increased capabilities (Zaremba & Sutskever, 2015; Graves et al., 2016). In contrast, we utilize a simple architecture based on a standard recurrent neural network (RNN) with LSTM cells (Hochreiter & Schmidhuber, 1997) as depicted in Figure 1. At each episode, the environment is initialized with a latent state h, unknown to the agent, which determines s1 and the subsequent state transition and reward functions. Once the agent observes s1
as the input to the RNN, the network outputs a distribution πθ(a1 | s1), from which an action a1 is sampled. This action is applied to the environment, and the agent receives a new state observation s2. The state s2 and the previous action a1 are then fed into the RNN and the process repeats until the end of the episode. Upon termination, a reward signal is received.
3 LEARNING A POLICY BY MAXIMIZING EXPECTED REWARD
We start by discussing the most common form of policy gradient, REINFORCE (Williams, 1992), and its entropy regularized variant (Williams & Peng, 1991). REINFORCE has been applied to model-free policy-based learning with neural networks and algorithmic domains (Zaremba & Sutskever, 2015; Graves et al., 2016).
The goal is to learn a policy πθ that, given an observed state st at step t, estimates a distribution over the next action at, denoted πθ(at |st). The environment is initialized with a latent vector, h, which determines the initial observed state s1 = g(h), and the transition function st+1 = f(st,at | h). Note that the use of nondeterministic transitions f as in Markov decision processes (MDP) may be recovered by assuming that h includes the random seed for the any nondeterministic functions. Given a latent state h, and s1:T ≡ (s1, . . . , sT ), the model probability of an action sequence a1:T ≡ (a1, . . . ,aT ) is expressed as,
πθ(a1:T | h) = T∏ t=1 πθ(at | st) , where s1 = g(h), st+1 = f(st,at | h) for 1 ≤ t < T .
The environment provides a reward at the end of the episode, denoted r(a1:T | h). For ease of readability we drop the subscript from a1:T and simply write πθ(a | h) and r(a | h). The objective used to optimize the policy parameters, θ, consists of maximizing expected reward under actions drawn from the policy, plus an optional maximum entropy regularizer. Given a distribution over initial latent environment states p(h), we express the regularized expected reward as,
ORL(θ; τ) = Eh∼p(h) {∑
a∈A πθ(a | h)
[ r(a | h)− τ log πθ(a | h) ]} . (1)
When πθ is a non-linear function defined by a neural network, finding the global optimum of θ is challenging, and one often resorts to gradient-based methods to find a local optimum of ORL(θ; τ). Given that ddθπθ(a) = πθ(a) d dθ log πθ(a) for any a such that πθ(a) > 0, one can verify that,
d
dθ ORL(θ; τ | h) = ∑ a∈A πθ(a | h) d dθ log πθ(a | h) [ r(a | h)− τ log πθ(a | h)− τ ] . (2)
Because the space of possible actionsA is large, enumerating over all of the actions to compute this gradient is infeasible. Williams (1992) proposed to compute the stochastic gradient of the expected
reward by using Monte Carlo samples. Using Monte Carlo samples, one first drawsN i.i.d. samples from the latent environment states {h(n)}Nn=1, and then draws K i.i.d. samples {a(k)}Kk=1 from πθ(a | h(n)) to approximate the gradient of (1) by using (2) as,
d
dθ ORL(θ; τ) ≈
1
NK N∑ n=1 K∑ k=1 d dθ log πθ(a (k) | h(n)) [ r(a(k) | h(n))− τ log πθ(a(k) | h(n))− τ ] .
(3) This reparametrization of the gradients is the key to the REINFORCE algorithm. To reduce the variance of (3), one uses rewards r̂ that are shifted by some offset values,
r̂ (a(k) | h) = r(a(k) | h)− b(h) , (4) where b is known as a baseline or sometimes called a critic. Note that subtracting any offset from the rewards in (1) simply results in shifting the objective ORL by a constant. Unfortunately, directly maximizing expected reward (i.e., when τ = 0) is prone to getting trapped in a local optimum. To combat this tendency, Williams & Peng (1991) augmented the expected reward objective by including a maximum entropy regularizer (τ > 0) to promote greater exploration. We will refer to this variant of REINFORCE as MENT (maximum entropy exploration).
4 UNDER-APPRECIATED REWARD EXPLORATION (UREX)
To explain our novel form of policy gradient, we first note that the optimal policy π∗τ , which globally maximizes ORL(θ; τ | h) in (1) for any τ > 0, can be expressed as,
π∗τ (a | h) = 1
Z(h) exp {1 τ r(a | h) } , (5)
where Z(h) is a normalization constant making π∗τ a distribution over the space of action sequences A. One can verify this by first acknowledging that,
ORL(θ; τ | h) = −τ DKL (πθ(· | h) ‖ π∗τ (· | h)) . (6) Since DKL (p ‖ q) is non-negative and zero iff p = q, then π∗τ defined in (5) maximizes ORL. That said, given a particular form of πθ, finding θ that exactly characterizes π∗τ may not be feasible.
The KL divergence DKL (πθ ‖ π∗τ ) is known to be mode seeking (Murphy, 2012, Section 21.2.2) even with entropy regularization (τ > 0). Learning a policy by optimizing this direction of the KL is prone to falling into a local optimum resulting in a sub-optimal policy that omits some of the modes of π∗τ . Although entropy regularization helps mitigate the issues as confirmed in our experiments, it is not an effective exploration strategy as it is undirected and requires a small regularization coefficient τ to avoid too much random exploration. Instead, we propose a directed exploration strategy that improves the mean seeking behavior of policy gradient in a principled way.
We start by considering the alternate mean seeking direction of the KL divergence, DKL (π∗τ ‖ πθ). Norouzi et al. (2016) considered this direction of the KL to directly learn a policy by optimizing
ORAML(θ; τ) = Eh∼p(h) { τ ∑ a∈A π∗τ (a | h) log πθ(a | h) } , (7)
for structured prediction. This objective has the same optimal solution π∗τ as ORL since, ORAML(θ; τ | h) = −τ DKL (π∗τ (· | h) ‖ πθ(· | h)) + const . (8)
Norouzi et al. (2016) argue that in some structured prediction problems when one can draw samples from π∗τ , optimizing (7) is more effective than (1), since no sampling from a non-stationary policy πθ is required. If πθ is a log-linear model of a set of features,ORAML is convex in θ whereasORL is not, even in the log-linear case. Unfortunately, in scenarios that the reward landscape is unknown or computing the normalization constant Z(h) is intractable, sampling from π∗τ is not straightforward.
In RL problems, the reward landscape is completely unknown, hence sampling from π∗τ is intractable. This paper proposes to approximate the expectation with respect to π∗τ by using selfnormalized importance sampling (Owen, 2013), where the proposal distribution is πθ and the reference distribution is π∗τ . For importance sampling, one draws K i.i.d. samples {a(k)}Kk=1 from
πθ(a | h) and computes a set of normalized importance weights to approximate ORAML(θ; τ | h) as,
ORAML(θ; τ | h) ≈ τ K∑ k=1 wτ (a (k) | h)∑K m=1 wτ (a (m) | h) log πθ(a (k) | h) , (9)
where wτ (a(k) | h) ∝ π∗τ/πθ denotes an importance weight defined by,
wτ (a (k) | h) = exp {1 τ r(a(k) | h)− log πθ(a(k) | h) } . (10)
One can view these importance weights as evaluating the discrepancy between scaled rewards r/τ and the policy’s log-probabilities log πθ. Among the K samples, a sample that is least appreciated by the model, i.e., has the largest r/τ − log πθ, receives the largest positive feedback in (9). In practice, we have found that just using the importance sampling RAML objective in (9) does not always yield promising solutions. Particularly, at the beginning of training, when πθ is still far away from π∗τ , the variance of importance weights is too large, and the self-normalized importance sampling procedure results in poor approximations. To stabilize early phases of training and ensure that the model distribution πθ achieves large expected reward scores, we combine the expected reward and RAML objectives to benefit from the best of their mode and mean seeking behaviors. Accordingly, we propose the following objective that we call under-appreciated reward exploration (UREX),
OUREX(θ; τ) = Eh∼p(h) {∑
a∈A
[ πθ(a | h) r(a | h) + τ π∗τ (a | h) log πθ(a | h) ]} , (11)
which is the sum of the expected reward and RAML objectives. In our preliminary experiments, we considered a composite objective of ORL + ORAML, but we found that removing the entropy term is beneficial. Hence, theOUREX objective does not include entropy regularization. Accordingly, the optimum policy for OUREX is no longer π∗τ , as it was for ORL and ORAML. Appendix A derives the optimal policy for OUREX as a function of the optimal policy for ORL. We find that the optimal policy of UREX is more sharply concentrated on the high reward regions of the action space, which may be an advantage for UREX, but we leave more analysis of this behavior to future work.
To compute the gradient ofOUREX(θ; τ), we use the self-normalized importance sampling estimate outlined in (9). We assume that the importance weights are constant and contribute no gradient to d dθOUREX(θ; τ). To approximate the gradient, one draws N i.i.d. samples from the latent environment states {h(n)}Nn=1, and then draws K i.i.d. samples {a(k)}Kk=1 from πθ(a |h (n)) to obtain
d
dθ OUREX(θ; τ) ≈
1
N N∑ n=1 K∑ k=1 d dθ log πθ(a (k) |h(n)) [ 1 K r̂ (a(k) | h(n))+τ wτ (a (k) |h(n))∑K m=1wτ (a (m) |h(n)) ] .
(12) As with REINFORCE, the rewards are shifted by an offset b(h). In this gradient, the model logprobability of a sample action sequence a(k) is reinforced if the corresponding reward is large, or the corresponding importance weights are large, meaning that the action sequence is under-appreciated. The normalized importance weights are computed using a softmax operator softmax(r/τ − log πθ).
5 RELATED WORK
Before presenting the experimental results, we briefly review some pieces of previous work that closely relate to the UREX approach.
Reward-Weighted Regression. Both RAML and UREX objectives bear some similarity to a method in continuous control known as Reward-Weighted Regression (RWR) (Peters & Schaal, 2007; Wierstra et al., 2008). Using our notation, the RWR objective is expressed as,
ORWR(θ; τ | h) = log ∑ a∈A π∗τ (a | h)πθ(a | h) (13)
≥ ∑ a∈A q(a | h) log π ∗ τ (a | h)πθ(a | h) q(a | h) . (14)
To optimize ORWR, Peters & Schaal (2007) propose a technique inspired by the EM algorithm to maximize a variational lower bound in (14) based on a variational distribution q(a | h). The RWR objective can be interpreted as a log of the correlation between π∗τ and πθ. By contrast, the RAML and UREX objectives are both based on a KL divergence between π∗τ and πθ.
To optimize the RWR objective, one formulates the gradient as, d
dθ ORWR(θ; τ | h) = ∑ a∈A π∗τ (a | h)πθ(a | h) C d dθ log πθ(a | h), (15)
where C denotes the normalization factor, i.e., C = ∑
a∈A π ∗ τ (a | h)πθ(a | h). The expectation
with respect to π∗τ (a | h)πθ(a | h)/C on the RHS can be approximated by self-normalized importance sampling,1 where the proposal distribution is πθ. Accordingly, one draws K Monte Carlo samples {a(k)}Kk=1 i.i.d. from πθ(a |h) and formulates the gradient as,
d
dθ ORWR(θ; τ | h) ≈
1
K K∑ k=1 u(a(k) | h)∑K m=1 u(a (m) | h) d dθ log πθ(a (k) | h), (16)
where u(a(k) | h) = exp{ 1τ r(a (k) | h)}. There is some similarity between (16) and (9) in that they both use self-normalized importance sampling, but note the critical difference that (16) and (9) estimate the gradients of two different objectives, and hence the importance weights in (16) do not correct for the sampling distribution πθ(a |h) as opposed to (9). Beyond important technical differences, the optimal policy of ORWR is a one hot distribution with all probability mass concentrated on an action sequence with maximal reward, whereas the optimal policies for RAML and UREX are everywhere nonzero, with the probability of different action sequences being assigned proportionally to their exponentiated reward (with UREX introducing an additional re-scaling; see Appendix A). Further, the notion of under-appreciated reward exploration evident in OUREX, which is key to UREX’s performance, is missing in the RWR formulation. Exploration. The RL literature contains many different attempts at incorporating exploration that may be compared with our method. The most common exploration strategy considered in valuebased RL is -greedy Q-learning, where at each step the agent either takes the best action according to its current value approximation or with probability takes an action sampled uniformly at random. Like entropy regularization, such an approach applies undirected exploration, but it has achieved recent success in game playing environments (Mnih et al., 2013; Van Hasselt et al., 2016; Mnih et al., 2016).
Prominent approaches to improving exploration beyond -greedy in value-based or model-based RL have focused on reducing uncertainty by prioritizing exploration toward states and actions where the agent knows the least. This basic intuition underlies work on counter and recency methods (Thrun, 1992), exploration methods based on uncertainty estimates of values (Kaelbling, 1993; Tokic, 2010), methods that prioritize learning environment dynamics (Kearns & Singh, 2002; Stadie et al., 2015), and methods that provide an intrinsic motivation or curiosity bonus for exploring unknown states (Schmidhuber, 2006; Bellemare et al., 2016).
In contrast to value-based methods, exploration for policy-based RL methods is often a by-product of the optimization algorithm itself. Since algorithms like REINFORCE and Thompson sampling choose actions according to a stochastic policy, sub-optimal actions are chosen with some non-zero probability. The Q-learning algorithm may also be modified to sample an action from the softmax of the Q values rather than the argmax (Sutton & Barto, 1998).
Asynchronous training has also been reported to have an exploration effect on both value- and policy-based methods. Mnih et al. (2016) report that asynchronous training can stabilize training by reducing the bias experienced by a single trainer. By using multiple separate trainers, an agent is less likely to become trapped at a policy found to be locally optimal only due to local conditions. In the same spirit, Osband et al. (2016) use multiple Q value approximators and sample only one to act for each episode as a way to implicitly incorporate exploration.
By relating the concepts of value and policy in RL, the exploration strategy we propose tries to bridge the discrepancy between the two. In particular, UREX can be viewed as a hybrid combination of value-based and policy-based exploration strategies that attempts to capture the benefits of each.
1Bornschein & Bengio (2014) apply the same trick to optimize the log-likelihood of latent variable models.
Per-step Reward. Finally, while we restrict ourselves to episodic settings where a reward is associated with an entire episode of states and actions, much work has been done to take advantage of environments that provide per-step rewards. These include policy-based methods such as actor-critic (Mnih et al., 2016; Schulman et al., 2016) and value-based approaches based on Qlearning (Van Hasselt et al., 2016; Schaul et al., 2016). Some of these value-based methods have proposed a softening of Q-values which can be interpreted as adding a form of maximum-entropy regularizer (Asadi & Littman, 2016; Azar et al., 2012; Fox et al., 2016; Ziebart, 2010). The episodic total-reward setting that we consider is naturally harder since the credit assignment to individual actions within an episode is unclear.
6 SIX ALGORITHMIC TASKS
We assess the effectiveness of the proposed approach on five algorithmic tasks from the OpenAI Gym (Brockman et al., 2016), as well as a new binary search problem. Each task is summarized below, with further details available on the Gym website2 or in the corresponding open-source code.3 In each case, the environment has a hidden tape and a hidden sequence. The agent observes the sequence via a pointer to a single character, which can be moved by a set of pointer control actions. Thus an action at is represented as a tuple (m,w, o) where m denotes how to move, w is a boolean denoting whether to write, and o is the output symbol to write.
1. Copy: The agent should emit a copy of the sequence. The pointer actions are move left and right. 2. DuplicatedInput: In the hidden tape, each character is repeated twice. The agent must dedupli-
cate the sequence and emit every other character. The pointer actions are move left and right.
3. RepeatCopy: The agent should emit the hidden sequence once, then emit the sequence in the reverse order, then emit the original sequence again. The pointer actions are move left and right.
4. Reverse: The agent should emit the hidden sequence in the reverse order. As before, the pointer actions are move left and right.
5. ReversedAddition: The hidden tape is a 2×n grid of digits representing two numbers in base 3 in little-endian order. The agent must emit the sum of the two numbers, in little-endian order. The allowed pointer actions are move left, right, up, or down.
The OpenAI Gym provides an additional harder task called ReversedAddition3, which involves adding three numbers. We omit this task, since none of the methods make much progress on it.
For these tasks, the input sequences encountered during training range from a length of 2 to 33 characters. A reward of 1 is given for each correct emission. On an incorrect emission, a small penalty of−0.5 is incurred and the episode is terminated. The agent is also terminated and penalized with a reward of −1 if the episode exceeds a certain number of steps. For the experiments using UREX and MENT, we associate an episodic sequence of actions with the total reward, defined as the sum of the per-step rewards. The experiments using Q-learning, on the other hand, used the per-step rewards. Each of the Gym tasks has a success threshold, which determines the required average reward over 100 episodes for the agent to be considered successful.
We also conduct experiments on an additional algorithmic task described below: 6. BinarySearch: Given an integer n, the environment has a hidden array of n distinct numbers
stored in ascending order. The environment also has a query number x unknown to the agent that is contained somewhere in the array. The goal of the agent is to find the query number in the array in a small number of actions. The environment has three integer registers initialized at (n, 0, 0). At each step, the agent can interact with the environment via the four following actions: • INC(i): increment the value of the register i for i ∈ {1, 2, 3}. • DIV(i): divide the value of the register i by 2 for i ∈ {1, 2, 3}. • AVG(i): replace the value of the register i with the average of the two other registers. • CMP(i): compare the value of the register i with x and receive a signal indicating which
value is greater. The agent succeeds when it calls CMP on an array cell holding the value x.
2gym.openai.com 3github.com/openai/gym
We set the maximum number of steps to 2n+1 to allow the agent to perform a full linear search. A policy performing full linear search achieves an average reward of 5, because x is chosen uniformly at random from the elements of the array. A policy employing binary search can find the number x in at most 2 log2 n + 1 steps. If n is selected uniformly at random from the range 32 ≤ n ≤ 512, binary search yields an optimal average reward above 9.55. We set the success threshold for this task to an average reward of 9.
7 EXPERIMENTS
We compare our policy gradient method using under-appreciated reward exploration (UREX) against two main RL baselines: (1) REINFORCE with entropy regularization termed MENT (Williams & Peng, 1991), where the value of τ determines the degree of regularization. When τ = 0, standard REINFORCE is obtained. (2) one-step double Q-learning based on bootstrapping one step future rewards.
7.1 ROBUSTNESS TO HYPER-PARAMETERS
Hyper-parameter tuning is often tedious for RL algorithms. We found that the proposed UREX method significantly improves robustness to changes in hyper-parameters when compared to MENT. For our experiments, we perform a careful grid search over a set of hyper-parameters for both MENT and UREX. For any hyper-parameter setting, we run the MENT and UREX methods 5 times with different random restarts. We explore the following main hyper-parameters:
• The learning rate denoted η chosen from a set of 3 possible values η ∈ {0.1, 0.01, 0.001}. • The maximum L2 norm of the gradients, beyond which the gradients are clipped. This parame-
ter, denoted c, matters for training RNNs. The value of c is selected from c ∈ {1, 10, 40, 100}. • The temperature parameter τ that controls the degree of exploration for both MENT and UREX.
For MENT, we use τ ∈ {0, 0.005, 0.01, 0.1}. For UREX, we only consider τ = 0.1, which consistently performs well across the tasks.
In all of the experiments, both MENT and UREX are treated exactly the same. In fact, the change of implementation is just a few lines of code. Given a value of τ , for each task, we run 60 training jobs comprising 3 learning rates, 4 clipping values, and 5 random restarts. We run each algorithm for a maximum number of steps determined based on the difficulty of the task. The training jobs for Copy, DuplicatedInput, RepeatCopy, Reverse, ReversedAddition, and BinarySearch are run for 2K, 500, 50K, 5K, 50K, and 2K stochastic gradient steps, respectively. We find that running a trainer job longer does not result in a better performance. Our policy network comprises a single LSTM layer with 128 nodes. We use the Adam optimizer (Kingma & Ba, 2015) for the experiments.
Table 1 shows the percentage of 60 trials on different hyper-parameters (η, c) and random restarts which successfully solve each of the algorithmic tasks. It is clear that UREX is more robust than
MENT to changes in hyper-parameters, even though we only report the results of UREX for a single temperature. See Appendix B for more detailed tables on hyper-parameter robustness.
7.2 RESULTS
Table 2 presents the number of successful attempts (out of 5 random restarts) and the expected reward values (averaged over 5 trials) for each RL algorithm given the best hyper-parameters. Onestep Q-learning results are also included in the table. We also present the training curves for MENT and UREX in Figure 2. It is clear that UREX outperforms the baselines on these tasks. On the more difficult tasks, such as Reverse and ReverseAddition, UREX is able to consistently find an appropriate algorithm, but MENT and Q-learning fall behind. Importantly, for the BinarySearch task, which exhibits many local maxima and necessitates smart exploration, UREX is the only method that can solve it consistently. The Q-learning baseline solves some of the simple tasks, but it makes little headway on the harder tasks. We believe that entropy regularization for policy gradient and - greedy for Q-learning are relatively weak exploration strategies in long episodic tasks with delayed rewards. On such tasks, one random exploratory step in the wrong direction can take the agent off the optimal policy, hampering its ability to learn. In contrast, UREX provides a form of adaptive and smart exploration. In fact, we observe that the variance of the importance weights decreases as the agent approaches the optimal policy, effectively reducing exploration when it is no longer necessary; see Appendix E.
7.3 GENERALIZATION TO LONGER SEQUENCES
To confirm whether our method is able to find the correct algorithm for multi-digit addition, we investigate its generalization to longer input sequences than provided during training. We evaluate the trained models on inputs up to a length of 2000 digits, even though training sequences were at most 33 characters. For each length, we test the model on 100 randomly generated inputs, stopping when the accuracy falls below 100%. Out of the 60 models trained on addition with UREX, we find that 5 models generalize to numbers up to 2000 digits without any observed mistakes. On the best UREX hyper-parameters, 2 out of the 5 random restarts are able to generalize successfully. For more detailed results on the generalization performance on 3 different tasks including Copy,
DuplicatedInput, and ReversedAddition, see Appendix C. During these evaluations, we take the action with largest probability from πθ(a | h) at each time step rather than sampling randomly. We also looked into the generalization of the models trained on the BinarySearch task. We found that none of the agents perform proper binary search. Rather, those that solved the task perform a hybrid of binary and linear search: first actions follow a binary search pattern, but then the agent switches to a linear search procedure once it narrows down the search space; see Appendix D for some execution traces for BinarySearch and ReversedAddition. Thus, on longer input sequences, the agent’s running time complexity approaches linear rather than logarithmic. We hope that future work will make more progress on this task. This task is especially interesting because the reward signal should incorporate both correctness and efficiency of the algorithm.
7.4 IMPLEMENTATION DETAILS
In all of the experiments, we make use of curriculum learning. The environment begins by only providing small inputs and moves on to longer sequences once the agent achieves close to maximal reward over a number of steps. For policy gradient methods including MENT and UREX, we only provide the agent with a reward at the end of the episode, and there is no notion of intermediate reward. For the value-based baseline, we implement one-step Q-learning as described in Mnih et al. (2016)-Alg. 1, employing double Q-learning with -greedy exploration. We use the same RNN in our policy-based approaches to estimate the Q values. A grid search over exploration rate, exploration rate decay, learning rate, and sync frequency (between online and target network) is conducted to find the best hyper-parameters. Unlike our other methods, the Q-learning baseline uses intermediate rewards, as given by the OpenAI Gym on a per-step basis. Hence, the Q-learning baseline has a slight advantage over the policy gradient methods.
In all of the tasks except Copy, our stochastic optimizer uses mini-batches comprising 400 policy samples from the model. These 400 samples correspond to 40 different random sequences drawn from the environment, and 10 random policy trajectories per sequence. In other words, we set K = 10 and N = 40 as defined in (3) and (12). For MENT, we use the 10 samples to subtract the mean of the coefficient of ddθ log πθ(a | h) which includes the contribution of the reward and entropy regularization. For UREX, we use the 10 trajectories to subtract the mean reward and normalize the importance sampling weights. We do not subtract the mean of the normalized importance weights. For the Copy task, we use mini-batches with 200 samples using K = 10 and N = 20. Experiments are conducted using Tensorflow (Abadi et al., 2016).
8 CONCLUSION
We present a variant of policy gradient, called UREX, which promotes the exploration of action sequences that yield rewards larger than what the model expects. This exploration strategy is the result of importance sampling from the optimal policy. Our experimental results demonstrate that UREX significantly outperforms other value and policy based methods, while being more robust
to changes of hyper-parameters. By using UREX, we can solve algorithmic tasks like multi-digit addition from only episodic reward, which other methods cannot reliably solve even given the best hyper-parameters. We introduce a new algorithmic task based on binary search to advocate more research in this area, especially when the computational complexity of the solution is also of interest. Solving these tasks is not only important for developing more human-like intelligence in learning algorithms, but also important for generic reinforcement learning, where smart and efficient exploration is the key to successful methods.
9 ACKNOWLEDGMENT
We thank Sergey Levine, Irwan Bello, Corey Lynch, George Tucker, Kelvin Xu, Volodymyr Mnih, and the Google Brain team for insightful comments and discussions.
A OPTIMAL POLICY FOR THE UREX OBJECTIVE
To derive the form of the optimal policy for the UREX objective (11), note that for each h one would like to maximize ∑
a∈A
[ πθ(a) r(a) + τ π ∗ τ (a) log πθ(a) ] , (17)
subject to the constraint ∑
a∈A πθ(a) = 1. To enforce the constraint, we introduce a Lagrange multiplier α and aim to maximize∑
a∈A
[ πθ(a) r(a) + τ π ∗ τ (a) log πθ(a)− απθ(a) ] + α . (18)
Since the gradient of the Lagrangian (18) with respect to θ is given by∑ a∈A dπθ(a) dθ [ r(a) + τ π∗τ (a) πθ(a) − α ] , (19)
the optimal choice for πθ is achieved by setting
πθ(a) = τ π∗τ (a)
α− r(a) for all a ∈ A , (20)
forcing the gradient to be zero. The Lagrange multiplier α can then be chosen so that ∑
a∈A πθ(a) = 1 while also satisfying α > maxa∈A r(a); see e.g. (Golub, 1987).
B ROBUSTNESS TO HYPER-PARAMETERS
Tables 3–8 provide more details on different cells of Table 1. Each table presents the results of MENT using the best temperature τ vs. UREX with τ = 0.1 on a variety of learning rates and clipping values. Each cell is the number of trials out of 5 random restarts that succeed at solving the task using a specific η and c.
C GENERALIZATION TO LONGER SEQUENCES
Table 9 provides a more detailed look into the generalization performance of the trained models on Copy, DuplicatedInput, and ReversedAddition. The tables show how the number of models which can solve the task correctly drops off as the length of the input increases.
D EXAMPLE EXECUTION TRACES
We provide the traces of two trained agents on the ReversedAddition task (Figure 3) and the BinarySearch task (Table 10).
E VARIANCE OF IMPORTANCE WEIGHTS
F A SIMPLE BANDIT TASK

overview:
This work proposes to link trajectory log-probabilities and rewards by defining under-appreciated rewards. This suggests that there is a linear relationship between trajectory rewards and their log-probability which can be exploited by measuring the resulting mismatch.
That is, when an action sequence under-appreciates its reward, its log-probability is increased.
This method is a simple modification to the well-known REINFORCE method, requiring only one extra hyperparameter \tau, and intuitively provides us with a better exploration mechanism than \epsilon-greedy or random exploration.
The method is tested on algorithmic environments, and compared to entropy-regularized REINFORCE and double Q-learning, and performs equally or better than those two baselines, especially in more complex environments.
remarks:
- the focus in the introduction on algorithmic tasks may be a double-edged sword. It is an interesting domain to test your hypothesis and benchmark your method. At the same time, it distracts the reader from the (IMO) generality of the proposed method.
- in the introduction you say the reward is sparse, in section 6, on tasks 1-5, you then say there is a reward at each correct emission, i.e. each time step. This is only 'corrected' to end-of-episode-reward in section 7.4, after having discussed results. I'd move or mention this in section 6.
- approach seems quite sensible to tau being in the same range as logpi(a|h), but you only try tau=0.1 for UREX. I'm not sure I understand nor agree with this experimentation choice.
- an alternative to grid search is random search (Bergstra&Bengio, 2012). It may illustrate better hyperparameter robustness, and allow you to explore more in the same number of experiment.
opinion:
- An interesting approach to policy-gradient, to be sure. It tackles the very important question of "how should agents explore?"
- I'm ambivalent to claiming that an algorithm is robust to hyperparmeters simply because it performs better on the selected hyperparameter range. All you really show is that it performs well some amount of time when the hyperparams lay in that range. Couldn't it be that MENT needs different hyperparameters? (Just being devil's advocate here)
- I see why matching 1/tau with logpi is the obvious choice, but it implies a very strong prior: that the reward (to a factor of 1/tau) lies in the same space as the log policy. One point of failure I see (but correct me if I'm wrong) is that as the length of the trajectory grows the reward is expected to grow linearly, so short ways to get some reward will be less explored than long ways of getting the same reward, creating an imbalance unless the reward is shaped such that shorter trajectories get more reward (which is only the case in task 6).
- It might have been good to also compare with methods explicitly trying to explore better with value-functions (e.g. prioritized experience replay, Schaul et al 2015)
- At the risk of repeating myself, tau plays a major role in this method, but there is little analysis on its effect on experiments.
The methodology and reasoning is clearly explained and I think this paper communicates its message very well.
That message is novel, albeit a minor modification to a well-known algorithm, it is well motivated and, I think, a welcome addition to literature concerning exploration in RL.
The experiments are chosen accordingly, and results seem to reflect the hypothesis of the authors.
I realize the tyranny of extensive experimentation and the scarcity of time, but I do think that this paper would benefit from more (or cleverer) experimentation, as well as demonstrating more explicitly the impact of the method on exploration.
Reading this paper convinced me that measuring mismatch between a trajectory's observed reward and its probability given the current policy is a clever (and well motivated) thing to do. Yet, I think that the paper could have a more convincing empirical argument, even if it is for toy tasks.