SURPRISE-BASED INTRINSIC MOTIVATION FOR DEEP REINFORCEMENT LEARNING
Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as -greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent’s surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the k-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.
1 INTRODUCTION
A reinforcement learning agent uses experiences obtained from interacting with an unknown environment to learn behavior that maximizes a reward signal. The optimality of the learned behavior is strongly dependent on how the agent approaches the exploration/exploitation trade-off in that environment. If it explores poorly or too little, it may never find rewards from which to learn, and its behavior will always remain suboptimal; if it does find rewards but exploits them too intensely, it may wind up prematurely converging to suboptimal behaviors, and fail to discover more rewarding opportunities. Although substantial theoretical work has been done on optimal exploration strategies for environments with finite state and action spaces, we are here concerned with problems that have continuous state and/or action spaces, where algorithms with theoretical guarantees admit no obvious generalization or are prohibitively impractical to implement.
Simple heuristic methods of exploring such as -greedy action selection and Gaussian control noise have been successful on a wide range of tasks, but are inadequate when rewards are especially sparse. For example, the Deep Q-Network approach of Mnih et al. [13] used -greedy exploration in training deep neural networks to play Atari games directly from raw pixels. On many games, the algorithm resulted in superhuman play; however, on games like Montezuma’s Revenge, where rewards are extremely sparse, DQN (and its variants [25], [26], [15], [12]) with -greedy exploration failed to achieve scores even at the level of a novice human. Similarly, in benchmarking deep reinforcement learning for continuous control, Duan et al.[5] found that policy optimization algorithms that explored by acting according to the current stochastic policy, including REINFORCE and Trust Region Policy Optimization (TRPO), could succeed across a diverse slate of simulated robotics control tasks with well-defined, non-sparse reward signals (like rewards proportional to the forward velocity of the robot). Yet, when tested in environments with sparse rewards—where the agent would only be able to attain rewards after first figuring out complex motion primitives without reinforcement—every algorithm failed to attain scores better than random agents. The failure modes in all of these cases pertained to the nature of the exploration: the agents encountered reward signals so infrequently that they were never able to learn reward-seeking behavior.
One approach to encourage better exploration is via intrinsic motivation, where an agent has a task-independent, often information-theoretic intrinsic reward function which it seeks to maximize in addition to the reward from the environment. Examples of intrinsic motivation include empowerment, where the agent enjoys the level of control it has about its future; surprise, where the agent is excited to see outcomes that run contrary to its understanding of the world; and novelty, where the agent is excited to see new states (which is tightly connected to surprise, as shown in [2]). For in-depth reviews of the different types of intrinsic motivation, we direct the reader to [1] and [17].
Recently, several applications of intrinsic motivation to the deep reinforcement learning setting (such as [2], [7], [22]) have found promising success. In this work, we build on that success by exploring scalable measures of surprise for intrinsic motivation in deep reinforcement learning. We formulate surprise as the KL-divergence of the true transition probability distribution from a transition model which is learned concurrently with the policy, and consider two approximations to this divergence which are easy to compute in practice. One of these approximations results in using the surprisal of a transition as an intrinsic reward; the other results in using a measure of learning progress which is closer to a Bayesian concept of surprise. Our contributions are as follows:
1. we investigate surprisal and learning progress as intrinsic rewards across a wide range of environments in the deep reinforcement learning setting, and demonstrate empirically that the incentives (especially surprisal) result in efficient exploration,
2. we evaluate the difficulty of the slate of sparse reward continuous control tasks introduced by Houthooft et al. [7] to benchmark exploration incentives, and introduce a new task to complement the slate,
3. and we present an efficient method for learning the dynamics model (transition probabilities) concurrently with a policy.
We distinguish our work from prior work in a number of implementation details: unlike Bellemare et al. [2], we learn a transition model as opposed to a state-action occupancy density; unlike Stadie et al. [22], our formulation naturally encompasses environments with stochastic dynamics; unlike Houthooft et al. [7], we avoid the overhead of maintaining a distribution over possible dynamics models, and learn a single deep dynamics model.
In our empirical evaluations, we compare the performance of our proposed intrinsic rewards with other heuristic intrinsic reward schemes and to recent results from the literature. In particular, we compare to Variational Information Maximizing Exploration (VIME) [7], a method which approximately maximizes Bayesian surprise and currently achieves state-of-the-art performance on continuous control with sparse rewards. We show that our incentives can perform on the level of VIME at a lower computational cost.
2 PRELIMINARIES
We begin by introducing notation which we will use throughout the paper. A Markov decision process (MDP) is a tuple, (S,A,R, P, µ), where S is the set of states, A is the set of actions, R : S × A × S → R is the reward function, P : S × A × S → [0, 1] is the transition probability function (where P (s′|s, a) is the probability of transitioning to state s′ given that the previous state was s and the agent took action a in s), and µ : S → [0, 1] is the starting state distribution. A policy π : S × A → [0, 1] is a distribution over actions per state, with π(a|s) the probability of selecting a in state s. We aim to select a policy π which maximizes a performance measure, L(π), which usually takes the form of expected finite-horizon total return (sum of rewards in a fixed time period), or expected infinite-horizon discounted total return (discounted sum of all rewards forever). In this paper, we use the finite-horizon total return formulation.
3 SURPRISE INCENTIVES
To train an agent with surprise-based exploration, we alternate between making an update step to a dynamics model (an approximator of the MDP’s transition probability function), and making a policy update step that maximizes a trade-off between policy performance and a surprise measure.
The dynamics model step makes progress on the optimization problem
min φ − 1 |D| ∑ (s,a,s′)∈D logPφ(s ′|s, a) + αf(φ), (1)
where D is is a dataset of transition tuples from the environment, Pφ is the model we are learning, f is a regularization function, and α > 0 is a regularization trade-off coefficient. The policy update step makes progress on an approximation to the optimization problem
max π L(π) + η E s,a∼π
[DKL(P ||Pφ)[s, a]] , (2)
where η > 0 is an explore-exploit trade-off coefficient. The exploration incentive in (2), which we select to be the on-policy average KL-divergence of Pφ from P , is intended to capture the agent’s surprise about its experience. The dynamics model Pφ should only be close to P on regions of the transition state space that the agent has already visited (because those transitions will appear in D and thus the model will be fit to them), and as a result, the KL divergence of Pφ and P will be higher in unfamiliar places. Essentially, this exploits the generalization in the model to encourage the agent to go where it has not gone before. The surprise incentive in (2) gives the net effect of performing a reward shaping of the form
r′(s, a, s′) = r(s, a, s′) + η (logP (s′|s, a)− logPφ(s′|s, a)) , (3) where r(s, a, s′) is the original reward and r′(s, a, s′) is the transformed reward, so ideally we could solve (2) by applying any reinforcement learning algorithm with these reshaped rewards. In practice, we cannot directly implement this reward reshaping because P is unknown. Instead, we consider two ways of finding an approximate solution to (2).
In one method, we approximate the KL-divergence by the cross-entropy, which is reasonable when H(P ) is finite (and small) and Pφ is sufficiently far from P 1; that is, denoting the cross-entropy by H(P, Pφ)[s, a] . = Es′∼P (·|s,a)[− logPφ(s′|s, a)], we assume
DKL(P ||Pφ)[s, a] = H(P, Pφ)[s, a]−H(P )[s, a] ≈ H(P, Pφ)[s, a].
(4)
This approximation results in a reward shaping of the form
r′(s, a, s′) = r(s, a, s′)− η logPφ(s′|s, a); (5) here, the intrinsic reward is the surprisal of s′ given the model Pφ and the context (s, a).
In the other method, we maximize a lower bound on the objective in (2) by lower bounding the surprise term:
DKL(P ||Pφ)[s, a] = DKL(P ||Pφ′)[s, a] + E s′∼P
[ log Pφ′(s ′|s, a)
Pφ(s′|s, a) ] ≥ E s′∼P [ log Pφ′(s ′|s, a)
Pφ(s′|s, a)
] .
(6)
The bound (6) results in a reward shaping of the form
r′(s, a, s′) = r(s, a, s′) + η (logPφ′(s ′|s, a)− logPφ(s′|s, a)) , (7)
which requires a choice of φ′. From (6), we can see that the bound becomes tighter by minimizing DKL(P ||Pφ′). As a result, we choose φ′ to be the parameters of the dynamics model after k updates based on (1), and φ to be the parameters from before the updates. Thus, at iteration t, the reshaped rewards are
r′(s, a, s′) = r(s, a, s′) + η ( logPφt(s ′|s, a)− logPφt−k(s′|s, a) ) ; (8)
here, the intrinsic reward is the k-step learning progress at (s, a, s′). It also bears a resemblance to Bayesian surprise; we expand on this similarity in the next section.
In our experiments, we investigate both the surprisal bonus (5) and the k-step learning progress bonus (8) (with varying values of k).
1On the other hand, if H(P )[s, a] is non-finite everywhere—for instance if the MDP has continuous states and deterministic transitions—then as long as it has the same sign everywhere, Es,a∼π[H(P )[s, a]] is a constant with respect to π and we can drop it from the optimization problem anyway.
3.1 DISCUSSION
Ideally, we would like the intrinsic rewards to vanish in the limit as Pφ → P , because in this case, the agent should have sufficiently explored the state space, and should primarily learn from extrinsic rewards. For the proposed intrinsic reward in (5), this is not the case, and it may result in poor performance in that limit. The thinking goes that when Pφ = P , the agent will be incentivized to seek out states with the noisiest transitions. However, we argue that this may not be an issue, because the intrinsic motivation seems mostly useful long before the dynamics model is fully learned. As long as the agent is able to find the extrinsic rewards before the intrinsic reward is just the entropy in P , the pathological noise-seeking behavior should not happen. On the other hand, the intrinsic reward in (8) should not suffer from this pathology, because in the limit, as the dynamics model converges, we should have Pφt ≈ Pφt−k . Then the intrinsic reward will vanish as desired. Next, we relate (8) to Bayesian surprise. The Bayesian surprise associated with a transition is the reduction in uncertainty over possibly dynamics models from observing it ([1],[8]):
DKL (P (φ|ht, at, st+1)||P (φ|ht)) . Here, P (φ|ht) is meant to represent a distribution over possible dynamics models parametrized by φ given the preceding history of observed states and actions ht (so ht includes st), and P (φ|ht, at, st+1) is the posterior distribution over dynamics models after observing (at, st+1). By Bayes’ rule, the dynamics prior and posterior are related to the model-based transition probabilities by
P (φ|ht, at, st+1) = P (φ|ht)P (st+1|ht, at, φ)
Eφ∼P (·|ht) [P (st+1|ht, at, φ)] ,
so the Bayesian surprise can be expressed as E
φ∼Pt+1 [logP (st+1|ht, at, φ)]− log E φ∼Pt [P (st+1|ht, at, φ)] , (9)
where Pt+1 = P (·|ht, at, st+1) is the posterior and Pt = P (·|ht) is the prior. In this form, the resemblance between (9) and (8) is clarified. Although the update from φt−k to φt is not Bayesian— and is performed in batch, instead of per transition sample—we can imagine (8) might contain similar information to (9).
3.2 IMPLEMENTATION DETAILS
Our implementation usesL2 regularization in the dynamics model fitting, and we impose an additional constraint to keep model iterates close in the KL-divergence sense. Denoting the average divergence as
D̄KL(Pφ′ ||Pφ) = 1 |D| ∑
(s,a)∈D
DKL(Pφ′ ||Pφ)[s, a], (10)
our dynamics model update is
φi+1 = arg min φ − 1 |D| ∑ (s,a,s′)∈D logPφ(s ′|s, a) + α‖φ‖22 : D̄KL(Pφ||Pφi) ≤ κ. (11)
The constraint value κ is a hyper-parameter of the algorithm. We solve this optimization problem approximately using a single second-order step with a line search, as described by [20]; full details are given in supplementary material. D is a FIFO replay memory, and at each iteration, instead of using the entirety of D for the update step we sub-sample a batch d ⊂ D. Also, similarly to [7], we adjust the bonus coefficient η at each iteration, to keep the average bonus magnitude upper-bounded (and usually fixed). Let η0 denote the desired average bonus, and r+(s, a, s′) denote the intrinsic reward; then, at each iteration, we set
η = η0 max (
1, 1|B| ∣∣∣∑(s,a,s′)∈B r+(s, a, s′)∣∣∣) , where B is the batch of data used for the policy update step. This normalization improves the stability of the algorithm by keeping the scale of the bonuses fixed with respect to the scale of the extrinsic rewards. Also, in environments where the agent can die, we avoid the possibility of the intrinsic rewards becoming a living cost by translating all bonuses so that the mean is nonnegative. The basic outline of the algorithm is given as Algorithm 1. In all experiments, we use fully-factored Gaussian distributions for the dynamics models, where the means and variances are the outputs of neural networks.
Algorithm 1 Reinforcement Learning with Surprise Incentive Input: Initial policy π0, dynamics model Pφ0 repeat
collect rollouts on current policy πi add rollout (s, a, s′) tuples to replay memory D compute reshaped rewards using (5) or (8) with dynamics model Pφi normalize η by the average intrinsic reward of the current batch of data update policy to πi+1 using any RL algorithm with the reshaped rewards update the dynamics model to Pφi+1 according to (11)
until training is completed
4 EXPERIMENTS
We evaluate our proposed surprise incentives on a wide range of benchmarks that are challenging for naive exploration methods, including continuous control and discrete control tasks. Our continuous control tasks include the slate of sparse reward tasks introduced by Houthooft et al. [7]: sparse MountainCar, sparse CartPoleSwingup, and sparse HalfCheetah, as well as a new sparse reward task that we introduce here: sparse Swimmer. (We refer to these environments with the prefix ‘sparse’ to differentiate them from other versions which appear in the literature, where agents receive non-sparse reward signals.) Additionally, we evaluate performance on a highly-challenging hierarchical sparse reward task introduced by Duan et al [5], SwimmerGather. The discrete action tasks are several games from the Atari RAM domain of the OpenAI Gym [4]: Pong, BankHeist, Freeway, and Venture.
Environments with deterministic and stochastic dynamics are represented in our benchmarks: the continuous control domains have deterministic dynamics, while the Gym Atari RAM games have stochastic dynamics. (In the Atari games, actions are repeated for a random number of frames.)
We use Trust Region Policy Optimization (TRPO) [20], a state-of-the-art policy gradient method, as our base reinforcement learning algorithm throughout our experiments, and we use the rllab implementations of TRPO and the continuous control tasks [5]. Full details for the experimental set-up are included in the appendix.
On all tasks, we compare against TRPO without intrinsic rewards, which we refer to as using naive exploration (in contrast to intrinsically motivated exploration). For the continuous control tasks, we also compare against intrinsic motivation using the L2 model prediction error,
r+(s, a, s ′) = ‖s′ − µφ(s, a)‖2, (12)
where µφ is the mean of the learned Gaussian distribution Pφ. The model prediction error was investigated as intrinsic motivation for deep reinforcement learning by Stadie et al [22], although they used a different method for learning the model µφ. This comparison helps us verify whether or not our proposed form of surprise, as a KL-divergence from the true dynamics model, is useful. Additionally, we compare our performance against the performance reported by Houthooft et al. [7] for Variational Information Maximizing Exploration (VIME), a method where the intrinsic reward associated with a transition approximates its Bayesian surprise using variational methods. Currently, VIME has achieved state-of-the-art results on intrinsic motivation for continuous control.
As a final check for the continuous control tasks, we benchmark the tasks themselves, by measuring the performance of the surprisal bonus without any dynamics learning: r+(s, a, s′) = − logPφ0(s′|s, a), where φ0 are the original random parameters of Pφ. This allows us to verify whether our benchmark tasks actually require surprise to solve at all, or if random exploration strategies successfully solve them.
4.1 CONTINUOUS CONTROL RESULTS
Median performance curves are shown in Figure 1 with interquartile ranges shown in shaded areas. Note that TRPO without intrinsic motivation failed on all tasks: the median score and upper quartile range for naive exploration were zero everywhere. Also note that TRPO with random exploration bonuses failed on most tasks, as shown separately in Figure 2. We found that surprise was not needed to solve MountainCar, but was necessary to perform well on the other tasks.
The surprisal bonus was especially robust across tasks, achieving good results in all domains and substantially exceeding the other baselines on the more challenging ones. The learning progress bonus for k = 1 was successful on CartpoleSwingup and HalfCheetah but it faltered in the others. Its weak performance in MountainCar was due to premature convergence of the dynamics model, which resulted in the agent receiving intrinsic rewards that were identically zero. (Given the simplicity of the environment, it is not surprising that the dynamics model converged so quickly.) In Swimmer, however, it seems that the learning progress bonuses did not inspire sufficient exploration. Because the Swimmer environment is effectively a stepping stone to the harder SwimmerGather, where the agent has to learn a motion primitive and collect target pellets, on SwimmerGather, we only evaluated the intrinsic rewards that had been successful on Swimmer.
Both surprisal and learning progress (with k = 1) exceeded the reported performance of VIME on HalfCheetah by learning to solve the task more quickly. On CartpoleSwingup, however, both were more susceptible to getting stuck in locally optimal policies, resulting in lower median scores than VIME. Surprisal performed comparably to VIME on SwimmerGather, the hardest task in the slate—in the sense that after 1000 iterations, they both reached approximately the same median score—although with greater variance than VIME.
Our results suggest that surprisal is a viable alternative to VIME in terms of performance, and is highly favorable in terms of computational cost. In VIME, a backwards pass through the dynamics model must be computed for every transition tuple separately to compute the intrinsic rewards, whereas our surprisal bonus only requires forward passes through the dynamics model for intrinsic
reward computation. (Limitations of current deep learning tool kits make it difficult to efficiently compute separate backwards passes, whereas almost all of them support highly parallel forward computations.) Furthermore, our dynamics model is substantially simpler than the Bayesian neural network dynamics model of VIME. To illustrate this point, in Figure 3 we show the results of a speed comparison making use of the open-source VIME code [6], with the settings described in the VIME paper. In our speed test, our bonus had a per-iteration speedup of a factor of 3 over VIME.2 We give a full analysis of the potential speedup in Appendix C.
4.2 ATARI RAM DOMAIN RESULTS
Median performance curves are shown in Figure 4, with tasks arranged from (a) to (d) roughly in order of increasing difficulty.
In Pong, naive exploration naturally succeeds, so we are not surprised to see that intrinsic motivation does not improve performance. However, this serves as a sanity check to verify that our intrinsic rewards do not degrade performance. (As an aside, we note that the performance here falls short of the standard score of 20 for this domain because we truncate play at 5000 timesteps.)
In BankHeist, we find that intrinsic motivation accelerates the learning significantly. The agents with surprisal incentives reached high levels of performance (scores > 1000) 10% sooner than naive exploration, while agents with learning progress incentives reached high levels almost 20% sooner.
In Freeway, the median performance for TRPO without intrinsic motivation was adequate, but the lower quartile range was quite poor—only 6 out of 10 runs ever found rewards. With the learning progress incentives, 8 out of 10 runs found rewards; with the surprisal incentive, all 10 did. Freeway is a game with very sparse rewards, where the agent effectively has to cross a long hallway before it can score a point, so naive exploration tends to exhibit random walk behavior and only rarely reaches the reward state. The intrinsic motivation helps the agent explore more purposefully.
2We compute this by comparing the marginal time cost incurred just by the bonus in each case: that is, if Tvime, Tsurprisal, and Tnobonus denote the times to 15 iterations, we obtain the speedup as
Tvime − Tnobonus Tsurprisal − Tnobonus .
In Venture, we obtain our strongest results in the Atari domain. Venture is extremely difficult because the agent has to navigate a large map to find very sparse rewards, and the agent can be killed by enemies interspersed throughout. We found that our intrinsic rewards were able to substantially improve performance over naive exploration in this challenging environment. Here, the best performance was again obtained by the surprisal incentive, which usually inspired the agent to reach scores greater than 500.
4.3 COMPARING INCENTIVES
Among our proposed incentives, we found that surprisal worked the best overall, achieving the most consistent performance across tasks. The learning progress-based incentives worked well on some domains, but generally not as well as surprisal. Interestingly, learning progress with k = 10 performed much worse on the continuous control tasks than with k = 1, but we observed virtually no difference in their performance on the Atari games; it is unclear why this should be the case.
Surprisal strongly outperformed the L2 error based incentive on the harder continuous control tasks, learning to solve them more quickly and without forgetting. Because we used fully-factored Gaussians for all of our dyanmics models, the surprisal had the form
− logPφ(s′|s, a) = n∑ i=1
( (s′i − µφ,i(s, a))2
2σ2φ,i(s, a) + log σφ,i(s, a)
) + k
2 log 2π,
which essentially includes the L2-squared error norm as a sub-expression. The relative difference in performance suggests that the variance terms confer additional useful information about the novelty of a state-action pair.
5 RELATED WORK
Substantial theoretical work has been done on optimal exploration in finite MDPs, resulting in algorithms such as E3 [10], R-max [3], and UCRL [9], which scale polynomially with MDP size. However, these works do not permit obvious generalizations to MDPs with continuous state and action spaces. C-PACE [18] provides a theoretical foundation for PAC-optimal exploration in MDPs with continuous state spaces, but it requires a metric on state spaces. Lopes et al. [11] investigated exploration driven by learning progress and proved theoretical guarantees for their approach in the finite MDP case, but they did not address the question of scaling their approach to continuous or high-dimensional MDPs. Also, although they formulated learning progress in the same way as (8), they formed intrinsic rewards differently. Conceptually and mathematically, our work is closest to prior work on curiosity and surprise [8, 19, 23, 24], although these works focus mainly on small finite MDPs.
Recently, several intrinsic motivation strategies that deal specifically with deep reinforcement learning have been proposed. Stadie et al. [22] learn deterministic dynamics models by minimizing Euclidean loss—whereas in our work, we learn stochastic dynamics with cross entropy loss—and use L2 prediction errors for intrinsic motivation. Houthooft et al. [7] train Bayesian neural networks to approximate posterior distributions over dynamics models given observed data, by maximizing a variational lower bound; they then use second-order approximations of the Bayesian surprise as intrinsic motivation. Bellemare et al. [2] derived pseudo-counts from CTS density models over states and used those to form intrinsic rewards, notably resulting in dramatic performance improvement on Montezuma’s Revenge, one of the hardest games in the Atari domain. Mohamed and Rezende [14] developed a scalable method of approximating empowerment, the mutual information between an agent’s actions and the future state of the environment, using variational methods. Oh et al. [16] estimated state visit frequency using Gaussian kernels to compare against a replay memory, and used these estimates for directed exploration.
6 CONCLUSIONS
In this work, we formulated surprise for intrinsic motivation as the KL-divergence of the true transition probabilities from learned model probabilities, and derived two approximations—surprisal and k-step
learning progress—that are scalable, computationally inexpensive, and suitable for application to high-dimensional and continuous control tasks. We showed that empirically, motivation by surprisal and 1-step learning progress resulted in efficient exploration on several hard deep reinforcement learning benchmarks. In particular, we found that surprisal was a robust and effective intrinsic motivator, outperforming other heuristics on a wide range of tasks, and competitive with the current state-of-the-art for intrinsic motivation in continuous control.
ACKNOWLEDGEMENTS
We thank Rein Houthooft for interesting discussions and for sharing data from the original VIME experiments. We also thank Rocky Duan, Carlos Florensa, Vicenc Rubies-Royo, Dexter Scobee, and Eric Mazumdar for insightful discussions and reviews of the preliminary manuscript.
This work is supported by TRUST (Team for Research in Ubiquitous Secure Technology) which receives support from NSF (award number CCF-0424422).
A SINGLE STEP SECOND-ORDER OPTIMIZATION
In our experiments, we approximately solve several optimization problems by using a single secondorder step with a line search. This section will describe the exact methodology, which was originally given by Schulman et al. [20].
We consider the optimization problem
p∗ = max θ L(θ) : D(θ) ≤ δ, (13)
where θ ∈ Rn, and for some θold we have D(θold) = 0,∇θD(θold) = 0, and∇2θD(θold) 0; also, ∀θ,D(θ) ≥ 0. We suppose that δ is small, so the optimal point will be close to θold. We also suppose that the curvature of the constraint is much greater than the curvature of the objective. As a result, we feel justified in approximating the objective to linear order and the constraint to quadratic order:
L(θ) ≈ L(θold) + gT (θ − θold) g . = ∇θL(θold)
D(θ) ≈ 1 2 (θ − θold)TA(θ − θold) A . = ∇2θD(θold).
We now consider the approximate optimization problem,
p∗ ≈ max θ gT (θ − θold) :
1 2 (θ − θold)TA(θ − θold) ≤ δ.
This optimization problem is convex as long as A 0, which is an assumption that we make. (If this assumption seems to be empirically invalid, then we repair the issue by using the substitution A→ A+ I , where I is the identity matrix, and > 0 is a small constant chosen so that we usually have A+ I 0.) This problem can be solved analytically by applying methods of duality, and its optimal point is
θ∗ = θold +
√ 2δ
gTA−1g A−1g. (14)
It is possible that the parameter update step given by (14) may not exactly solve the original optimization problem (13)—in fact, it may not even satisfy the constraint—so we perform a line search between θold and θ∗. Our update with the line search included is given by
θ = θold + s k
√ 2δ
gTA−1g A−1g, (15)
where s ∈ (0, 1) is a backtracking coefficient, and k is the smallest integer for which L(θ) ≥ L(θold) and D(θ) ≤ δ. We select k by checking each of k = 1, 2, ...,K, where K is the maximum number of backtracks. If there is no value of k in that range which satisfies the conditions, no update is performed.
Because the optimization problems we solve with this method tend to involve thousands of parameters, inverting A is prohibitively computationally expensive. Thus in the implementation of this algorithm that we use, the search direction x = A−1g is found by using the conjugate gradient method to solve Ax = g; this avoids the need to invert A.
When A and g are sample averages meant to stand in for expectations, we employ an additional trick to reduce the total number of computations necessary to solve Ax = g. The computation of A is more expensive than g, and so we use a smaller fraction of the population to estimate it quickly. Concretely, suppose that the original optimization problem’s objective is Ez∼P [L(θ, z)], and the constraint is Ez∼P [D(θ, z)] ≤ δ, where z is some random variable and P is its distribution; furthermore, suppose that we have a dataset of samples D = {zi}i=1,...,N drawn on P , and we form an approximate optimization problem using these samples. Defining g(z) .= ∇θL(θold, z) and A(z)
. = ∇2θD(θold, z), we would need to solve(
1 |D| ∑ z∈D A(z)
) x = 1
|D| ∑ z∈D g(z)
to obtain the search direction x. However, because the computation of the average Hessian is expensive, we sub-sample a batch b ⊂ D to form it. As long as b is a large enough set, then the approximation
1 |b| ∑ z∈b A(z) ≈ 1 |D| ∑ z∈D A(z) ≈ E z∼P [A(z)]
is good, and the search direction we obtain by solving( 1
|b| ∑ z∈b A(z)
) x = 1
|D| ∑ z∈D g(z)
is reasonable. The sub-sample ratio |b|/|D| is a hyperparameter of the algorithm.
B EXPERIMENT DETAILS
B.1 ENVIRONMENTS
The environments have the following state and action spaces: for the sparse MountainCar environment, S ⊆ R2, A ⊆ R1; for the sparse CartpoleSwingup task, S ⊆ R4, A ⊆ R1; for the sparse HalfCheetah
task, S ⊂ R20, A ⊆ R6; for the sparse Swimmer task, S ⊆ R13, A ⊆ R2; for the SwimmerGather task, S ⊆ R33, A ⊆ R2; for the Atari RAM domain, S ⊆ R128, A ⊆ {1, ..., 18}. For the sparse MountainCar task, the agent receives a reward of 1 only when it escapes the valley. For the sparse CartpoleSwingup task, the agent receives a reward of 1 only when cos(β) > 0.8, with β the pole angle. For the sparse HalfCheetah task, the agent receives a reward of 1 when xbody ≥ 5. For the sparse Swimmer task, the agent receives a reward of 1 + |vbody| when |xbody| ≥ 2. Atari RAM states, by default, take on values from 0 to 256 in integer intervals. We use a simple preprocessing step to map them onto values in (−1/3, 1/3). Let x denote the raw RAM state, and s the preprocessed RAM state:
s = 1
3 ( x 128 − 1 ) .
B.2 POLICY AND VALUE FUNCTIONS
For all continuous control tasks we used fully-factored Gaussian policies, where the means of the action distributions were the outputs of neural networks, and the variances were separate trainable parameters. For the sparse MountainCar and sparse CartpoleSwingup tasks, the policy mean networks had a single hidden layer of 32 units. For sparse HalfCheetah, sparse Swimmer, and SwimmerGather, the policy mean networks were of size (64, 32). For the Atari RAM tasks, we used categorical distributions over actions, produced by neural networks of size (64, 32).
The value functions used for the sparse MountainCar and sparse CartpoleSwingup tasks were neural networks with a single hidden layer of 32 units. For sparse HalfCheetah, sparse Swimmer, and SwimmerGather, time-varying linear value functions were used, as described by Duan et al. [5]. For the Atari RAM tasks, the value functions were neural networks of size (64, 32). The neural network value functions were learned via single second-order step optimization; the linear baselines were obtained by least-squares fit at each iteration.
All neural networks were feed-forward, fully-connected networks with tanh activation units.
B.3 TRPO HYPERPARAMETERS
For all tasks, the MDP discount factor γ was fixed to 0.995, and generalized advantage estimators (GAE) [21] were used, with the GAE λ parameter fixed to 0.95.
In the table below, we show several other TRPO hyperparameters. Batch size refers to steps of experience collected at each iteration. The sub-sample factor is for the second-order optimization step, as detailed in Appendix A.
B.4 EXPLORATION HYPERPARAMETERS
For all tasks, fully-factored Gaussian distributions were used as dynamics models, where the means and variances of the distributions were the outputs of neural networks.
For the sparse MountainCar and sparse CartpoleSwingup tasks, the means and variances were parametrized by single hidden layer neural networks with 32 units. For all other tasks, the means and variances were parametrized by neural networks with two hidden layers of size 64 units each. All networks used tanh activation functions.
For all continuous control tasks except SwimmerGather, we used replay memories of size 5, 000, 000, and a KL-divergence step size of κ = 0.001. For SwimmerGather, the replay memory was the same size, but we set the KL-divergence size to κ = 0.005. For the Atari RAM domain tasks, we used replay memories of size 1, 000, 000, and a KL-divergence step size of κ = 0.01.
For all tasks except SwimmerGather and Venture, 5000 time steps of experience were sampled from the replay memory at each iteration of dynamics model learning to take a stochastic step on (11), and a sub-sample factor of 1 was used in the second-order step optimizer. For SwimmerGather and Venture, 10, 000 time steps of experience were sampled at each iteration, and a sub-sample factor of 0.5 was used in the optimizer.
For all continuous control tasks, the L2 penalty coefficient was set to α = 1. For the Atari RAM tasks except for Venture, it was set to α = 0.01. For Venture, it was set to α = 0.1.
For all continuous control tasks except SwimmerGather, η0 = 0.001. For SwimmerGather, η0 = 0.0001. For the Atari RAM tasks, η0 = 0.005.
C ANALYSIS OF SPEEDUP COMPARED TO VIME
In this section, we provide an analysis of the time cost incurred by using VIME or our bonuses, and derive the potential magnitude of speedup attained by our bonuses versus VIME.
At each iteration, bonuses based on learned dynamics models incur two primary costs:
• the time cost of fitting the dynamics model, • and the time cost of computing the rewards.
We denote the dynamics fitting costs for VIME and our methods as T fitvime and T fit ours. Although the Bayesian neural network dynamics model for VIME is more complex than our model, the fit times can work out to be similar depending on the choice of fitting algorithm. In our speed test, the fit times were nearly equivalent, but used different algorithms.
For the time cost of computing rewards, we first introduce the following quantities:
• n: the number of CPU threads available, • tf : time for a forward pass through the model, • tb: time for a backward pass through the model, • N : batch size (number of samples per iteration), • k: the number of forward passes that can be performed simultaneously.
For our method, the time cost of computing rewards is
T rewours = Ntf kn .
For VIME, things are more complex. Each reward requires the computation of a gradient through its model, which necessitates a forward and a backward pass. Because gradient calculations cannot be efficiently parallelized by any deep learning toolkits currently available3, each (s, a, s′) tuple requires its own forward/backward pass. As a result, the time cost of computing rewards for VIME is:
T rewvime = N(tf + tb)
n .
The speedup of our method over VIME is therefore
T fitvime + N(tf+tb) n
T fitours + Ntf kn
.
In the limit of large N , and with the approximation that tf ≈ tb, the speedup is a factor of ∼ 2k. 3If this is not correct, please contact the authors so that we can issue a correction! But to the best of our knowledge, this is currently true, at time of publication.

This paper explores the topic of intrinsic motivation in the context of deep RL. It proposes a couple of variants derived from an auxiliary model-learning process (prediction error, surprise and learning progress), and shows that those can help exploration on a number of continuous control tasks (and the Atari game “venture”, maybe).
Novelty: none of the proposed types of intrinsic motivation are novel, and it’s arguable whether the application to deep RL is novel (see e.g. Kompella et al 2012).
Potential: the idea of seeking out states where a transition model is uncertain is sensible, but also limited -- I would encourage the authors to also discuss the limitations. For example in a game like Go the transition model is trivially learned, so this approach would revert to random exploration. So other forms of learning progress or surprise derived from the agent’s competence instead might be more promising in the long run? See also Srivastava et al 2012 for further thoughts.
Computation time: I find the paper’s claimed superiority over VIME to be overblown: the gain seems to stem almost exclusively from a faster initialization, but have very similar per-step cost? So given that VIME is also performing very competitively, what arguments can you advance for your own method(s)?


The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks. In order to substantiate our theoretical analysis, we perform targeted experiments to verify our assumptions, illustrate our claims, and quantify the phenomena. This paper is divided into three sections. The first section introduces the problem at hand. The second section is dedicated to studying and proving rigorously the problems including instability and saturation that arize when training generative adversarial networks. The third section examines a practical and theoretically grounded direction towards solving these problems, while introducing new tools to study them.
1 INTRODUCTION
Generative adversarial networks (GANs)(Goodfellow et al., 2014a) have achieved great success at generating realistic and sharp looking images. However, they are widely general methods, now starting to be applied to several other important problems, such as semisupervised learning, stabilizing sequence learning methods for speech and language, and 3D modelling. (Denton et al., 2015; Radford et al., 2015; Salimans et al., 2016; Lamb et al., 2016; Wu et al., 2016)
However, they still remain remarkably difficult to train, with most current papers dedicated to heuristically finding stable architectures. (Radford et al., 2015; Salimans et al., 2016)
Despite their success, there is little to no theory explaining the unstable behaviour of GAN training. Furthermore, approaches to attacking this problem still rely on heuristics that are extremely sensitive to modifications. This makes it extremely hard to experiment with new variants, or to use them in new domains, which limits their applicability drastically. This paper aims to change that, by providing a solid understanding of these issues, and creating principled research directions towards adressing them.
It is interesting to note that the architecture of the generator used by GANs doesn’t differ significantly from other approaches like variational autoencoders (Kingma & Welling, 2013). After all, at the core of it we first sample from a simple prior z ∼ p(z), and then output our final sample gθ(z), sometimes adding noise in the end. Always, gθ is a neural network parameterized by θ, and the main difference is how gθ is trained.
Traditional approaches to generative modeling relied on maximizing likelihood, or equivalently minimizing the Kullback-Leibler (KL) divergence between our unknown data distribution Pr and our generator’s distribution Pg (that depends of course on θ). If we assume that both distributions are continuous with densities Pr and Pg , then these methods try to minimize
KL(Pr‖Pg) = ∫ X Pr(x) log Pr(x) Pg(x) dx
This cost function has the good property that it has a unique minimum at Pg = Pr, and it doesn’t require knowledge of the unknown Pr(x) to optimize it (only samples). However, it is interesting to see how this divergence is not symetrical between Pr and Pg:
• If Pr(x) > Pg(x), then x is a point with higher probability of coming from the data than being a generated sample. This is the core of the phenomenon commonly described as ‘mode dropping’: when there are large regions with high values of Pr, but small or zero values in Pg . It is important to note that when Pr(x) > 0 but Pg(x) → 0, the integrand inside the KL grows quickly to infinity, meaning that this cost function assigns an extremely high cost to a generator’s distribution not covering parts of the data.
• If Pr(x) < Pg(x), then x has low probability of being a data point, but high probability of being generated by our model. This is the case when we see our generator outputting an image that doesn’t look real. In this case, when Pr(x)→ 0 and Pg(x) > 0, we see that the value inside the KL goes to 0, meaning that this cost function will pay extremely low cost for generating fake looking samples.
Clearly, if we would minimize KL(Pg‖Pr) instead, the weighting of these errors would be reversed, meaning that this cost function would pay a high cost for generating not plausibly looking pictures. Generative adversarial networks have been shown to optimize (in its original formulation), the Jensen-shannon divergence, a symmetric middle ground to this two cost functions
JSD(Pr‖Pg) = 1
2 KL(Pr‖PA) +
1 2 KL(Pg‖PA)
where PA is the ‘average’ distribution, with density Pr+Pg2 . An impressive experimental analysis of the similarities, uses and differences of these divergences in practice can be seen at Theis et al. (2016). It is indeed conjectured that the reason of GANs success at producing reallistically looking images is due to the switch from the traditional maximum likelihood approaches. (Theis et al., 2016; Huszar, 2015). However, the problem is far from closed.
Generative adversarial networks are formulated in two steps. We first train a discriminator D to maximize
L(D, gθ) = Ex∼Pr [logD(x)] + Ex∼Pg [log(1−D(x))] (1)
One can show easily that the optimal discriminator has the shape
D∗(x) = Pr(x)
Pr(x) + Pg(x) (2)
and that L(D∗, gθ) = 2JSD(Pr‖Pg)− 2 log 2, so minimizing equation (1) as a function of θ yields minimizing the Jensen-Shannon divergence when the discriminator is optimal. In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the cost function on θ better approximates the JSD), and then do gradient steps on θ, alternating these two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates to the generator get consistently worse. The original GAN paper argued that this issue arose from saturation, and switched to another similar cost function that doesn’t have this problem. However, even with this new cost function, updates tend to get worse and optimization gets massively unstable. Therefore, several questions arize:
• Why do updates get worse as the discriminator gets better? Both in the original and the new cost function.
• Why is GAN training massively unstable?
• Is the new cost function following a similar divergence to the JSD? If so, what are its properties?
• Is there a way to avoid some of these issues?
The fundamental contributions of this paper are the answer to all these questions, and perhaps more importantly, to introduce the tools to analyze them properly. We provide a new direction designed to avoid the instability issues in GANs, and examine in depth the theory behind it. Finally, we state a series of open questions and problems, that determine several new directions of research that begin with our methods.
2 SOURCES OF INSTABILITY
The theory tells us that the trained discriminator will have cost at most 2 log 2 − 2JSD(Pr‖Pg). However, in practice, if we just train D till convergence, its error will go to 0, as observed in Figure 1, pointing to the fact that the JSD between them is maxed out. The only way this can happen is if the distributions are not continuous1, or they have disjoint supports.
One possible cause for the distributions not to be continuous is if their supports lie on low dimensional manifolds. There is strong empirical and theoretical evidence to believe that Pr is indeed extremely concentrated on a low dimensional manifold (Narayanan & Mitter, 2010). As of Pg , we will prove soon that such is the case as well.
In the case of GANs, Pg is defined via sampling from a simple prior z ∼ p(z), and then applying a function g : Z → X , so the support of Pg has to be contained in g(Z). If the dimensionality of Z is less than the dimension of X (as is typically the case), then it’s imposible for Pg to be continuous. This is because in most cases g(Z) will be contained in a union of low dimensional manifolds, and therefore have measure 0 in X . Note that while intuitive, this is highly nontrivial, since having an ndimensional parameterization does absolutely not imply that the image will lie on an n-dimensional manifold. In fact, there are many easy counterexamples, such as Peano curves, lemniscates, and many more. In order to show this for our case, we rely heavily on g being a neural network, since we are able to leverage that g is made by composing very well behaved functions. We now state this properly in the following Lemma: Lemma 1. Let g : Z → X be a function composed by affine transformations and pointwise nonlinearities, which can either be rectifiers, leaky rectifiers, or smooth strictly increasing functions (such as the sigmoid, tanh, softplus, etc). Then, g(Z) is contained in a countable union of manifolds of dimension at most dimZ . Therefore, if the dimension of Z is less than the one of X , g(Z) will be a set of measure 0 in X .
Proof. See Appendix A.
Driven by this, this section shows that if the supports of Pr and Pg are disjoint or lie in low dimensional manifolds, there is always a perfect discriminator between them, and we explain exactly how and why this leads to an unreliable training of the generator.
2.1 THE PERFECT DISCRIMINATION THEOREMS
For simplicity, and to introduce the methods, we will first explain the case where Pr and Pg have disjoint supports. We say that a discriminator D : X → [0, 1] has accuracy 1 if it takes the value 1 on a set that contains the support of Pr and value 0 on a set that contains the support of Pg . Namely, Pr[D(x) = 1] = 1 and Pg[D(x) = 0] = 1. Theorem 2.1. If two distributions Pr and Pg have support contained on two disjoint compact subsetsM and P respectively, then there is a smooth optimal discrimator D∗ : X → [0, 1] that has accuracy 1 and ∇xD∗(x) = 0 for all x ∈M∪P .
Proof. The discriminator is trained to maximize Ex∼Pr [logD(x)] + Ex∼Pg [log(1−D(x))] SinceM and P are compact and disjoint, 0 < δ = d(P,M) the distance between both sets. We now define
M̂ = {x : d(x,M) ≤ δ/3} P̂ = {x : d(x, P ) ≤ δ/3}
By definition of δ we have that P̂ and M̂ are clearly disjoint compact sets. Therefore, by Urysohn’s smooth lemma there exists a smooth function D∗ : X → [0, 1] such that D∗|M̂ ≡ 1 and D∗|P̂ ≡ 0. Since logD∗(x) = 0 for all x in the support of Pr and log(1−D∗(x)) = 0 for all x in the support of Pg , the discriminator is completely optimal and has accuracy 1. Furthermore, let x be inM∪P . If we assume that x ∈ M, there is an open ball B = B(x, δ/3) on which D∗|B is constant. This shows that∇xD∗(x) ≡ 0. Taking x ∈ P and working analogously we finish the proof.
1By continuous we will actually refer to an absolutely continuous random variable (i.e. one that has a density), as it typically done. For further clarification see Appendix B.
In the next theorem, we take away the disjoint assumption, to make it general to the case of two different manifolds. However, if the two manifolds match perfectly on a big part of the space, no discriminator could separate them. Intuitively, the chances of two low dimensional manifolds having this property is rather dim: for two curves to match in space in a specific segment, they couldn’t be perturbed in any arbitrarilly small way and still satisfy this property. To do this, we will define the notion of two manifolds perfectly aligning, and show that this property never holds with probability 1 under any arbitrarilly small perturbations.
Definition 2.1. We first need to recall the definition of transversallity. LetM andP be two boundary free regular submanifolds of F , which in our cases will simply be F = Rd. Let x ∈ M ∩ P be an intersection point of the two manifolds. We say that M and P intersect transversally in x if TxM+ TxP = TxF , where TxM means the tangent space ofM around x.
Definition 2.2. We say that two manifolds without boundaryM and P perfectly align if there is an x ∈M∩P such thatM and P don’t intersect transversally in x. We shall note the boundary and interior of a manifoldM by ∂M and Int M respectively. We say that two manifoldsM and P (with or without boundary) perfectly align if any of the boundary free manifold pairs (IntM, Int P), (IntM, ∂P), (∂M, Int P) or (∂M, ∂P) perfectly align.
The interesting thing is that we can safely assume in practice that any two manifolds never perfectly align. This can be done since an arbitrarilly small random perturbation on two manifolds will lead them to intersect transversally or don’t intersect at all. This is precisely stated and proven in Lemma 2.
As stated by Lemma 3, if two manifolds don’t perfectly align, their intersection L = M∩ P will be a finite union of manifolds with dimensions strictly lower than both the dimension ofM and the one of P . Lemma 2. LetM andP be two regular submanifolds of Rd that don’t have full dimension. Let η, η′ be arbitrary independent continuous random variables. We therefore define the perturbed manifolds as M̃ =M+ η and P̃ = P + η′. Then
Pη,η′(M̃ does not perfectly align with P̃) = 1
Proof. See Appendix A.
Lemma 3. Let M and P be two regular submanifolds of Rd that don’t perfectly align and don’t have full dimension. Let L =M∩P . IfM and P don’t have boundary, then L is also a manifold, and has strictly lower dimension than both the one ofM and the one of P . If they have boundary, L is a union of at most 4 strictly lower dimensional manifolds. In both cases, L has measure 0 in bothM and P .
Proof. See Appendix A.
We now state our perfect discrimination result for the case of two manifolds. Theorem 2.2. Let Pr and Pg be two distributions that have support contained in two closed manifoldsM and P that don’t perfectly align and don’t have full dimension. We further assume that Pr and Pg are continuous in their respective manifolds, meaning that if there is a set A with measure 0 in M, then Pr(A) = 0 (and analogously for Pg). Then, there exists an optimal discriminator D∗ : X → [0, 1] that has accuracy 1 and for almost any x inM or P , D∗ is smooth in a neighbourhood of x and ∇xD∗(x) = 0.
Proof. By Lemma 3 we know that L = M∩P is strictly lower dimensional than bothM and P , and has measure 0 on both of them. By continuity, Pr(L) = 0 and Pg(L) = 0. Note that this implies the support of Pr is contained inM\L and the support of Pg is contained in P \ L. Let x ∈ M \ L. Therefore, x ∈ Pc (the complement of P) which is an open set, so there exists a ball of radius x such that B(x, x) ∩ P = ∅. This way, we define
M̂ = ⋃
x∈M\L
B(x, x/3)
We define P̂ analogously. Note that by construction these are both open sets on Rd. SinceM\L ⊆ M̂, and P \ L ⊆ P̂ , the support of Pr and Pg is contained in M̂ and P̂ respectively. As well by construction, M̂ ∩ P̂ = ∅. Let us defineD∗(x) = 1 for all x ∈ M̂, and 0 elsewhere (clearly including P̂ . Since logD∗(x) = 0 for all x in the support of Pr and log(1−D∗(x)) = 0 for all x in the support of Pg , the discriminator is completely optimal and has accuracy 1. Furthermore, let x ∈ M̂. Since M̂ is an open set and D∗ is constant on M̂, then ∇xD∗|M̂ ≡ 0. Analogously, ∇xD∗|P̂ ≡ 0. Therefore, the set of points where D∗ is non-smooth or has non-zero gradient insideM∪P is contained in L, which has null-measure in both manifolds, therefore concluding the theorem.
These two theorems tell us that there are perfect discriminators which are smooth and constant almost everywhere inM and P . The fact that the discriminator is constant in both manifolds points to the fact that we won’t really be able to learn anything by backproping through it, as we shall see in the next subsection. To conclude this general statement, we state the following theorem on the divergences of Pr and Pg , whose proof is trivial and left as an exercise to the reader. Theorem 2.3. Let Pr and Pg be two distributions whose support lies in two manifoldsM and P that don’t have full dimension and don’t perfectly align. We further assume that Pr and Pg are continuous in their respective manifolds. Then,
JSD(Pr‖Pg) = log 2 KL(Pr‖Pg) = +∞ KL(Pg‖Pr) = +∞
Note that these divergences will be maxed out even if the two manifolds lie arbitrarilly close to each other. The samples of our generator might look impressively good, yet both KL divergences will be infinity. Therefore, Theorem 2.3 points us to the fact that attempting to use divergences out of the box to test similarities between the distributions we typically consider might be a terrible idea. Needless to say, if these divergencies are always maxed out attempting to minimize them by gradient descent isn’t really possible. We would like to have a perhaps softer measure, that incorporates a notion of distance between the points in the manifolds. We will come back to this topic later in section 3, where we explain an alternative metric and provide bounds on it that we are able to analyze and optimize.
2.2 THE CONSEQUENCES, AND THE PROBLEMS OF EACH COST FUNCTION
Theorems 2.1 and 2.2 showed one very important fact. If the two distributions we care about have supports that are disjoint or lie on low dimensional manifolds, the optimal discriminator will be perfect and its gradient will be zero almost everywhere.
2.2.1 THE ORIGINAL COST FUNCTION
We will now explore what happens when we pass gradients to the generator through a discriminator. One crucial difference with the typical analysis done so far is that we will develop the theory for an approximation to the optimal discriminator, instead of working with the (unknown) true discriminator. We will prove that as the approximaton gets better, either we see vanishing gradients or the massively unstable behaviour we see in practice, depending on which cost function we use.
In what follows, we denote by ‖D‖ the norm
‖D‖ = sup x∈X |D(x)|+ ‖∇xD(x)‖2
The use of this norm is to make the proofs simpler, but could have been done in another Sobolev norm ‖ · ‖1,p for p < ∞ covered by the universal approximation theorem in the sense that we can guarantee a neural network approximation in this norm (Hornik, 1991).
Theorem 2.4 (Vanishing gradients on the generator). Let gθ : Z → X be a differentiable function that induces a distribution Pg . Let Pr be the real data distribution. Let D be a differentiable discriminator. If the conditions of Theorems 2.1 or 2.2 are satisfied, ‖D − D∗‖ < , and Ez∼p(z) [ ‖Jθgθ(z)‖22 ] ≤M2, 2 then
‖∇θEz∼p(z)[log(1−D(gθ(z)))]‖2 < M
1−
Proof. In both proofs of Theorems 2.1 and 2.2 we showed that D∗ is locally 0 on the support of Pg . Then, using Jensen’s inequality and the chain rule on this support we have
‖∇θEz∼p(z)[log(1−D(gθ(z)))]‖22 ≤ Ez∼p(z) [ ‖∇θD(gθ(z))‖22 |1−D(gθ(z))|2 ] ≤ Ez∼p(z) [ ‖∇xD(gθ(z))‖22‖Jθgθ(z)‖22
|1−D(gθ(z))|2 ] < Ez∼p(z) [ (‖∇xD∗(gθ(z))‖2 + )2 ‖Jθgθ(z)‖22
(|1−D∗(gθ(z))| − )2
]
= Ez∼p(z)
[ 2‖Jθgθ(z)‖22
(1− )2
]
≤M2 2
(1− )2
Taking square root of everything we get
‖∇θEz∼p(z)[log(1−D(gθ(z)))]‖2 < M
1− finishing the proof
Corollary 2.1. Under the same assumptions of Theorem 2.4
lim ‖D−D∗‖→0
∇θEz∼p(z)[log(1−D(gθ(z)))] = 0
2Since M can depend on θ, this condition is trivially verified for a uniform prior and a neural network. The case of a Gaussian prior requires more work because we need to bound the growth on z, but is also true for current architectures.
This shows that as our discriminator gets better, the gradient of the generator vanishes. For completeness, this was experimentally verified in Figure 2. The fact that this happens is terrible, since the fact that the generator’s cost function being close to the Jensen Shannon divergence depends on the quality of this approximation. This points us to a fundamental: either our updates to the discriminator will be inacurate, or they will vanish. This makes it difficult to train using this cost function, or leave up to the user to decide the precise amount of training dedicated to the discriminator, which can make GAN training extremely hard.
2.2.2 THE − logD ALTERNATIVE
To avoid gradients vanishing when the discriminator is very confident, people have chosen to use a different gradient step for the generator.
∆θ = ∇θEz∼p(z) [− logD(gθ(z))]
We now state and prove for the first time which cost function is being optimized by this gradient step. Later, we prove that while this gradient doesn’t necessarily suffer from vanishing gradients, it does cause massively unstable updates (that have been widely experienced in practice) under the prescence of a noisy approximation to the optimal discriminator. Theorem 2.5. Let Pr and Pgθ be two continuous distributions, with densities Pr and Pgθ respectively. Let D∗ = PrPgθ0 +Pr be the optimal discriminator, fixed for a value θ03. Therefore,
Ez∼p(z) [−∇θ logD∗(gθ(z))|θ=θ0 ] = ∇θ [KL(Pgθ‖Pr)− 2JSD(Pgθ‖Pr)] |θ=θ0 (3)
Before diving into the proof, let’s look at equation (3) for a second. This is the inverted KL minus two JSD. First of all, the JSDs are in the opposite sign, which means they are pushing for the distributions to be different, which seems like a fault in the update. Second, the KL appearing in the equation is KL(Pg‖Pr), not the one equivalent to maximum likelihood. As we know, this KL assigns an extremely high cost to generating fake looking samples, and an extremely low cost on mode dropping; and the JSD is symetrical so it shouldn’t alter this behaviour. This explains what we see in practice, that GANs (when stabilized) create good looking samples, and justifies what is commonly conjectured, that GANs suffer from an extensive amount of mode dropping.
3This is important since when backpropagating to the generator, the discriminator is assumed fixed
Proof. We already know by Goodfellow et al. (2014a) that
Ez∼p(z) [∇θ log(1−D∗(gθ(z)))|θ=θ0 ] = ∇θ2JSD(Pgθ‖Pr)|θ=θ0 Furthermore, as remarked by Huszar (2016),
KL(Pgθ‖Pr) = Ex∼Pgθ
[ log Pgθ (x)
Pr(x) ] = Ex∼Pgθ [ log Pgθ0 (x)
Pr(x)
] − Ex∼Pgθ [ log Pgθ (x)
Pgθ0 (x)
]
= −Ex∼Pgθ
[ log D∗(x)
1−D∗(x)
] −KL(Pgθ‖Pgθ0 )
= −Ez∼p(z) [ log D∗(gθ(z))
1−D∗(gθ(z))
] −KL(Pgθ‖Pgθ0 )
Taking derivatives in θ at θ0 we get ∇θKL(Pgθ‖Pr)|θ=θ0 = −∇θEz∼p(z) [ log D∗(gθ(z))
1−D∗(gθ(z))
] |θ=θ0 −∇θKL(Pgθ‖Pgθ0 )|θ=θ0
= Ez∼p(z) [ −∇θ log D∗(gθ(z))
1−D∗(gθ(z))
] |θ=θ0
Substracting this last equation with the result for the JSD, we obtain our desired result.
We now turn to our result regarding the instability of a noisy version of the true distriminator.
Theorem 2.6 (Instability of generator gradient updates). Let gθ : Z → X be a differentiable function that induces a distribution Pg . Let Pr be the real data distribution, with either conditions of Theorems 2.1 or 2.2 satisfied. Let D be a discriminator such that D∗ − D = is a centered Gaussian process indexed by x and independent for every x (popularly known as white noise) and ∇xD∗ −∇xD = r another independent centered Gaussian process indexed by x and independent for every x. Then, each coordinate of
Ez∼p(z) [−∇θ logD(gθ(z))]
is a centered Cauchy distribution with infinite expectation and variance.4
Proof. Let us remember again that in this case D is locally constant equal to 0 on the support of Pg . We denote r(z), (z) the random variables r(gθ(z)), (gθ(z)). By the chain rule and the definition of r, , we get
Ez∼p(z) [−∇θ logD(gθ(z))] = Ez∼p(z) [ −Jθgθ(z)∇xD(gθ(z))
D(gθ(z)) ] = Ez∼p(z) [ −Jθgθ(z)r(z)
(z)
]
Since r(z) is a centered Gaussian distribution, multiplying by a matrix doesn’t change this fact. Furthermore, when we divide by (z), a centered Gaussian independent from the numerator, we get a centered Cauchy random variable on every coordinate. Averaging over z the different independent Cauchy random variables again yields a centered Cauchy distribution. 5
4Note that the theorem holds regardless of the variance of r and . As the approximation gets better, this error looks more and more as centered random noise due to the finite precision.
5A note on technicality: when is defined as such, the remaining process is not measurable in x, so we can’t take the expectation in z trivially. This is commonly bypassed, and can be formally worked out by stating the expectation as the result of a stochastic differential equation.
Note that even if we ignore the fact that the updates have infinite variance, we still arrive to the fact that the distribution of the updates is centered, meaning that if we bound the updates the expected update will be 0, providing no feedback to the gradient.
Since the assumption that the noises of D and ∇D are decorrelated is albeit too strong, we show in Figure 3 how the norm of the gradient grows drastically as we train the discriminator closer to optimality, at any stage in training of a well stabilized DCGAN except when it has already converged. In all cases, using this updates lead to a notorious decrease in sample quality. The noise in the curves also shows that the variance of the gradients is increasing, which is known to delve into slower convergence and more unstable behaviour in the optimization (Bottou et al., 2016).
3 TOWARDS SOFTER METRICS AND DISTRIBUTIONS
An important question now is how to fix the instability and vanishing gradients issues. Something we can do to break the assumptions of these theorems is add continuous noise to the inputs of the discriminator, therefore smoothening the distribution of the probability mass. Theorem 3.1. If X has distribution PX with support on M and is an aboslutely continuous distribution with density P , then PX+ is absolutely continuous with density
PX+ (x) = Ey∼PX [P (x− y)]
= ∫ M P (x− y) dPX(y)
Proof. See Appendix A.
Corollary 3.1. • If ∼ N (0, σ2I) then
PX+ (x) = 1
Z ∫ M e− ‖y−x‖2 2σ2 dPX(y)
• If ∼ N (0,Σ) then PX+ (x) = 1
Z Ey∼PX
[ e− 1 2‖y−x‖ 2 Σ−1 ]
• If P (x) ∝ 1‖x‖d+1 then
PX+ (x) = 1
Z Ey∼PX
[ 1
‖x− y‖d+1
]
This theorem therefore tells us that the density PX+ (x) is inversely proportional to the average distance to points in the support of PX , weighted by the probability of these points. In the case of the support of PX being a manifold, we will have the weighted average of the distance to the points along the manifold. How we choose the distribution of the noise will impact the notion of distance we are choosing. In our corolary, for example, we can see the effect of changing the covariance matrix by altering the norm inside the exponential. Different noises with different types of decays can therefore be used.
Now, the optimal discriminator between Pg+ and Pr+ is
D∗(x) = Pr+ (x)
Pr+ (x) + Pg+ (x)
and we want to calculate what the gradient passed to the generator is.
Theorem 3.2. Let Pr and Pg be two distributions with support on M and P respectively, with ∼ N (0, σ2I). Then, the gradient passed to the generator has the form
Ez∼p(z) [∇θ log(1−D∗(gθ(z)))] (4) = Ez∼p(z) [ a(z) ∫ M P (gθ(z)− y)∇θ‖gθ(z)− y‖2 dPr(y)
− b(z) ∫ P P (gθ(z)− y)∇θ‖gθ(z)− y‖2 dPg(y) ] where a(z) and b(z) are positive functions. Furthermore, b > a if and only if Pr+ > Pg+ , and b < a if and only if Pr+ < Pg+ .
This theorem proves that we will drive our samples gθ(z) towards points along the data manifold, weighted by their probability and the distance from our samples. Furthermore, the second term drives our points away from high probability samples, again, weighted by the sample manifold and distance to these samples. This is similar in spirit to contrastive divergence, where we lower the free energy of our samples and increase the free energy of data points. The importance of this term is seen more clearly when we have samples that have higher probability of coming from Pg than from Pr. In this case, we will have b > a and the second term will have the strength to lower the probability of this too likely samples. Finally, if there’s an area around x that has the same probability to come from Pg than Pr, the gradient contributions between the two terms will cancel, therefore stabilizing the gradient when Pr is similar to Pg .
There is one important problem with taking gradient steps exactly of the form (4), which is that in that case, D will disregards errors that lie exactly in g(Z), since this is a set of measure 0. However, g will be optimizing its cost only on that space. This will make the discriminator extremely susceptible to adversarial examples, and will render low cost on the generator without high cost on the discriminator, and lousy meaningless samples. This is easilly seen when we realize the term inside the expectation of equation (4) will be a positive scalar times ∇x log(1 − D∗(x))∇θgθ(z), which is the directional derivative towards the exact adversarial term of Goodfellow et al. (2014b). Because of this, it is important to backprop through noisy samples in the generator as well. This will yield a crucial benefit: the generator’s backprop term will be through samples on a set of positive measure that the discriminator will care about. Formalizing this notion, the actual gradient through the generator will now be proportional to ∇θJSD(Pr+ ‖Pg+ ), which will make the two noisy distributions match. As we anneal the noise, this will make Pr and Pg match as well. For completeness, we show the smooth gradient we get in this case. The proof is identical to the one of Theorem 3.2, so we leave it to the reader.
Corollary 3.2. Let , ′ ∼ N (0, σ2I) and g̃θ(z) = gθ(z) + ′, then
Ez∼p(z), ′ [∇θ log(1−D∗(g̃θ(z)))] (5) = Ez∼p(z), ′ [ a(z) ∫ M P (g̃θ(z)− y)∇θ‖g̃θ(z)− y‖2 dPr(y)
− b(z) ∫ P P (g̃θ(z)− y)∇θ‖g̃θ(z)− y‖2 dPg(y) ] = 2∇θJSD(Pr+ ‖Pg+ )
In the same as with Theorem 3.2, a and b will have the same properties. The main difference is that we will be moving all our noisy samples towards the data manifold, which can be thought of as moving a small neighbourhood of samples towards it. This will protect the discriminator against measure 0 adversarial examples.
Proof of theorem 3.2. Since the discriminator is assumed fixed when backproping to the generator, the only thing that depends on θ is gθ(z) for every z. By taking derivatives on our cost function
Ez∼p(z) [∇θ log(1−D∗(gθ(z)))] = Ez∼p(z) [ ∇θ log Pg+ (gθ(z))
Pr+ (gθ(z)) + Pg+ (gθ(z)) ] = Ez∼p(z) [∇θ logPg+ (gθ(z))−∇θ log (Pg+ (gθ(z)) + Pr+ (gθ(z)))]
= Ez∼p(z) [ ∇θPg+ (gθ(z)) Pg+ (gθ(z)) − ∇θPg+ (gθ(z)) +∇θPr+ (gθ(z)) Pg+ (gθ(z)) + Pr+ (gθ(z)) ] = Ez∼p(z) [ 1
Pg+ (gθ(z)) + Pr+ (gθ(z)) ∇θ[−Pr+ (gθ(z))]−
1
Pg+ (gθ(z)) + Pr+ (gθ(z))
Pr+ (gθ(z)) Pg+ (gθ(z)) ∇θ[−Pg+ (gθ(z))] ] Let the density of be 1Z e − ‖x‖ 2 2σ2 . We now define
a(z) = 1 2σ2 1
Pg+ (gθ(z)) + Pr+ (gθ(z))
b(z) = 1 2σ2 1
Pg+ (gθ(z)) + Pr+ (gθ(z))
Pr+ (gθ(z))
Pg+ (gθ(z))
Trivially, a and b are positive functions. Since b = aPr+ Pg+ , we know that b > a if and only if Pr+ > Pg+ , and b < a if and only if Pr+ < Pg+ as we wanted. Continuing the proof, we know
Ez∼p(z) [∇θ log(1−D∗(gθ(z)))] = Ez∼p(z) [ 2σ2a(z)∇θ[−Pr+ (gθ(z))]− 2σ2b(z)∇θ[−Pg+ (gθ(z)) ] = Ez∼p(z) [ 2σ2a(z) ∫ M −∇θ 1 Z e −‖gθ(z)−y‖ 2 2 2σ2 dPr(y)− 2σ2b(z) ∫ P −∇θ 1 Z e −‖gθ(z)−y‖ 2 2 2σ2 dPg(y) ]
= Ez∼p(z) [ a(z) ∫ M 1 Z e −‖gθ(z)−y‖ 2 2 2σ2 ∇θ‖gθ(z)− y‖2 dPr(y)
− b(z) ∫ P 1 Z e −‖gθ(z)−y‖ 2 2 2σ2 ∇θ‖gθ(z)− y‖2 dPg(y) ]
= Ez∼p(z) [ a(z) ∫ M P (gθ(z)− y)∇θ‖gθ(z)− y‖2 dPr(y)
− b(z) ∫ P P (gθ(z)− y)∇θ‖gθ(z)− y‖2 dPg(y) ] Finishing the proof.
An interesting observation is that if we have two distributions Pr and Pg with support on manifolds that are close, the noise terms will make the noisy distributions Pr+ and Pg+ almost overlap, and the JSD between them will be small. This is in drastic contrast to the noiseless variants Pr and Pg , where all the divergences are maxed out, regardless of the closeness of the manifolds. We could argue to use the JSD of the noisy variants to measure a similarity between the original distributions, but this would depend on the amount of noise, and is not an intrinsic measure of Pr and Pg . Luckilly, there are alternatives.
Definition 3.1. We recall the definition of the Wasserstein metric W (P,Q) for P and Q two distributions over X . Namely,
W (P,Q) = inf γ∈Γ ∫ X×X ‖x− y‖2dγ(x, y)
where Γ is the set of all possible joints on X × X that have marginals P and Q.
The Wasserstein distance also goes by other names, most commonly the transportation metric and the earth mover’s distance. This last name is most explicative: it’s the minimum cost of transporting the whole probability mass of P from its support to match the probability mass ofQ onQ’s support. This identification of transporting points from P toQ is done via the coupling γ. We refer the reader to Villani (2009) for an in-depth explanation of these ideas. It is easy to see now that the Wasserstein metric incorporates the notion of distance (as also seen inside the integral) between the elements in the support of P and the ones in the support of Q, and that as the supports of P and Q get closer and closer, the metric will go to 0, inducing as well a notion of distance between manifolds.
Intuitively, as we decrease the noise, PX and PX+ become more similar. However, it is easy to see again that JSD(PX‖PX+ ) is maxed out, regardless of the amount of noise. The following Lemma shows that this is not the case for the Wasserstein metric, and that it goes to 0 smoothly when we decrease the variance of the noise.
Lemma 4. If is a random vector with mean 0, then we have
W (PX ,PX+ ) ≤ V 1 2
where V = E[‖ ‖22] is the variance of .
Proof. Let x ∼ PX , and y = x+ with independent from x. We call γ the joint of (x, y), which clearly has marginals PX and PX+ . Therefore,
W (PX ,PX+ ) ≤ ∫ ‖x− y‖2dγ(x, y)
= Ex∼PXEy∼x+ [‖x− y‖2] = Ex∼PXEy∼x+ [‖ ‖2] = Ex∼PXE [‖ ‖2] = E [‖ ‖2]
≤ E [‖ ‖22] 1 2 = V 1 2
where the last inequality was due to Jensen.
We now turn to one of our main results. We are interested in studying the distance between Pr and Pg without any noise, even when their supports lie on different manifolds, since (for example) the closer these manifolds are, the closer to actual points on the data manifold the samples will be. Furthermore, we eventually want a way to evaluate generative models, regardless of whether they are continuous (as in a VAE) or not (as in a GAN), a problem that has for now been completely unsolved. The next theorem relates the Wasserstein distance of Pr and Pg , without any noise or modification, to the divergence of Pr+ and Pg+ , and the variance of the noise. Since Pr+ and Pg+ are continuous distributions, this divergence is a sensible estimate, which can even be attempted to minimize, since a discriminator trained on those distributions will approximate the JSD between them, and provide smooth gradients as per Corolary 3.2.
Theorem 3.3. Let Pr and Pg be any two distributions, and be a random vector with mean 0 and variance V . If Pr+ and Pg+ have support contained on a ball of diameter C, then 6
W (Pr,Pg) ≤ 2V 1 2 + 2C √ JSD(Pr+ ‖Pg+ ) (6)
Proof.
W (Pr,Pg) ≤W (Pr,Pr+ ) +W (Pr+ ,Pg+ ) +W (Pg+ ,Pg)
≤ 2V 12 +W (Pr+ ,Pg+ )
≤ 2V 12 + Cδ(Pr+ ,Pg+ )
≤ 2V 12 + C (δ(Pr+ ,Pm) + δ(Pg+ ,Pm))
≤ 2V 12 + C
(√ 1
2 KL(Pr+ ‖Pm) +
√ 1
2 KL(Pg+ ‖Pm) ) ≤ 2V 12 + 2C √ JSD(Pr+ ‖Pg+ )
We first used the Lemma 4 to bound everything but the middle term as a function of V . After that, we followed by the fact that W (P,Q) ≤ Cδ(P,Q) wih δ the total variation, which is a popular Lemma arizing from the Kantorovich-Rubinstein duality. After that, we used the triangular inequality on δ and Pm the mixture distribution between Pg+ and Pr+ . Finally, we used Pinsker’s inequality and later the fact that each individual KL is only one of the non-negative sumands of the JSD.
Theorem 3.3 points us to an interesting idea. The two terms in equation (6) can be controlled. The first term can be decreased by annealing the noise, and the second term can be minimized by a GAN when the discriminator is trained on the noisy inputs, since it will be approximating the JSD between the two continuous distributions. One great advantage of this is that we no longer have to worry about training schedules. Because of the noise, we can train the discriminator till optimality without any problems and get smooth interpretable gradients by Corollary 3.2. All this while still minimizing the distance between Pr and Pg , the two noiseless distributions we in the end care about.
ACKNOWLEDGMENTS
The first author would like to especially thank Luis Scoccola for help with the proof of Lemma 1.
The authors would also like to thank Ishmael Belghazi, Yoshua Bengio, Gerry Che, Soumith Chintala, Caglar Gulcehre, Daniel Jiwoong Im, Alex Lamb, Luis Scoccola, Pablo Sprechmann, Arthur Szlam, Jake Zhao for insightful comments and advice.
A PROOFS OF THINGS
Proof of Lemma 1. We first consider the case where the nonlinearities are rectifiers or leaky rectifiers of the form σ(x) = 1[x < 0]c1x + 1[x ≥ 0]c2x for some c1, c2 ∈ R. In this case, g(z) = DnWn . . .D1W1z, where Wi are affine transformations and Di are some diagonal matrices dependent on z that have diagonal entries c1 or c2. If we consider D to be the (finite) set of all diagonal matrices with diagonal entries c1 or c2, then g(Z) ⊆ ⋃ Di∈DDnWn . . .D1W1Z , which is a finite union of linear manifolds.
The proof for the second case is technical and slightly more involved. When σ is a pointwise smooth strictly increasing nonlinearity, then applying it vectorwise it’s a diffeomorphism to its image. Therefore, it sends a countable union of manifolds of dimension d to a countable union of manifolds of dimension d. If we can prove the same thing for affine transformations we will be finished, since g(Z) is just a composition of these applied to a dimZ dimensional manifold. Of course, it suffices to prove that an affine transformation sends a manifold to a countable union of manifolds without increasing dimension, since a countable union of countable unions is still a countable union. Furthermore, we only need to show this for linear transformations, since applying a bias term is a diffeomorphism.
Let W ∈ Rn×m be a matrix. Note that by the singular value decomposition, W = UΣV, where Σ is a square diagonal matrix with diagonal positive entries and U,V are compositions of changes of basis, inclusions (meaning adding 0s to new coordinates) and projections to a subset of the coordinates. Multiplying by Σ and applying a change of basis are diffeomorphisms, and adding 0s to new coordinates is a manifold embedding, so we only need to prove our statement for projections onto a subset of the coordinates. Let π : Rn+k → Rn, where π(x1, . . . , xn+k) = (x1, . . . , xn) be our projection andM⊆ Rn+k our d-dimensional manifold. If n ≤ d, we are done since the image of π is contained in all Rn, a manifold with at most dimension d. We now turn to the case where n > d. Let πi(x) = xi be the projection onto the i-th coordinate. If x is a critical point of π, since the coordinates of π are independent, then x has to be a critical point of a πi. By a consequence of the Morse Lemma, the critical points of πi are isolated, and therefore so are the ones of π, meaning that there is at most a countable number of them. Since π maps the non-critical points onto a d dimensional manifold (because it acts as an embedding) and the countable number of critical points into a countable number of points (or 0 dimensional manifolds), the proof is finished.
Proof of Lemma 2. For now we assume thatM and P are without boundary. If dimM+ dimP ≥ d it is known that under arbitrarilly small perturbations defined as the ones in the statement of this Lemma, the two dimensions will intersect only transversally with probability 1 by the General Position Lemma. If dimM+ dimP < d, we will show that with probability 1,M+ η and P + η′ will not intersect, thereby getting our desired result. Let us then assume dimM+ dimP < d. Note that M̂ ∩ P̂ 6= ∅ if and only if there are x ∈ M, y ∈ P such that x + η = y + η′, or equivalently x − y = η′ − η. Therefore, M̂ and P̂ intersect if and only if η′ − η ∈ M − P . Since η, η′ are independent continuous random variables, the difference is also continuous. IfM−P has measure 0 in Rd then P(η′ − η ∈ M− P) = 0, concluding the proof. We will therefore show thatM−P has measure 0. Let f :M×P → Rd be f(x, y) = x− y. If m and p are the dimensions ofM and P , then f is a smooth function between an m + p-dimensional manifold and a d dimensional one. Clearly, the image of f isM−P . Therefore,
M−P = f({z ∈M×P|rank(dzf) < m+ p}) ∪ f({z ∈M×P|rank(dzf) = m+ p})
The first set is the image of the critical points, namely the critical values. By Sard’s Lemma, this set has measure 0. Let’s call A = {z ∈ M × P|rank(dzf) = m + p}. Let z be an element of A. By the inverse function theorem, there is a neighbourhood Uz ⊆ M × P of z such that f |Uz is an embedding. Since every manifold has a countable topological basis, we can cover A by countable sets Uzn , where n ∈ N. We will just note them by Un. Since f |Un is an embedding, f(Un) is an m+ p-dimensional manifold, and since m+ p < d, this set has measure 0 in Rd. Now, f(A) = ⋃ n∈N f(Un), which therefore has measure 0 in Rd, finishing the proof of the boundary free case.
Now we consider the case whereM and P are manifolds with boundary. By a simple union bound, Pη,η′(M̃ perfectly aligns with P̃) ≤ Pη,η′(Int M̃ perfectly aligns with Int P̃)
+ Pη,η′(Int M̃ perfectly aligns with ∂P̃) + Pη,η′(∂M̃ perfectly aligns with Int P̃) + Pη,η′(∂M̃ perfectly aligns with ∂P̃) = 0
where the last equality arizes when combining the facts that ˜IntM = η + IntM = Int (η +M) = Int M̃ (and analogously for the boundary and P), that the boundary and interiors ofM and P are boundary free regular submanifolds of Rd without full dimension, and then applying the boundary free case of the proof.
Proof of Lemma 3. Let m = dimM and p = dimP . We again consider first the case whereM and P are manifolds without boundary. If m+p < d, then L = ∅ so the statement is obviously true. If m+ p ≥ d, thenM and P intersect transversally. This implies that L is a manifold of dimension m + p − d < m, p. Since L is a submanifold of bothM and P that has lower dimension, it has measure 0 on both of them.
We now tackle the case whereM and P have boundaries. Let us remember thatM = IntM∪∂M and the union is disjoint (and analogously for P). By using elementary properties of sets, we can trivially see that
L =M∩P = (IntM∩ Int P) ∪ (IntM∩ ∂P) ∪ (∂M∩ Int P) ∪ (∂M∩ ∂P)
where the unions are disjoint. This is the disjoint union of 4 strictly lower dimensional manifolds, by using the first part of the proof. Since each one of these intersections has measure 0 on either the interior or boundary ofM (again, by the first part of the proof), and interior and boundary are contained in M, each one of the four intersections has measure 0 in M. Analogously, they have measure 0 in P , and by a simple union bound we see that L has measure 0 inM and P finishing the remaining case of the proof.
Proof of Theorem 3.1. We first need to show that PX+ is absolutely continuous. Let A be a Borel set with Lebesgue measure 0. Then, by the fact that and X are independent, we know by Fubini
PX+ (A) = ∫ Rd P (A− x) dPX(x)
= ∫ Rd 0 dPX(x) = 0
Where we used the fact that if A has Lebesgue measure zero, then so does A − x and since P is absolutely continuous, P (A− x) = 0. Now we calculate the density of PX+ . Again, by using the independence of X and , for any Borel set B we know
PX+ (B) = ∫ Rd P (B − y) dPX(y)
= Ey∼PX [P (B − y)] = Ey∼Px [∫
B−y P (x)dx ] = Ey∼Px [∫ B P (x− y)dx ]
= ∫ B Ey∼Px [P (x− y)] dx
Therefore, PX+ (B) = ∫ B PX+ (x)dx for our proposed PX+ and all Borel sets B. By the uniqueness of the Radon-Nikodym theorem, this implies the proposed PX+ is the density of PX+ . The equivalence of the formula changing the expectation for ∫ M PX is trivial by the definition of expectation and the fact that the support of PX lies onM.
B FURTHER CLARIFICATIONS
In this appendix we further explain some of the terms and ideas mentioned in the paper, which due to space constrains, and to keep the flow of the paper, couldn’t be extremely developed in the main text. Some of these have to do with notation, others with technical elements of the proofs. On the latter case, we try to convey more intuition than we previously could. We present these clarifications in a very informal fashion in the following item list.
• There are two different but very related properties a random variable can have. A random variable X is said to be continuous if P (X = x) = 0 for all single points x ∈ X . Note that a random variable concentrated on a low dimensional manifold such as a plane can have this property. However, an absolutely continuous random variable has the following property: if a set A has Lebesgue measure 0, then P (X ∈ A) = 0. Since points have measure 0 with the Lebesgue measure, absolute continuity implies continuity. A random variable that’s supported on a low dimensional manifold therefore will not be absolutely continuous: let M a low dimensional manifold be the support of X . Since a low dimensional manifold has 0 Lebesgue measure, this would imply P (X ∈ M) = 0, which is an absurd since M was the support of X . The property of X being absolutely continuous can be shown to be equivalent to X having a density: the existence of a function f : X → R such that P (X ∈ A) = ∫ A f(x) dx (this is a consequence of the Radon-Nikodym theorem).
The annoying part is that in everyday paper writing when we talk about continuous random variables, we omit the ”absolutely” word to keep the text concise and actually talk about absolutely continuous random variables (ones that have a density), this is done through almost all sciences and throughout mathematics as well, annoying as it is. However we made the clarification in here since it’s relevant to our paper not to mistake the two terms. • The notation Pr[D(x) = 1] = 1 is the abbreviation of Pr[{x ∈ X : D(x) = 1}] = 1 for a
measure Pr. Another way of expressing this more formally is Pr[D−1(1)] = 1. • In the proof of Theorem 2.1, the distance between sets d(A,B) is defined as the usual
distance between sets in a metric space
d(A,B) = inf x∈A,y∈B d(x, y)
where d(x, y) is the distance between points (in our case the Euclidean distance). • Note that not everything that’s outside of the support of Pr has to be a generated image.
Generated images are only things that lie in the support of Pg , and there are things that don’t need to be in the support of either Pr or Pg (these could be places where 0 < D < 1 for example). This is because the discriminator is not trained to discriminate Pr from all things that are not Pr, but to distinguish Pr from Pg . Points that don’t lie in the support of Pr or Pg are not important to the performance of the discriminator (as is easily evidenced in its cost). Why we define accuracy 1 as is done in the text is to avoid the identification of a single ‘tight’ support, since this typically leads to problems (if I take a measure 0 set from any support it still is the support of the distribution). In the end, what we aim for is:
– We want D(x) = 1 with probability 1 when x ∼ Pr. – We want D(x) = 0 with probability 1 when x ∼ Pg . – Whatever happens elsewhere is irrelevant (as it is also reflected by the cost of the
discriminator) • We say that a discriminator D∗ is optimal for gθ (or its corresponding Pg) if for all mea-
surable functions D : X → [0, 1] we have
L(D∗, gθ) ≥ L(D, gθ)
for L defined as in equation (1).

SUMMARY
This paper addresses important questions about the difficulties in training generative adversarial networks. It discusses consequences of using an asymmetric divergence function and sources of instability in training GANs. Then it proposes an alternative using a smoothening approach.
PROS
Theory, good questions, nice answers.
Makes an interesting use of concepts form analysis and differential topology.
Proposes avenues to avoid instability in GANs.
CONS
A bit too long, technical. Some parts and consequences still need to be further developed (which is perfectly fine for future work).
MINOR COMMENTS
- Section 2.1 Maybe shorten this section a bit. E.g., move all proofs to the appendix.
- Section 3 provides a nice, intuitive, simple solution.
- On page 2 second bullet. This also means that P_g is smaller than the data distribution in some other x, which in turn will make the KL divergence non zero.
- On page 2, ``for not generating plausibly looking pictures'' should be ``for generating not plausibly looking pictures''.
- Lemma 1 would also hold in more generality.
- Theorem 2.1 seems to be basic analysis. (In other words, a reference could spare the proof).
- In Theorem 2.4, it would be good to remind the reader about p(z).
- Lemma 2 seems to be basic analysis. (In other words, a reference could spare the proof).
Specify the domain of the random variables.
- relly - > rely
- Theorem 2.2 the closed manifolds have boundary or not? (already in the questions)
- Corollary 2.1, ``assumptions of Theorem 1.3''. I could not find Theorem 1.3.
- Theorem 2.5 ``Therefore'' -> `Then'?
- Theorem 2.6 ``Is a... '' -> `is a' ?
- The number of the theorems is confusing.