f -GANs in an Information Geometric Nutshell
Nowozin et al showed last year how to extend the GAN principle to all fdivergences. The approach is elegant but falls short of a full description of the supervised game, and says little about the key player, the generator: for example, what does the generator actually converge to if solving the GAN game means convergence in some space of parameters? How does that provide hints on the generator's design and compare to the flourishing but almost exclusively experimental literature on the subject? In this paper, we unveil a broad class of distributions for which such convergence happens -namely, deformed exponential families, a wide superset of exponential families -. We show that current deep architectures are able to factorize a very large number of such densities using an especially compact design, hence displaying the power of deep architectures and their concinnity in the f -GAN game. This result holds given a sufficient condition on activation functions -which turns out to be satisfied by popular choices. The key to our results is a variational generalization of an old theorem that relates the KL divergence between regular exponential families and divergences between their natural parameters. We complete this picture with additional results and experimental insights on how these results may be used to ground further improvements of GAN architectures, via (i) a principled design of the activation functions in the generator and (ii) an explicit integration of proper composite losses' link function in the discriminator.
The authors identify several interesting GAN-related questions, such as to what extend solving the GAN problem implies convergence in parameter space, what the generator is actually fitting when convergence occurs and (perhaps most relevant from a network architectural point of view) how to choose the output activation function of the discriminator so as to ensure proper compositeness of the loss function.
The authors set out to address these questions within the (information theoretic) framework of deformed exponential distributions, from which they derive among other things the following theoretical results:
They present a variational generalization (amenable to f-GAN formulation) of a known theorem that relates an f-divergence between distributions to a corresponding Bregman divergence between the parameters of such distributions. As such, this theorem provides an interesting connection between the information-theoretic view point of measuring dissimilarity between probability distributions and the information-geometric perspective of measuring dissimilarity between the corresponding parameters. They show that under a reversibility assumption, deep generative networks factor as so called escorts of deformed exponential distributions. Their theoretical investigation furthermore suggests that a careful choice of the hidden activation functions of the generator as well as a proper selection of the output activation function of the discriminator could potentially help to further improve GANs. They also briefly mention an alternative interpretation of the GAN game in the context of expected utility theory.
Overall, I find that the authors present some interesting theoretical results, however, I do have several concerns regarding the practical relevance and usefulness of their results in the present form.
Questions & concerns:
To begin with, many of their theorems and resulting insights hold for the specific case that the data and model distributions P and Q are members of the deformed exponential family. Can the authors justify this assumption, i.e. elaborate on whether it is met in practice and explain whether similar results (such as Theorems 4 & 6) could be derived without this assumption?
One of the author’s main contributions, the information-geometric f-GAN identity in Eq.(7), relates the well-known variational f-divergence formulation over distributions to an information-geometric optimization problem over parameters. Can the authors explain what we gain from this parameter-based point of view? Is it possible to implement GANs in terms of this parameter-based optimization and what would be the benefits? I would really have liked to see experimental results comparing the two approaches to optimization of GANs. Somewhat surprising, the authors don’t seem to make use of the right hand side of this identity, other than to assert that solving the GAN game implies convergence in parameter space (provided the residual J(Q) is small). And why is this implication not obvious? Can the authors give a realistic scenario where convergence in the variational f-divergence formulation over distributions does not imply convergence in parameter space?
As an interesting practical implication of their theoretical investigation, the authors show that the choice of the output activation function g_f (see section 2.4 in Ref.[34]) matters in order for the GAN loss to have the desirable proper compositeness property. I found this result particularly interesting and would certainly have liked to see how their theoretically derived g_f (as the composition of f′ and the link function) compares experimentally against the heuristic choice provided in Ref. [34].
Consequences for deep learning and experimental results: Under the assumption that the generator network is reversible, the authors show that there exists an activation function v such that the generator’s hidden layers factor exactly as escorts for the deformed exponential family. In Table 2 (left), the authors compare several generator architectures against different choices of activation functions (that may or may not comply with this escort factorization). The plot for the GAN MLP, for instance, indicates that the “theoretically superior” μ-ReLU performs actually worse than the baseline ReLU (limiting case μ → 1). From their analysis, I would however expect the invertible μ-ReLU (satisfying reversibility assumption) to perform better than the non-invertible baseline (not satisfying reversibility assumption). Can the authors explain why the μ-ReLU performs worse? Can they comment on whether the DCGAN architecture satisfies the reversibility assumption and how these results compare against the Wasserstein GAN which is not based on f-divergences (and for which I therefore do not expect their theoretical analysis to hold)? Finally, as far as I can tell, their results on whether one activation (Table 2 center) or link function (Table 2 right) performs better than the others are all within error bars. I don’t think this provides sufficient support for their theoretical results to be relevant in practice.
Minor concerns:
Many of their results are derived within the framework of deformed exponential densities. As the general ML community might not yet be familiar with this, I would welcome it if the authors could provide more intuition as to what a deformation (or signature) and an escort is for instance.
In Theorem 8, the authors derive several upper bounds for J(Q). How do these bounds compare against each other and what are the implications for which activation functions to use? They also mention that this upper bound on J(Q) is decreasing with the normalization parameter of the escort Z. Can they elaborate a bit more on why this is a good thing and how we can leverage this?
In light of these comments, I believe the insights gained from the theoretical analysis are not substantial enough from the point of view of a GAN practitioner.