summaries: The paper develops an information-theoretic training scheme for Variational Auto-Encoders (VAEs). This scheme is tailored for addressing the well-known disentanglement problem of VAEs where an over-capacity encoder sometimes manages to both maximize data fit and shrink the KL divergence between the approximate posterior and prior to zero. Consequently, the latent representations of the observations become independent, making them unusable for any downstream task. Overview: This paper describes the Variational InfoMax AutoEncoder (VIMAE), which is based on the learning principle of the Capacity Constrained InfoMax. The core idea behind VIMAE is that the encoding information is not bounded while network capacity is. The issue that VIMAE can handle, and where VAE fails, is that representations are not informative of input data, due to the information bottleneck idea that VAE is built upon. The authors describe InfoVAE and β-VAE, which both attempt to solve this problem. The theory behind VIMAE is then described and tested against VAE and β-VAE, in their abilities to evaluate the entropy of Z, in reconstruction and generative performance, and in robustness to noise and generalization.  I went over this work multiple times and had a really hard time judging the novelty of this work. The paper seems to be a summary of existing work reinterpreting variational autoencoding objectives from an information theoretic standpoint. In particular, the paper seems to follow the same analysis as in Wasserstein Autoencoders (Tolstikhin et al., 2017) and InfoVAE (Zhao et al., 2017).  It is unfair to say that the objectives were "derived independently" since these works are from a couple of years ago. 
abstractText: The Variational AutoEncoder (VAE) can learn simultaneously a representation and a generative model, but often, in order to generate sharper samples, a useless representation is learnt. Starting from the observation that a meaningful representation is an informative one, in this work we analyse the VAE from an information theoretic perspective. We define the concept of capacity for the variational network, and we associate the two tasks of VAE: generation and representation quality, to two information properties of the network: information of the generative model and capacity of the network. Then we suggest that an optimal generative model is the one optimising the Capacity-Constrained InfoMax (CCIM), a theoretical objective learning the maximal informative generative model while maintaining bounded the network capacity. The theoretical assumptions are confirmed by the computational experiments, where between the different families of VAE, the one optimising a variational lower bound of the CCIM generates sharper samples and learns a clustered and robust representation. We call this the Variational InfoMax AutoEncoder (VIMAE).
authors.name: INFOMAX AUTOENCODER
references.authors: Alessandro Achille Tom Eccles Loic Matthey Chris Burgess Nicholas Watters Alexander Lerchner Irina Higgins
references.title: Life-long disentangled representation learning with cross-domain latent homologies
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Alexander A Alemi Ian Fischer Joshua V Dillon Kevin Murphy
references.title: Deep variational information bottleneck
references.venue: arXiv preprint arXiv:1612.00410,
references.authors: Alexander A Alemi Ben Poole Ian Fischer Joshua V Dillon Rif A Saurous Kevin Murphy
references.title: Fixing a broken elbo
references.venue: arXiv preprint arXiv:1711.00464,
references.authors: Pierre Baldi Kurt Hornik
references.title: Neural networks and principal component analysis: Learning from examples without local minima
references.venue: Neural networks,
references.authors: Anthony J Bell Terrence J Sejnowski
references.title: The independent components of natural scenes are edge filters
references.venue: Vision research,
references.authors: Yoshua Bengio Aaron Courville Pascal Vincent
references.title: Representation learning: A review and new perspectives
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Christopher P Burgess Irina Higgins Arka Pal Loic Matthey Nick Watters Guillaume Desjardins Alexander Lerchner
references.title: Understanding disentangling in β-vae
references.venue: arXiv preprint arXiv:1804.03599,
references.authors: Xi Chen Diederik P Kingma Tim Salimans Yan Duan Prafulla Dhariwal John Schulman Ilya Sutskever Pieter Abbeel
references.title: Variational lossy autoencoder
references.venue: arXiv preprint arXiv:1611.02731,
references.authors: Laurent Dinh Jascha Sohl-Dickstein Samy Bengio
references.title: Density estimation using real nvp
references.venue: arXiv preprint arXiv:1605.08803,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in neural information processing systems,
references.authors: Aditya Grover Stefano Ermon
references.title: Uncertainty autoencoders: Learning compressed representations via variational information maximization
references.venue: arXiv preprint arXiv:1812.10539,
references.authors: Irina Higgins Loic Matthey Arka Pal Christopher Burgess Xavier Glorot Matthew Botvinick Shakir Mohamed Alexander Lerchner
references.title: beta-vae: Learning basic visual concepts with a constrained variational framework
references.venue: In International Conference on Learning Representations,
references.authors: Geoffrey E Hinton Simon Osindero Yee-Whye Teh
references.title: A fast learning algorithm for deep belief nets
references.venue: Neural computation,
references.authors: Aapo Hyvärinen Jarmo Hurri Patrick O Hoyer
references.title: Natural image statistics: A probabilistic approach to early computational vision., volume 39
references.venue: Springer Science & Business Media,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: arXiv preprint arXiv:1502.03167,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114,
references.authors: Durk P Kingma Shakir Mohamed Danilo Jimenez Rezende Max Welling
references.title: Semi-supervised learning with deep generative models
references.venue: In Advances in neural information processing systems,
references.authors: Durk P Kingma Tim Salimans Rafal Jozefowicz Xi Chen Ilya Sutskever Max Welling
references.title: Improved variational inference with inverse autoregressive flow
references.venue: In Advances in neural information processing systems,
references.authors: Ralph Linsker
references.title: An application of the principle of maximum information preservation to linear systems
references.venue: In Advances in neural information processing systems,
references.authors: Ziwei Liu Ping Luo Xiaogang Wang Xiaoou Tang
references.title: Deep learning face attributes in the wild
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Chris J Maddison John Lawson George Tucker Nicolas Heess Mohammad Norouzi Andriy Mnih Arnaud Doucet Yee Teh
references.title: Filtering variational objectives
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Emile Mathieu Tom Rainforth Siddharth Narayanaswamy Yee Whye Teh
references.title: Disentangling disentanglement
references.venue: arXiv preprint arXiv:1812.02833,
references.authors: Alec Radford Luke Metz Soumith Chintala
references.title: Unsupervised representation learning with deep convolutional generative adversarial networks
references.venue: arXiv preprint arXiv:1511.06434,
references.authors: Jason Ramapuram Magda Gregorova Alexandros Kalousis
references.title: Lifelong generative modeling
references.venue: arXiv preprint arXiv:1705.09847,
references.authors: Danilo Jimenez Rezende Shakir Mohamed
references.title: Variational inference with normalizing flows
references.venue: arXiv preprint arXiv:1505.05770,
references.authors: Danilo Jimenez Rezende Shakir Mohamed Daan Wierstra
references.title: Stochastic backpropagation and approximate inference in deep generative models
references.venue: arXiv preprint arXiv:1401.4082,
references.authors: Salah Rifai Grégoire Mesnil Pascal Vincent Xavier Muller Yoshua Bengio Yann Dauphin Xavier Glorot
references.title: Higher order contractive auto-encoder
references.venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,
references.authors: Andrew Michael Saxe Yamini Bansal Joel Dapello Madhu Advani Artemy Kolchinsky Brendan Daniel Tracey David Daniel Cox
references.title: On the information bottleneck theory of deep
references.authors: Naftali Tishby Fernando C Pereira William Bialek
references.title: The information bottleneck method
references.venue: arXiv preprint physics/0004057,
references.authors: Pascal Vincent Hugo Larochelle Yoshua Bengio Pierre-Antoine Manzagol
references.title: Extracting and composing robust features with denoising autoencoders
references.venue: In Proceedings of the 25th international conference on Machine learning,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap koray kavukcuoglu Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: Advances in Neural Information Processing Systems
references.authors: Shengjia Zhao Jiaming Song Stefano Ermon
references.title: Infovae: Information maximizing variational autoencoders
references.venue: arXiv preprint arXiv:1706.02262,
sections.heading: 1 INTRODUCTION
sections.text: A common assumption in machine learning is that any visible data x ∈ X is completely described by some generative factor o, living in a smaller hidden space O, i.e. x = g(o) with g a (possibly stochastic) generative function. The aim of unsupervised representation learning research is to find a representation z of the generative factor o living in a known space Z describing, as well as o, the visible data x. This is particularly relevant because the learnt small representation z is task agnostic and, in principle, can be used as input for networks performing different tasks, leading to faster and more robust learning (generalisation property), (Rifai et al., 2011).
  Many models fφ : X → Z trying to learn such representations have been proposed (Dinh et al., 2016; Hinton et al., 2006; Maddison et al., 2017; Radford et al., 2015), but recently in order to solve this problem it was proposed to consider a dual problem: define a priori z and find a generator map gθ, such that for any z, gθ(z) is an element of X . In particular, two families of probabilistic generative models have become dominant: Variational AutoEncoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) and Generative Adversarial Network (GAN) (Goodfellow et al., 2014). The common idea of the two approaches is that a good generator pθ(x|z) is the one able to generate the data that is as close as possible to the visible one, i.e. that with respect a certain metric D, the distance between the marginal pθ(x) = Ep(z)[pθ(x|z)] and the visible distribution pD(x) is minimal. In this manuscript we restrict our attention to the VAE model, since by its architecture, it is the only one where the learnt representation, possibly from different datasets, can be used as input for networks performing different tasks, (Achille et al., 2018; Ramapuram et al., 2017). Although VAE, by its training robustness and general good generative performance is the most popular model for representation learning, in particular cases it suffers from the uninformative representation issue: the representation is entangled and the generative model tends to be independent of z, i.e. pθ(x|z) ≈ pθ(x). As highlighted in the next section such behaviour is intrinsic in the variational loss, the Evidence Lower BOund (ELBO), encouraging a less informative representation.
  Following the direction suggested in (Alemi et al., 2017; Zhao et al., 2017) we describe the VAE from an information theoretic perspective. Such description lead us to two observations: the WAE (Tolstikhin et al., 2017) and InfoVAE (Zhao et al., 2017) models are actually maximising a lower bound of the mutual information associated to a generator pθ(x|z) belonging in a certain family
  P , the Variational InfoMax (VIM); and given that the capacity of the network is a function of the entropy of the prior p(z), the VIM is the variational expression of the Capacity-Constrained InfoMax (CCIM). Following the analogies between the CCIM and the Information Bottleneck (IB), the theoretical principle associated with the ELBO, we deduce that the representation quality is associated to the capacity term.
  The theoretical arguments are confirmed by the performed experiments where we observe that, differently from what was argued in previous works (Higgins et al., 2017; Burgess et al., 2018), it is possible to train a model that is able to learn good (able to generalise) representations while maintaining optimal generative performance. The main contributions of the paper are summarised in the following points:
  • derivation of a variational lower bound for the maximal mutual information of a generaive model belonging in a certain family, see equation 7;
  • definition and bounds estimation for the network capacity for a variational autoencoder, see equation 9;
  • association of the two main properties of VAE, generation quality and good representation, to two different information concepts, respectively Mutual Information and network capacity;
  • proposal of a new learning principle for unsupervised models: the Capacity-Constrained InfoMax, see equation 10, that allows both to learn a good representation while maintaining optimal generative performance.
  The work is divided as follows: in the second section we describe briefly the VAE and its variants; in the third and fourth sections we describe the variational infomax method and related work. We conclude the paper with the experimental results and the final observations.
sections.heading: 2 BACKGROUND
sections.text: The aim of this section is to describe VAE, understand principal issues of the ELBO objective and describe the three most relevant approaches to overcome such issues.
sections.heading: 2.1 NOTATION AND PRELIMINARY DEFINITIONS
sections.text: We use calligraphic letters (i.e. X ) for sets, capital letters (i.e. X) for random variables, and lower case letters (i.e. x) for their samples. With abuse of notation we denote both the probability and the corresponding density with the lower case letters (i.e. p(x)).
  KL divergence Given two random distributions p(x) and q(x), the Kullback-Leibler (KL) divergence
  DKL(p(x)||q(x)) = ∫ log (p(y) q(y) ) p(y)dy (1)
  is an (intuitive) measure of the distance between the distributions p and q.
  Mutual Information and Capacity Given a channel Z → X with X and Z random variables, jointly distributed according to p(x, z) and with marginals p(x) and p(z). The mutual information
  I(X,Z) = DKL(p(x, z)||p(x)p(z)),
  is a measure of the reduction of uncertainty in X due to the knowledge of Z, and the capacity
  C(X,Z) = sup p(z)∈P I(X,Z)
  is the maximal information that can be shared for a fixed generator p(x|z).
sections.heading: 2.2 VARIATIONAL AUTOENCODER
sections.text: From now on let us assume that the unknown distribution of the data p(x) coincides with the empirical one pD(x), and that the distribution of the latent representation p(z) is known. In this context the VAE is a model solving the following optimisation problem: find the generative model pθ(x, z) ∈ Pθ, specified by the parameters θ of the associated neural network, maximising the ELBO objective
  ELBOθ,φ = Ep(x)[−DKL(qφ(z|x)||p(z)) + Eq(z|x)[log p(x|z)]], (2)
  a lower bound of the unfeasible-to-compute marginal likelihood Ep(x)[log pθ(x)]. The ELBO objective is optimised by a regularised autoencoder, with encoder and decoder parametetrising, respectively, the inference and generative distributions, qφ(z|x) and pθ(x|z), with φ ∈ Φ, θ ∈ Θ and regulariser defined by the rate term DKL(qφ(z|x)||p(z)), measuring the excess number of bits required to encode samples from the encoder using the optimal code designed for p(z).
sections.heading: 2.3 UNINFORMATIVE REPRESENTATION ISSUE
sections.text: As underlined in the introduction, the main issue of VAE is that the representations are not really informative of the input data and in the worst case, it is learned a Z-independent generative model pθ(x|z) = pθ(x). Such issues are intrinsic in the ELBO objective, equation 2, that can reach the optimum when DKL(qφ(z|x)||p(z)) = 0 (Zhao et al., 2017). The latter case means that the representation is completely uninformative, indeed the rate term, which can be rewritten as
  DKL(qφ(z|x)||p(z)) = Iq(X,Z) +DKL(qφ(z)||p(z)),
  is a penalty on the encoding information, and is zero when Iq(X,Z) = 0, with qφ(z|x) = qφ(z) = p(z), i.e. when qφ does not encode any information about the input x.
  We now describe the three most relevant models that try to overcome the uninformative representation issue.
  InfoVAE In (Zhao et al., 2017) the InfoVAE family of models was proposed, a generalisation of the VAE model optimising the objective
  −αIq(X,Z)− λDKL(qφ(z)||p(z)) + Ep(x)[Eq(z|x)[log p(x|z)]],
  with α and λ two real positive hyper-parameters.
  The main advantage of this definition is that it is possible to consider separately the two components of the rate term. In particular, in (Zhao et al., 2017) it was observed that by eliminating the information penalty (α = 0), the generative performance of the model improves and the representation results are more informative.
  β-VAE In (Higgins et al., 2017), starting from the observation that the optimal case is rare, but most of the learned features by VAE are not disentangled, it is proposed an opposite approach: put a high penalty to the rate term, in order to constrain the model to learn the most informative property of the data, and then have a disentangled representation of the data. The β-VAE family is a particular case of InfoVAE where α = λ 1. This idea, that at first sight looks counter-intuitive, is based on the observation that by the additive property of the KL-divergence
  DKL(qφ(z|x)||p(z)) = dim(Z)∑ i=1 DKL(qφ(zi|x)||p(zi)) (3)
  pushing the penalty associated with the rate is equivalent to penalising the informativeness of most features, leaving few features containing the relevant information. Starting from a bits-back coding argument, a similar conclusion was derived in (Chen et al., 2016).
  Minimal rate bound In order to avoid the null rate issue, and maintain semantic informative representation, in (Alemi et al., 2017) was suggested to optimise the following variant of ELBO:
  Ep(x)[Eq(z|x)[log p(x|z)]]− β|R0 −DKL(qφ(z|x)||p(z))|, (4)
  where R0 > 0, defines both the lower bound of the rate and the upper bound of the encoding information Iq(X,Z). If, as we will see below, the objective in equation 4 is a generalisation of both InfoVAE (R0 = maxθ Iθ(X,Z)) and β-VAE (R0 = 0), the correct R0 is unknown and its optimisitation is not an easy task.
sections.heading: 3 THE MODEL
sections.heading: 3.1 THE VARIATIONAL INFOMAX
sections.text: Assuming the distribution associated to the two random variables p(x) and p(z) is known, the InfoMax objective is defined as: find the joint distribution pθ(x, z) ∈ Pθ := {pθ(x, z) : Ep(z)[pθ(x|z)] = p(x), Ep(x)[pθ(z|x)] = p(z)} maximising the mutual information Iθ(X,Z) = DKL(pθ(x, z)||p(x)p(z)), i.e. find θ∗ ∈ Θ s.t. Iθ∗ ≥ Iθ for any θ ∈ Θ. Since the definition via KL divergence is computationally intractable, it is necessary to re-write the mutual information as Iθ(X,Z) = hθ(X)− hθ(X|Z), (5) where hθ(X) = −Epθ(x)[log pθ(x)] is the entropy of X , and hθ(X|Z) = −Epθ(x,z)[log pθ(x|z)] is the conditional entropy hθ(X|Z). Since pθ(x, z) ∈ Pθ the entropy hθ(X) = h(X) is constant, and in order to maximise the mutual information it is sufficient to minimise the conditional entropy.
  Excluding some special cases (Bell & Sejnowski, 1997), minimising the conditional entropy is unfeasible, so it is necessary to consider an associated variational problem: for any qφ(z|x) such that qφ(z) = p(z) and φ ∈ Φ = Θ, learn the generative model pθ(x|z) minimising the reconstruction accuracy term Ep(x)[Eqφ(z|x)[log(pθ(x|z))]]. Indeed, the following variational objective:
  Iθ,φ(X,Z) = h(X) + Ep(x)Eq(z|x)[log pθ(x|z)] s.t. qφ(z) = p(z) (6)
  is a lower bound of Iθ∗(X,Z) and is maximal when q(z|x) = pθ(z|x) = pθ∗(z|x), (see the Appendix).
  Unfortunately, the formulation in equation 6 is still unfeasible to compute, because it requires that qφ(z) = p(z), but by the butterfly architecture of the autoencoder, qφ(z) tends to be uniformly distributed on the space Z . For this reason, the model is trained maximising the following relaxed form: V IMθ,φ = Ep(x)Eq(z|x)[log pθ(x|z)]− λD(qφ(z)||p(z)), (7) where it is introduced a term D(qφ(z)||p(z)) encouraging the empirical distribution qφ(z) to be close, according to the metric D, to p(z). In the following we assume D = DKL, and in order to avoid any confusion the variational autoencoder trained maximising equation 7 will be dubbed VIMAE.
  Encoding channel In VAE we observed that an uninformative representation was caused by the non-informativeness of the encoding map qφ(z|x). Since from equation 7 it is not clear how qφ(z|x) behaves, we consider an equivalent representation, (see the Appendix):
  V IMθ,φ = −DKL(p(x)||pθ(x))− (λ− 1)DKL(qφ(z)||p(z)) + Iθ,φ(X,Z). (8)
  From equation 8 we see that the infomax objective, equation 7, can be read as a composition of three sub-objectives: find a generative model pθ(x|z), with marginal resembling the visible distribution p(x) (first term); maximise the (unbounded) variational mutual information (third term); and learn an inferred distribution qφ(z, x) close to the generative model pθ(x, z). Then the optimum is obtained by qφ(x, z) = pθ(x, z) such that Iθ(X,Z) is maximal, confirming the validity of the approximation made above.
sections.heading: 3.2 CHANNEL CAPACITY
sections.text: In a channel with variational mutual information Iθ,φ as defined in equation 6 the (variational) capacity Cθ,φ(X,Z), is defined as
  Cθ,φ(X,Z) = sup θ,φ,p(z)∈P Iθ,φ(X,Z). (9)
  If P is the space of all distributions on Z , the capacity of the network coincides with the variational mutual information of the model trained minimising only the reconstruction loss. In the latter case, as observed above, it is not guaranteed to learn the generator with pθ(z) = p(z). This is because, given two equally informative generative models pθ(x|z) and pθ′(x|z), with encoder respectively qφ(z|x) and qφ′(z|x), with h(qφ(z)) < h(qφ′(z)), then Iθ,φ < Iθ′,φ′ . From such observation we deduce that bounding the entropy of the representation Z is the way to bound the capacity without penalising the encoding information, and that the penalty introduced in equation 7 is actually a bound to the capacity itself. So, the VIM objective can be defined as a variational approximation of the Capacity-Constrained InfoMax:
  max Z∈Z
  I(X,Z)− λC(X,Z), (10)
  which given a set of equally informative generators, learns the one having the minimal capacity. The idea of the CCIM is similar to the idea of the IB, from which was derived the β-VAE: constrain the capacity of the network in order to learn only the relevant features of the input data. The difference with the IB,
  max Z∈Z
  I(X,Z)− λIq(X,Z) (11)
  lies in the second term: the network capacity instead of the encoding information. This choice allows the network to learn a good representation (small capacity) while maintaining good generative performance (high mutual information). Indeed, as shown in equation 8, the generative performance is associated to the informativeness of qφ(z|x) and pθ(x|z). In order to test the assumption that it is sufficient to bound the entropy of Z, instead of the encoding mutual information, to learn a good representation, in the experiments (see below) we consider the cases Z is Normal (VIMAE-n) or Logistic (VIMAE-l) distributed. We choose to compare the popular Normal distribution with the Logistic one for two reasons: the Logistic has less entropy than a Gaussian distribution and because it is a common assumption in natural science to suppose that the hidden factors of the visible data are logistically distributed (Hyvärinen et al., 2009).
  Entropy vs Rate According to what written above the choice of the distribution p(z) is a way to bound the maximal information Iθ of the generative model as well as R0 in equation 4, indeed, if we choose R0 = C(X,Z) the theoretical solutions of equation 4 and equation 7 coincide. Although the two approaches are similar, both defines a target Iθ for the generator pθ(x|z), they differ in one fundamental aspect: the ratio between the information Iθ and network capacity C. For any given choice of the prior p(z) the generative model maximising VIM is the one with Iθ = C, instead for any R0 < C the information Iθ associated to the generative model learned, by equation 4, coincides with R0, i.e. Iθ = R0 < C. That implies an high sensitivity to the hyper-parameter R0 for the model in equation 4. In fact, as suggested in (Mathieu et al., 2018), a ratio Iθ/C ≈ 1, guarantees an appropriate level of overlap in the latent space between latent encoding, and then a better representation quality.
sections.heading: 4 RELATED WORK
sections.text: Autoencoder literature Autoencoder models are one of the most used family of neural networks to extract features in an unsupervised way (Bengio et al., 2013), and their relationship with Information Theory is well-established from the first unregularised autoencoders (Baldi & Hornik, 1989). The classical unregularised autoencoders, minimising the reconstruction loss Ep(x)[Eqφ(z|x)[− log pθ(x|z)]], are maximising an unbounded information, i.e. they are looking for a solution in the space P̃θ = {pθ : pθ(x) = p(x)}. In general, a solution in this wide space is good only for reconstruction performance because Z contains all the possible information that can be stored in the space Z , and is not robust to input noise (Vincent et al., 2008); but, as observed in (Grover & Ermon, 2018), if qφ(z|x) ∼ N (µ(x), σ(x)) the model pθ(x|z) is robust to noise and is a Gaussian generator. In this context the uninformative issue is avoided, but the price to pay is the impossibility to sample directly from a prior p(z) that is not defined; indeed, the model described in (Grover & Ermon, 2018) requires running relatively expensive Markov Chain to obtain samples.
  Many regularised models have been proposed, but the most well known is VAE, that minimises the expected code length of communicating x. As we observed in the previous sections, it is not
  guaranteed that the method finds a useful representation. Such issue can be solved both controlling the information of the model, or considering a more flexible prior p(z),(Rezende & Mohamed, 2015; Kingma et al., 2016; Dinh et al., 2016). In this manuscript we do not consider the latter approach, because by the model structure, it requires quite expensive computation (autoregressive models are necessary) and because the learned representation is not suitable for task different than the generation such as lifelong context, for which the standard VAE is used with success (Achille et al., 2018).
  The objective in equation 7 was firstly derived in (Tolstikhin et al., 2017) and (Zhao et al., 2017). Particularly relevant is the derivation in (Tolstikhin et al., 2017) because it allows us to describe an informative model pθ(x, z) as the one minimising the transport cost between the original and generated data.
  Finally, we underline that in case we wish to consider a Jensen-Shannon divergence in equation 7 it is necessary to consider an adversarial network model, discriminating the true samples z ∼ p(z) from the fake sampled by qφ(z) (Goodfellow et al., 2014). In the latter case the obtained model is equivalent to the Adversarial AutoEncoder (Makhzani et al., 2015). We conclude by remarking that in all the cases cited above the Infomax objective was never maximised using a prior p(z) different from a Gaussian.
  Information theoretic literature Information theory is strongly related with neural networks, and not only with autoencoders. Originally the InfoMax objective was applied to a self-organised system with a single hidden layer, (Bell & Sejnowski, 1997; Linsker, 1989) where the bound in the capacity was given by the numbers of hidden neurons. More recently, the (naive) InfoMax has given way to a new information-theoretic principle: the Information-Bottleneck (Tishby et al., 2000). The idea of this principle is that a feed-forward neural network trained for task T tends to learn a minimal sufficient representation of the data, maximising the following objective:
  max Z
  I(Z, T )− βI(X,Z). (12)
  Although it was shown that in the general case this principle does not hold true (Saxe et al., 2018), the principle was used as a regularisation technique with success both in unsupervised (Alemi et al., 2017; Higgins et al., 2017) and supervised (Alemi et al., 2016) settings. We observe that the VIM, equation 7, and IB, equation 12, coincide in the case of a deterministic encoder, where the encoding information is the entropy of Z.
sections.heading: 5 EXPERIMENTS
sections.text: Here we empirically evaluate the VIMAE. The section is divided into three parts: in the first part we compare the ability of the described models to infer the representation, z ∼ N (0, I). Such an experiment is to evaluate the entropy of Z ∼ q(Z) and then, as observed in section 3.2, an indirect way to estimate capacity of the network (C(X,Z) ∝ h(Z)). In the second part we evaluate the reconstruction and generative performance of the models. Indeed, the combination of the two tasks is estimation of the mutual information of the generative model Iθ(X,Z), see equation 8.
  In the third part we evaluate the robustness to noise and generalisation property of the learnt representation, observing that an informative model with small capacity is the best one for these tasks.
  In all the described experiments, the divergence DKL(q(z)||p(z)) in equation 7 is approximated via the Maximum Mean Discrepancy (Zhao et al., 2017) defined as:
  MMD(q(z), p(z)) = sup f :‖f‖Hk≤1 Ep(z)[f(Z)]− Eq(z)[f(Z)] (13)
  where Hk is the Reproducing Kernel Hilbert Space associated to a positive definite kernel k(·, ·) : Z×Z → R+. Moreover, by difficulties to compute the objective equation 4, as suggested in (Alemi et al., 2017) we decided to optimise a β-VAE, denoted βA-VAE, with β < 1; in order to avoid any confusion, the original version proposed in (Higgins et al., 2017) with β 1 will be renamed βH -VAE.
sections.heading: 5.1 THE ENTROPY OF Z
sections.text: Experiments in this part are performed with an autoencoder trained with the MNIST data-set, a collection of 70k monocromatic handwritten digits, where both the inference and generative distribution are modelled by 3-layer deep neural nets with 256 hidden units in each layer and Z = R2. From the 2d hidden distribution learnt by the different methods in figure 1, we observe that the ELBO-based models are not able to learn a distribution q(z) that is close to the prior p(z); and, as we will see in the following experiments, βH -VAE, the only ELBO-model having comparing results to the VIMAE, has to penalise drastically the decoding information.
  Thanks to this experiment we see that, in general, the ELBO-based models are not seeking for a small capacity network, since h(q(z)) h(p(z)), and that the divergence penalty introduced in equation 7 is a bound for the entropy h(q(z)), and then of the channel capacity C.
sections.heading: 5.2 THE MODEL INFORMATION
sections.text: The experiments in these final sections were performed with the same settings and autoencoder models used in (Tolstikhin et al., 2017), an architecture similar to the DCGAN (Radford et al., 2015) with batch normalization (Ioffe & Szegedy, 2015) (more details given in the Appendix). We consider four data-sets: MNIST and CIFAR10, two standard data-sets with ground-truth labels; Omniglot, a data-set of 1623 characters from 50 alphabets, 30 training and 20 evaluation, where each character appears 80 times, to evaluate the informativeness of the model and the quality of the learned representation; in the Appendix, we also consider CelebA (Liu et al., 2015), consisting of roughly of 203k faces of 64 × 64 resolution, in order to compare the generative quality of the pictures. After considering many parameters for βH , βA and λ, we choose, in accordance with what was suggested in (Tolstikhin et al., 2017), βH = λ = 10, and βA = 0.2 for MNIST and Omniglot and β = λ = 100 and βA = 0.4 for CelebA and CIFAR10 experiments.
  The goal of this section is to evaluate the informativeness of the learnt generative model pθ(x|z). From what was described above, the reconstruction loss or the generation quality alone are not reliable metrics, because the reconstruction loss is an estimation of the variational mutual information Iθ,φ(X,Z), and the generation quality is an estimation of DKL(pθ(x)||p(x)) that, as observed in section 2.3, it is possible to minimise with an uninformative generator. But, according to equation 8 the combination of the two task performances is a good empirical estimation of Iθ(X,Z); indeed, by generative experiments, we require that q(z) = p(z), so the KL divergence term in equation 8 comes for free.
  Reconstruction and generative performances According to what was asserted above, we observe from figure 2 (top) and figure 3 that an high penalty on the rate term (small encoding information) in the ELBO based model coincides with poor reconstruction. Indeed, the βH -VAE, is the one with worst reconstruction, and instead βA-VAE is the model that behaves similarly to VIMAE, that is optimising an encoding information-free objective. Such behaviour is consistent with the rate term associated to each model, see caption of each figure, where we observe that VIMAE, have the highest rate and in particular this is twice the one associated to βA-VAE, confirming that VIMAE learns a maximal informative decoder, and that βA-VAE is only theoretically analogous to VIMAE.
  The models that we are considering are defined as generative models, so giving a sample z ∼ p(z) they should be able to generate a new data x similar to the original one. As we observe from the
  generated samples of Omniglot in figure 2 (bottom) and Negative LogLikelihood (NLL) listed in table 1, both VAE and βH -VAE do not generate good samples, confirming that sharper samples are associated to informative generators. Such qualities on Omniglot are confirmed by the experiments with the CIFAR10 and CelebA data-set (Liu et al., 2015), see the Appendix, where it is observed that if the generative difference between the two VIMAEs is small, the difference with the VAE counterparts is high. Such behaviour is in agreement with what was observed until now: the ELBO based model does not learn a good generative network, and the good reconstruction is simply associated to a large entropy of Z, (encoding information) instead of an informative generative model pθ(x|z).
  Table 1: NLL for generated samples on Omniglot (smaler is better)
  Method NLL
  VAE 1224 βH -VAE 1254 βA-VAE 1228 VIMAE-n 1190 VIMAE-l 1223
  Similar experiments were conducted with the MNIST dataset and are discussed in the appendix.
sections.heading: 5.3 GENERALISATION PROPERTY
sections.text: We defined a good representation as the one containing the relevant properties of the visible data and able to generalise from the task for which was trained. In order to evaluate such quality, following the approach proposed in (Rifai et al., 2011), we evaluate the accuracy of an SVM directly trained on the learned features of the data. Proceeding as in (Zhao et al., 2017), we train the M1+TSVM (Kingma et al., 2014) and use the semi-supervised performance over 1000 (100 for Omniglot) samples as an approximate metric to verify the relevance and the quality of the learned representation. In order to evaluate the robustness of the learned features, we performed the same algorithm on the representation associated to corrupted data, i.e. z ∼ q(z|x+ν), considering two types of noise: Gaussian and mask. In the Gaussian case, we add to each pixel a ν value sampled from N (0, σ2) with σ ∈ {0.2, 0.3, 0.4}, and in the masking
  Table 3: Semi-supervised classification, MNIST.
  accuracy (%)
  Method ν = 0 ν = N (0, σ2) ν = B(p) 0.2 0.4 0.2 0.5
  VAE 80 77 70 72 52 βH -VAE 92 86 82 91 84 βA-VAE 93 66 13 85 65 VIMAE-n 93 92 86 92 86 VIMAE-l 93 92 88 92 87
  Table 4: Semi-supervised classification, Omniglot (random sampling: 20%).
  accuracy (%)
  ν = 0 ν = N (0, σ2) ν = B(p) 0.2 0.4 0.2 0.5
  22 22 17 22 16 21 21 22 19 17 22 22 21 21 24 22 23 24 22 22 24 23 20 23 22
  case a fraction ν of the elements is forced to be 0: each pixel is masked according to a Bernoulli distribution B(p), p ∈ {0.2, 0.5}. Higher classification performance suggests that the learned representation contains the relevant information and, in case of corrupted input data, that it is robust. In the Omniglot case by the challenge of the task (the test alphabet was never seen in the training) we consider a 5-character data-set, split into 300 (60× 5) for training and 100 for evaluation. From the classification scores listed in tables 2- 4, we see that the ELBO-based model learnt good representations for clean data, but not when corrupted data is given as input. This is particularly clear in the Bernoulli case, that is a noise different from the one seen in the training. Particularly relevant is the behaviour of the two VIMAEs: they are comparable in the cases of clean data and small noise, but the one with big capacity, VIMAE-n, suffers in large noise, while the one with small capacity, VIMAE-l, is the most robust and in some challenging cases, see table 2, the noise helps to improve the model accuracy. Such a result is consistent with the idea that a small capacity network is learning the relevant factors of the input data, that are the only ones robust to the input noise.
sections.heading: 6 CONCLUSION
sections.text: We observe, via an information theoretic description of VAE, that it is possible to learn a good generative model while maintaining a meaningful hidden representation, and that goal can be reached by optimising the CCIM, an objective that separates out the two properties of a network: the generative information and its capacity. We compare both theoretically and computationally the CCIM objective with the IB and its variation (Alemi et al., 2017), observing that the CCIM optimal solution, in particular cases, coincides with the IB one, but that in the practical case the difference is substantial.
  Future work includes the extension of the VIMAE model to a lifelong task, a scenario where it is necessary to learn both an informative representation while maintaining bounded the capacity of the network, in order to avoid the catastrophic forgetting issue (Achille et al., 2018).
sections.heading: RELATIONSHIP BETWEEN ENCODING, DECODING AND VARIATIONAL INFORMATION
sections.text: Defined q(z, x) := qφ(z|x)p(x) the encoding distribution and pθ(x, z) := pθ(x|z)p(z) the decoding one. The encoding Iq(X,Z), decoding Iθ(X,Z) and the variational Iθ,φ(X,Z) information are defined respectively:
  Iq(X,Z) = DKL(q(z, x)||q(z)p(x)) = h(X)− hq(X|Z) Iθ(X,Z) = DKL(pθ(z, x)||p(z)p(x)) = h(X)− hθ(X|Z)
  Iθ,φ(X,Z) = h(X)− Eq(z,x)[− log pθ(x|z)]
  Assuming θ∗ ∈ Θ is the parameter associated to the maximal decoding information, Iθ(X,Z) ≤ Iθ∗(X,Z) for any θ ∈ Θ, it follows that for any qφ(x, z) ∈ Pθ, i.e. for any qφ(z) = Ep(x)[qφ(x|z)] = p(z) and φ ∈ Φ ⊂ Θ,
  Iθ∗(X,Z) ≥ Iq(X,Z).
  Then a lower bound of Iq is a lower bound of Iθ∗ . By property of KL-divergence we have that for any pθ(x|z) the following relationship holds:
  Eq(z,x)[− log pθ(x|z)] = hq(X|Z) + Eq(z)[DKL(qφ(x|z)||pθ(x|z))] (14)
  From equation 14 and the definition of the variational information Iθ,φ we deduce that:
  Iθ∗(X,Z) ≥ Iq(X,Z) ≥ Iθ,φ(X,Z)
  We conclude observing that if Θ = Φ, at optimum the three information terms above are equal, and then qφ(x, z) = pθ(x, z).
  Observation: If Φ ⊂ Θ, the variational mutual information can be at most equal to Iq∗ , the maximal Iq , but it is not guaranteed that Iq∗ = Iθ∗ .
sections.heading: DERIVATION OF EQUATION 8
sections.text: In (Zhao et al., 2017), it is observed that equation 7 can be written as follows:
  V IMθ,φ =−DKL(p(x)||pθ(x))− Ep(x)[DKL(qφ(z|x)||pθ(z|x))]− − (λ− 1)DKL(qφ(z)||p(z)) + Iq(X,Z).
  From equation above, to verify that equation 8 is correct, it is sufficient to show that
  Iθ,φ(X,Z) = Iq(X,Z)− Ep(x)[DKL(qφ(z|x)||pθ(z|x))]. (15)
  The equation 15 follows by the property of the autoencoder and equation 15.
  More precisely, by equation 14 we have that
  Iθ,φ(X,Z) = Iq(X,Z)− Eq(z)DKL(qφ(x|z)||pθ(x|z))
  and, by AutoEncoder architecture, as observed in section 3, pθ(z) = Ep(x)[pθ(z|x)] = q(z). Then the following equation holds
  Eq(z)DKL(qφ(x|z)||pθ(x|z)) = Ep(x)DKL(qφ(z|x)||pθ(z|x)).
  Indeed, Eq(z)DKL(qφ(x|z)||pθ(x|z)) = ∫ q(z) ∫ q(x|z) log q(x|z)
  pθ(x|z) dxdz
  = ∫ ∫ q(x)q(z|x) log q(x|z)q(z)
  pθ(x|z)(z) dxdz =
  ∫ p(x) ∫ q(z|x) log q(z|x)
  pθ(z|x) dxdz =
  = Ep(x)DKL(qφ(z|x)||pθ(z|x)).
sections.heading: FURTHER DETAILS ON EXPERIMENTS
sections.text: In all the experiments in section 5.2 we considered the latent space Z = Rd, for all the models we choose the prior p(z) to be a Gaussian with zero mean and identity covariance, only in VIMAE-l we choose the prior p(z) to be a logistic with mean zero and identity variance. We choose pθ(x|z) to be similar to DCGAN with batch normalization and qφ(z|x) to be a convolutional deep neural network. The entire models are trained end to end by Adam (Kingma & Ba, 2014) with α = 10−3, β1 = 0.5, β2 = 0.999. We considered a deterministic decoder and we approximate the reconstruction loss with the L2 loss, i.e. Ep(x)[Eq(z|x)[− log pθ(x|z)]] = ‖x − xg‖22, with xg indicating the generated datum. In VIMAE case, while training we were adding a pixel-wise Gaussian noise truncated at 0.01 to all the images before feeding them to encoder, in order to make the encoder random. In VAE and β−VAE case, instead we used the standard reparameterization trick (Kingma & Welling, 2013). In the following we describe the data-sets considered and the associated neural networks, we follow the same description given in (Tolstikhin et al., 2017) since we used the same neural nets.
sections.heading: MNIST AND OMNIGLOT
sections.text: MNIST is a data-set containing 70k grey-scale handwritten digits and associated labels of resolution 28× 28, subdivided in three subsets: train (50k), validation (10k) and test (10k). Omniglot is a data-set containing 1623 different handwritten characters of resolution 28 × 28 from 50 different alphabets. Each of the 1623 characters was drawn online via Amazon’s Mechanical Turk by 20 different people. In the same fashion as done in (Vinyals et al., 2016) we considered an augmented version where each character is rotated respectively by 90, 180, 270 degrees, in this way each character appears 80 times. The Omniglot data-set although has the same resolution of the MNIST, for this reason we use the same network, it is more challenging because, it is more entropic then MNIST, in fact the classes move from 10 to 1623 and the test classes are never seen in the training.
  We chooseZ = R8, and β = λ = 10, we used mini-batches of size 100 and trained the model for 80 epochs. Both encoder and decoder used fully convolutional architectures with 4 × 4 convolutional filters. Encoder:
  x ∈ R28×28 → Conv128 → BN→ ReLu → Conv256 → BN→ ReLu → Conv512 → BN→ ReLu → Conv1024 → BN→ ReLu
  Decoder:
  z ∈ R8 → FC7×7×1024 → FSConv512 → BN→ ReLu → FSConv256 → BN→ ReLu→ FSConv1
  Where Convk stands for a convolution with k filters, FSConvk for the fractional strided convolution with k filters, BN for batch normalization, ReLU for the rectified linear units, and FCk for the fully connected layer mapping to Rk. All the convolutions in the encoder used vertical and horizontal strides 2 and SAME padding.
sections.heading: CELEBA AND CIFAR10
sections.text: CelebA is a data-set with 202 599 faces images. We preprocessed the images by first taking a 140 × 140 center crops and then resizing to the 64 × 64 resolution and we consider the last 20k images as test subset. CIFAR10 is a dataset consisting of of 60k 32× 32 colour images in 10 classes, with 6k images per class. There are 50k training images and 10k test images.
  For these data-sets we choose the same network with the same hyper-parameters, λ = 100 and Z = R64. We used mini-batches of size 100 and trained the model for 60 epochs. Both encoder and decoder used a fully convolutional architectures with 5× 5 convolutioanal filters. Encoder:
  x ∈ R64×64×3 → Conv128 → BN→ ReLu → Conv256 → BN→ ReLu → Conv512 → BN→ ReLu → Conv1024 → BN→ ReLu
  Decoder:
  z ∈ R64 → FC8×8×1024 → FSConv512 → BN→ ReLu → FSConv256 → BN→ ReLu → FSConv128 → BN→ ReLu→ FSConv1
sections.heading: CELEBA AND OMNIGLOT EXPERIMENTS
sections.text: In section 5.2 we evaluate the generative performance on relatively simple grey-scale data-set. In order to consider a more challenging data-set and quantitatively compare the generative performances of the trained models we evaluate the Frechet Inception Distance (FID) on CelebA and CIFAR10 based on 104 samples. From table 5, we observe, in agreement on what observed in section 5.2 and figures 4 and 5 that the difference between the two VIMAE models is minimal, instead it is big the difference with the ELBO based models (βH -VAE is not listed in table 5, because it does not converge).
sections.heading: MNIST EXPERIMENTS
sections.text: The MNIST dataset is a classical benchmark but quite simple, indeed as we see from figure 6 all the models considered are able to reconstruct without big differences, although as we see from the L2 reconstruction loss in the VIMAE models is smaller than the ELBO-based counterpart. That results combined with the high value of the rate term, suggest that the VIMAE are actually learning a maximal informative decoder. We conclude observing that, according to what seen until now, the high rate value for βA-VAE is associated to an high entropy h(q(z)), this is well visible by the generated samples from p(z) in figure 7, where the samples associated to βA-VAE seem to belong to the same cluster.