summaries: The paper tries to minimize the difference of the PMI between related and unrelated pairs of multimodal data but arrives at a very different objective with many approximations. It can be plugged into existing VAE based methods and improve learning performance and data efficiency. My major concern is about the derivation and the connection between motivation and the final objective.

summaries: The paper proposes a contrastive objective that (1) minimizes the distance between "related" samples while (2) maximizing the distance between randomly paired samples. Existing multimodal VAEs optimize (1) via different multimodal ELBOs. The novelty lies in the optimization of (2) which can further benefit from unimodal samples for which no "related" samples of the other modality are available---this can be viewed as a semi-supervised approach for weakly-supervised multimodal data.  For the estimation of (2), the paper experiments with two different estimators, IWAE and CUBO.

summaries: This work presents a generative model for multimodal learning. The paper maximizes or minimizes the pointwise mutual information between data from two modalities considering a novel random variable relatedness to dictates if data are related or not.  This is realized by casting multimodal learning as max-margin optimization with the contrastive loss for the objective. For the optimization, the paper considers the IWAE estimator. As per the experiments, the paper considers MNIST-SVHN and CUB Image-Captions dataset and perform evaluations across four metrics. Using the experiments, the paper demonstrates that the proposed approach improves multimodal learning, data-efficient learning, and label propagation.

abstractText: Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of “related” multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between “related” and “unrelated” multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal variational autoencoder (VAE) models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.

authors:
A. Blum
S. Chawla

title: Learning from labeled and unlabeled data using graph mincuts

venue: Proceedings of the Eighteenth International Conference on Machine Learning (ICML

authors:
A. Blum
T. Mitchell

title: Combining labeled and unlabeled data with co-training

venue: In Proceedings of the eleventh annual conference on Computational learning theory,

authors:
Y. Burda
R.B. Grosse
R. Salakhutdinov

title: Importance weighted autoencoders

venue: 4th International Conference on Learning Representations,

authors:
M.C. Burkhart
K. Shan

title: Deep low-density separation for semi-supervised classification

venue: In International Conference on Computational Science,

authors:
T. Chen
S. Kornblith
M. Norouzi
G.E. Hinton

title: A simple framework for contrastive learning of visual representations

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
A.P. Dempster
N.M. Laird
D.B. Rubin

title: Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society series b-methodological

authors:
A.B. Dieng
D. Tran
R. Ranganath
J.W. Paisley
D.M. Blei

title: Variational inference via χ upper bound minimization

venue: In 31st Annual Conference on Neural Information Processing Systems,

authors:
W. Guo
J. Wang
S. Wang

title: Deep multimodal representation learning: A survey

venue: IEEE Access,

authors:
R. Hadsell
S. Chopra
Y. LeCun

title: Dimensionality reduction by learning an invariant mapping

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06),

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep residual learning for image recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
K. He
H. Fan
Y. Wu
S. Xie
R.B. Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,

title: Data-efficient image recognition with contrastive predictive coding

authors:
R.D. Hjelm
A. Fedorov
S. Lavoie-Marchildon
K. Grewal
P. Bachman
A. Trischler
Y. Bengio

title: Learning deep representations by mutual information estimation and maximization

venue: In 7th International Conference on Learning Representations,

authors:
C.-C. Hsu
C.-W. Lin
W.-T. Su
G. Cheung

title: Sigan: Siamese generative adversarial network for identitypreserving face hallucination

venue: IEEE Transactions on Image Processing,

title: Transductive inference for text classification using support vector machines

venue: Proceedings of the Sixteenth International Conference on Machine Learning,

authors:
D.J. Miller
H.S. Uyar

title: A mixture of experts classifier with learning based on both labelled and unlabelled data

venue: In Advances in Neural Information Processing Systems

authors:
A. Mnih
K. Kavukcuoglu

title: Learning word embeddings efficiently with noise-contrastive estimation

venue: Advances in Neural Information Processing Systems

authors:
K. Nigam
A.K. McCallum
S. Thrun
T. Mitchell

title: Text classification from labeled and unlabeled documents using em

venue: Machine Learning,

authors:
S.E. Reed
Z. Akata
H. Lee
B. Schiele

title: Learning deep representations of fine-grained visual descriptions

authors:
Y. Shi
S. Narayanaswamy
B. Paige
P.H.S. Torr

title: Variational mixture-of-experts autoencoders for multimodal deep generative models

venue: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems

authors:
H.O. Song
Y. Xiang
S. Jegelka
S. Savarese

title: Deep metric learning via lifted structured feature embedding

authors:
Y.H. Tsai
P.P. Liang
A. Zadeh
L. Morency
R. Salakhutdinov

title: Learning factorized multimodal representations

authors:
A. van den Oord
Y. Li
O. Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
R. Vedantam
I. Fischer
J. Huang
K. Murphy

title: Generative models of visually grounded imagination

venue: In 6th International Conference on Learning Representations,

authors:
T. Wang
P. Isola

title: Understanding contrastive representation learning through alignment and uniformity on the hypersphere

authors:
K.Q. Weinberger
J. Blitzer
L.K. Saul

title: Distance metric learning for large margin nearest neighbor classification

venue: In Advances in Neural Information Processing Systems 18 [Neural Information Processing Systems,

authors:
P. Welinder
S. Branson
T. Mita
C. Wah
F. Schroff
S. Belongie
P. Perona

title: Caltech-UCSD Birds 200

venue: Technical Report CNS-TR-2010-001,

title: Caltech-UCSD Birds 200

authors:
M. Wu
N.D. Goodman

title: Multimodal generative models for scalable weakly-supervised learning

venue: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems

authors:
G. Yildirim
N. Jetchev
U. Bergmann

title: Unlabeled disentangling of gans with guided siamese

title: From perception to conception: Learning multisensory representations

authors:
C. Zhang
Z. Yang
X. He
L. Deng

title: Multimodal intelligence: Representation learning, information fusion, and applications

venue: IEEE Journal of Selected Topics in Signal Processing,

authors:
D. Zhou
O. Bousquet
T.N. Lal
J. Weston
B. Schölkopf

title: Learning with local and global consistency

venue: Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems,

authors:
Z.-H. Zhou
M. Li

title: Tri-training: exploiting unlabeled data using three classifiers

venue: IEEE Transactions on Knowledge and Data Engineering,

title: Semi-supervised learning by disagreement

venue: Knowledge and Information Systems,

authors:
X. Zhu
Z. Ghahramani
J.D. Lafferty

title: Semi-supervised learning using gaussian fields and harmonic functions

venue: Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003),

text:
1 INTRODUCTION
To comprehensively describe concepts in the real world, humans collect multiple perceptual signals of the same object such as image, sound, text and video. We refer to each of these media as a modality, and a collection of different media featuring the same underlying concept is characterised as multimodal data. Learning from multiple modalities has been shown to yield more generalisable representations (Zhang et al., 2020; Guo et al., 2019; Yildirim, 2014), as different modalities are often complimentary in content while overlapping for common abstract concept.
Despite the motivation, it is worth noting that the multimodal framework
is not exactly data-efficient—constructing a suitable dataset requires a lot of “annotated” unimodal data, as we need to ensure that each multimodal pair is related in a meaningful way. The situation is worse when we consider more complicated multimodal settings such as language–vision, where one-to-one or one-to-many correspondence between instances of the two datasets are required, due to the difficulty in categorising data such that commonality amongst samples is preserved within categories. See Figure 1 for an example from the CUB dataset (Welinder et al., a); although the same species of bird is featured in both image-caption pairs, their content differs considerably. It would be unreasonable to apply the caption from one to describe the bird depicted in the other, necessitating one-to-one correspondence between images and captions.
However, the scope of multimodal learning has been limited to leveraging the commonality between “related” pairs, while largely ignoring “unrelated” samples potentially available in any multimodal dataset—constructed through random pairing between modalities (Figure 3). We posit that if a distinction can be established between the “related” and “unrelated” observations within a multimodal dataset, we could greatly reduce the amount of related data required for effective learning. Figure 2 formalises this proposal. Multimodal generative models in previous work (Figure 2a) typically assumes one latent variable z that always generates related multimodal pair (x,y). In this work (Figure 2b), we introduce an additional Bernoulli random variable r that dictates the “relatedness” between x and y through z, where x and y are related when r = 1, and unrelated when r = 0.
∗work done while at Oxford
(a) Previous (b) Ours
Figure 2: Graphical models for multimodal generative process.
While r can encode different dependencies, here we make the simplifying assumption that the Pointwise Mutual Information (PMI) between x and y should be high when r = 1, and low when r = 0. Intuitively, this can be achieved by adopting a max-margin metric. We therefore propose to train the generative moels with a novel contrastive-style loss (Hadsell et al., 2006; Weinberger et al., 2005), and demonstrate the effectiveness of our proposed method from a few different perspectives: Improved multimodal learning: showing improved multimodal learning for various state-of-the-art multimodal
generative models on two challenging multimodal datasets. This is evaluated on four different metrics (Shi et al., 2019) summarised in § 4.2; Data efficiency: learning generative models under the contrastive framework requires only 20% of the data needed in baseline methods to achieve similar performance—holding true across different models, datasets and metrics; Label propagation: the contrastive loss encourages a larger discrepency between related and unrelated data, making it possible to directly identify related samples using the PMI between observations. We show that these data pairs can be used to further improve the learning of the generative model.

heading: 2 RELATED WORK

text:
Contrastive loss Our work aims to encourage data-efficient multimodal generative-model learning using a popular representation learning metric—contrastive loss (Hadsell et al., 2006; Weinberger et al., 2005). There has been many successful applications of contrastive loss to a range of different tasks, such as contrastive predictive coding for time series data (van den Oord et al., 2018), image classification (Hénaff, 2020), noise contrastive estimation for vector embeddings of words (Mnih and Kavukcuoglu, 2013), as well as a range of frameworks such as DIM (Hjelm et al., 2019), MoCo (He et al., 2020), SimCLR (Chen et al., 2020) for more general visual-representation learning. The features learned by contrastive loss also perform well when applied to different downstream tasks — their ability to generalise is further analysed in (Wang and Isola, 2020) using quantifiable metrics for alignment and uniformity.
Contrastive methods have also been employed under a generative-model setting, but typically on generative adversarial networks (GANs) to either preserve or identify factors-of-variations in their inputs. For instance, SiGAN (Hsu et al., 2019) uses a contrastive loss to preserve identity for faceimage hallucination from low-resolution photos, while (Yildirim et al., 2018) uses a contrastive loss to disentangle the factors of variations in the latent code of GANs. We here employ a contrastive loss in a distinct setting of multimodal generative model learning, that, as we will show with our experiments and analyses promotes better, more robust representation learning. Multimodal VAEs We also demonstrate that our approach is applicable across different approaches to learning multimodal generative models. To do so, we first summarise past work on multimodal VAE into two categories based on the modelling choice of approximated posterior qΦ(z|x,y):
a) Explicit joint models: qΦ as single joint encoder qΦ(z|x,y). Example work in this area include JMVAE (Suzuki et al.), triple ELBO (Vedantam et al., 2018) and MFM (Tsai et al., 2019). Since the joint encoder require multimodal pair (x,y) as input, these approaches typically require additional modelling components and/or inference steps to deal with missing modality at test time; in fact, all three approaches propose to train unimodal VAEs on top of the joint model that handles data from each modality independently.
b) Factorised joint models: qΦ as factored encoders qΦ(z|x,y) = f ( qφx(z|x), qφy (z|y) ) .
This was first seen in Wu and Goodman (2018), as the MVAE model with f defined as a product of experts (POE), i.e. qΦ(z|x,y) = qφx(z|x)qφy (z|y)p(z), allowing for cross-modality generation without extra modelling components. Particularly, the MVAE caters to settings where data was not guaranteed to be always related, and where additional modalities were, in terms of information content, subsets of a primary data source—such as images and their class labels. Alternately, Shi et al. (2019) explored an approach that explicitly leveraged the availability of related/paired data, motivated by arguments from embodied cognition of the world. They propose the MMVAE model, which additionally differs from the MVAE model in its choice of posterior approximation—where f is modelled as the mixture of experts (MOE) of unimodal posteriors—to ameliorate shortcomings to do with precision miscalibration of the POE. Furthermore, Shi et al.
(2019) also posit four criteria that a multimodal VAE should satisfy, which we adopt in this work to evaluate the performance of our models.
Weakly-supervised learning Using generative models for label propagation (see § 4.5) is a form of weak supervision. Commonly seen approaches for weakly-supervised training with incomplete data include (1) graph-based methods (such as minimum cut) (Zhou et al., 2003; Zhu et al., 2003; Blum and Chawla, 2001), (2) low-density separation methods (Joachims, 1999; Burkhart and Shan, 2020) and (3) disagreement-based models (Blum and Mitchell, 1998; Zhou and Li, 2005; 2010). However, (1) and (2) suffers from scalability issues due to computational inefficiency and optimisation complexity, while (3) works well for many different tasks but requires training an ensemble of learners.
The use of generative models for weakly supervised learning has also been explored in Nigam et al. (2000); Miller and Uyar (1996), where labels are estimated using expectation-maximisation (EM) (Dempster et al., 1977) for instances that are unlabelled. Notably, models trained with our contrastive objective does not need EM to leverage unlabelled data (see § 4.5) to determine the relatedness of two examples we only need to compute a threshold (estimation of PMI) using the trained model.

heading: 3 METHODOLOGY

text:
Given data over observations from two modalties (x,y), one can learn a multimodal VAE targetting pΘ(x,y, z) = p(z)pθx(x|z)pθy (y|z), where pθ(·|z) are deep neural networks (decoders) parametrised by Θ = {θx, θy}. To maximise the joint marginal likelihood log pΘ(x,y), one approximates the intractable model posterior pΘ(z|x,y) with a variational posterior qΦ(z|x,y), allowing us to optimise a variational evidence lower bound (ELBO), defined as
log pΘ(x,y) ≥ Ez∼ qΦ(z|x,y) [ log pΘ(z,x,y)
qΦ(z | x,y)
] = ELBO(x,y). (1)
This leaves open the question of how to model the approximatd posterior qΦ(z|x,y). As mentioned in § 2, there are two schools of thinking, namely explicit joint model such as JMVAE (Suzuki et al.) and factorised joint model including MVAE (Wu and Goodman, 2018) and MMVAE (Shi et al., 2019). In this work we demonstrate the effectiveness of our approach across all these models.

heading: 3.1 CONTRASTIVE LOSS FOR “RELATEDNESS” LEARNING

text:
Where prior work always assumes (x,y) to be related, we introduce a relatedness variable r explicitly capturing this aspect. Our approach is motivated by the characteristics of pointwise mutual information (PMI) between related and unrelated observations across modalities:
Hypothesis 3.1. Let (x,y) ∼ pΘ(x,y) be a related data pair from two modalities, and let y′ denote a data point unrelated to x. Then, the pointwise mutual information I(x,y) > I(x,y′).
(a) Related (b) Unrelated
in PMI between related and unrelated pairs, i.e. I(x,y)− I(x,y′). We show in Appendix A that this is equivalent to maximising the difference between the joint marginal likelihoods of related and unrelated pairs (Figure 3a and 3b). This casts multimodal learning as max-margin optimisation, with the contrastive (triplet) loss as a natural choice of objective: LC(x,y,y′) = d(x,y)−d(x,y′) +m. Intuitively, LC attempts to make distance d between a positive pair (x,y) smaller than the distance between a negative pair (x,y′) by margin m. We adopt this loss to our objective by omitting margin m and replacing d with the (negative) joint marginal likelihood − log pΘ. Following Song et al. (2016), with N negative samples {y′i}Ni=1, we have
LC(x, Y ) = − log pΘ(x,y) + log N∑ i=1 pΘ(x,y ′ i). (2)
We choose to put the sum overN within the log following conventions in previous work on contrastive loss (van den Oord et al., 2018; Hjelm et al., 2019; Chen et al., 2020). As the loss is asymmetric, one can average over LC(y, X) and LC(x, Y ) to ensure negative samples in both modalities are accounted for—giving our contrastive objective:
LC(x,y)= 1
2 {LC(x, Y )+LC(y, X)}=−log pΘ(x,y)︸ ︷︷ ︸
1
+ 1
2 log N∑ x′i=1 pΘ(x ′,y) + log N∑ y′i=1 pΘ(x,y ′)  ︸ ︷︷ ︸
2
(3)
Note that since only the joint marginal likelihood terms are needed in (3), the contrastive learning framework can be directly applied to any multimodal generative model without needing extra components. We also show our contrastive framework applies when the number of modalities is more than two (cf. Appendix C).
Figure 4: log p(x,y) of imitation, unrelated digits and random noise, where p = N (mx,my).
Dissecting LC Although similar to the VAE (1), our objective (3) directly maximises pΘ(x,y) in 1 . LC by itself is not a effective objective as 2 in LC minimises pΘ(x,y), which can overpower the effect of 1 during training.
We intuit this phenomenon using a simple example in Figure 4 with natural images, showing the log likelihood of log p(x,y) in column 2, 3, 4 (green) on a Gaussian distribution N ((mx,my); c), with the
images in the first column of Figure 4 as means and with constant variance c. While achieving high log p(x,y) requires matching (mx,my) (col 2), we see both unrelated digits (col 3) and noise (col 4) can lead to (comparatively) poor joint log likelihoods. This indicates that the generative model need not generate valid, unrelated images to minimise 2 —generating noise would have roughly the same effect on log likelihood. As a result, the model can trivially minimise LC by generating noise that minimises 2 instead of accurate reconstruction that maximises 1 .
This learning dynamic is verified empirically, as we show in Figure 9 in Appendix D: optimising LC by itself results in the loss approaching 0 within the first 100 iterations, and both 1 and 2 takes on extremely low values, resulting in a model that generates random noise. Final objective To mitigate this issue, we need to ensure that minimising 2 does not overpower maximising 1 . We hence introduce a hyperparameter γ on 1 to upweight the maximisation of the marginal likelihood, with the final objective to minimise
LC(x,y) = −γ log pΘ(x,y)︸ ︷︷ ︸ 1
+ 1
2 log N∑ x′i=1 pΘ(x ′,y) + log N∑ y′i=1 pΘ(x,y ′)  ︸ ︷︷ ︸
2
, γ > 1. (4)
We conduct ablation studies on the effect of γ in Appendix E, noting that larger γ encourages better quality of generation and more stable training in some cases, while models trained with smaller γ are better at predicting “relatedness” between multimodal samples. We also note that optimising (4) maximises the pointwise mutual information I(x,y); see Appendix B for a proof.

heading: 3.2 OPTIMISING THE OBJECTIVE

text:
Since in VAEs we do not have access to the exact marginal likelihood, we have to optimise an approximated version of the true contrastive objective in (4). In (5) we list a few possible candidates of estimators and their relationships to the (log) joint marginal likelihood:
ELBO ≤ E{zk}K1 ∼qΦ
[ log 1
K K∑ k=1 pΘ(zk,x,y) qΦ(zk | x,y) ] ︸ ︷︷ ︸
IWAE
≤ log pΘ(x,y) ≤ E{zk}K1 ∼qΦ log 2 √√√√ 1 K K∑ k=1 ( pΘ(zk,x,y) qΦ(zk | x,y) )2 ︸ ︷︷ ︸
CUBO
(5)
with the ELBO from Eq. (1), the importance weighted autoencoder (IWAE), a K-sample lower bound estimator that can compute an arbitrarily tight bound with increasing K (Burda et al., 2016), and the χ upper bound (CUBO), an upper-bound estimator (Dieng et al., 2017).
We now discuss our choice of approximation of log pΘ(x,y) for each term in equation (4):
1 Minimising this term maximises the joint marginal likelihood, which can hence be approximated with a valid lower-bound estimator (5); the IWAE estimator being the preferred choice.
2 For this term, we consider two choices of estimators:
1) CUBO: Since 2 of (4) minimises the joint marginal likelihoods, to ensure an upper-bound estimator to LC in (4), we need to employ an upper-bound estimator. Here we propose to use CUBO in (5). While such a bounded approximation is indeed desirable, existing upperbound estimators tend to have rather large bias/variance and can therefore yield poor quality approximations.
2) IWAE: We therefore also propose to estimate 2 with the IWAE estimator, as it provides an arbitrarily tight, low variance lower-bound (Burda et al., 2016) to log pΘ(x,y). Although this no longer ensure a valid bound on the objective, we hope that having a more accurate approximation to the marginal likelihood (and by extension, the contrastive loss) can affect the performance of the model positively.
We report results using both IWAE and CUBO estimators in § 4, denoted cI and cC respectively.

heading: 4 EXPERIMENTS

text: As stated in § 1, we analyse the suitability of contrastive learning for multimodal generative models from three persepctives—improved multimodal learning (§ 4.3), data efficiency (§ 4.4) and label propagation (§ 4.5). Throughout the experiments, we takeN = 5 negative samples for the contrastive objective, set γ = 2 based on analyses of ablations in appendix E, and take K = 30 samples for our IWAE estimators. We now introduce the datasets and metrics used for our experiments.

heading: 4.1 DATASETS

text: MNIST-SVHN The dataset is designed to separate conceptual complexity, i.e. digit, from perceptual complexity, i.e. color, style, size. Each data pair contains 2 samples of the same digit, one from each dataset (see examples in Figure 3a). We construct the dataset such that each instance from one dataset is paired with 30 instances of the same digit from the other dataset. Although both datasets are simple and well-studied, the many-to-many pairing between samples creates matching of different writing styles vs. backgrounds and colors, making it a challenging multimodal dataset. CUB Image-Captions We also consider a more challenging language-vision multimodal dataset, Caltech-UCSD Birds (CUB) (Welinder et al., b; Reed et al., 2016). The dataset contains 11,788 photos of birds, paired with 10 captions describing the bird’s physical characteristics, collected through Amazon Mechanical Turk (AMT). See CUB image-caption pair in Figure 1.

heading: 4.2 METRICS

text:
Shi et al. (2019) proposed four criteria for multimodal generative models (Figure 5, left), that we summarise and unify as metrics to evaluate these criteria for different generative models (Figure 5, right). We now introduce each criterion and its corresponding metric in detail. (a) Latent accuracy (Figure 5a) Criterion: latent space factors into “shared” and “private” subspaces across modalities. We fit a linear classifier on the samples from z ∼ qΦ(z|x,y) to classify the information shared between the two modalities. For MNIST-SVHN, this can be the digit label as shown in Figure 5a (right). We check if l̂z is the same as the digit label of the original inputs x and y, with the intuition that extracting the commonality between x and y from latent representation using a linear transform, supports the claim that the latent space has factored as desired.
(b) Joint coherence (Figure 5b) Criterion: model generates paired samples that preserves the commonality observed in data. Again taking MNIST-SVHN as an example, this can be verified by taking pre-trained MNIST and SVHN digit classifiers and applying them on the multimodal observations generated from the same prior sample z. Coherence is computed by how often generations x̂ and ŷ classify to the same digit, i.e. whether l̂x̂ = l̂ŷ . (c) Cross coherence (Figure 5c) Criterion: model generates data in one modality conditioned on the other while preserving shared commonality. To compute cross coherence, we generate observations x̂ using latent from unimodal marginal posterior z ∼ qφ(z|y) and ŷ from z ∼ qφ(z|x). Similar to joint coherence, the criterion here is evaluated by predicting the label of the cross-generated samples x̂, ŷ using off-the-shelf MNIST and SVHN classifiers. In Figure 5c (right), the cross coherence is the frequency of which l̂x̂ = ly and l̂ŷ = lx. (d) Synergy coherence (Figure 5d) Criterion: models learnt across multiple modalities should be no worse than those learnt from just one. For consistency, we evaluate this criterion from a coherence perspective. Given generations x̂ and ŷ from z ∼ qΦ(z|x,y), we again examine if generated and original labels match; i.e. if l̂x̂ = ly = l̂ŷ = lx. See Appendix F for details on architecture. All quantitative results are reported over 5 runs. In addition to these quantitative metrics, we also showcase the qualitative results on both datasets in Appendix G and Appendix H.

heading: 4.3 IMPROVED MULTIMODAL LEARNING

text:
Finding: Contrastive learning improves multimodal learning across all models and datasets. MNIST-SVHN See Table 1 (top, 100% data used) for results on the full MNIST-SVHN dataset. Note that for MMVAE, since the joint posterior qΦ is factorised as the mixture of unimodal posteriors qφx and qφy , the model never directly takes sample from the explicit form of the joint posterior. Instead, it takes equal number of samples from each unimodal posteriors, reflective of the equal weighting of the mixture. As a result, it is not meaningful to compute synergy coherence for MMVAE as it is exactly the same as the coherence of any single-way generation.
From Table 1 (top, 100% data used), we see that models trained on our contrastive objective (cI<MODEL> and cC-<MODEL>) improves multimodal learning performance significantly for all three generative models evaluated on the metrics. The results showcase the robustness of our approach from the perspectives of modelling choice and metric of interests. In particular, note that for the best performing model MMVAE, using IWAE estimator for 2 of Eq (4) (cI-MMVAE) yields slightly better results than CUBO (cC-MMVAE), while for the two other models the performance for different estimators are similar.
We also include qualitative results for MNIST-SVHN, including generative results, marginal likelihood table and diversity analysis in Appendix G.
CUB Following Shi et al. (2019), for the images in CUB, we observe and generate in feature space instead of pixel space by preprocessing the images using a pre-trained ResNet-101 (He et al., 2016). A nearest-neighbour lookup among all the features in the test set is used to project the feature generations of the model back to image space. This helps circumvent CUB image complexities to some extent—as the primary goal here is to learn good models and representations of multimodal data, rather than a focus on pixel-level image quality of generations.
The metrics listed in § 4.2 can also be applied to CUB with some modifications. Since bird-species classes are disjoint for the train and test sets, and as we show in Figure 1 contains substantial in-class variance, it is not constructive to evaluate these metrics using bird categories as labels. In Shi et al. (2019), the authors propose to use Canonical Correlation Analysis (CCA)—used by Massiceti et al. as a reliable vision-language correlation baseline—to compute coherence scores between generated image-caption pairs; which we employ (i.e. (b), (c), (d) in Figure 5) for CUB.
We show the results in Table 2 (top, 100% data used). We see that our contrastive approach (both cI-<MODEL> and cC-<MODEL>) is even more effective on this challenging vision-language dataset, with significant improvements to the correlation of generated image-caption pairs. It is also on this more complicated dataset where the advantages of using the stable, low-variance IWAE estimator are highlighted — for both MVAE and JMVAE, the contrastive objective with CUBO estimator suffers from numerical stability issues, yielding close-to-zero correlations for all metrics in Table 2. Results for these models are therefore omitted.
We also show the qualitative results for CUB in Appendix H.

heading: 4.4 DATA EFFICIENCY

text:
Finding: Contrastive learning on 20% of data matches baseline models on full data.
We plot the quantitative performance of MMVAE with and without contrastive learning against the percentage of the original dataset used, as seen in Figure 6. We observe that the performance of contrastive MMVAE with the IWAE (cI-MMVAE, red) and CUBO estimators (cC-MMVAE, yellow) are consistently better than the baselines (MMVAE, blue), and that baseline performance using all related data is matched by the contrastive MMVAE using just 10—20% of data. The partial datasets used here are constructed by first taking n% of each unimodal dataset, then pairing to create multimodal datasets (§ 4.1)—ensuring it contains the requisite amount of “related” samples. In addition, we reproduce results generated from using 100% of the data in MNIST-SVHN and CUB (Tables 1 and 2) using only 20% of the original multimodal datasets (Tables 1 and 2 (bottom)). Comparing results between top vs. bottom in Tables 1 and 2 shows that this finding holds across the models, on both MNIST-SVHN and CUB datasets. This shows that the efficiency gains from a contrastive approach is invariant to VAE type, data, and metrics used, underscoring its effectiveness.

heading: 4.5 LABEL PROPAGATION

text:
Finding: Generative models learned contrastively are good predictors of “relatedness”, enabling label propagation and matching baseline performance on full datasets, using only 10% of data.
Here, we show that our contrastive framework encourages a larger discrepancy between the PMI of related vs. unrelated data, as set out in hypothesis 3.1, allowing one to first train the model on a small subset of related data, and subsequently construct a classifier using PMI that identifies related samples in the remaining data. We now introduce our pipeline for label propagation in details.
Pipeline As showing in Figure 7, we first construct a full dataset by randomly matching instances in MNIST and SVHN, and denote the related pairs by Fr (full, related). We further assume access to only n% of Fr, denoted as Sr (small, related), and denote the rest as Fm, containing a mix of related and unrelated pairs. Next, we train a generative model g on Sr. To find a relatedness threshold, we construct a small, mixed dataset Sm by randomly matching samples across modalities in Sr. Given relatedness ground-truth for Sm, we can compute the PMI I(x,y) = log pΘ(x,y)− log pθx(x)pθy (y) for all pairs (x,y) in Sm and estimate an optimal threshold. This threshold can now be applied to the full, mixed dataset Fm to identify related pairs giving us a new related dataset Fr̂, which can be used to further improve the performance of the generative model g. Results In Figure 8 (a-e), we plot the performance of baseline MMVAE (blue) and contrastive MMVAE with IWAE (cI-MMVAE, red) and CUBO (cC-MMVAE, yellow) estimators for term 2 of Eq (4), trained with (solid curves) and without (dotted curves) label propagation. Here, the x-axis is the proportion in size of Sr to Fr, i.e. the percentage of related data used to pretrain the generative model before label propagation. We compare these results to MMVAE trained on all related data Fr (cyan horizontal line) as a “best case scenario” of these training regimes.
Clearly, label propagation using a contrastive model with IWAE estimator is helpful, and in general the improvement is greater when less data is available; Figure 8 also shows that when Sr is 10% of Fr, cI-MMVAE is competitive with the performance of baseline MMVAE trained on Fr.
For baselines MMVAE and cC-MMVAE however, label propagation hurts performance no matter the size of Sr, as shown by the blue and yellow curves in Figure 8 (a-e). The advantages of cIMMVAE is further demonstrated in Figure 8f, where we compute the precision, recall, and F1 score of relatedness prediction on Fm, for models trained on 10% of all related data. We also compare to a simple label-propagation baseline, where the relatedness of Fm is predicted using a siamese network (Hadsell et al., 2006) trained on the same 10% dataset. Notably, while the Siamese baseline in Figure 8f is a competitive predictor of relatedness, cI-MMVAE has the highest F1 score amongst all four, making it the most reliable indicator of relatedness. Beyond that, note that with the contrastive MMVAE, relatedness can be predicted without additional training and only requires a simple threshold computation directly computed using the generative model. The fact that the cI-MMVAE’s relatedness-prediction performance is the only one that matches the Siamese baseline strongly supports the view that the contrastive loss encourages generative models to utilise and better learn the relatedness between multimodal pairs; in addition, the poor performance of cC-MMVAE shows that the advantages of having a bounded estimation to the contrastive objective by using an
upper-bound for 2 is overshadowed by the high bias of CUBO, and that one may benefit more from choosing a low variance lower-bound like IWAE.

heading: 5 CONCLUSION

text: We introduced a contrastive-style objective for multimodal VAE, aiming at reducing the amount of multimodal data needed by exploiting the distinction between "related" and "unrelated" multimodal pairs. We showed that this objective improves multimodal training, drastically reduce the amount of multimodal data needed, and establishes a strong sense of "relatedness" for the generative model. These findings hold true across a multitude of datasets, models and metrics. The positive results of our method indicates that it is beneficial to utilise the relatedness information when training on multimodal data, which has been largely ignored in previous work. While we propose to utilise it implicitly through contrastive loss, one may consider relatedness as a random variable in the graphical model and see if explicit dependency on relatedness can be useful. It is also possible to extend this idea to Generative adversarial networks (GANs), by employing an additional discriminator that evaluates relatedness between generations across modalities. We will leave these interesting directions to be explored by future work.

heading: ACKNOWLEDGEMENTS

text: YS and PHST were supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme, EPSRC/MURI grant EP/N019474/1 and FiveAI. YS was additionally supported by Remarkdip through their PhD Scholarship Programme. BP is supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1. Special thanks to Elise van der Pol for helpful discussions on contrastive learning.

heading: A FROM POINTWISE MUTUAL INFORMATION TO JOINT MARGINAL LIKELIHOOD

text:
In this section, we show that for the purpose of utilising the relatedness between mutlimodal pairs, maximising the difference between pointwise mutual information between related points x,y and unrelated points x,y′ is equivalent to maximising the difference between their log joint marginal likelihoods.
To see this, we can expand I(x,y)− I(x,y′) as
I(x,y)− I(x,y′) = [log pΘ(x,y)− log pΘ(x)− log pΘ(y)]− [log pΘ(x,y′)− log pΘ(x)− log pΘ(y′)] = log pΘ(x,y)− log pΘ(x,y′)︸ ︷︷ ︸
1
+ log pΘ(y ′)− log pΘ(y)︸ ︷︷ ︸
2
. (6)
In (6) the PMI difference is decomposed as two terms: 1 the difference between joint marginal likelihoods and 2 the difference between marginal likelihoods of different instances of y-s. It is clear that since 2 involves only one modality and only accounts for the difference in values between y and y′, it is not relevant to the relatedness of x, y.
Therefore, for the purpose of utilising relatedness information, we only need to maximise I(x,y)− I(x,y′) through maximising term 1 , i.e. the difference between joint marginal likelihoods.

heading: B CONNECTION OF FINAL OBJECTIVE TO POINTWISE MUTUAL INFORMATION

text:
Here, we show that minimising the objective in (4) maximises the PMI between x and y:
L(x,y) = −γ log pΘ(x,y) + 1
2 log ∑ x′∈X pΘ(x ′,y) + log ∑ y′∈Y pΘ(x,y ′)  = −(γ − 1
2 ) log pΘ(x,y)−
1 2 log pΘ(x,y)∑ y′∈Y pΘ(x,y ′) ∑ x′∈X pΘ(y,x ′)
≈ −(γ − 1 2 ) log pΘ(x,y)− 1 2 log pΘ(x,y)
pΘ(x)pΘ(y)︸ ︷︷ ︸ I(x,y)
(7)
We see in (7) that minimising L can be decomposed to maximising both the joint marginal likelihood pΘ(x,y) and an approximation of PMI I(x,y). Note that since γ > 1, we can be sure that the joint marginal likelihood weighting γ − 12 is non-negative.

heading: C GENERALISATION TO M > 2 MODALITIES

text:
In this section we show how the contrastive loss generalise to cases where number of modalities considered M is greater than 2.
Given observations from M modalities D = {X1, X2, · · · , Xm, · · · , XM}, where Xm denotes unimodal dataset of modalit m of size Nm, i.e. Xm = {x(i)m }Nmi=1. Similar to (2), we can write the assymetrical contrastive loss for any observation x(i)m from modality m, where negative samples are taken for all (M − 1) other modalities:
LC(x(i)m , Dm̃) = − log pΘ(x (i) 1:M ) + log M∑ d=1 d 6=m N∑ j=1 j 6=i pΘ(x (i) 1:(d−1),(d+1):M ,x (j) d ). (8)
We can therefore rewrite (3) as:
LC(x(i)1:M ) = 1
M M∑ m=1 LC(x(i)m , Dm̃) (9)
= − log pΘ(x(i)1:M ) + 1
M M∑ m=1 log M∑ d=1 d 6=m N∑ j=1 j 6=i pΘ(x (i) 1:(d−1),(d+1):M ,x (j) d )  , (10) where N is the number of negative samples, all log pΘ(x1:M ) are approximated by the following joint ELBO for M modalities:
log pΘ(x1:M ) ≥ Ez∼ qΦ(z|x1:M ) [ log pΘ(z,x1:M )
qΦ(z | x1:M )
] = ELBO(x1:M ). (11)
While the above gives us the true generalisation of (3), we note that the number of times where ELBO needs to be evaluated in (10) is O(M2N), making it difficult to implement this objective in practice, especially on more complicated datasets. We therefore propose a simplified version of the objective, where we estimate the second term of (10) with N sets of random samples from all modalities. Specifically, we can precompute the following M ×N random index matrix J :
J =  j11 j12 · · · j1N j21 j22 · · · j2N
... ...
jM1 jM2 · · · jMN  , (12) where each entry of J is a random integer taken from range [1, Nm]. We can then replace the second term of (10) random samples selected by the indices in J , giving us
LC(x(i)1:M ) ≈ − log pΘ(x (i) 1:M ) +
( log
N∑ n=1 pΘ(x (J1n) 1 ,x (J2n) 2 , · · ·x (JMn) M )
) . (13)
The number of times ELBO needs to be computed is now O(N), and is no longer relevant to the number of modalities M .
We can now also generalise the final objective in (4) to M modalities:
L(x(i)1:M ) = −γELBO(x (i) 1:M ) +
( logsumexp
n∈N ELBO(x(J1n)1 ,x (J2n) 2 , · · ·x (JMn) M )
) . (14)

heading: D THE INEFFECTIVENESS OF TRAINING WITH LC ONLY

text:
We demonstrate why training with the contrastive loss proposed in (3) is ineffective, and why additional ELBO term is needed for the final objective. As we show in Figure 9, when training with LC only, while the contrastive loss (green) quickly drops to zero, both term 1 and 2 in (3) also reduces drastically. This means the joint marginal likelihood of any generation log pΘ(x,y) is small regardless the relatedness of (x,y).
In comparison, we also plot the training curve for model trained on the final objective in (4), which upweights term 1 in (3) by γ. We see in Figure 10 that by setting γ = 2, the joint marginal likelihood (yellow and blue curve) improves during training, while LC (green curve) gets minimised.
E ABLATION STUDY OF γ
In § 3, we specified that γ needs to be greater than 1 to offset the negative effect of minimising ELBO through term 2 in (4). Here, we study the effect of γ in details.
Figure 11 compares latent accuracy, cross coherence and joint coherence of MMVAE on MNISTSVHN dataset trained on different values of γ. Note that here we only consider cases where γ ≥ 1, since the minimum value of γ is 1. In this case, the loss reduces to the original contrastive objective in (3).
A few interesting observations from the plot are as follows: First, when γ = 1, the model is trained using the contrastive loss only, and as we showed is an ineffective objective for generative model learning. This is verified again in Figure 11 — when γ = 1, both coherences and latent accuracies take on extremely low values; interestingly, there is a significant boost of performance across all metrics by simply increasing the value of γ from 1 to 1.1; after that, as the value of γ increases, performance on most metrics decreases monotonically (joint coherence being the only exception), and eventually converges to baseline MMVAE (dotted lines in Figure 11). This is unsurprising, since the final objective in (4) reduces to the original joint ELBO as γ approaches infinity.
Figure 11 seems to suggest that 1.1 is the optimal value for hyperparameter γ, however close inspection of the qualitative generative results shows that this might not be the case. See Figure 12 for a comparison of the model’s generation between MMVAE models trained on (from left to right) γ = 1.1, γ = 2 and γ = +∞ (i.e. original MMVAE). Although γ = 1.1 yields model with high coherence scores, we can clearly see from the left-most column of Figure 12 that the generation of the model seems deprecated, especially for the SVHN modality, where the backgrounds of model’s generation appear to be unnaturally spotty and deformed. This problem is mitigated by increasing γ — as shown in Figure 11, the image generation quality of γ = 2 (middle column) is not visibly different from that of γ = +∞ (right column).
To verify this observation, we also compute the marginal log likelihood log pΘ(x,y) to quantify the quality of generations. We compute this for all γs considered in Figure 11, and take the average over the entire test set. From the results in Figure 13, we can see a significant increase of the log likelihood between γ = 1.1 to γ = 1. This gain in image generation quality then slows down as γ further increases, and as all other metrics converges to the original MMVAE model.

heading: F ARCHITECTURE

text: We use architectures listed in Table 3 for the unimodal encoder and decoder for MMVAE, MVAE and JMVAE. For JMVAE we use an extra joint encoder, the architecture of which is described in Table 4.

heading: G QUALITATIVE RESULTS ON MNIST-SVHN

text:
G.1 GENERATIVE RESULTS & MARGINAL LIKELIHOODS ON MNIST-SVHN G.1.1 MMVAE
G.1.2 MVAE MVAE cI-MVAE cC-MVAE
G.1.3 JMVAE JMVAE cI-JMVAE cC-JMVAE
G.2 GENERATION DIVERSITY To further demonstrate that the improvements from our contrastive objective did not come at a price of sacrifising generation diversity, in Figure 17 we show histograms of the number of examples generated for each class with samples from prior. Similar to how we compute joint coherence, the class label of generated images are determined using classifiers trained on the original MNIST and SVHN dataset.
We can see that the contrastive models (cI-∗ and cI-∗) are capable of generating examples from different classes, and for MMVAE and MVAE the histogram of contrastive models are more uniform than the original models (less variance between class counts).

heading: H QUALITATIVE RESULTS ON CUB

text: The generative results of MMVAE, cI-MMVAE and cC-MMVAE on CUB Image-Caption dataset are as shown in Figure 18, Figure 19 and Figure 20. Note that for the generation in the vision modality, we reconstruct and generate features from ResNet101 and perform nearest neighbour search in all the features in train set to showcase our generation results.


summaries: The authors highlight an important problem in VAE - the prior-hole problem - which is that the approximate posterior and the simple gaussian prior do not match in spite of the KL term in the ELBO which makes sampling an issue - leading to the prior putting probability mass on latents that are not decoded to high probability mass regions in data manifold. Prior approaches have overcome this problem by increasing the expressivity of the prior through autoregressive models, and/or using hierarchical latents, EBMs with MCMC sampling. This paper proposes a very simple two stage method - (1) train a regular VAE, (2) train a binary classifier in NCE style to distinguish samples from prior and approx. posterior; use the re-weighting term from the NCE score to sample from a better re-weighted prior - either through langevin dynamics or re-sampling. The authors combine this approach with the use of hierarchical latents and produce really good performing generative models on a host of benchmarks with good looking samples.

summaries: Authors approach the "hole problem" of variational autoencoders where the aggregate posterior fails to match the prior, causing some areas of the prior distribution to be left out. Consequently, the decoder is not trained properly to operate in such regions, and the whole generate models is then subject to suboptimal performance. To attack this problem authors introduce two changes:

summaries: The goal of the paper is to model the marginal over latents in VAEs in such a way to minimize the mismatch with the aggregated posterior. The paper proposes a new class of marginal distributions over the latent space that is a product of two experts: the first expert is a non-trainable probability distribution, and the second expert is an unnormalized probability distribution parameterized using neural networks. Since training a product of experts requires to apply an approximate inference (e.g., MCMC sampling), the authors propose to use the likelihood ratio trick. Eventually, a VAE is trained in two stages. First, they assume the marginal over z's to be simply the non-trainable distribution, and the VAE is trained. At the second stage, they propose to train the second expert (i.e., the binary classifier that distinguishes z ~ q(z) and z ~ p(z)) in order to obtain the final NCP that better matches the aggregated posterior. Further, the idea is extended to hierarchical VAEs, and a separate binary classifier is trained per each stochastic level.

abstractText: Variational autoencoders (VAEs) are one of the powerful likelihood-based generative models with applications in various domains. However, they struggle to generate high-quality images, especially when samples are obtained from the prior without any tempering. One explanation for VAEs’ poor generative quality is the prior hole problem: the prior distribution fails to match the aggregate approximate posterior. Due to this mismatch, there exist areas in the latent space with high density under the prior that do not correspond to any encoded image. Samples from those areas are decoded to corrupted images. To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior. We train the reweighting factor by noise contrastive estimation, and we generalize it to hierarchical VAEs with many latent variable groups. Our experiments confirm that the proposed noise contrastive priors improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets.

authors:
Jyoti Aneja
Harsh Agrawal
Dhruv Batra
Alexander Schwing

title: Sequential latent spaces for modeling the intention during diverse image captioning

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Samaneh Azadi
Catherine Olsson
Trevor Darrell
Ian Goodfellow
Augustus Odena

title: Discriminator rejection sampling

venue: In International Conference on Learning Representations,

authors:
Matthias Bauer
Andriy Mnih

title: Resampled priors for variational autoencoders

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale gan training for high fidelity natural image synthesis

venue: arXiv preprint arXiv:1809.11096,

authors:
Tong Che
Ruixiang Zhang
Jascha Sohl-Dickstein
Hugo Larochelle
Liam Paull
Yuan Cao
Yoshua Bengio

title: Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling

venue: arXiv preprint arXiv:2003.06060,

authors:
Xi Chen
Diederik P Kingma
Tim Salimans
Yan Duan
Prafulla Dhariwal
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Variational lossy autoencoder

venue: arXiv preprint arXiv:1611.02731,

authors:
Bin Dai
David Wipf

title: Diagnosing and enhancing vae models

authors:
Aditya Deshpande
Jyoti Aneja
Liwei Wang
Alexander G Schwing
David Forsyth

title: Fast, diverse and accurate image captioning guided by part-of-speech

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Prafulla Dhariwal
Heewoo Jun
Christine Payne
Jong Wook Kim
Alec Radford
Ilya Sutskever

title: Jukebox: A generative model for music

venue: arXiv preprint arXiv:2005.00341,

authors:
Yilun Du
Igor Mordatch

title: Implicit generation and modeling with energy based models

venue: In Advances in Neural Information Processing Systems,

authors:
Jesse Engel
Matthew Hoffman
Adam Roberts

title: Latent constraints: Learning to generate conditionally from unconditional generative models

authors:
Vincent Fortuin
Matthias Hüser
Francesco Locatello
Heiko Strathmann
Gunnar Rätsch

title: Som-vae: Interpretable discrete representation learning on time series

authors:
Ruiqi Gao
Erik Nijkamp
Diederik P Kingma
Zhen Xu
Andrew M Dai
Ying Nian Wu

title: Flow contrastive estimation of energy-based models

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Partha Ghosh
Mehdi S.M. Sajjadi
Antonio Vergari
Michael Black
Bernhard

title: Scholkopf. From variational to deterministic autoencoders

authors:
Karol Gregor
Frederic Besse
Danilo Jimenez Rezende
Ivo Danihelka
Daan Wierstra

title: Towards conceptual compression

venue: In Advances In Neural Information Processing Systems,

authors:
Ishaan Gulrajani
Kundan Kumar
Faruk Ahmed
Adrien Ali Taiga
Francesco Visin
David Vazquez
Aaron Courville

title: PixelVAE: A latent variable model for natural images

venue: arXiv preprint arXiv:1611.05013,

authors:
Michael Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

venue: In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,

authors:
Tian Han
Erik Nijkamp
Xiaolin Fang
Mitch Hill
Song-Chun Zhu
Ying Nian Wu

title: Divergence triangle for joint training of generator model, energy-based model, and inferential model

authors:
Tian Han
Erik Nijkamp
Linqi Zhou
Bo Pang
Song-Chun Zhu
Ying Nian Wu

title: Joint training of variational auto-encoder and latent energy-based model

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

venue: In Advances in neural information processing systems,

title: Training products of experts by minimizing contrastive divergence

venue: Neural computation,

authors:
Geoffrey E Hinton
Terrence J Sejnowski

title: Learning and relearning in boltzmann machines. Parallel distributed processing: Explorations in the microstructure of cognition

authors:
Geoffrey E Hinton
Simon Osindero
Yee-Whye Teh

title: A fast learning algorithm for deep belief nets

authors:
Jonathan Ho
Ajay Jain
Pieter Abbeel

title: Denoising diffusion probabilistic models

venue: arXiv preprint arxiv:2006.11239,

authors:
Matthew D Hoffman
Matthew J Johnson

title: Elbo surgery: yet another way to carve up the variational evidence lower bound

venue: In Workshop in Advances in Approximate Bayesian Inference,

authors:
Jie Hu
Li Shen
Gang Sun

title: Squeeze-and-excitation networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: In International Conference on Machine Learning,

authors:
Pavel Izmailov
Polina Kirichenko
Marc Finzi
Andrew Gordon Wilson

title: Semi-supervised learning with normalizing flows

venue: In ICML,

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

venue: arXiv preprint arXiv:1710.10196,

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

authors:
Tero Karras
Miika Aittala
Janne Hellsten
Samuli Laine
Jaakko Lehtinen
Timo Aila

title: Training generative adversarial networks with limited data

venue: arXiv preprint arXiv:2006.06676,

authors:
Diederik Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: In The International Conference on Learning Representations (ICLR),

authors:
Diederik P Kingma
Shakir Mohamed
Danilo Jimenez Rezende
Max Welling

title: Semi-supervised learning with deep generative models

authors:
Diederik P Kingma
Tim Salimans
Rafal Jozefowicz
Xi Chen
Ilya Sutskever
Max Welling

title: Improved variational inference with inverse autoregressive flow

authors:
Durk P Kingma
Prafulla Dhariwal

title: Glow: Generative flow with invertible 1x1 convolutions

authors:
Alexej Klushyn
Nutan Chen
Richard Kurle
Botond Cseke
Patrick van der Smagt

title: Learning hierarchical priors in vaes

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny

authors:
Hugo Larochelle
Yoshua Bengio

title: Classification using discriminative restricted boltzmann machines

venue: In Proceedings of the 25th international conference on Machine learning,

authors:
John Lawson
George Tucker
Bo Dai
Rajesh Ranganath

title: Energy-inspired models: Learning with sampler-induced distributions

title: The mnist database of handwritten digits. http://yann

venue: lecun. com/exdb/mnist/,

authors:
Chieh Hubert Lin
Chia-Che Chang
Yu-Sheng Chen
Da-Cheng Juan
Wei Wei
Hwann-Tzong Chen

title: Coco-gan: generation by parts via conditional coordinating

authors:
Ziwei Liu
Ping Luo
Xiaogang Wang
Xiaoou Tang

title: Deep learning face attributes in the wild

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Ilya Loshchilov
Frank Hutter

title: Sgdr: Stochastic gradient descent with warm restarts

venue: arXiv preprint arXiv:1608.03983,

authors:
Lars Maaløe
Marco Fraccaro
Valentin Liévin
Ole Winther

title: BIVA: A very deep hierarchy of latent variables for generative modeling

authors:
Lars Mescheder
S Nowozin
Andreas Geiger

title: Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks

venue: In 34th International Conference on Machine Learning (ICML),

authors:
Charlie Nash
Conor Durkan

title: Autoregressive energy machines

venue: arXiv preprint arXiv:1904.05626,

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Monte Carlo theory, methods and examples

authors:
Bo Pang
Tian Han
Erik Nijkamp
Song-Chun Zhu
Ying Nian Wu

title: Learning latent space energy-based prior model

venue: arXiv preprint arXiv:2006.08205,

authors:
Kancharla Parimala
Sumohana Channappayya

title: Quality aware generative adversarial networks

authors:
Stanislav Pidhorskyi
Donald A Adjeroh
Gianfranco Doretto

title: Adversarial latent autoencoders

authors:
Wei Ping
Kainan Peng
Kexin Zhao
Zhao Song

title: Waveflow: A compact flow-based model for raw audio

authors:
Prajit Ramachandran
Barret Zoph
Quoc V Le

title: Searching for activation functions

venue: arXiv preprint arXiv:1710.05941,

authors:
Ali Razavi
Aaron van den Oord
Oriol Vinyals

title: Generating diverse high-fidelity images with vq-vae-2

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

title: Stochastic backpropagation and approximate inference in deep generative models

title: Discrete variational autoencoders

venue: arXiv preprint arXiv:1609.02200,

authors:
Mihaela Rosca
Balaji Lakshminarayanan
Shakir Mohamed

title: Distribution matching in variational inference

venue: arXiv preprint arXiv:1802.06847,

authors:
Ruslan Salakhutdinov
Geoffrey Hinton

title: Deep boltzmann machines

venue: In Artificial intelligence and statistics,

authors:
Casper Kaae Sønderby
Tapani Raiko
Lars Maaløe
Søren Kaae Sønderby
Ole Winther

title: Ladder variational autoencoders

authors:
Hiroshi Takahashi
Tomoharu Iwata
Yuki Yamanaka
Masanori Yamada
Satoshi Yagi

title: Variational autoencoder with implicit optimal priors

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
I Tolstikhin
O Bousquet
S Gelly
B Schölkopf

title: Wasserstein auto-encoders

venue: In International Conference on Learning Representations (ICLR 2018). OpenReview

authors:
Jakub Tomczak
Max Welling

title: Vae with a vampprior

venue: In International Conference on Artificial Intelligence and Statistics,

authors:
Ryan Turner
Jane Hung
Eric Frank
Yunus Saatchi
Jason Yosinski

title: Metropolis-hastings generative adversarial networks

authors:
Arash Vahdat
Jan Kautz

title: NVAE: A deep hierarchical variational autoencoder

venue: In Neural Information Processing Systems (NeurIPS),

authors:
Arash Vahdat
Evgeny Andriyash
William G Macready

title: DVAE#: Discrete variational autoencoders with relaxed Boltzmann priors

venue: In Neural Information Processing Systems,

authors:
Arash Vahdat
William G. Macready
Zhengbing Bian
Amir Khoshaman
Evgeny Andriyash

title: DVAE++: Discrete variational autoencoders with overlapping transformations

venue: In International Conference on Machine Learning (ICML),

title: Undirected graphical models as approximate posteriors

authors:
Aäron Van Den Oord
Nal Kalchbrenner
Koray Kavukcuoglu

title: Pixel recurrent neural networks

venue: In Proceedings of the 33rd International Conference on International Conference on Machine Learning,

authors:
Aaron Van Den Oord
Oriol Vinyals

title: Neural discrete representation learning

authors:
Haowen Xu
Wenxiao Chen
Jinlin Lai
Zhihan Li
Youjian Zhao
Dan Pei

title: On the necessity and effectiveness of learning the prior of variational auto-encoder

authors:
Zijun Zhang
Ruixiang Zhang
Zongpeng Li
Yoshua Bengio
Liam Paull

title: Perceptual generative autoencoders

authors:
Adam (Kingma
Ba

title: Learning Rate Initialize at 1e-3, CosineAnnealing (Loshchilov & Hutter, 2016

text:
1 INTRODUCTION q(z)
p(z)
r(z)
Figure 1: We propose an EBM prior using the product of a base prior p(z) and a reweighting factor r(z), designed to bring the base prior closer to the aggregate posterior q(z).
Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) are one of the powerful likelihood-based generative models that have applications in image generation (Brock et al., 2018; Karras et al., 2019; Razavi et al., 2019), music synthesis (Dhariwal et al., 2020), speech generation (Oord et al., 2016; Ping et al., 2020), image captioning (Aneja et al., 2019; Deshpande et al., 2019; Aneja et al., 2018), semi-supervised learning (Kingma et al., 2014; Izmailov et al., 2020), and representation learning (Van Den Oord et al., 2017; Fortuin et al., 2018). Although there has been tremendous progress in improving the expressivity of the approximate posterior, several studies have observed that VAE priors fail to match the aggregate (approximate) posterior (Rosca et al., 2018; Hoffman & Johnson, 2016). This phenomenon is sometimes described as holes in the prior, referring to regions in the latent space that are not decoded to data-like samples. Such regions often have a high density under the prior but have a low density under the aggregate approximate posterior.
The prior hole problem is commonly tackled by increasing the flexibility of the prior via hierarchical priors (Klushyn et al., 2019), autoregressive models (Gulrajani et al., 2016), a mixture of approximate posteriors (Tomczak & Welling, 2018), normalizing flows (Xu et al., 2019; Chen et al., 2016), resampled priors (Bauer & Mnih, 2019), and energy-based models (Pang et al., 2020; Vahdat et al., 2018b;a; 2020). Among them, energy-based models (EBMs) (Du & Mordatch, 2019; Pang et al., 2020) have shown promising results in learning expressive priors. However, they require running iterative MCMC steps during training which is computationally expensive, especially when the energy function is represented by a
neural network. Moreover, they scale poorly to hierarchical models where an EBM is defined on each group of latent variables.
Our key insight in this paper is that a trainable prior is brought as close as possible to the aggregate posterior as a result of training a VAE. The mismatch between the prior and the aggregate posterior can be reduced by simply reweighting the prior to re-adjust its likelihood in the area of mismatch with the aggregate posterior. To represent this reweighting mechanism, we formulate the prior using an EBM that is defined by the product of a reweighting factor and a base trainable prior as shown in Fig. 1. We represent the reweighting factor using neural networks and the base prior using Normal distributions.
Instead of expensive MCMC sampling, we use noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010) for training the EBM prior. We show that NCE naturally trains the reweighting factor in our prior by learning a binary classifier to distinguish samples from a target distribution (i.e., samples from the approximate posterior) vs. samples from a noise distribution (i.e., the base trainable prior). However, since NCE’s success depends on how close the noise distribution is to the target distribution, we first train the VAE with the base prior to bring it close to the aggregate posterior. And then, we train the EBM prior using NCE.
In this paper, we make the following contributions: i) We propose an EBM prior termed noise contrastive prior (NCP) which is trained by contrasting samples from the aggregate posterior to samples from a base prior. NCPs are learned as a post-training mechanism to replace the original prior with a more flexible prior, which can improve the generative performance of VAEs with any structure. ii) We also show how NCPs are trained on hierarchical VAEs with many latent variable groups. We show that training hierarchical NCPs scales easily to many groups, as they are trained for each latent variable group in parallel. iii) Finally, we demonstrate that NCPs improve the generative quality of VAEs by a large margin across datasets.

heading: 2 BACKGROUND

text:
We first review VAEs, their extension to hierarchical VAEs, and the prior hole problem.
Variational Autoencoders: VAEs learn a generative distribution p(x, z) = p(z)p(x|z) where p(z) is a prior distribution over the latent variable z and p(x|z) is a likelihood function that generates the data x given z. VAEs are trained by maximizing a variational lower bound on the log-likelihood log p(x):
log p(x) ≥ Ez∼q(z|x)[log p(x|z)]− KL(q(z|x)||p(z)) := LVAE(x), (1) where q(z|x) is an approximate posterior and KL is the Kullback–Leibler divergence. The final training objective is formulated by Epd(x)[LVAE(x)] where pd(x) is the data distribution (Kingma & Welling, 2014). Hierarchical VAEs (HVAEs): To increase the expressivity of both prior and approximate posterior, earlier work adapted a hierarchical latent variable structure (Vahdat & Kautz, 2020; Kingma et al., 2016; Sønderby et al., 2016; Gregor et al., 2016). In HVAEs, the latent variable z is divided into K separate groups, z = {z1, . . . , zK}. The approximate posterior and the prior distributions are then defined by q(z|x) = ∏K k=1 q(zk|z<k,x) and p(z) = ∏K k=1 p(zk|z<k). Using these, the training objective becomes:
LHVAE(x) := Eq(z|x)[log p(x|z)]− K∑
k=1
Eq(z<k|x) [KL(q(zk|z<k,x)||p(zk|z<k))] , (2)
where q(z<k|x) = ∏k−1 i=1 q(zi|z<i,x) is the approximate posterior up to the (k − 1)th group1.
The Prior Hole Problem: Let q(z) , Epd(x)[q(z|x)] denote the aggregate (approximate) posterior. In Appendix B.1, we show that maximizing Epd(x)[LVAE(x)] with respect to the prior parameters corresponds to bringing the prior as close as possible to the aggregate posterior by minimizing KL(q(z)||p(z)) w.r.t. p(z). Formally, the prior hole problem refers to the phenomenon that p(z) fails to match q(z).
1For k = 1, the expectation inside the summation is simplified to KL(q(z1|x)||p(z1)).

heading: 3 NOISE CONTRASTIVE PRIORS (NCPS)

text: One of the main causes of the prior hole problem is the limited expressivity of the prior that prevents it from matching the aggregate posterior. Recently, energy-based models have shown promising results in representing complex distributions. Motivated by their success, we introduce the noise contrastive prior (NCP) pNCP(z) = 1Z r(z)p(z), where p(z) is a base prior distribution, e.g., a Normal, r(z) is a reweighting factor, and Z = ∫ r(z)p(z)dz is the normalization constant. The function r : Rn → R+ maps the latent variable z ∈ Rn to a positive scalar, and can be implemented using neural nets. The reweighting factor r(z) can be trained using MCMC sampling as discussed in Appendix A. However, MCMC requires expensive sampling iterations that scale poorly to hierarchical VAEs. To address this, we describe a noise contrastive estimation based approach to train pNCP(z) without MCMC sampling.

heading: 3.1 LEARNING THE REWEIGHTING FACTOR WITH NOISE CONTRASTIVE ESTIMATION

text:
Recall that training VAEs closes the gap between the prior and the aggregate posterior by minimizing KL(q(z)||p(z)) with respect to prior. Assuming the base prior p(z) to be fixed, KL(q(z)||pNCP(z)) is zero when r(z) = q(z)/p(z). However, since we do not have the density function for q(z), we cannot compute the ratio explicitly. Instead, in this paper, we propose to estimate r(z) using noise contrastive estimation (Gutmann & Hyvärinen, 2010), also known as the likelihood ratio trick that has been popularized in machine learning by predictive coding (Oord et al., 2018) and generative adversarial networks (GANs) (Goodfellow et al., 2014). Since, we can generate samples from both p(z) and q(z)2, we train a binary classifier to distinguish samples from q(z) and samples from the base prior p(z) by minimizing the binary cross-entropy loss:
min D − Ez∼q(z)[logD(z)]− Ez∼p(z)[log(1−D(z))]. (3)
Here, D : Rn → (0, 1) is a binary classifier that generates the classification prediction probabilities. Eq. (3) is minimized when D(z) = q(z)q(z)+p(z) . Denoting the classifier at optimality by D ∗(z), we estimate the reweighting factor r(z) = q(z)p(z) ≈ D∗(z)
1−D∗(z) . The appealing advantage of this estimator is that it is obtained by simply training a binary classifier rather than using expensive MCMC sampling.

heading: 3.2 TWO-STAGE TRAINING FOR NOISE CONTRASTIVE PRIORS

text:
To properly learn the reweighting factor, NCE training requires the base prior distribution to be close to the target distribution. Intuitively, if p(z) is very close to q(z) (i.e., p(z) ≈ q(z)), the optimal classifier will have a large loss value in Eq. (3), and we will have r(z) ≈ 1. If p(z) is instead far from q(z), the binary classifier will easily learn to distinguish samples from the two distributions and it will not learn the likelihood ratios correctly. If p(z) is roughly close to q(z), then the binary classifier can learn the ratios.
To ensure that the base prior distribution is close to the target aggregate posterior distribution, we propose a two-stage training algorithm. In the first stage, we train the VAE with only the base prior p(z). From Appendix B.1, we know that at the end of training, p(z) is as close as possible to q(z). In the second stage, we freeze the VAE model including the approximate posterior q(z|x), the base prior p(z), and the likelihood p(x|z), and we only train the reweighting factor r(z) using Eq. (3). The second stage can be thought of as replacing the base distribution p(z) with a more expressive distribution of the form pNCP(z) ∝ r(z)p(z). Hence, NCP matches the prior to the aggregate posterior q(z) by using the reweighting factors. Note that our proposed method is generic as it only assumes that we can draw samples from q(z) and p(z), which applies to any VAE. Our training is illustrated in Fig. 2.
2We generate samples from the aggregate posterior q(z) = Epd(x)[q(z|x)] via ancestral sampling: draw data from the training set (x ∼ pd(x)) and then sample from z ∼ q(z|x).

heading: 3.3 TEST TIME SAMPLING

text:
To sample from a VAE with an NCP, we first generate samples from the NCP before passing them to the decoder to generate output samples (shown in Fig. 2). We propose two methods for sampling from NCPs.
Sampling-Importance-Resampling (SIR): We first generate M samples from the base prior distribution {z(m)}Mm=1 ∼ p(z). We then resample one of the M proposed samples using importance weights proportional to w(m) = pNCP(z(m))/p(z(m)) = r(z(m)). The benefit of this technique: both proposal generation and the evaluation of r on the samples are done in parallel.
Langevin Dynamics (LD): Since our NCP is an EBM, we can use LD for sampling. Denoting the energy function by E(z) = − log r(z) − log p(z), we initialize a sample z0 by drawing from p(z) and update the sample iteratively using: zt+1 = zt − 0.5 λ∇zE(z) + √ λ t where t ∼ N (0, 1) and λ is the step size. LD is run for a finite number of iterations, and in contrast to SIR, it can be slow given its sequential form.

heading: 3.4 GENERALIZATION TO HIERARCHICAL VAES

text:
The state-of-the-art VAE (Vahdat & Kautz, 2020) uses a hierarchical q(z|x) and p(z). Here p(z) is chosen to be a Gaussian distribution. Appendix B.2 shows that training a HVAE encourages the prior to minimize Eq(z<k) [KL(q(zk|z<k)||p(zk|z<k))] for each conditional, where q(z<k) , Epd(x)[q(z<K |x)] is the aggregate posterior up to the (k − 1)th group, and q(zk|z<k) , Epd(x)[q(zk|z<k,x)] is the aggregate conditional for the kth group. Given this observation, we extend NCPs to hierarchical models to match each conditional in the prior with q(zk|z<k). Formally, we define hierarchical NCPs by pNCP(z) = 1 Z ∏K k=1 r(zk|z<k)p(zk|z<k) where each factor is an EBM. pNCP(z) resembles energy machines with an autoregressive structure among groups (Nash & Durkan, 2019).
In the first stage, we train the HVAE with prior ∏K
k=1 p(zk|z<k). For the second stage, we use K binary classifiers, each for a hierarchical group. Following Appendix C, we train each classifier via:
min Dk
Epd(x)q(z<k|x) [ − Eq(zk|z<k,x)[logDk(zk, c(z<k))]− Ep(zk|z<k)[log(1−Dk(zk, c(z<k)))] ] , (4)
where the outer expectation samples from groups up to the (k−1)th group, and the inner expectations sample from approximate posterior and base prior for the kth group, conditioned on the same z<k. The discriminator Dk classifies samples zk while conditioning its prediction on z<k using a shared context feature c(z<k).
The NCE training in Eq. (4) is minimized when Dk(zk, c(z<k)) = q(zk|z<k)
q(zk|z<k)+p(zk|z<k) . Denoting the
classifier at optimality by D∗k(z, c(z<k)), we obtain the reweighting factor r(zk|z<k) ≈ D∗k(zk,c(z<k))
1−D∗k(zk,c(z<k)) in the second stage. Given our hierarchical NCP, we use ancestral sampling to sample from the prior. For sampling from each group, we can use SIR or LD as discussed before.
The context feature c(z<k) extracts a representation from z<k. Instead of learning a new representation at stage two, we simply use the representation that is extracted from z<k in the hierarchical prior, trained in the first stage. Note that the binary classifiers in the second stage are trained in parallel for all groups.

heading: 4 RELATED WORK

text:
In this section, we review prior works related to the proposed method.
Energy-based Models (EBMs): Early work on EBMs for generative learning goes back to 1980s (Ackley et al., 1985; Hinton et al., 1986). Prior to the modern deep learning era, most attempts for building generative models using EBMs were centered around Boltzmann machines (Hinton, 2002; Hinton et al., 2006) and their “deep” extensions (Salakhutdinov & Hinton, 2009; Larochelle & Bengio, 2008). Although the energy function in these models is restricted to simple bilinear functions, they have been proven effective for representing the prior in discrete VAEs (Rolfe, 2016; Vahdat et al., 2018a;b; 2020). Recently, EBMs with neural energy functions have gained popularity for representing complex data distribution (Du & Mordatch, 2019). Pang et al. (2020) have shown that neural EBMs can represent expressive prior distributions. However, in this case, the prior is trained using MCMC sampling, and it has been limited to a single group of latent variables. To eliminate MCMC sampling, NCE (Gutmann & Hyvärinen, 2010) has recently been used for training a normalizing flow on data distributions (Gao et al., 2020). Moreover, Han et al. (2019; 2020) have used divergence triangulation to sidesteps MCMC sampling. In contrast, we use NCE to train an EBM prior where a noise distribution is easily available through a pre-trained VAE.
Adversarial Training: Similar to NCE, generative adversarial networks (GANs) (Goodfellow et al., 2014) also rely on a discriminator to learn the likelihood ratio between noise and real images. However, GANs use the discriminator to update the generator, whereas in NCE, the noise generator is fixed. In spirit similar are recent works (Azadi et al., 2018; Turner et al., 2019; Che et al., 2020) that link GANs, defined in the pixels space, to EBMs. We apply the likelihood ratio trick to the latent space of VAEs. The main difference: the base prior and approximate posterior are trained with the VAE objective rather than the adversarial loss. Adversarial loss has been used for training implicit encoders in VAEs (Makhzani et al., 2015; Mescheder et al., 2017; Engel et al., 2018). But, they have not been linked to energy-based priors as we do explicitly.
Prior Hole Problem: Among prior works on this problem, VampPrior (Tomczak & Welling, 2018) uses a mixture of encoders to represent the prior. However, this requires storing training data or pseudo-data to generate samples at test time. Takahashi et al. (2019) use the likelihood ratio estimator to train a simple prior distribution. However at test time, the aggregate posterior is used for sampling in the latent space. Bauer & Mnih (2019) propose a reweighting factor similar to ours, but it is trained via importance sampling. Recently, Lawson et al. (2019) introduced energy-inspired models (EIMs) that define distributions induced by the sampling processes used by (Bauer & Mnih, 2019) as well as our SIR sampling. Although, EIMs have the advantage of end-to-end training and can be used as either prior or decoder in VAEs, they require multiple samples during training (up to 1K). This can make application of EIMs to deep hierarchical models such as NVAEs very challenging as these models are often very memory intensive and are trained with a few training samples per GPU.
Two-stage VAEs: VQ-VAE (Van Den Oord et al., 2017; Razavi et al., 2019) first trains an autoencoder and then fits an autoregressive PixelCNN (Van Den Oord et al., 2016) prior to the latent variables which is slow to sample from. Two-stage VAE (2s-VAE) (Dai & Wipf, 2018) trains a VAE on the data, and then, trains another VAE in the latent space. Regularized autoencoders (RAE) (Ghosh et al., 2020) train an autoencoder, and subsequently a Gaussian mixture model on latent codes. In contrast, we train the model with the original VAE objective in the first stage, and we improve the expressivity of the prior using an EBM.

heading: 5 EXPERIMENTS

text: Our implementation of NCP-VAE builds upon NVAE (Vahdat & Kautz, 2020), the state-of-the-art hierarchical VAE. We examine NCP-VAE on four datasets including dynamically binarized MNIST (LeCun, 1998), CIFAR-10 (Krizhevsky et al., 2009), CelebA-64 (Liu et al., 2015) and CelebA-HQ-256 (Karras et al., 2017). For CIFAR-10 and CelebA-64, the model has 30 groups, and for CelebA-HQ-256 it has 20 groups. We sample from NCP-VAE using SIR with 5K proposal samples at the test (image generation) time. On these datasets, we measure the sample quality using the Fréchet Inception Distance (FID) score (Heusel et al., 2017) with 50,000 samples, as computing the log-likelihood requires estimating the intractable normalization constant. To report log-likelihood results, we train an NVAE model with a small latent space on MNIST with 10 groups of 4 × 4 latent variables. We estimate log normalization constant in NCPs using 1000 importance weighted samples. We intentionally limit the latent space to ensure that we can estimate the normalization constant correctly (standard deviation of logZ estimation ≤ 0.23). Thus, on this dataset, we report negative log-likelihood (NLL). Implementation details are provided in Appendix E.

heading: 5.1 QUANTITATIVE RESULTS

text:
The quantitative results are reported in Tab. 1, Tab. 2, Tab. 3, and Tab. 4. On all four datasets, our model improves upon state-of-the-art NVAE, and it reduces the gap with GANs by a large margin. On CelebA 64, we improve NVAE from an FID of 13.48 to 5.25, comparable to GANs. On CIFAR-10, NCP-VAE
Table 1: Generative performance on CelebA-64
Model FID↓ NCP-VAE (ours) 5.25 NVAE (Vahdat & Kautz, 2020) 13.48 RAE (Ghosh et al., 2020) 40.95 2s-VAE (Dai & Wipf, 2018) 44.4 WAE (Tolstikhin et al., 2018) 35 Perceptial AE(Zhang et al., 2020) 13.8 Latent EBM (Pang et al., 2020) 37.87 COCO-GAN (Lin et al., 2019) 4.0 QA-GAN (Parimala & Channappayya, 2019) 6.42 NVAE-Recon (Vahdat & Kautz, 2020) 1.03
Table 3: Generative results on CelebA-HQ-256
Model FID↓ NCP-VAE (ours) 24.79 NVAE (Vahdat & Kautz, 2020) 40.26 GLOW (Kingma & Dhariwal, 2018) 68.93 Advers. LAE (Pidhorskyi et al., 2020) 19.21 PGGAN (Karras et al., 2017) 8.03 NVAE-Recon (Vahdat & Kautz, 2020) 0.45
Table 4: Likelihood results on MNIST in nats
Model NLL↓ NCP-VAE (ours) 78.10 NVAE-small (Vahdat & Kautz, 2020) 78.67 BIVA (Maaløe et al., 2019) 78.41 DAVE++ (Vahdat et al., 2018b) 78.49 IAF-VAE (Kingma et al., 2016) 79.10 VampPrior AR dec. (Tomczak & Welling) 78.45
improves the NVAE FID of 51.71 to 24.08. On MNIST, although our latent space is much smaller, our model outperforms previous VAEs. NVAE has reported 78.01 nats on this dataset with a larger latent space.
Reconstruction: In the last row of Tab. 1, Tab. 2, and Tab. 3, we report the FID score for the reconstructed images of the NVAE baseline. Note how reconstruction FID is much lower than our FID, indicating that our model is far from memorizing the training data. In Appendix G, we also provide nearest neighbours from training data for generated samples. In Appendix F, we present our NCP applied to vanilla VAEs.

heading: 5.2 QUALITATIVE RESULTS

text: We visualize samples generated by NCP-VAE in Fig. 3 without any manual intervention. We adopt the common practice of reducing the temperature of the base prior p(z) by scaling down the standard-deviation of the conditional Normal distributions (Kingma & Dhariwal, 2018)3. Brock et al. (2018); Vahdat & Kautz (2020) also observe that re-adjusting the batch-normalization (BN), given a temperature applied to the prior, improves the generative quality. Similarly, we achieve diverse, high-quality images by re-adjusting the BN statistics as described by Vahdat & Kautz (2020). Additional qualitative results are shown in Appendix H.

heading: 5.3 ADDITIONAL STUDIES

text:
We perform additional experiments to study i) how hierarchical NCPs perform as the number of latent groups increases, ii) the impact of SIR and LD hyperparameters, and iii) what the classification loss in NCE training conveys about p(z) and q(z). All experiments in this section are performed on CelebA-64 data.
3Lowering of the temperature is only used to obtain qualitative samples. It’s not used when computing any of the quantitative results in Sec. 5.1.
Number of latent variable groups: Tab. 5 shows the generative performance of hierarchical NCP with different amounts of latent variable groups. As we increase the number of groups, the FID score of both NVAE and our model improves. This demonstrates the efficacy of our NCPs, even with expressive hierarchical priors and in the presence of many groups.
SIR and LD parameters: The computationally complexity of SIR is similar to LD if we set the number of proposal samples in SIR equal to the number LD iterations. Tab. 6 reports the impact of these parameters. We observe that increasing both the number of proposal samples in SIR and the LD iterations leads to a noticeable improvement in FID score. For SIR, the proposal generation and the evaluation of r(z) are parallelizable. Hence, as shown in Tab. 6, image generation is faster with SIR than with LD (LD iterations are sequential). However, GPU memory usage scales with the number of SIR proposals, but not with the number of LD iterations. Interestingly, SIR, albeit simple, performs better than LD when using about the same compute.
Classification loss in NCE: We can draw a direct connection between the classification loss in Eq. 3 and the similarity of p(z) and q(z). Denoting the classification loss in Eq. 3 at optimality by L∗, Goodfellow et al. (2014) show that JSD(p(z)||q(z)) = log 2 − 0.5 × L∗ where JSD denotes the Jensen–Shannon divergence between two distributions. Fig. 4(a) plots the classification loss (Eq. 4) for each classifier for a 15-group NCP trained on the CelebA-64 dataset. Assume that the classifier loss at the end of training is a good approximation of L∗. We observe that 8 out of 15 groups have L∗ ≥ 0.4, indicating a good overlap between p(z) and q(z) for those groups. To further assess the impact of the distribution match on SIR sampling, in Fig. 4(b), we visualize the effective sample size (ESS)4 in SIR vs. L∗ for the same group. We observe a strong correlation between L∗ and the effective sample size. SIR is more reliable on the same 8 groups that have high classification loss. These groups are mostly at the top of the NVAE hierarchy which have been shown to control the global structure of generated samples (see B.6 in Vahdat & Kautz (2020)).
4ESS measures reliability of SIR via 1/ ∑
m(ŵ (m))2, where ŵ(m) = r(z(m))/ ∑ m′ r(z (m′)) (Owen, 2013).

heading: 5.4 ADDITIONAL COMPARISON TO PRIOR ART

text:
In our previous experiments, we rely on NVAE as the VAE backbone. Although this shows that our approach can be applied to large scale models successfully, this may make comparison against the prior art unfair as they are often applied to smaller models. To provide a fair comparison, in this section, we apply NCP to several commonly used small VAE models.
Comparison against RAE, 2s-VAE, and WAE: In Tab. 7 we show the generative performance of our approach applied to the VAE architecture in RAE (Ghosh et al., 2020). Note that this VAE architecture has only one latent variable group. The same base architecture was used in the implementation of 2s-VAE (Dai & Wipf, 2018) and WAE (Tolstikhin et al., 2018). We borrow the training setup from Ghosh et al. (2020) on the CelebA-64 dataset, and compare to the baselines reported in this work. We apply our NCP-VAE on top of vanilla VAE with a Gaussian prior as well as a 10-component Gaussian mixture model (GMM) prior that was proposed in RAEs. Our NCP-VAE improves the performance on the base VAE, improving the FID score to 41.28 from 48.12. Additionally, when NCP is applied to the VAE with GMM prior (the RAE model), it improves its performance from 40.95 to the FID score of 39.00.
Comparison against LARS and SNIS: In Tab. 8, we compare NCP-VAE to LARS (Bauer & Mnih, 2019) and SNIS (Lawson et al., 2019) priors on MNIST. We implement our NCP-VAE using the VAE and energyfunction architectures that were used by Lawson et al. (2019). We closely follow the training hyperparameters used by Lawson et al. (2019) as well as their approach for obtaining a lower bound on the log likelihood. NCP-VAE obtains the negative log-likelihood (NLL) of 82.82, comparable to Lawson et al. (2019), while outperforming LARS (Bauer & Mnih, 2019).

heading: 6 CONCLUSIONS

text: The prior hole problem is one of the main reasons for VAEs’ poor generative quality. In this paper, we tackled this problem by introducing the noise contrastive prior (NCP), defined by the product of a reweighting factor and a base prior. We showed how the reweighting factor is trained by contrasting samples from the aggregate posterior with samples from the base prior. Our proposal is simple and can be applied to any VAE to increase its prior’s expressivity. We also showed how NCP training scales to large hierarchical VAEs, as it can be done in parallel simultaneously for all the groups. Finally, we demonstrated that NCPs improve the generative performance of state-of-the-art NVAEs by a large margin, closing the gap to GANs.

heading: A TRAINING ENERGY-BASED PRIORS USING MCMC

text:
In this section, we show how a VAE with energy-based model in its prior can be trained. Assuming that the prior is in the form pEBM(z) = 1Z r(z)p(z), the variational bound is of the form:
Epd(x)[LVAE] = Epd(x) [ Eq(z|x)[log p(x|z)]− KL(q(z|x)||pEBM(z)) ] = Epd(x) [ Eq(z|x)[log p(x|z)− log q(z|x) + log r(z) + log p(z)] ] − logZ,
where the expectation term, similar to VAEs, can be trained using the reparameterization trick. The only problematic term is the log-normalization constant logZ, which captures the gradient with respect to the parameters of the prior pEBM(z). Denoting these parameters by θ, the gradient of logZ is obtained by:
∂
∂θ logZ =
1
Z
∫ ∂(r(z)p(z))
∂θ dz =
∫ r(z)p(z)
Z
∂ log(r(z)p(z))
∂θ dz = EPEBM (z)[
∂ log(r(z)p(z))
∂θ ], (5)
where the expectation can be estimated using MCMC sampling from the EBM prior.

heading: B MAXIMIZING THE VARIATIONAL BOUND FROM THE PRIOR’S PERSPECTIVE

text:
In this section, we discuss how maximizing the variational bound in VAEs from the prior’s perspective corresponds to minimizing a KL divergence from the aggregate posterior to the prior. Note that this relation has been explored by Hoffman & Johnson (2016); Rezende & Viola (2018); Tomczak & Welling (2018) and we include it here for completeness.
B.1 VAE WITH A SINGLE GROUP OF LATENT VARIABLES
Denote the aggregate (approximate) posterior by q(z) , Epd(x)[q(z|x)]. Here, we show that maximizing the Epd(x)[LVAE(x)] with respect to the prior parameters corresponds to learning the prior by minimizing KL(q(z)||p(z)). To see this, note that the prior p(z) only participates in the KL term in LVAE (Eq. 1). We hence have:
argmax p(z) Epd(x)[LVAE(x)] = argmin p(z) Epd(x)[KL(q(z|x)||p(z))]
= argmin p(z) −Epd(x)[H(q(z|x))]− Eq(z)[log p(z)]
= argmin p(z)
−H(q(z))− Eq(z)[log p(z)]
= argmin p(z)
KL(q(z)||p(z)),
where H(.) denotes the entropy. Above, we replaced the expected entropy Epd(x)[H(q(z|x))] with H(q(z)) as the minimization is with respect to the parameters of the prior p(z).
B.2 HIERARCHICAL VAES Denote hierarchical approximate posterior and prior distributions by: q(z|x) = ∏K
k=1 q(zk|z<k,x) and p(z) = ∏K k=1 p(zk|z<k). The hierarchical VAE objective becomes:
LHVAE(x) = Eq(z|x)[log p(x|z)]− K∑
k=1
Eq(z<k|x) [KL(q(zk|z<k,x)||p(zk|z<k))] , (6)
where q(z<k|x) = ∏k−1
i=1 q(zi|z<i,x) is the approximate posterior up to the (k − 1)th group. Denote the aggregate posterior up to the (K − 1)th group by q(z<k) , Epd(x)[q(z<K |x)] and the aggregate conditional for the kth group given the previous groups q(zk|z<k) , Epd(x)[q(zk|z<k,x)].
Here, we show that maximizing Epd(x)[LHVAE(x)] with respect to the prior corresponds to learning the prior by minimizing Eq(z<k) [KL(q(zk|z<k)||p(zk|z<k))] for each conditional:
argmax p(zk|z<k) Epd(x)[LHVAE(x)] = argmin p(zk|z<k)
Epd(x) [ Eq(z<k|x) [KL(q(zk|z<k,x)||p(zk|z<k))] ] = argmin
p(zk|z<k) −Epd(x)q(z<k|x)q(zk|z<k,x) [log p(zk|z<k)]
= argmin p(zk|z<k) −Eq(zk,z<k) [log p(zk|z<k)]
= argmin p(zk|z<k)
−Eq(z<k) [ Eq(zk|z<k) [log p(zk|z<k)] ] = argmin
p(zk|z<k) Eq(z<k)
[ −H(q(zk|z<k))− Eq(zk|z<k) [log p(zk|z<k)] ] = argmin
p(zk|z<k) Eq(z<k) [KL(q(zk|z<k)||p(zk|z<k))] . (7)

heading: C CONDITIONAL NCE FOR HIERARCHICAL VAES

text:
In this section, we describe how we derive the NCE training objective for hierarchical VAEs given in Eq. (4). Our goal is to learn the likelihood ratio between the aggregate conditional q(zk|z<k) and the prior p(zk|z<k). We can define the NCE objective to train the discriminator Dk(zk, z<k) that classifies zk given samples from the previous groups z<k using:
min Dk − Eq(zk|z<k)[logDk(zk, z<k)]− Ep(zk|z<k)[log(1−Dk(zk, z<k))] ∀z<k. (8)
Since z<k is in a high dimensional space, we cannot apply the minimization ∀z<k. Instead, we sample from z<k using the aggregate approximate posterior q(z<k) as done for the KL in a hierarchical model (Eq. (7)):
min Dk
Eq(z<k) [ − Eq(zk|z<k)[logDk(zk, z<k)]− Ep(zk|z<k)[log(1−Dk(zk, z<k))] ] . (9)
Since q(z<k)q(zk|z<k) = q(zk, z<k) = Epd(x)[q(z<k|x)q(zk|z<k,x)], we have:
min Dk
Epd(x)q(z<k|x) [ − Eq(zk|z<k,x)[logDk(zk, z<k)]− Ep(zk|z<k)[log(1−Dk(zk, z<k))] ] . (10)
Finally, instead of passing all the samples from the previous latent variables groups to D, we can pass the context feature c(z<k) that extracts a representation from all the previous groups:
min Dk
Epd(x)q(z<k|x) [ − Eq(zk|z<k,x)[logDk(zk, c(z<k))]− Ep(zk|z<k)[log(1−Dk(zk, c(z<k)))] ] . (11)

heading: D NVAE BASED MODEL AND CONTEXT FEATURE

text:
Context Feature: The base model NVAE (Vahdat & Kautz, 2020) is hierarchical. To encode the information from the lower levels of the hierarchy to the higher levels, during training of the binary classifiers, we concatenate the context feature c(z<k) to the samples from both p(z) and q(z). The context feature for each group is the output of the residual cell of the top-down model and encodes a representation from z<k.
Image Decoder p(x|z): The base NVAE (Vahdat & Kautz, 2020) uses a mixture of discretized logistic distributions for all the datasets but MNIST, for which it uses a Bernoulli distribution. In our model, we observe that replacing this with a Normal distribution for the RGB image datasets leads to significant improvements in the base model performance. This is also reflected in the gains of our approach.
E IMPLEMENTATION DETAILS
The binary classifier is composed of two types of residual blocks as in Fig. 5. The residual blocks use batchnormalization (Ioffe & Szegedy, 2015), the Swish activation function (Ramachandran et al., 2017), and the Squeeze-and-Excitation (SE) block (Hu et al., 2018). SE performs a squeeze operation (e.g., mean) to obtain a single value for each channel. An excitation operation (non-linear transformation) is applied to these values to get per-channel weights. The Residual-Block-B differs from Residual-Block-A in that it doubles the number of channels (C → 2C), while down-sampling the other spatial dimensions. It therefore also includes a factorized reduction with 1×1 convolutions along the skip-connection. The complete architecture of the classifier is:

heading: F PERFORMANCE ON SINGLE GROUP VAE

text: To demonstrate the efficacy of our approach on any off-the-shelf VAE, we, apply our NCE based approach to the VAE in (Ghosh et al., 2020). Note we use the vanilla VAE with Normal prior provided by the authors. The FID for CelebA 64 improves from 48.12 to 41.28. Note that the FID for reconstruction is reported as 39.12. Single group VAEs are known to perform poorly on the task of image geeration, which is reflected in the high FID value of reconstruction.

heading: G NEAREST NEIGHBORS FROM THE TRAINING DATASET

text:
To highlight that hierarchical NCP generates unseen samples at test time rather than memorizing the training dataset, Figures 6-7 visualize samples from the model along with a few training images that are most similar
to them (nearest neighbors). To get the similarity score for a pair of images, we downsample to 64 × 64, center crop to 40 × 40 and compute the Euclidean distance. The KD-tree algorithm is used to fetch the nearest neighbors. We note that the generated samples are quite distinct from the training images.
Query Image Nearest neighbors from the training dataset .
Query Image Nearest neighbors from the training dataset .

heading: H ADDITIONAL QUALITATIVE EXAMPLES

heading: I EXPERIMENT ON SYNTHETIC DATA

text:
In Fig. 11 we demonstrate the efficacy of our approach on the 25-Gaussians dataset, that is generated by a mixture of 25 two-dimensional Gaussian distributions that are arranged on a grid. The encoder and decoder of the VAE have 4 fully connected layers with 256 hidden units, with 20 dimensional latent variables. The discriminator has 4 fully connected layers with 256 hidden units. Note that the samples decoded from prior p(z) Fig. 11(b)) without the NCP approach generates many points from the the low density regions in the data distribution. These are removed by using our NCP approach (Fig. 11(c)).
We use 50k samples from the true distribution to estimate the log-likelihood. Our NCP approach obtains an average log-likelihood of −0.954 nats compared to the log-likelihood obtained by sampling from the Gaussian prior, −2.753 nats. We use 20k Monte Carlo samples to estimate the log partition function for the calculation of log-likelihood.

heading: J ADDITIONAL QUALITATIVE EXAMPLES

text: In Fig. 12, we show additional examples of images generated by NVAE (Vahdat & Kautz, 2020) and our NCP-VAE. We use temperature=0.7 for both. Visually corrupt images are highlighted with a red square.