summaries: This paper proposes to learn a joint model consisting of a VAE and a latent space diffusion. The training process is augmented with a contrastive loss.  The authors show that this approach makes possible a few-shot generation of images with desired properties. To achieve this,  one needs to train a classifier in the latent space and after that sample latent codes with high classifier scores in any way.  Also, the case of semantic editing is evaluated. The manipulation is implemented using a Langevin-like approach in the latent space. The paper describes two methods: 1) how to incorporate self-supervised learning in VAE to have a meaningful encoded representation, 2) how to use diffusion model over later to bridge the gap between posterior and prior for unconditional generation, 3) how to use the given setup to do few-shot conditional generation. Empirically, authors show highly improved representation quality, which can be attributed to the incorporation of self-supervised learning. Additionally, experiments show improved FID scores for face datasets, but not for CIFAR. Authors don’t show results on ImageNet. The authors show improved FID scores for the conditional generation task. There is no FID comparison against StyleGAN2 (or any other GAN-based model), even if the authors claim to come very close in sample quality to such models. The qualitative depiction of image manipulation tasks is very impressive. The authors present Diffusion Decoding Models with Contrastive Representations (D2C) - a special VAE model with diffusion modeling in the latent space combined with contrastive learning based training of the inference network. The authors claim that D2C helps avoid the prior hole issue with VAEs due to diffusion modeling of the latent space, while contrastive learning helps learn meaningful representations allowing Few-Shot conditional generation. They further demonstrate that D2C outperforms NVAE and DDIM in few-shot conditional generation while remaining comparable to DDIM in unconditional generation.  D2C proposes three main contributions to diffusion models, a recently popularized class of likelihood-based generative models based on parameterized Markov chains, particularly successful when viewed from a denoising score matching lens. First, this paper learns a prior for the *latent space* of an image VAE (NVAE), while past work mostly applied diffusion to the data space. Second, D2C adds a SimCLR-style contrastive loss to image latents in order to improve the quality of latent representations for classification. Third, binary attribute classifiers are learned conditioned on these latents and used to guide prior sampling with rejection sampling or Langevin dynamics. Experimental results qualitatively and quantitatively show some success in conditional image generation and manipulation.
abstractText: Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive selfsupervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% − 60% of the human evaluators in a double-blind study. We release our code at https: //github.com/jiamings/d2c.
authors.name: Abhishek Sinha
authors.name: Jiaming Song
authors.name: Stefano Ermon
references.authors: Alexander A Alemi Ben Poole Ian Fischer Joshua V Dillon Rif A Saurous Kevin Murphy
references.title: Fixing a broken ELBO
references.venue: arXiv preprint arXiv:1711.00464,
references.authors: Jyoti Aneja Harsh Agrawal Dhruv Batra Alexander Schwing
references.title: Sequential latent spaces for modeling the intention during diverse image captioning
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Jyoti Aneja Alexander Schwing Jan Kautz Arash Vahdat
references.title: NCP-VAE: Variational autoencoders with noise contrastive priors
references.venue: arXiv preprint arXiv:2010.02917,
references.authors: Muhammad Asim Ali Ahmed Paul Hand
references.title: Invertible generative models for inverse problems: mitigating representation error and dataset bias
references.venue: arXiv preprint arXiv:1905.11672,
references.authors: Sergey Bartunov Dmitry Vetrov
references.title: Few-shot generative modelling with generative matching networks
references.venue: Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics,
references.authors: David Bau Jun-Yan Zhu Jonas Wulff William Peebles Hendrik Strobelt Bolei Zhou Antonio Torralba
references.title: Seeing what a gan cannot generate
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Andrew Brock Jeff Donahue Karen Simonyan
references.title: Large scale GAN training for high fidelity natural image synthesis
references.venue: arXiv preprint arXiv:1809.11096,
references.authors: Andrew Brock Theodore Lim James M Ritchie Nick Weston
references.title: Neural photo editing with introspective adversarial networks
references.venue: arXiv preprint arXiv:1609.07093,
references.authors: Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei
references.title: Language models are Few-Shot learners
references.venue: arXiv preprint arXiv:2005.14165,
references.authors: Emmanuel J Candes Justin K Romberg Terence Tao
references.title: Stable signal recovery from incomplete and inaccurate measurements
references.venue: Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences,
references.authors: Qifeng Chen Vladlen Koltun
references.title: Photographic image synthesis with cascaded refinement networks
references.venue: In ICCV,
references.authors: Ricky T Q Chen Yulia Rubanova Jesse Bettencourt David Duvenaud
references.title: Neural ordinary differential equations
references.venue: arXiv preprint arXiv:1806.07366,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: arXiv preprint arXiv:2002.05709,
references.authors: Xi Chen Diederik P Kingma Tim Salimans Yan Duan Prafulla Dhariwal John Schulman Ilya Sutskever Pieter Abbeel
references.title: Variational lossy autoencoder
references.venue: arXiv preprint arXiv:1611.02731,
references.authors: Xinlei Chen Haoqi Fan Ross Girshick Kaiming He
references.title: Improved baselines with momentum contrastive learning
references.venue: arXiv preprint arXiv:2003.04297,
references.authors: Xinlei Chen Kaiming He
references.title: Exploring simple siamese representation learning
references.venue: arXiv preprint arXiv:2011.10566,
references.authors: Xinlei Chen Saining Xie Kaiming He
references.title: An empirical study of training Self-Supervised vision transformers
references.venue: arXiv preprint arXiv:2104.02057,
references.authors: Rewon Child
references.title: Very deep vaes generalize autoregressive models and can outperform them on images
references.venue: arXiv preprint arXiv:2011.10650,
references.authors: Kristy Choi Aditya Grover Trisha Singh Rui Shu Stefano Ermon
references.title: Fair generative modeling via weak supervision
references.venue: In International Conference on Machine Learning,
references.authors: Gordon Christie Neil Fendley James Wilson Ryan Mukherjee
references.title: Functional map of the world
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Louis Clouâtre Marc Demers
references.title: Figr: Few-shot image generation with reptile
references.venue: arXiv preprint arXiv:1901.02199,
references.authors: Bin Dai David Wipf
references.title: Diagnosing and enhancing VAE models
references.venue: arXiv preprint arXiv:1903.05789,
references.authors: Giannis Daras Joseph Dean Ajil Jalal Alexandros G Dimakis
references.title: Intermediate layer optimization for inverse problems using deep generative models
references.venue: arXiv preprint arXiv:2102.07364,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of deep bidirectional transformers for language understanding
references.venue: arXiv preprint arXiv:1810.04805,
references.authors: Prafulla Dhariwal Heewoo Jun Christine Payne Jong Wook Kim Alec Radford Ilya Sutskever
references.title: Jukebox: A generative model for music
references.venue: arXiv preprint arXiv:2005.00341,
references.authors: Prafulla Dhariwal Alex Nichol
references.title: Diffusion models beat GANs on image synthesis
references.venue: arXiv preprint arXiv:2105.05233,
references.authors: L Dinh D Krueger Y Bengio
references.title: NICE: Non-linear independent components estimation
references.venue: arXiv preprint arXiv:1410.8516,
references.authors: Laurent Dinh Jascha Sohl-Dickstein Samy Bengio
references.title: Density estimation using real NVP
references.venue: arXiv preprint arXiv:1605.08803,
references.authors: Jeff Donahue Philipp Krähenbühl Trevor Darrell
references.title: Adversarial feature learning
references.venue: arXiv preprint arXiv:1605.09782,
references.authors: Yilun Du Igor Mordatch
references.title: Implicit generation and generalization in Energy-Based models
references.venue: arXiv preprint arXiv:1903.08689,
references.authors: Vincent Dumoulin Ishmael Belghazi Ben Poole Olivier Mastropietro Alex Lamb Martin Arjovsky Aaron Courville
references.title: Adversarially learned inference
references.venue: arXiv preprint arXiv:1606.00704,
references.authors: Charles Elkan Keith Noto
references.title: Learning classifiers from only positive and unlabeled data
references.venue: In 14th ACM SIGKDD,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: Advances in Neural Information Processing Systems
references.authors: Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko
references.title: Bootstrap your own latent: A new approach to Self-Supervised learning
references.venue: arXiv preprint arXiv:2006.07733,
references.authors: Aditya Grover Aaron Zweig Stefano Ermon
references.title: Graphite: Iterative generative modeling of graphs
references.venue: In International conference on machine learning,
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: arXiv preprint arXiv:1911.05722,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: arXiv preprint arXiv:1512.03385,
references.authors: Olivier Henaff
references.title: Data-efficient image recognition with contrastive predictive coding
references.venue: In International Conference on Machine Learning,
references.authors: Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Sepp Hochreiter
references.title: GANs trained by a two Time-Scale update rule converge to a local nash equilibrium
references.venue: arXiv preprint arXiv:1706.08500,
references.authors: Geoffrey E Hinton
references.title: Training products of experts by minimizing contrastive divergence
references.venue: Neural computation,
references.authors: Jonathan Ho Ajay Jain Pieter Abbeel
references.title: Denoising diffusion probabilistic models
references.venue: arXiv preprint arXiv:2006.11239,
references.authors: Matthew D Hoffman Matthew J Johnson
references.title: Elbo surgery: yet another way to carve up the variational evidence lower bound
references.venue: In Workshop in Advances in Approximate Bayesian Inference, NIPS,
references.authors: Drew A Hudson C Lawrence Zitnick
references.title: Generative adversarial transformers
references.venue: arXiv preprint arXiv:2103.01209,
references.authors: Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A Efros
references.title: Image-to-image translation with conditional adversarial networks
references.authors: Erik Jones Shiori Sagawa Pang Wei Koh Ananya Kumar Percy Liang
references.title: Selective classification can magnify disparities across groups
references.venue: arXiv preprint arXiv:2010.14134,
references.authors: Zahra Kadkhodaie Eero P Simoncelli
references.title: Solving linear inverse problems using the prior implicit in a denoiser
references.venue: arXiv preprint arXiv:2007.13640,
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of GANs for improved quality, stability, and variation
references.venue: arXiv preprint arXiv:1710.10196,
references.authors: Tero Karras Samuli Laine Timo Aila
references.title: A Style-Based generator architecture for generative adversarial networks
references.venue: arXiv preprint arXiv:1812.04948,
references.authors: Tero Karras Samuli Laine Miika Aittala Janne Hellsten Jaakko Lehtinen Timo Aila
references.title: Analyzing and improving the image quality of stylegan
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Diederik P Kingma Tim Salimans Rafal Jozefowicz Xi Chen Ilya Sutskever Max Welling
references.title: Improved variational inference with inverse autoregressive flow
references.venue: Advances in Neural Information Processing Systems
references.authors: Diederik P Kingma Max Welling
references.title: Auto-Encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114v10,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: ImageNet classification with deep convolutional neural networks
references.venue: Advances in Neural Information Processing Systems
references.authors: Anders Boesen Lindbo Larsen Søren Kaae Sønderby Hugo Larochelle Ole Winther
references.title: Autoencoding beyond pixels using a learned similarity metric
references.venue: In International conference on machine learning,
references.authors: C Li H Liu C Chen Y Pu L Chen others
references.title: Alice: Towards understanding adversarial learning for joint distribution matching
references.venue: Advances in neural information processing systems,
references.authors: Chunyuan Li Xiang Gao Yuan Li Baolin Peng Xiujun Li Yizhe Zhang Jianfeng Gao
references.title: Optimus: Organizing sentences via pre-trained modeling of a latent space
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: In European conference on computer vision,
references.authors: Ziwei Liu Ping Luo Xiaogang Wang Xiaoou Tang
references.title: Deep learning face attributes in the wild
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Elman Mansimov Emilio Parisotto Jimmy Lei Ba Ruslan Salakhutdinov
references.title: Generating images from captions with attention
references.venue: arXiv preprint arXiv:1511.02793,
references.authors: Mehdi Mirza Simon Osindero
references.title: Conditional generative adversarial nets
references.venue: arXiv preprint arXiv:1411.1784,
references.authors: Gautam Mittal Jesse Engel Curtis Hawthorne Ian Simon
references.title: Symbolic music generation with diffusion models
references.venue: arXiv preprint arXiv:2103.16091,
references.authors: Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida
references.title: Spectral normalization for generative adversarial networks
references.venue: arXiv preprint arXiv:1802.05957,
references.authors: Shakir Mohamed Balaji Lakshminarayanan
references.title: Learning in implicit generative models
references.venue: arXiv preprint arXiv:1610.03483,
references.authors: Alex Najibi
references.title: Racial Discrimination in Face Recognition Technology, 2020
references.authors: Anh Nguyen Jeff Clune Yoshua Bengio Alexey Dosovitskiy Jason Yosinski
references.title: Plug & play generative networks: Conditional iterative generation of images in latent space
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Chenhao Niu Yang Song Jiaming Song Shengjia Zhao Aditya Grover Stefano Ermon
references.title: Permutation invariant graph generation via score-based generative modeling
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: Mehdi Noroozi Paolo Favaro
references.title: Unsupervised learning of visual representations by solving jigsaw puzzles
references.venue: In European conference on computer vision,
references.authors: Taesung Park Jun-Yan Zhu Oliver Wang Jingwan Lu Eli Shechtman Alexei A Efros Richard Zhang
references.title: Swapping autoencoder for deep image manipulation
references.venue: arXiv preprint arXiv:2007.00653,
references.authors: Or Patashnik Zongze Wu Eli Shechtman Daniel Cohen-Or Dani Lischinski
references.title: StyleCLIP: Text-Driven manipulation of StyleGAN imagery
references.venue: arXiv preprint arXiv:2103.17249,
references.authors: Wei Ping Kainan Peng Kexin Zhao Zhao Song
references.title: Waveflow: A compact flow-based model for raw audio
references.venue: In International Conference on Machine Learning,
references.authors: Tiziano Portenier Qiyang Hu Attila Szabó Siavash Arjomand Bigdeli Paolo Favaro Matthias Zwicker
references.title: Faceshop: Deep sketch-based face image editing
references.venue: ACM Transactions on Graphics,
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever
references.title: Learning transferable visual models from natural language supervision
references.venue: arXiv preprint arXiv:2103.00020,
references.authors: Aditya Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever
references.title: Zero-Shot Text-to-Image generation
references.venue: arXiv preprint arXiv:2102.12092,
references.authors: Ali Razavi Aaron van den Oord Oriol Vinyals
references.title: Generating diverse high-fidelity images with vq-vae-2
references.venue: arXiv preprint arXiv:1906.00446,
references.authors: Scott Reed Zeynep Akata Xinchen Yan Lajanugen Logeswaran Bernt Schiele Honglak Lee
references.title: Generative adversarial text to image synthesis
references.venue: In International Conference on Machine Learning,
references.authors: Danilo Jimenez Rezende Shakir Mohamed
references.title: Variational inference with normalizing flows
references.venue: arXiv preprint arXiv:1505.05770,
references.authors: Gareth O Roberts Jeffrey S Rosenthal
references.title: Optimal scaling of discrete approximations to langevin diffusions
references.venue: Journal of the Royal Statistical Society: Series B (Statistical Methodology),
references.authors: Mihaela Rosca Balaji Lakshminarayanan Shakir Mohamed
references.title: Distribution matching in variational inference
references.venue: arXiv preprint arXiv:1802.06847,
references.authors: Yujun Shen Jinjin Gu Xiaoou Tang Bolei Zhou
references.title: Interpreting the latent space of gans for semantic face editing
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jascha Sohl-Dickstein Eric A Weiss Niru Maheswaranathan Surya Ganguli
references.title: Deep unsupervised learning using nonequilibrium thermodynamics
references.venue: arXiv preprint arXiv:1503.03585,
references.authors: Jiaming Song Stefano Ermon
references.title: Multi-label contrastive predictive coding
references.venue: arXiv preprint arXiv:2007.09852,
references.authors: Jiaming Song Pratyusha Kalluri Aditya Grover Shengjia Zhao Stefano Ermon
references.title: Learning controllable fair representations
references.venue: arXiv preprint arXiv:1812.04218,
references.authors: Jiaming Song Chenlin Meng Stefano Ermon
references.title: Denoising diffusion implicit models
references.venue: arXiv preprint arXiv:2010.02502,
references.authors: Yang Song Stefano Ermon
references.title: Generative modeling by estimating gradients of the data distribution
references.venue: arXiv preprint arXiv:1907.05600,
references.authors: Yang Song Jascha Sohl-Dickstein Diederik P Kingma Abhishek Kumar Stefano Ermon Ben Poole
references.title: Score-based generative modeling through stochastic differential equations
references.venue: arXiv preprint arXiv:2011.13456,
references.authors: Hiroshi Takahashi Tomoharu Iwata Yuki Yamanaka Masanori Yamada Satoshi Yagi
references.title: Variational autoencoder with implicit optimal priors
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Jakub M Tomczak Max Welling
references.title: Improving variational auto-encoders using householder flow
references.venue: arXiv preprint arXiv:1611.09630,
references.authors: Jakub M Tomczak Max Welling
references.title: VAE with a VampPrior
references.venue: arXiv preprint arXiv:1705.07120,
references.authors: Arash Vahdat Jan Kautz
references.title: NVAE: A deep hierarchical variational autoencoder
references.venue: arXiv preprint arXiv:2007.03898,
references.authors: Arash Vahdat Karsten Kreis Jan Kautz
references.title: Score-based generative modeling in latent space
references.venue: arXiv preprint arXiv:2106.05931,
references.authors: Aaron van den Oord Sander Dieleman Heiga Zen Karen Simonyan Oriol Vinyals Alex Graves Nal Kalchbrenner Andrew Senior Koray Kavukcuoglu
references.title: WaveNet: A generative model for raw audio
references.venue: arXiv preprint arXiv:1609.03499,
references.authors: Aaron van den Oord Nal Kalchbrenner Koray Kavukcuoglu
references.title: Pixel recurrent neural networks
references.venue: arXiv preprint arXiv:1601.06759,
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding
references.venue: arXiv preprint arXiv:1807.03748,
references.authors: Aaron van den Oord Oriol Vinyals Koray Kavukcuoglu
references.title: Neural discrete representation learning
references.venue: arXiv preprint arXiv:1711.00937,
references.authors: Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Andrew Tao Jan Kautz Bryan Catanzaro
references.title: High-resolution image synthesis and semantic manipulation with conditional gans
references.authors: Weihao Xia Yujiu Yang Jing-Hao Xue Baoyuan Wu
references.title: TediGAN: Text-Guided diverse face image generation and manipulation
references.authors: Zhe Xie Chengxuan Liu Yichi Zhang Hongtao Lu Dong Wang Yue Ding
references.title: Adversarial and contrastive variational autoencoder for sequential recommendation
references.venue: arXiv preprint arXiv:2103.10693,
references.authors: Yinghao Xu Yujun Shen Jiapeng Zhu Ceyuan Yang Bolei Zhou
references.title: Generative hierarchical features from synthesizing images
references.authors: Jiaxuan You Bowen Liu Rex Ying Vijay Pande Jure Leskovec
references.title: Graph convolutional policy network for Goal-Directed molecular graph generation
references.venue: arXiv preprint arXiv:1806.02473,
references.authors: Lantao Yu Weinan Zhang Jun Wang Yong Yu
references.title: Seqgan: Sequence generative adversarial nets with policy gradient
references.venue: In Proceedings of the AAAI conference on artificial intelligence,
references.authors: Shengjia Zhao Jiaming Song Stefano Ermon
references.title: InfoVAE: Information maximizing variational autoencoders
references.venue: arXiv preprint arXiv:1706.02262,
references.authors: Shengjia Zhao Jiaming Song Stefano Ermon
references.title: Towards deeper understanding of variational autoencoding models
references.venue: arXiv preprint arXiv:1702.08658,
references.authors: Shengjia Zhao Jiaming Song Stefano Ermon
references.title: A lagrangian perspective on latent variable generative models
references.venue: In Proc. 34th Conference on Uncertainty in Artificial Intelligence,
references.authors: Jiapeng Zhu Yujun Shen Deli Zhao Bolei Zhou
references.title: In-domain gan inversion for real image editing
references.venue: In European Conference on Computer Vision,
references.authors: Jiapeng Zhu Deli Zhao Bolei Zhou Bo Zhang
references.title: Lia: Latently invertible autoencoder with adversarial learning
sections.text: Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive selfsupervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% − 60% of the human evaluators in a double-blind study. We release our code at https: //github.com/jiamings/d2c.
sections.heading: 1 Introduction
sections.text: Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67]. However, downstream applications of generative models are often based on various conditioning signals, such as labels [61], text descriptions [60], reward values [101], or similarity with existing data [45]. While it is possible to directly train conditional models, this often requires large amounts of paired data [57, 74] that are costly to acquire. Hence, it would be desirable to learn conditional generative models using large amounts of unlabeled data and as little paired data as possible.
  Contrastive self-supervised learning (SSL) methods can greatly reduce the need for labeled data in discriminative tasks by learning effective representations from unlabeled data [95, 37, 35], and have also been shown to improve few-shot learning [39]. It is therefore natural to ask if they can also be used to improve few-shot generation. Latent variable generative models (LVGM) are a natural candidate for this, since they already involve a low-dimensional, structured latent representation of the data they generate. However, generative adversarial networks (GANs, [34, 50]) and diffusion
  ∗Equal contribution.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  models [42, 84], lack explicit tractable functions to map inputs to representations, making it difficult to optimize latent variables with SSL. Variational autoencoders (VAEs, [52, 77]), on the other hand, can naturally adopt SSL through their encoder model, but they typically have worse sample quality.
  In this paper, we propose Diffusion-Decoding models with Contrastive representations (D2C), a special VAE that is suitable for conditional few-shot generation. D2C uses contrastive self-supervised learning methods to obtain a latent space that inherits the transferrability and few-shot capabilities of self-supervised representations. Unlike other VAEs, D2C learns a diffusion model over the latent representations. This latent diffusion model ensures that D2C uses the same latent distribution for both training and generation. We provide a formal argument to explain why this approach leads to better sample quality than existing hierarchical VAEs. We further discuss how to apply D2C to few-shot conditional generation where the conditions are defined through labeled examples or manipulation constraints. Our approach combines a discriminative model providing conditioning signal and generative diffusion model over the latent space, and is computationally more efficient than methods that act directly over the image space (Figure 1).
  We evaluate and compare D2C with several state-of-the-art generative models over 6 datasets. On unconditional generation, D2C outperforms state-of-the-art VAEs and is competitive with diffusion models under similar computational budgets. On conditional generation with 100 labeled examples, D2C significantly outperforms state-of-the-art VAE [91] and diffusion models [84]. D2C can also learn to perform certain image manipulation tasks from as few as 100 labeled examples. Notably, for manipulating images, D2C is two orders of magnitude faster than StyleGAN2 [106] and preferred by 50%− 60% of human evaluations, which to our best knowledge is the first for any VAE model.
sections.heading: 2 Background
sections.text: Latent variable generative models A latent variable generative model (LVGM) is posed as a conditional distribution pθ : Z → P(X ) from a latent variable z to a generated sample x, parametrized by θ. To acquire new samples, LVGMs draw random latent variables z from some distribution p(z) and map them to image samples through x ∼ pθ(x|z). Most LVGMs are built on top of four paradigms: variational autoencoders (VAEs, [52, 77]), Normalizing Flows (NFs, [28, 29]), Generative Adversarial Networks (GANs, [34]), and diffusion / score-based generative models [42, 85].
  Particularly, VAEs use an inference model from x to z for training. Denoting the inference distribution from x to z as qφ(z|x), the generative distribution from z to x as pθ(x|z), VAEs are trained by minimizing the following upper bound of negative log-likelihood:
  LVAE = Ex∼pdata [Ez∼qφ(z|x)[− log p(x|z)] +DKL(qφ(z|x)‖p(z))] (1)
  where pdata is the data distribution and DKL is the KL-divergence.
  Diffusion models Diffusion models [81, 42, 84] produce samples by reversing a Gaussian diffusion process. We use the index α ∈ [0, 1] to denote the particular noise level of an noisy observation x(α) = √ αx + √ 1− α , where x is the clean observation and ∼ N (0, I) is a standard Gaussian distribution; as α→ 0, the distribution of x(α) converges to N (0, I). Diffusion models are typically parametrized as reverse noise models θ(x(α), α) that predict the noise component of x(α) given a noise level α, and trained to minimize ‖ θ(x(α), α) − ‖22, the mean squared error loss between the true noise and predicted noise. Given any non-increasing series {αi}Ti=0 between 0 and 1, the diffusion objective for a clean sample from the data x is:
  `diff(x;w, θ) := T∑ i=1 w(αi)E ∼N (0,I)[‖ − θ(x(αi), αi)‖ 2 2], x (αi) := √ αix + √ 1− αi (2)
  where w : {αi}Ti=1 → R+ controls the loss weights for each α. When w(α) = 1 for all α, we recover the denoising score matching objective for training score-based generative models [85].
  Given an initial sample x0 ∼ N (0, I), diffusion models acquires clean samples (i.e., samples of x1) through a gradual denoising process, where samples with reducing noise levels α are produced (e.g., x0 → x0.3 → x0.7 → x1). In particular, Denoising Diffusion Implicit Models (DDIMs, [84]) uses an Euler discretization of some neural ODE [13] to produce samples (Figure 2, left).
  We provide a more detailed description for training diffusion models in Appendix A.1 and sampling from DDIM in Appendix A.2. For conciseness, we use the notation p(α)(x(α)) to denote the marginal distribution of x(α) under the diffusion model, and p(α1,α2)(x(α2) | x(α1)) to denote the diffusion sampling process from x(α1) to x(α2) (assuming α1 < α2).
  Self-supervised learning of representations In self-supervised learning (SSL), representations are learned by completing certain pretext tasks that do not require extra manual labeling [68, 25]; these representations can then be applied to other downstream tasks, often in few-shot or zero-shot scenarios. In particular, contrastive representation learning encourages representations to be closer between “positive” pairs and further between “negative” pairs; contrastive predictive coding (CPC, [95]), based on multi-class classification, have been commonly used in state-of-the-art methods [37, 16, 18, 14, 82]. Other non-contrastive methods exist, such as BYOL [35] and SimSiam [17], but they usually require additional care to prevent the representation network from collapsing.
sections.heading: 3 Problem Statement
sections.text: Few-shot conditional generation Our goal is to learn an unconditional generative model pθ(x) such that it is suitable for conditional generation. Let C(x, c, f) describe an event that “f(x) = c”, where c is a property value and f(x) is a property function that is unknown at training. In conditional generation, our goal is to sample x such that the event C(x, c, f) occurs for a chosen c. If we have access to some “ground-truth” model that gives us p(C|x) := p(f(x) = c|x), then the conditional model can be derived from Bayes’ rule: pθ(x|C) ∝ p(C|x)pθ(x). These properties c include (but are not limited to2) labels [61], text descriptions [60, 76], noisy or partial observations [11, 5, 47, 24], and manipulation constraints [69]. In many cases, we do not have direct access to the true f(x), so we need to learn an accurate model from labeled data [6] (e.g., (c,x) pairs).
  Desiderata Many existing methods are optimized for some known condition (e.g., labels in conditional GANs [8]) or assume abundant pairs between images and conditions that can be used for pretraining (e.g., DALL-E [74] and CLIP [73] over image-text pairs). Neither is the case in this paper, as we do not expect to train over paired data.
  While high-quality latent representations are not essential to unconditional image generation (e.g., autoregressive [94], energy-based [31], and some diffusion models [42]), they can be beneficial when we wish to specify certain conditions with limited supervision signals, similar to how SSL representations can reduce labeling efforts in downstream tasks. A compelling use case is detecting and removing biases in datasets via image manipulation, where we should not only address wellknown biases a-priori but also address other hard-to-anticipate biases, adapting to societal needs [65].
  2When C refers to an event that is always true, we recover unconditioned generation.
  Therefore, a desirable generative model should not only have high sample quality but also contain informative latent representations. While VAEs are ideal for learning rich latent representations due to being able to incorporate SSL within the encoder, they generally do not achieve the same level of sample quality as GANs and diffusion models.
sections.heading: 4 Diffusion-Decoding Generative Models with Contrastive Learning
sections.text: In this paper, we present Diffusion-Decoding generative models with Contrastive Learning (D2C), an extension to VAEs with high-quality samples and high-quality latent representations, and are thus well suited to few-shot conditional generation. Moreover, unlike GAN-based methods, D2C does not involve unstable adversarial training (Table 1).
  As its name suggests, the generative model for D2C has two components – diffusion and decoding; the diffusion component operates over the latent space and the decoding component maps from latent representations to images. Let us use the α index notation for diffusion random variables: z(0) ∼ p(0)(z(0)) := N (0, I) is the “noisy” latent variable with α = 0, and z(1) is the “clean” latent variable with α = 1. The generative process of D2C, which we denote pθ(x|z(0)), is then defined as:
  z(0) ∼ p(0)(z(0)), z(1) ∼ p(0,1)θ (z (1)|z(0))︸ ︷︷ ︸
  diffusion
  , x ∼ pθ(x|z(1))︸ ︷︷ ︸ decoding , (3)
  where p(0)(z(0)) = N (0, I) is the prior distribution for the diffusion model, p(0,1)θ (z(1)|z(0)) is the diffusion process from z(0) to z(1), and pθ(x|z(1)) is the decoder from z(1) to x. Intuitively, D2C models produce samples by drawing z(1) from a diffusion process and then decoding x from z(1).
  In order to train a D2C model, we use an inference model qφ(z(1)|x) that predicts proper z(1) latent variables from x; this can directly incorporate SSL methods [99], leading to the following objective:
  LD2C(θ, φ;w) := LD2(θ, φ;w) + λLC(qφ), (4)
  LD2(θ, φ;w) := Ex∼pdata,z(1)∼qφ(z(1)|x)[− log pθ(x|z (1)) + `diff(z (1);w, θ)], (5)
  where `diff is defined as in Eq.(2), LC(qφ) denotes any contrastive predictive coding objective [95] with rich data augmentations [37, 16, 18, 14, 82] (details in Appendix A.3) and λ > 0 is a weight hyperparameter. We illustrate D2C in Figure 2, and its training procedure in Appendix A.4.
sections.heading: 4.1 Relationship to maximum likelihood
sections.text: The D2 objective (LD2) appears similar to the original VAE objective (LVAE). Here, we make an informal statement that the D2 objective function is deeply connected to the variational lower bound of log-likelihood; we present the full statement and proof in Appendix B.1. Theorem 1. (informal) For any valid {αi}Ti=0, there exists some weights ŵ : {αi}Ti=1 → R+ for the diffusion objective such that −LD2 is a variational lower bound to the log-likelihood, i.e.,
  −LD2(θ, φ; ŵ) ≤ Epdata [log pθ(x)], (6) where pθ(x) := Ez0∼p(0)(z(0))[pθ(x|z (0))] is the marginal probability of x under the D2C model.
  16
  Proof. (sketch) The diffusion term `diff upper bounds the KL divergence between qφ(z1|x) and p
  (1) θ (z (1)) for suitable weights [42, 84], which recovers a VAE objective.
  Moreover, since some contrastive losses are lower bounds to mutual information [82], D2C can be treated as an implementation of the InfoVAE approach [103], where the latent variables are encouraged to have higher mutual information with the observations. Since the evidence lower bound (ELBO) for VAEs do not necessarily promote informative latent representations [15], D2C comes with the benefit of being able to learn more informative latents despite not exactly optimizing an ELBO.
sections.heading: 4.2 D2C models address latent posterior mismatch in VAEs
sections.text: While D2C is a special case of VAE, we argue that D2C is non-trivial in the sense that it addresses a long-standing problem in VAE methods [90, 87], namely the mismatch between the prior distribution pθ(z) and the aggregate (approximate) posterior distribution qφ(z) := Epdata(x)[qφ(z|x)]. A mismatch could create “holes” [79, 43, 3] in the prior that the aggregate posterior fails to cover during training, resulting in worse sample quality, as many latent variables used during generation are likely to never have been trained on. We formalize this notion in the following definition.
  Definition 1 (Prior hole). Let p(z), q(z) be two distributions with supp(q) ⊆ supp(p). We say that q has an ( , δ)-prior hole with respect to (the prior) p for , δ ∈ (0, 1), δ > , if there exists a set S ∈ supp(P ), such that ∫ S p(z)dz ≥ δ and ∫ S q(z)dz ≤ .
  Intuitively, if qφ(z) has a prior hole with large δ and small (e.g., inversely proportional to the number of training samples), then it is very likely that latent variables within the hole are never seen during training (small ), yet frequently used to produce samples (large δ). Most existing methods address this problem by optimizing certain statistical divergences between qφ(z) and pθ(z), such as the KL divergence or Wasserstein distance [88]. However, we argue in the following statement that prior holes might not be eliminated even if we optimize certain divergence values to be reasonably low, especially when qφ(z) is very flexible. We present the formal statement and proof in Appendix B.2.
  p(z) q(z)Theorem 2. (informal) Let pθ(z) = N (0, 1). For any > 0, there exists a qφ(z) with an ( , 0.49)-prior hole, such that DKL(qφ‖pθ) ≤ log 23and W2(qφ, pθ) < γ for any γ > 0, where W2 is the 2-Wasserstein distance.
  3This is reasonably low for realistic VAE models (NVAE [91] reports a KL divergence of around 2810 nats).
  Proof. (sketch) We construct a qφ that satisfies these properties (top-right figure). First, we truncate the Gaussian and divide them into regions with same probability mass; then we support qφ over half of these regions (so δ > 0.49); finally, we show that the divergences are small enough.
  In contrast to addressing prior holes by optimization, diffusion models eliminate prior holes by construction, since the diffusion process from z(1) to z(0) is constructed such that the distribution of z(α) always converges to a standard Gaussian as α → 0. As a result, the distribution of latent variables used during training is arbitrarily close to that used in generation4, which is also the case in GANs. Therefore, our argument provides an explanation as to why we observe better sample quality results from GANs and diffusion models than VAEs and NFs.
sections.heading: 5 Few-shot Conditional Generation with D2C
sections.text: In this section, we discuss how D2C can be used to learn to perform conditional generation from few-shot supervision. We note that D2C is only trained on images and not with any other data modalities (e.g., image-text pairs [74]) or supervision techniques (e.g., meta-learning [22, 6]).
  Algorithm 1 Conditional generation with D2C 1: Input n examples {(xi, ci)}ni=1, property c. 2: Acquire latents z(1)i ∼ qφ(z
  (1)|xi) for i ∈ [n]; 3: Train model rψ(c|z(1)) over {(z(1)i , ci)} n i=1 4: Sample latents with ẑ(1) ∼ rψ(c|z(1)) · p(1)θ (z (1))
  (unnormalized); 5: Decode x̂ ∼ pθ(x|ẑ(1)). 6: Output x̂. Algorithm We describe the general algorithm for conditional generation from a few images in Algorithm 1, and detailed implementations in Appendix C. With a model over the latent space (denoted as rψ(c|z(1))), we draw conditional latents from an unnormalized distribution with the diffusion prior (line 4). This can be implemented in many ways such as rejection sampling or Langevin dynamics [66, 86, 27].
  Conditions from labeled examples Given a few labeled examples, we wish to produce diverse samples with a certain label. For labeled examples we can directly train a classifier over the latent space, which we denote as rψ(c|z(1)) with c being the class label and z(1) being the latent representation of x from qφ(z(1)|x). If these examples do not have labels (i.e., we merely want to generate new samples similar to given ones), we can train a positive-unlabeled (PU) classifier [33] where we assign “positive” to the new examples and “unlabeled” to training data. Then we use the classifier with the diffusion model pθ(z(1)|z(0)) to produce suitable values of z(1), such as by rejecting samples from the diffusion model that has a small rψ(c|z(1)).
  Conditions from manipulation constraints Given a few labeled examples, here we wish to learn how to manipulate images. Specifically, we condition over the event that “x has label c but is similar to image x̄”. Here rψ(c|z(1)) is the unnormalized product between the classifier conditional probability and closeness to the latent z̄(1) of x̄ (e.g., measured with RBF kernel). We implement line 4 of Alg. 1 with a Lanvegin-like procedure where we take a gradient step with respect to the classifier probability and then correct this gradient step with the diffusion model. Unlike many GAN-based methods [12, 72, 97, 45, 98], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value is much faster to compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.
sections.heading: 6 Related Work
sections.text: Latent variable generative models Most deep generative models explicitly define a latent representation, except for some energy-based models [41, 31] and autoregressive models [94, 93, 10]. Unlike VAEs and NFs, GANs do not explicitly define an inference model and instead optimize a two-player game. In terms of sample quality, GANs currently achieve superior performance over VAEs and NFs, but they can be difficult to invert even with additional optimization [48, 100, 7]. This can be partially addressed by training reconstruction-based losses with GANs [54, 55]. Moreover,
  4We expand this argument in Appendix B.2.
  the GAN training procedure can be unstable [9, 8, 63], lack a informative objective for measuring progress [4], and struggle with discrete data [102]. Diffusion models [27] achieves high sample quality without adversarial training, but its latent dimension must be equal to the image dimension. A concurrent work [92] applied score-based generative modeling on the latent space of an VAE, and achieved similar improvements compared to regular VAEs.
  Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior. In Section 4.2, we explain why these approaches do not necessarily remove large “prior holes”, so their sample qualities remain relatively poor even after many layers [91, 19]. Other methods adopt a “two-stage” approach [23], which fits a generative model over the latent space of autoencoders [96, 75, 26, 74]. [62] have applied diffusion models directly on latent spaces learned by a music VAE.
  Conditional generation with unconditional models To perform conditional generation over an unconditional LVGM, most methods assume access to a discriminative model (e.g., a classifier); the latent space of the generator is then modified to change the outputs of the discriminative model. The disciminative model can operate on either the image space [66, 70, 27] or the latent space [80, 98]. For image space discriminative models, plug-and-play generative networks [66] control the attributes of generated images via Langevin dynamics [78]; these ideas are also explored in diffusion models [86]. Image manipulation methods are based on GANs often operate with latent space discriminators [80, 98]. However, these methods have some trouble manipulating real images because of imperfect reconstruction [107, 7]. This is not a problem in D2C since a reconstruction objective is optimized.
sections.heading: 7 Experiments
sections.text: We examine the conditional and unconditional generation qualities of D2C over CIFAR-10 [53], CIFAR-100 [53], fMoW [21], CelebA-64 [58], CelebA-HQ-256 [48], and FFHQ-256 [49]. Our D2C implementation is based on the state-of-the-art NVAE [91] autoencoder structure, the U-Net diffusion model [42], and the MoCo-v2 contrastive representation learning method [16]. We also consider the D2 objective where we do not applying the contrastive loss in the NVAE autoencoder; we also tried adding contrastive loss directly to the VAE objective, but we were unable to achieve satisfactory generation results (reconstruction MSE remains high). This is possibly due to the many regularizations needed for NVAE to work well, which could conflict with contrastive learning5.
  We keep the diffusion series hyperparameter {αi}Ti=1 identical to ensure a fair comparison with different diffusion models. For the contrastive weight λ in Equation (4), we consider the value of λ = 10−4 based on the relative scale between the LC and LD2; we find that the results are relatively insensitive to λ. We use 100 diffusion steps for DDIM and D2C unless mentioned otherwise, as running with longer steps is not computationally economical despite tiny gains in FID [84]. We include additional training details, such as architectures, optimizers and learning rates in Appendix C.
sections.heading: 7.1 Unconditional generation
sections.text: For unconditional generation, we measure the sample quality of images using the Frechet Inception Distance (FID, [40]) with 50,000 images. In particular, we extensively evaluate NVAE [91] and
  5See https://github.com/NVlabs/NVAE#known-issues for a detailed description.
  DDIM [84], a competitive VAE model and a competitive diffusion model as baselines because we can directly obtain features from them without additional optimization steps6. For them, we report mean-squared reconstruction error (MSE, summed over all pixels, pixels normalized to [0, 1]) and linear classification accuracy (Acc., measured in percentage) over z1 features for the test set.
  We report sample quality results7 in Tables 2, and 3. For FID, we outperform NVAE in all datasets and outperform DDIM on CelebA-64 and CelebA-HQ-256, which suggests our results are competitive with state-of-the-art non-adversarial generative models. In Table 2, we additionally compare NVAE, DDIM and D2C in terms of reconstruction and linear classification accuracy. As all three methods contain reconstruction losses, the MSE values are low and comparable. However, D2C enjoys much better linear classification accuracy than the other two thanks to the contrastive SSL component. We further note that training the same contrastive SSL method without LD2 achieves slightly higher 78.3% accuracy on CIFAR-10. We tried improving this via ResNet [38] encoders, but this significantly increased reconstruction error, possibly due to loss of information in average pooling layers.
sections.heading: 7.2 Few-shot conditional generation from examples
sections.text: We demonstrate the advantage of D2C representations by performing few-shot conditional generation over labels. We consider two types of labeled examples: one has binary labels for which we train a binary classifier; the other is positive-only labeled (e.g., images of female class) for which we train a PU classifier. Our goal here is to generate a diverse group of images with a certain label. We evaluate and compare three models: D2C, NVAE and DDIM. We train a classifier rψ(c|z) over the latent space of these models; we also train a image space classifier and use it with DDIM (denoted as DDIM-I). We run Algorithm 1 for these models, where line 4 is implemented via rejection sampling. As our goal is to compare different models, we leave more sophisticated methods [27] as future work.
  First, we consider performing 8 conditional generation tasks over CelebA-64 with 2 binary classifiers (trained over 100 samples, 50 for each class) and 4 PU classifiers (trained over 100 positively labeled and 10k unlabeled samples). We also report a “naive” approach where we use all the training images (regardless of labels) and compute its FID with the corresponding subset of images (e.g., all images versus blond images). In Table 4, we report the FID score between generated images (5k samples) and real images of the corresponding label.
  Next, we perform a similar experiment on CIFAR-10, but with 50 labels for each of the 10 classes. For each label, we evaluate the FID between the real images and generated images under that label
  6For DDIM, the latent representations x(0) are obtained by reversing the neural ODE process. 7Due to space limits, we place additional CIFAR-10 results in Appendix D.
  and take the average FID over the 10 labels. The average FID for D2C, DDIM, and NVAE are 32.57, 82.78, and 95.45 respectively (note that FIDs are expected to be higher because it is evaluated with 5k real samples per class instead of the commonly used 50k). This shows that D2C performs significantly better than DDIM and NVAE on the few-shot conditional generation task for both datasets, illustrating the advantage of contrastive representations for few-shot conditional generation. We demonstrate some figures in the appendix.
  Fair generative modeling with D2C We illustrate some conditional generation results with D2C (from 100 “blond” labels) in Figure 4 (left), which exhibits a biased gender ratio (3 examples out of 36). This kind of dataset bias in CelebA is well-documented in previous work [46]; among all the images with the “blond” attribute, only 5.6 are labeled as “male”. In fact, we observe that the D2C samples have a similar gender bias as the original dataset since D2C has learned the data distribution quite well. To mitigate the presence of dataset bias in D2C’s samples, we leverage a conditional generation approach. This is in contrast to previous methods that try to learn a fair generative model via a carefully designed representative dataset [20]. We give a concrete instantiation of our method in the case of gender bias that can be generalized to all attributes. Suppose our goal is to achieve a balanced gender ratio among all the generated images with “blond” attribute; we can do so by generating the same number of samples conditioned on the (1) “blond, male” attributes and the (2) “blond, female” attributes. Figure 4 (right) demonstrates that D2C can be used for fair generative modeling by simply specifying our desired constraints with a handful of labels.
  17
sections.heading: 7.3 Few-shot conditional generation from manipulation constraints
sections.text: Finally, we consider image manipulation where we use binary classifiers that are learned over 50 labeled instances for each class. To manipulate an image, we first acquire its latent code, move its latent code towards the decision boundary of the classifier and then use the diffusion prior to make the
  sample more realistic. We perform Amazon Mechanical Turk (AMT) evaluations over two attributes in the CelebA-256 dataset, blond and red lipstick, over D2C, DDIM, NVAE and StyleGAN2 [50] (see Figure 5). The evaluation is double-blinded: neither we nor the evaluators know the correspondence between generated image and underlying model during the study. We include more details (algorithm, setup and human evaluation) in Appendix C and additional qualitative results (such as beard and gender attributes) in Appendix D.
  In Figure 8, we show the percentage of manipulations preferred by AMT evaluators for each model; D2C slightly outperforms StyleGAN2 for blond and significantly outperforms StyleGAN2 for red lipstick. When we compare D2C with only StyleGAN2, D2C is preferred over 51.5% for blond and 60.8% for red lipstick. An additional advantage of D2C is that the manipulation is much faster than StyleGAN2, since the latter requires additional optimization over the latent space to improve reconstruction [106]. On the same Nvidia 1080Ti GPU, it takes 0.013 seconds to obtain the latent code in D2C, while the same takes 8 seconds [106] for StyleGAN2 (615× slower). As decoding is very fast for both models, D2C generations are around two orders of magnitude faster to produce.
sections.heading: 8 Discussions and Limitations
sections.text: We introduced D2C, a VAE-based generative model with a latent space suitable for few-shot conditional generation. To our best knowledge, our model is the first unconditional VAE to demonstrate superior image manipulation performance than StyleGAN2, which is surprising given our use of a regular NVAE architecture. We believe that with better architectures, such as designs from StyleGAN2 or Transformers [44], D2C can achieve even better performance. It is also interesting to formally investigate the integration between D2C and other types of conditions on the latent space, as well as training D2C in conjunction with other domains and data modalities, such as text [74], in a fashion that is similar to semi-supervised learning. Nevertheless, we note that our model have to be used properly in order to mitigate potential negative societal impacts, such as deep fakes.
sections.heading: Acknowledgements
sections.text: We thank the anonymous reviewers and Kristy Choi for insightful discussions and feedback.
  Funding transparency statement. This research was supported by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1-2145), AFOSR (FA9550-19-1-0024), ARO (W911NF-21-1-0125), Sloan Fellowship, Amazon AWS, Stanford Institute for Human-Centered Artificial Intelligence (HAI), and Google Cloud.
title: D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation