summaries: The paper describes the novel approach to generate high-resolution images in progressive manner. The approach is based on set of conditional flows that take the image generated from previous stage as conditioning factor and generate the image with the higher resolution. The quality of the method is compared to the reference approaches, mainly focusing on WaveletFlow as the most similar method. The paper proposes a multi-resolution variant of continuous normalizing flows for images. They show empirical results for image sizes upto 64 x 64 where they seem to be better than regular continuous normalizing flows. The key proposed benefit seems to be in number of parameters and training times. In this work, the authors propose a multi-resolution strategy for continuous normalizing flows. The proposed approach consists of a general wavelet based decomposition/downsampling where the transformation obeys some useful mathematical properties such as volume and range preservation. Empirical results indicate likelihood estimation performance better or on par with benchmarked methods. The proposed model also trains significantly faster with fewer parameters. This paper proposes a new architecture for continuous normalizing flows that explicitly models images at multiple resolutions. The authors achieve this by learning an unconditional distribution of coarse images and then learning conditional distributions at progressively finer resolutions, each with a continuous normalizing flow. In contrast to the typical “squeeze” layers used in normalizing flows, where the noise is only passed at the first layer, the proposed model injects noise at several resolutions (which has already proved successful with other generative models). The authors introduce a new (invertible) transformation to go from a coarse image to a finer image which preserves the range of the image while having unit determinant (implying the transformation leaves the log likelihood unchanged). The whole model is then trained via maximum likelihood.  The authors evaluate their method on standard image datasets and perform several ablations to test the contributions of their model.  The main contributions of the paper in my eyes are then: - The introduction of a new architecture and layer for learning multi resolution normalizing flows - Demonstration that the proposed model can be trained with limited computational resources even on fairly large scale datasets (such as Imagenet128) - Ablation studies evaluating the improvements from each aspect of the proposed model 
abstractText: Recent work has shown that Neural Ordinary Differential Equations (ODEs) can serve as generative models of images using the perspective of Continuous Normalizing Flows (CNFs). Such models offer exact likelihood calculation, and invertible generation/density estimation. In this work we introduce a Multi-Resolution variant of such models (MRCNF), by characterizing the conditional distribution over the additional information required to generate a fine image that is consistent with the coarse image. We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, using orders of magnitude fewer parameters than the prior methods, in significantly less training time, using only one GPU.
references.authors: Edward H Adelson Charles H Anderson James R Bergen Peter J Burt Joan M Ogden
references.title: Pyramid methods in image processing
references.venue: RCA engineer,
references.authors: Martin Arjovsky Léon Bottou
references.title: Towards principled methods for training generative adversarial networks
references.venue: arXiv preprint arXiv:1701.04862,
references.authors: Jens Behrmann Will Grathwohl Ricky TQ Chen David Duvenaud Jörn-Henrik Jacobsen
references.title: Invertible residual networks
references.venue: In International Conference on Machine Learning,
references.authors: Hugo Berard Gauthier Gidel Amjad Almahairi Pascal Vincent Simon Lacoste-Julien
references.title: A closer look at the optimization landscapes of generative adversarial networks
references.venue: In International Conference on Machine Learning,
references.authors: Andrew Brock Jeff Donahue Karen Simonyan
references.title: Large scale GAN training for high fidelity natural image synthesis
references.venue: In International Conference on Learning Representations,
references.authors: Peter Burt Edward Adelson
references.title: The laplacian pyramid as a compact image code
references.venue: IEEE Transactions on communications,
references.authors: Peter J Burt
references.title: Fast filter transform for image processing
references.venue: Computer graphics and image processing,
references.authors: Jianfei Chen Cheng Lu Biqi Chenli Jun Zhu Tian Tian
references.title: Vflow: More expressive generative flows with variational data augmentation
references.venue: In International Conference on Machine Learning,
references.authors: Ricky T.Q. Chen Yulia Rubanova Jesse Bettencourt David Duvenaud
references.title: Neural ordinary differential equations
references.venue: Advances in Neural Information Processing Systems,
references.authors: Ricky TQ Chen Jens Behrmann David K Duvenaud Jörn-Henrik Jacobsen
references.title: Residual flows for invertible generative modeling
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Xi Chen Nikhil Mishra Mostafa Rohaninejad Pieter Abbeel
references.title: Pixelsnail: An improved autoregressive generative model
references.venue: In International Conference on Machine Learning,
references.authors: Rewon Child Scott Gray Alec Radford Ilya Sutskever
references.title: Generating long sequences with sparse transformers
references.venue: arXiv preprint arXiv:1904.10509,
references.authors: Hyunsun Choi Eric Jang Alexander A Alemi
references.title: Waic, but why? generative ensembles for robust anomaly detection
references.venue: arXiv preprint arXiv:1810.01392,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: Emily L Denton Soumith Chintala Rob Fergus
references.title: Deep generative image models using a laplacian pyramid of adversarial networks
references.venue: In Advances in neural information processing systems,
references.authors: Laurent Dinh Jascha Sohl-Dickstein Samy Bengio
references.title: Density estimation using real nvp
references.venue: In International Conference on Learned Representations,
references.authors: Conor Durkan Artur Bekasov Iain Murray George Papamakarios
references.title: Neural spline flows
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Chris Finlay Jörn-Henrik Jacobsen Levon Nurbekyan Adam Oberman
references.title: How to train your neural ode: the world of jacobian and kinetic regularization
references.venue: International Conference on Machine Learning,
references.authors: Arnab Ghosh Harkirat Singh Behl Emilien Dupont Philip HS Torr Vinay Namboodiri. Steer
references.title: Simple temporal regularization for neural odes
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Will Grathwohl Ricky T.Q. Chen Jesse Bettencourt Ilya Sutskever David Duvenaud
references.title: Ffjord: Free-form continuous dynamics for scalable reversible generative models
references.venue: International Conference on Learning Representations,
references.authors: Matej Grcić Ivan Grubišić Siniša Šegvić
references.title: Densely connected normalizing flows
references.venue: arXiv preprint,
references.authors: Dan Hendrycks Kevin Gimpel
references.title: A baseline for detecting misclassified and out-of-distribution examples in neural networks
references.venue: In International Conference on Learning Representations,
references.authors: Dan Hendrycks Mantas Mazeika Thomas Dietterich
references.title: Deep anomaly detection with outlier exposure
references.venue: In International Conference on Learning Representations,
references.authors: Jonathan Ho Xi Chen Aravind Srinivas Yan Duan Pieter Abbeel
references.title: Flow++: Improving flow-based generative models with variational dequantization and architecture design
references.venue: In International Conference on Machine Learning,
references.authors: Jonathan Ho Nal Kalchbrenner Dirk Weissenborn Tim Salimans
references.title: Axial attention in multidimensional transformers
references.venue: arXiv preprint arXiv:1912.12180,
references.authors: Emiel Hoogeboom Rianne van den Berg Max Welling
references.title: Emerging convolutions for generative normalizing flows
references.venue: In International Conference on Machine Learning,
references.authors: Emiel Hoogeboom Jorn Peters Rianne van den Berg Max Welling
references.title: Integer discrete flows and lossless compression
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Emiel Hoogeboom Victor Garcia Satorras Jakub Tomczak Max Welling
references.title: The convolution exponential and generalized sylvester flows
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Anders Høst-Madsen Elyas Sabeti Chad Walton
references.title: Data discovery and anomaly detection using atypicality: Theory
references.venue: IEEE Transactions on Information Theory,
references.authors: Chin-Wei Huang Laurent Dinh Aaron Courville
references.title: Augmented normalizing flows: Bridging the gap between generative flows and latent variable models
references.venue: arXiv preprint arXiv:2002.07101,
references.authors: Han-Hsien Huang Mi-Yen Yeh
references.title: Accelerating continuous normalizing flow with trajectory polynomial regularization
references.venue: AAAI Conference on Artificial Intelligence,
references.authors: Danilo Jimenez Rezende Shakir Mohamed
references.title: Variational inference with normalizing flows
references.venue: In International Conference on Machine Learning,
references.authors: Heewoo Jun Rewon Child Mark Chen John Schulman Aditya Ramesh Alec Radford Ilya Sutskever
references.title: Distribution augmentation for generative modeling
references.venue: In International Conference on Machine Learning,
references.authors: Mahdi Karami Dale Schuurmans Jascha Sohl-Dickstein Laurent Dinh Daniel Duckworth
references.title: Invertible convolutional flow
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Animesh Karnewar Oliver Wang
references.title: Msg-gan: Multi-scale gradients for generative adversarial networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of gans for improved quality, stability, and variation
references.venue: In International Conference on Learned Representations,
references.authors: Tero Karras Samuli Laine Miika Aittala Janne Hellsten Jaakko Lehtinen Timo Aila
references.title: Analyzing and improving the image quality of stylegan
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jacob Kelly Jesse Bettencourt Matthew James Johnson David Duvenaud
references.title: Learning differential equations that are easy to solve
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114,
references.authors: Diederik P. Kingma Tim Salimans Rafal Jozefowicz Xi Chen Ilya Sutskever Max Welling
references.title: Improving variational inference with inverse autoregressive flow. 2016
references.venue: URL http://arxiv.org/abs/1606
references.authors: Durk P Kingma Prafulla Dhariwal
references.title: Glow: Generative flow with invertible 1x1 convolutions
references.venue: In Advances in neural information processing systems,
references.authors: Polina Kirichenko Pavel Izmailov Andrew Gordon Wilson
references.title: Why normalizing flows fail to detect out-of-distribution data
references.venue: In Advances in neural information processing systems,
references.authors: Ivan Kobyzev Simon Prince Marcus Brubaker
references.title: Normalizing flows: An introduction and review of current methods
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical Report, University of Toronto,
references.authors: Kimin Lee Kibok Lee Honglak Lee Jinwoo Shin
references.title: A simple unified framework for detecting out-ofdistribution samples and adversarial attacks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Sang-gil Lee Sungwon Kim Sungroh Yoon
references.title: Nanoflow: Scalable normalizing flows with sublinear parameter complexity
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Shiyu Liang Yixuan Li Rayadurgam Srikant
references.title: Enhancing the reliability of out-of-distribution image detection in neural networks
references.venue: In International Conference on Learning Representations,
references.authors: Zinan Lin Ashish Khetan Giulia Fanti Sewoong Oh
references.title: Pacgan: The power of two samples in generative adversarial networks
references.venue: In Advances in neural information processing systems,
references.authors: Tony Lindeberg
references.title: Scale-space for discrete signals
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Jonathan Long Evan Shelhamer Trevor Darrell
references.title: Fully convolutional networks for semantic segmentation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: You Lu Bert Huang
references.title: Woodbury transformations for deep generative flows
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Xuezhe Ma Xiang Kong Shanghang Zhang Eduard Hovy
references.title: Macow: Masked convolutional generative flow
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Stephane G Mallat
references.title: A theory for multiresolution signal decomposition: the wavelet representation
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Stéphane G. Mallat Gabriel Peyré
references.title: A wavelet tour of signal processing: the sparse way
references.authors: David Marr
references.title: Vision: A computational investigation into the human representation and processing of visual information
references.venue: MIT press,
references.authors: Jacob Menick Nal Kalchbrenner
references.title: Generating high fidelity images with subscale pixel networks and multidimensional upscaling
references.venue: In International Conference on Learning Representations,
references.authors: Eric Nalisnick Akihiro Matsukawa Yee Whye Teh Dilan Gorur Balaji Lakshminarayanan
references.title: Do deep generative models know what they don’t know
references.venue: In International Conference on Learning Representations,
references.authors: Eric Nalisnick Akihiro Matsukawa Yee Whye Teh Balaji Lakshminarayanan
references.title: Detecting out-ofdistribution inputs to deep generative models using a test for typicality
references.venue: arXiv preprint arXiv:1906.02994,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y. Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: NIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Didrik Nielsen Ole Winther
references.title: Closing the dequantization gap: Pixelcnn as a single-layer flow
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Derek Onken Samy Wu Fung Xingjian Li Lars Ruthotto
references.title: Ot-flow: Fast and accurate continuous normalizing flows via optimal transport
references.venue: AAAI Conference on Artificial Intelligence,
references.authors: Aaron van den Oord Nal Kalchbrenner Koray Kavukcuoglu
references.title: Pixel recurrent neural networks
references.venue: International Conference on Machine Learning,
references.authors: George Papamakarios Eric Nalisnick Danilo Jimenez Rezende Shakir Mohamed Balaji Lakshminarayanan
references.title: Normalizing flows for probabilistic modeling and inference
references.authors: Razvan Pascanu Tomas Mikolov Yoshua Bengio
references.title: On the difficulty of training recurrent neural networks
references.venue: In International conference on machine learning,
references.authors: Alec Radford Luke Metz Soumith Chintala
references.title: Unsupervised representation learning with deep convolutional generative adversarial networks
references.venue: arXiv preprint arXiv:1511.06434,
references.authors: Ali Razavi Aaron van den Oord Oriol Vinyals
references.title: Generating diverse high-fidelity images with vq-vae-2
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Scott Reed Aäron van den Oord Nal Kalchbrenner Sergio Gómez Colmenarejo Ziyu Wang Dan Belov Nando De Freitas
references.title: Parallel multiscale autoregressive density estimation
references.venue: In International Conference on Machine Learning,
references.authors: Elyas Sabeti Anders Høst-Madsen
references.title: Data discovery and anomaly detection using atypicality for real-valued data
references.authors: Joan Serrà David Álvarez Vicenç Gómez Olga Slizovskaia José F Núñez Jordi Luque
references.title: Input complexity and out-of-distribution detection with likelihood-based generative models
references.venue: In International Conference on Learning Representations,
references.authors: Tamar Rott Shaham Tali Dekel Tomer Michaeli
references.title: Singan: Learning a generative model from a single natural image
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Jon Sneyers Pieter
references.title: Wuille. Flif: Free lossless image format based on maniac compression
references.venue: In 2016 IEEE International Conference on Image Processing (ICIP),
references.authors: Yang Song Chenlin Meng Stefano Ermon
references.title: Mintnet: Building invertible neural networks with masked convolutions
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Esteban G Tabak Cristina V Turner
references.title: A family of nonparametric density estimation algorithms
references.venue: Communications on Pure and Applied Mathematics,
references.authors: Lucas Theis Aäron van den Oord Matthias Bethge
references.title: A note on the evaluation of generative models
references.venue: In International Conference on Learning Representations,
references.authors: Arash Vahdat Jan Kautz
references.title: Nvae: A deep hierarchical variational autoencoder
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Aaron Van den Oord Nal Kalchbrenner Lasse Espeholt Oriol Vinyals Alex Graves
references.title: Conditional image generation with pixelcnn decoders
references.venue: In Advances in neural information processing systems,
references.authors: Richard S Varga
references.title: On a discrete maximum principle
references.venue: SIAM Journal on Numerical Analysis,
references.authors: Andrew P Witkin
references.title: Scale-space filtering
references.venue: In Readings in Computer Vision, pp. 329–332
references.authors: Changyi Xiao Ligang Liu
references.title: Generative flows with matrix exponential
references.venue: In International Conference on Machine Learning,
references.authors: Hanshu Yan Jiawei Du Vincent Y.F. Tan Jiashi Feng
references.title: On robustness of neural ordinary differential equations
references.venue: International Conference on Learning Representations,
references.authors: Jason Yu Konstantinos Derpanis Marcus Brubaker
references.title: Wavelet flow: Fast training of high resolution normalizing flows
references.venue: In Advances in Neural Information Processing Systems,
sections.heading: 1 INTRODUCTION
sections.text: Reversible generative models derived through the use of the change of variables technique (Dinh et al., 2017; Kingma & Dhariwal, 2018; Ho et al., 2019a; Yu et al., 2020) are growing in interest as alternatives to generative models based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2016) and Variational Autoencoders (VAEs) (Kingma & Welling, 2013). While GANs and VAEs have been able to produce visually impressive samples of images, they have a number of limitations. A change of variables approach facilitates the transformation of a simple base probability distribution into a more complex model distribution. Reversible generative models using this technique are attractive because they enable efficient density estimation, efficient sampling, and computation of exact likelihoods.
  A promising variation of the change-of-variable approach is based on the use of a continuous time variant of normalizing flows (Chen et al., 2018a; Grathwohl et al., 2019; Finlay et al., 2020), which uses an integral over continuous time dynamics to transform a base distribution into the model distribution, called Continuous Normalizing Flows (CNF). This approach uses ordinary differential equations (ODEs) specified by a neural network, or Neural ODEs. CNFs have been shown to be capable of modelling complex distributions such as those associated with images.
  While this new paradigm for the generative modelling of images is not as mature as GANs or VAEs in terms of the generated image quality, it is a promising direction of research as it does not have some key shortcomings associated with GANs and VAEs. Specifically, GANs are known to suffer from mode-collapse (Lin et al., 2018), and are notoriously difficult to train (Arjovsky & Bottou, 2017) compared to feed forward networks because their adversarial loss seeks a saddle point instead of a local minimum (Berard et al., 2020). CNFs are trained by mapping images to noise, and their reversible architecture allows images to be generated by going in reverse, from noise to images. This leads to fewer issues related to mode collapse, since any input example in the dataset can be recovered from the flow using the reverse of the transformation learned during training. VAEs only provide a lower bound on the marginal likelihood whereas CNFs provide exact likelihoods. Despite the many advantages of reversible generative models built with CNFs, quantitatively such methods still do not match the widely used Fréchet Inception Distance (FID) scores of GANs or VAEs. However their other advantages motivate us to explore them further.
  Furthermore, state-of-the art GANs and VAEs exploit the multi-resolution properties of images, and recent top-performing methods also inject noise at each resolution (Brock et al., 2019; Shaham et al., 2019; Karras et al., 2020; Vahdat & Kautz, 2020). While shaping noise is fundamental to normalizing flows, only recently have normalizing flows exploited the multi-resolution properties of images. For example, WaveletFlow (Yu et al., 2020) splits an image into multiple resolutions using the Discrete Wavelet Transform, and models the average image at each resolution using a normalizing flow. While this method has advantages, it suffers from many issues such as high parameter count and long training time.
  In this work, we consider a non-trivial multi-resolution approach to continuous normalizing flows, which fixes many of these issues. A high-level view of our approach is shown in Figure 1. Our main contributions are:
  1. We propose a multi-resolution transformation that does not add cost in terms of likelihood.
  2. We introduce Multi-Resolution Continuous Normalizing Flows (MRCNF). 3. We achieve comparable Bits-per-dimension (BPD) (negative log likelihood per pixel) on image
  datasets using fewer model parameters and significantly less training time with only one GPU.
sections.heading: 2 BACKGROUND
sections.heading: 2.1 NORMALIZING FLOWS
sections.text: Normalizing flows (Tabak & Turner, 2013; Jimenez Rezende & Mohamed, 2015; Dinh et al., 2017; Papamakarios et al., 2019; Kobyzev et al., 2020) are generative models that map a complex data distribution, such as real images, to a known noise distribution. They are trained by maximizing the log likelihood of their input images. Suppose a normalizing flow g produces output z from an input x i.e. z = g(x). The change-of-variables formula provides the likelihood of the image under this transformation as:
  log p(x) = log ∣∣∣∣det dgdx ∣∣∣∣+ log p(z) (1)
  The first term on the right (log determinant of the Jacobian) is often intractable, however, previous works on normalizing flows have found ways to estimate this efficiently. The second term, log p(z), is computed as the log probability of z under a known noise distribution, typically the standard Gaussian N (0, I).
sections.heading: 2.2 WAVELET FLOW (YU ET AL., 2020)
sections.text: WaveletFlow decomposes an image using the Discrete Wavelet Transformation, and maps the average image at each resolution to noise using a normalizing flow. The WaveletFlow model builds on the Glow (Kingma & Dhariwal, 2018) architecture, with a high parameter count and a long training time. It uses an orthogonal transformation, which does not preserve the range of the image, and adds a constant term to the log likelihood at each resolution. We aim to fix these issues using our MRCNF.
sections.heading: 2.3 CONTINUOUS NORMALIZING FLOWS
sections.text: Continuous Normalizing Flows (CNF) (Chen et al., 2018a; Grathwohl et al., 2019; Finlay et al., 2020) are a variant of normalizing flows that operate in the continuous domain. A CNF creates a geometric flow between the input and target (noise) distributions, by assuming that the state transition is governed by an Ordinary Differential Equation (ODE). It further assumes that the differential function is parameterized by a neural network, this model is called a Neural ODE (Chen et al., 2018a). Suppose CNF g transforms its state v(t) using a Neural ODE, with neural network f defining the differential. Here, v(t0) = x is, say, an image, and at the final time step v(t1) = z is a sample from a known noise distribution.
  dv(t)
  dt = f(v(t), t) =⇒ v(t1) = g(v(t0)) = v(t0) + ∫ t1 t0 f(v(t), t) dt (2)
  This integration is typically performed by an ODE solver. Since this integration can be run backwards as well to obtain the same v(t0) from v(t1), a CNF is a reversible model.
  Equation 1 can be used to compute the change in log-probability induced by the CNF. However, Chen et al. (2018a) and Grathwohl et al. (2019) proposed a more efficient variant in the context of CNFs, called the instantaneous change-of-variables formula:
  ∂ log p(v(t))
  ∂t = −Tr
  ( ∂f
  ∂v(t)
  ) =⇒ ∆ log pv(t0)→v(t1) = − ∫ t1 t0 Tr ( ∂f ∂v(t) ) dt (3)
  Hence, the change in log-probability of the state of the Neural ODE i.e. ∆ log pv is expressed as another differential equation. The ODE solver now solves both differential equations eq. (2) and eq. (3) by augmenting the original state with the above. Thus, a CNF provides both the final state v(t1) as well as the change in log probability ∆ log pv(t0)→v(t1) together.
  Prior works (Grathwohl et al., 2019; Finlay et al., 2020; Ghosh et al., 2020; Onken et al., 2021; Huang & Yeh, 2021) have trained CNFs as reversible generative models of images, by maximizing the likelihood of the images under the model:
  z = g(x) ; log p(x) = ∆ log px→z + log p(z) (4)
  where x is an image, z and ∆ log px→z are computed by the CNF using eq. (2) and eq. (3), and log p(z) is the likelihood of the computed z under a known noise distribution, typically the standard Gaussian N (0, I). Novel images are generated by sampling z from the known noise distribution, and running it through the CNF in reverse.
sections.heading: 3 OUR METHOD
sections.text: Our method is a reversible generative model of images that builds on top of CNFs. We introduce the notion of multiple resolutions in images, and connect the different resolutions in an autoregressive fashion. This helps generate images faster, with better likelihood values at higher resolutions, using only one GPU in all our experiments. We call this model Multi-Resolution Continuous Normalizing Flow (MRCNF).
sections.heading: 3.1 MULTI-RESOLUTION IMAGE REPRESENTATION
sections.text: Multi-resolution representations of images have been explored in computer vision for decades (Burt, 1981; Marr, 2010; Witkin, 1987; Burt & Adelson, 1983; Mallat, 1989; Lindeberg, 1990). This implies that much of the content of an image at a resolution is a composition of low-level information captured at coarser resolutions, and some high-level information not present in the coarser images. We take advantage of this property by first decomposing an image in resolution space i.e. by expressing it as a series of S images at decreasing resolutions: x → (x1,x2, . . . ,xS), where x1 = x is the finest image, xS is the coarsest, and every xs+1 is the average image of xs. This called an image pyramid, or a Gaussian Pyramid if the upsampling-downsampling operations include a Gaussian filter (Burt, 1981; Burt & Adelson, 1983; Adelson et al., 1984; Witkin, 1987; Lindeberg, 1990). In this work, we obtain a coarser image simply by averaging pixels in every 2×2 patch, thereby halving the width and height. However, this representation is redundant since much of the information in x1 is contained in xs>1. Instead, we express x as a series of high-level information ys not present in the immediate coarser images xs+1, and a final coarse image xS :
  x→ (y1,x2)→ (y1,y2,x3)→ · · · → (y1,y2, . . . ,yS−1,xS) (5) Our overall method is to map these S terms to S noise samples using S CNFs.
  3.2 DEFINING THE HIGH-LEVEL INFORMATION ys
  We choose to design a linear transformation with the following properties: 1) invertible i.e. it should be possible to deterministically obtain xs from ys and xs+1, and vice versa ; 2) volume preserving i.e. determinant is 1, change in log-likelihood is 0 ; 3) angle preserving ; and 4) range preserving (under the notion of the maximum principle (Varga, 1966)).
  Consider the simplest case of 2 resolutions where x1 is a 2×2 image with pixel values x1, x2, x3, x4, and x2 is a 1×1 image with pixel value x̄ = 14 (x1 + x2 + x3 + x4). We require three values (y1, y2, y3) = y1 that contain information not present in x2, such that x1 is obtained when y1 and x2 are combined.
  This could be viewed as a problem of finding a matrix M such that: [x1, x2, x3, x4]> = M [y1, y2, y3, x̄]>. We fix the last column of M as [1, 1, 1, 1]>, since every pixel value in x1 depends on x̄. Finding the rest of the parameters can be viewed as requiring four 3D vectors that are spaced such that they do not degenerate the number of dimensions of their span. These can be considered as the four corners of a tetrahedron in 3D space, under any configuration (rotated in 3D space), and any scaling of the vectors (see Figure 2).
  Out of the many possibilities for this tetrahedron, we could choose the matrix that performs the Discrete Haar Wavelet Transform (Mallat, 1989; Mallat & Peyré, 2009):x1x2x3
  x4
   =  1 2
  1 2
  1 2 1
  1 2 − 1 2 − 1 2 1
  − 12 1 2 − 1 2 1 − 12 − 1 2 1 2 1
   y1y2y3 x̄  ⇐⇒ y1y2y3 x̄  =  1 2
  1 2 − 1 2 − 1 2
  1 2 − 1 2
  1 2 − 1 2
  1 2 − 1 2 − 1 2
  1 2
  1 4
  1 4
  1 4
  1 4
   x1x2x3 x4  (6) However, this has log
  ∣∣det(M−1)∣∣ = log(1/2) (eq. (6)), and is therefore not volume preserving. Other simple scaling of eq. (6) has been used in the past, for example multiplying the last row of eq. (6) by 2, yielding an orthogonal transformation, such as in WaveletFlow (Yu et al., 2020). However, this transformation neither preserves the volume i.e. the log determinant is not 0, nor the maximum i.e. the range of xs changes.
  We wish to find a transformation M where: one of the results is the average of the inputs, x̄; it is unit determinant; the columns are orthogonal; and it preserves the range of x̄. Fortunately such a matrix exists – although we have not seen it discussed in prior literature. It can be seen as a variant of the Discrete Haar Wavelet Transformation matrix that is unimodular, i.e. has a determinant of 1 (and is therefore volume preserving), while also preserving the range of the images for the input and its average:x1x2x3
  x4
   = 1 a  c c c ac −c −c a−c c −c a −c −c c a  y1y2y3 x̄ ⇐⇒ y1y2y3 x̄  = c −1 c−1 −c−1 −c−1 c−1 −c−1 c−1 −c−1 c−1 −c−1 −c−1 c−1 a−1 a−1 a−1 a−1  x1x2x3 x4  (7) where c = 22/3, a = 4. Hence, log
  ∣∣det(M−1)∣∣ = log(1) = 0. This can be scaled up to larger spatial regions by performing the same calculation for each 2×2 patch. Let M be the function that uses matrix M from above and combines every pixel in xs+1 with the three corresponding pixels in ys to make the 2×2 patch at that location in xs using eq. (7): xs = M(ys,xs+1) ⇐⇒ ys,xs+1 = M−1(xs) (8) Equation 1 can be used to compute the change in log likelihood from this transformation xs → (ys,xs+1):
  log p(xs) = ∆ log pxs→(ys,xs+1) + log p(ys,xs+1) = log ∣∣det(M−1)∣∣+ log p(ys | xs+1) + log p(xs+1) (9)
  where log ∣∣det(M−1)∣∣ = dims(xs+1) log(1/2) in the case of eq. (6), where “dims” is the number of pixels
  times the number of channels (typically 3) in the image, and log ∣∣det(M−1)∣∣ = 0 for eq. (7). Although a model could learn to compensate for the additional log-likelihood due to eq. (6), we find that using eq. (7) improves log likelihood values and decreases training time.
sections.heading: 3.3 MULTI-RESOLUTION CONTINUOUS NORMALIZING FLOWS
sections.text: Using the multi-resolution image representation in eq. (5), we characterize the conditional distribution over the additional information (ys) required to generate a higher resolution image (xs) that is consistent with the average (xs+1) over the pixel space. This framework ensures that one coarse image could generate several potential fine images, but these fine images have the same coarse image as their average. This fact is preserved across resolutions. Note that the 3 additional pixels in ys per pixel in xs+1 are generated conditioned on the entire coarser image xs+1, thus maintaining consistency using the full context.
  At each resolution s, we use a CNF to reversibly map between ys (or xS when s=S) and a sample zs from a known noise distribution.
  In principle, any generative model could be used to map between the multi-resolution image and noise. Normalizing flows are good candidates for this as they are probabilistic generative models that perform exact likelihood estimates, and can be run in reverse to generate novel data from the model’s distribution. This allows model comparison and measurement of generalization to unseen data. We choose the CNF variant of normalizing flows at each resolution. CNFs have recently been shown to be effective in modeling image distributions using a fraction of the number of parameters typically used in normalizing flows (and non flow-based approaches), and their underlying framework of Neural ODEs have been shown to be more robust than convolutional layers (Yan et al., 2020).
  Training: We train an MRCNF by maximizing the average log-likelihood of the images in the training dataset under the model. The log probability of each image log p(x) can be estimated recursively from eq. (9) as:
  log p(x) = ∆ log px1→(y1,x2) + log p(y1,x2) = ∆ log px1→(y1,x2) + log p(y1 | x2) + log p(x2)
  = S−1∑ s=1 ( ∆ log pxs→(ys,xs+1) + log p(ys | xs+1) ) + log p(xS) (10)
  where ∆ log pxs→(ys,xs+1) is log ∣∣det(M−1)∣∣, log p(ys | xs+1) and log p(xS) are given by eq. (4):
  zs = gs(ys | xs+1) ; log p(ys | xs+1) = ∆ log p(ys→zs)|xs+1 + log p(zs) (11) zS = gS(xS) ; log p(xS) = ∆ log pxS→zS + log p(zS) (12)
  The coarsest resolution S can be chosen such that the last CNF operates on the image distribution at a small enough resolution that is easy to model unconditionally. All other CNFs are conditioned on the immediate coarser image. The conditioning itself is achieved by concatenating the input image of the CNF with the coarser image. This model could be seen as a stack of CNFs connected in an autoregressive fashion.
  Typically, likelihood-based generative models are compared using the metric of bits-per-dimension (BPD), i.e. the negative log likelihood per pixel in the image. Hence, we train our MRCNF to minimize the average BPD of the images in the training dataset, computed using eq. (13):
  BPD(x) = − log p(x)/dims(x) (13)
  We use FFJORD (Grathwohl et al., 2019) as the baseline model for our CNFs. In addition, we use to two regularization terms introduced by RNODE (Finlay et al., 2020) to speed up the training of FFJORD models by stabilizing the learnt dynamics: the kinetic energy of the flow K(θ), and the Jacobian norm B(θ):
  K(θ) = ∫ t1 t0 ‖f(v(t), t, θ)‖22 dt; B(θ) = ∫ t1 t0 ‖ >∇zf(v(t), t, θ)‖22 dt, ∼ N (0, I) (14)
  Parallel training: Note that although the final log likelihood log p(x) involves sequentially summing over values returned by all S CNFs, the log likelihood term of each CNF is independent of the others. Conditioning is done using ground truth images. Hence, each CNF can be trained independently, in parallel.
  Generation: Given an S-resolution model, we first sample zs, s = 1, . . . , S from the latent noise distributions. The CNF gs at resolution s transforms the noise sample zs to high-level information ys conditioned on the immediate coarse image xs+1 (except gS which is unconditioned). ys and xs+1 are then combined to form xs using M from eq. (7). This process is repeated progressively from coarser to finer resolutions, until the finest resolution image x1 is computed (see Figure 1). It is to be noted that the generated image at one resolution is used to condition the CNF at the finer resolution.{
  xS = g −1 S (zS) s = S ys = g −1 s (zs | xs+1); xs = M(ys,xs+1) s = S-1→ 1
  (15)
sections.heading: 3.4 MULTI-RESOLUTION NOISE
sections.text: We further decompose the noise image as well into its respective coarser components. This means that ultimately we use only one noise image at the finest level, but it is decomposed into multiple resolutions using eq. (7). xs+1 is mapped to noise of a quarter variance, while ys is mapped to noise of c-factored variance (see fig. 1). Although this is optional, it preserves interpretation between the single- and multi-resolution models.
sections.heading: 4 RELATED WORK
sections.text: Multi-resolution approaches already serve as a key component of state-of-the-art GAN (Denton et al., 2015; Karras et al., 2018; Karnewar & Wang, 2020) and VAE (Razavi et al., 2019; Vahdat & Kautz, 2020) based deep generative models. Deconvolutional CNNs (Long et al., 2015; Radford et al., 2015) use upsampling layers to generate images more effectively. Modern state-of-the-art generative models have also injected noise at different levels to improve sample quality (Brock et al., 2019; Karras et al., 2020; Vahdat & Kautz, 2020).
  Several prior works on normalizing flows (Kingma & Dhariwal, 2018; Hoogeboom et al., 2019a;b; Song et al., 2019; Ma et al., 2019; Durkan et al., 2019; Chen et al., 2020; Ho et al., 2019a; Lee et al., 2020; Yu et al., 2020) build on RealNVP (Dinh et al., 2017). Although they achieve great results in terms of BPD and image quality, they nonetheless report results from significantly higher number of parameters (some with 100x!), and several times GPU hours of training. However, many prior works don’t report these metrics.
  STEER (Ghosh et al., 2020) introduced temporal regularization to CNFs by making the final time of integration stochastic. However, we found that this increased training time without significant BPD improvement.
  Comparison to WaveletFlow: We emphasize that there are important and crucial differences between our MRCNF and WaveletFlow. We generalize the notion of a multi-resolution image representation (section 3.2), and show that Wavelets are one case of this general formulation. WaveletFlow builds on the Glow (Kingma & Dhariwal, 2018) architecture, while ours builds on CNFs (Grathwohl et al., 2019; Finlay et al., 2020). We also make use of the notion of multi-resolution decomposition of the noise, which is optional, but is not taken into account by WaveletFlow. WaveletFlow uses an orthogonal transformation which does not preserve range ; our MRCNF uses eq. (7) which is volume-preserving and range-preserving. Finally, WaveletFlow applies special sampling techniques to obtain better samples from its model. We have so far not used such techniques for generation, but we believe they can potentially help our models as well. By making these important changes, we fix many of the previously discussed issues with WaveletFlow. For a more detailed ablation study, please check subsection 5.1.
  “Multiple scales” in prior normalizing flows: Normalizing flows (Dinh et al., 2017; Kingma & Dhariwal, 2018; Grathwohl et al., 2019) try to be “multi-scale” by transforming the input in a smart way (squeezing operation) such that the width of the features progressively reduces in the direction of image to noise, while maintaining the total dimensions. This happens while operating at a single resolution. In contrast, our model stacks normalizing flows at multiple resolutions in an autoregressive fashion by conditioning on the images at coarser resolutions.
sections.heading: 5 EXPERIMENTAL RESULTS
sections.text: We train MRCNF models on the CIFAR10 (Krizhevsky et al., 2009) dataset at finest resolution of 32x32, and the ImageNet (Deng et al., 2009) dataset at 32x32, 64x64, 128x128. We build on top of the code provided in Finlay et al. (2020)1. Our code is attached in the supplementary material. In all cases, we train using only one NVIDIA RTX 20280 Ti GPU with 11GB.
  In Table 1, we compare our results with prior work in terms of (lower is better in all cases) the BPD of the images of the test datasets under the trained models, the number of parameters used by the model, and the number of GPU hours taken to train. The most relevant models for comparison are the 1-resolution FFJORD (Grathwohl et al., 2019) models, and their regularized version RNODE (Finlay et al., 2020), since our model directly converts their architecture into multi-resolution. Other relevant comparisons are previous flow-based methods (Dinh et al., 2017; Kingma & Dhariwal, 2018; Song et al., 2019; Ho et al., 2019a; Yu et al., 2020), however their core architecture (RealNVP (Dinh et al., 2017)) is quite different from FFJORD.
  BPD: At lower resolution spaces, we achieve comparable BPDs in lesser time with far fewer parameters than previous normalizing flows (and non flow-based approaches). However, the power of the multi-resolution formulation is more evident at higher resolutions: we achieve better BPD for ImageNet64 and ImageNet128 (Table 2) with significantly fewer parameters and lower time using only one GPU.
  It is to be noted that we were not able to reproduce the same BPD as provided by STEER (Ghosh et al., 2020), we report the results of our re-implementation in Table 1. A more complete table can be found in the appendix. We also include analysis of out-of-distribution properties in the appendix.
  1https://github.com/cfinlay/ffjord-rnode
  Train time: All our experiments used only one GPU, and took significantly less time to train than 1-resolution CNFs, and all prior works including flow-based and non-flow-based models. Since all the CNFs can be trained in parallel, the actual training time in practice could be much lower than reported.
  Super-resolution: Our formulation also allows for super-resolution of images (Figure 3) free of cost since our framework is autoregressive in resolution.
  Progressive training: We trained an MRCNF model on ImageNet128 by training only the finest resolution (128×128) conditioned on the immediate coarser (64×64) images, and attached it to a 3-resolution model trained on ImageNet64. The resultant 4-resolution Ima-
  geNet128 model gives a BPD of 3.31 (Table 2) with just 2.74M parameters in ≈60 GPU hours.
sections.heading: 5.1 ABLATION STUDY
sections.text: Our MRCNF method differs from WaveletFlow in three respects: 1. we use CNFs, 2. we use eq. (7) instead of eq. (6) as used by WaveletFlow, 3. we use multi-resolution noise. We check the individual effects of these changes in an ablation study in Table 3, and conclude that:
  Thus, our MRCNF model is not a trivial replacement of normalizing flows with CNFs in WaveletFlow. We generalize the notion of multi-resolution image representation, in which the Discrete Wavelet Transform is one of many possibilities. We then derived a unimodular transformation that adds no change in likelihood.
sections.heading: 6 CONCLUSION
sections.text: We have presented a Multi-Resolution approach to Continuous Normalizing Flows (MRCNF). MRCNF models achieve comparable or better performance in significantly less training time, training on a single GPU, with a fraction of the number of parameters of other competitive models. We achieved this by designing a general formulation for a multi-resolution image representation, and derived a unimodular transformation that adds no cost in terms of log-likelihood. Although the likelihood values for 32×32 resolution datasets such as CIFAR10 and ImageNet32 do not improve significantly over the baseline, ImageNet64 and above see a marked improvement. The performance is better for higher resolutions, as seen in the case of ImageNet128.
  We also conducted an ablation study to note the effects of each change introduced in the formulation. We found that our unimodular transformation has a significant contribution towards improving BPD and decreasing training time.
  ETHICS STATEMENT
  In terms of broader social impacts of this work, generative models of images can be used to generate so-called fake images, and this issue has been discussed at length in other works. We have tried to stay away from particularly concerning use of our work. We have not performed experiments on face-specific datasets. Although the ImageNet dataset contains human face images, we have only tested our method on an unconditional model, and have made no effort towards controllable generation. An increasing concern in the machine learning community is that of increased use of computational resources and its impact on the environment. We emphasize lower computational budgets, and achieve comparable performance with far fewer parameters and significantly less training time.
  REPRODUCIBILITY STATEMENT
  To ensure reproducibility of our work, we have attached anonymized version of our code repository in the supplementary material. We have mentioned that our code builds on top of a public code repository of the most relevant prior method: https://github.com/cfinlay/ffjord-rnode. We have included the exact mathematical equations governing our method in Section 3.3, which can be verified in our code. We have mentioned other details of our models in the appendix : number of channels, number of blocks, corresponding total number of parameters. We have also mentioned in the appendix that we perform gradient clipping, and added relevant details there. Our code logs and plots the BPD, number of parameters, and training time for every experiment, and saves samples of generated images at regular intervals. It also plots other values of interest such as change in log likelihood, log probability of mapped noise samples, etc. The code also includes options for launching experiments with configurations of prior methods including FFJORD (Grathwohl et al., 2019), RNODE (Finlay et al., 2020), STEER (Ghosh et al., 2020), and WaveletFlow (Yu et al., 2020) using CNFs.
sections.heading: A OUT-OF-DISTRIBUTION PROPERTIES OF (MR)CNF
sections.text: The derivation of likelihood-based models suggests that the density of an image under the model is an effective measure of its likelihood of being in distribution. However, recent works (Theis et al., 2016; Nalisnick et al., 2019a; Serrà et al., 2020; Nalisnick et al., 2019b) have pointed out that it is possible that images drawn from other distributions have higher model likelihood. Examples have been shown where normalizing flow models (such as Glow) trained on CIFAR10 images assign higher likelihood to SVHN (Netzer et al., 2011) images. This could have serious implications on the practical applicability of these models. Some also note that likelihood-based models do not generate images with good sample quality as they avoid assigning small probability to out-of-distribution (OoD) data points, hence
  using model likelihood (-BPD) for detecting OoD data is not effective.
  We conduct the same experiments with (MR)CNFs, and find that similar conclusions can be drawn. Figure 4 plots the histogram of log likelihood per dimension (-BPD) of OoD images (SVHN, TinyImageNet) under MRCNF models trained on CIFAR10. It can be observed that the likelihood of the OoD SVHN is higher than CIFAR10 for MRCNF, similar to the findings for Glow, PixelCNN, VAE in earlier works (Nalisnick et al., 2019a; Choi et al., 2018; Serrà et al., 2020; Nalisnick et al., 2019b; Kirichenko et al., 2020).
  One possible explanation put forward by Nalisnick et al. (2019b) is that “typical” images are less “likely” than constant images, which is a consequence of the distribution of a Gaussian in high dimensions. Indeed, as our Figure 4 shows, constant images have the highest likelihood under MRCNFs, while randomly generated (uniformly distributed) pixels have the least likelihood (not shown in figure due to space constraints).
  Choi et al. (2018); Nalisnick et al. (2019b) suggest using “typicality” as a better measure of OoD. However, Serrà et al. (2020) observe that the complexity of an image plays a significant role in the training of likelihoodbased generative models. They propose a new metric S as an out-of-distribution detector:
  S(x) = bpd(x)− L(x) (16)
  where L(x) is the complexity of an image x measured as the length of the best compressed version of x (we use FLIF (Sneyers & Wuille, 2016) following Serrà et al. (2020)) normalized by the number of dimensions.
  We perform a similar analysis as Serrà et al. (2020) to test how S compares with -bpd for OoD detection. For different MRCNF models trained on CIFAR10, we compute the area under the receiver operating characteristic curve (auROC) using -bpd and S as standard evaluation for the OoD detection task (Hendrycks et al., 2019; Serrà et al., 2020).
  Table 4 shows that S does perform better than -bpd in the case of (MR)CNFs, similar to the findings in Serrà et al. (2020) for
  Glow and PixelCNN++. It seems that SVHN is easier to detect as OoD for Glow than MRCNFs. However, OoD detection performance is about the same for TinyImageNet. We also observe that MRCNFs are better at OoD than CNFs.
  Other OoD methods (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Sabeti & Høst-Madsen, 2019; Høst-Madsen et al., 2019; Hendrycks et al., 2019) are not suitable in our case, as identified in Serrà et al. (2020).
  A.1 SHUFFLED IN-DISTRIBUTION IMAGES
  Kirichenko et al. (2020) conclude that normalizing flows do not represent images based on their semantic contents, but rather directly encode their visual appearance. We verify this for continuous normalizing flows by estimating the density of in-distribution test images, but with patches of pixels randomly shuffled. Figure 5 (a) shows an example of images of shuffled patches of varying size, Figure 5 (b) shows the graph of the their log-likelihoods.
  That shuffling pixel patches would render the image semantically meaningless is reflected in the Fréchet Inception Distance (FID) between CIFAR10-Train and these sets of shuffled images — 1x1: 340.42, 2x2: 299.99, 4x4: 235.22, 8x8: 101.36, 16x16: 33.06, 32x32 (i.e. CIFAR10-Test): 3.15. However, we see that images with large pixel patches shuffled are quite close in likelihood to the unshuffled images, suggesting that since their visual content has not changed much they are almost as likely as unshuffled images under MRCNFs.
  B FULL TABLE 1
sections.heading: C QUALITATIVE SAMPLES
sections.heading: D SIMPLE EXAMPLE OF DENSITY ESTIMATION
sections.text: For example, if we use Euler method as our ODE solver, for density estimation Equation 2 reduces to:
  v(t1) = v(t0) + (t1 − t0)fs(v(t0), t0 | c) (17)
  where fs is a neural network, t0 represents the "time" at which the state is image x, and t1 is when the state is noise z. We start at scale S with an image sample xS , and assume t0 and t1 are 0 and 1 respectively: zS = xS + fS(xS , t0 | xS−1) zS−1 = xS−1 + fS−1(xS−1, t0 | xS−2) ... z1 = x1 + f1(x1, t0 | x0) z0 = x0 + f0(x0, t0) (18)
sections.heading: E SIMPLE EXAMPLE OF GENERATION
sections.text: For example, if we use Euler method as our ODE solver, for generation Equation 2 reduces to:
  v(t0) = v(t1) + (t0 − t1)fs(v(t1), t1 | c) (19)
  i.e. the state is integrated backwards from t1 (i.e. zs) to t0 (i.e. xs). We start at scale 0 with a noise sample z0, and assume t0 and t1 are 0 and 1 respectively: x0 = z0 − f0(z0, t1) x1 = z1 − f1(z1, t1 | x0) ... xS−1 = zS−1 − fS−1(zS−1, t1 | xS−2) xS = zS − fS(zS , t1 | xS−1) (20)
sections.heading: F MODELS
sections.text: We used the same neural network architecture as in RNODE (Finlay et al., 2020). The CNF at each resolution consists of a stack of bl blocks of a 4-layer deep convolutional network comprised of 3x3 kernels and softplus activation functions, with 64 hidden dimensions, and time t concatenated to the spatial input. In addition, except at the coarsest resolution, the immediate coarser image is also concatenated with the state. The integration time of each piece is [0, 1]. The number of blocks bl and the corresponding total number of parameters are given in Table 6.
sections.heading: G GRADIENT NORM
sections.text: In order to avoid exploding gradients, We clipped the norm of the gradients (Pascanu et al., 2013) by a maximum value of 100.0. In case of using adversarial loss, we first clip the gradients provided by the adversarial loss by 50.0, sum up the gradients provided by the log-likelihood loss, and then clip the summed gradients by 100.0.
  H 8-BIT TO UNIFORM
  The change-of-variables formula gives the change in probability due to the transformation of u to v:
  log p(u) = log p(v) + log ∣∣∣∣det dvdu ∣∣∣∣
  Specifically, the change of variables from an 8-bit image to an image with pixel values in range [0, 1] is:
  b (p) S =
  a (p) S
  256 =⇒ log p(aS) = log p(bS) + log ∣∣∣∣det dbda ∣∣∣∣ =⇒ log p(aS) = log p(bS) + log ( 1
  256 )DS =⇒ log p(aS) = log p(bS)−DS log 256
  =⇒ bpd(aS) = − log p(aS) DS log 2
  = −(log p(bS)−DS log 256)
  DS log 2
  = − log p(bS) DS log 2 + log 256 log 2
  = bpd(x) + 8
  where bpd(x) is given from Equation 13.
sections.heading: I FID V/S TEMPERATURE
sections.text: Table 7 lists the FID values of generated images from MRCNF models trained on CIFAR10, with different temperature settings on the Gaussian.
title: MULTI-RESOLUTION CONTINUOUS NORMALIZING FLOWS