summaries: The paper describes the novel approach to generate high-resolution images in progressive manner. The approach is based on set of conditional flows that take the image generated from previous stage as conditioning factor and generate the image with the higher resolution. The quality of the method is compared to the reference approaches, mainly focusing on WaveletFlow as the most similar method.

summaries: The paper proposes a multi-resolution variant of continuous normalizing flows for images. They show empirical results for image sizes upto 64 x 64 where they seem to be better than regular continuous normalizing flows. The key proposed benefit seems to be in number of parameters and training times.

summaries: In this work, the authors propose a multi-resolution strategy for continuous normalizing flows. The proposed approach consists of a general wavelet based decomposition/downsampling where the transformation obeys some useful mathematical properties such as volume and range preservation. Empirical results indicate likelihood estimation performance better or on par with benchmarked methods. The proposed model also trains significantly faster with fewer parameters.

summaries: This paper proposes a new architecture for continuous normalizing flows that explicitly models images at multiple resolutions. The authors achieve this by learning an unconditional distribution of coarse images and then learning conditional distributions at progressively finer resolutions, each with a continuous normalizing flow. In contrast to the typical “squeeze” layers used in normalizing flows, where the noise is only passed at the first layer, the proposed model injects noise at several resolutions (which has already proved successful with other generative models). The authors introduce a new (invertible) transformation to go from a coarse image to a finer image which preserves the range of the image while having unit determinant (implying the transformation leaves the log likelihood unchanged). The whole model is then trained via maximum likelihood.  The authors evaluate their method on standard image datasets and perform several ablations to test the contributions of their model.  The main contributions of the paper in my eyes are then: - The introduction of a new architecture and layer for learning multi resolution normalizing flows - Demonstration that the proposed model can be trained with limited computational resources even on fairly large scale datasets (such as Imagenet128) - Ablation studies evaluating the improvements from each aspect of the proposed model

abstractText: Recent work has shown that Neural Ordinary Differential Equations (ODEs) can serve as generative models of images using the perspective of Continuous Normalizing Flows (CNFs). Such models offer exact likelihood calculation, and invertible generation/density estimation. In this work we introduce a Multi-Resolution variant of such models (MRCNF), by characterizing the conditional distribution over the additional information required to generate a fine image that is consistent with the coarse image. We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, using orders of magnitude fewer parameters than the prior methods, in significantly less training time, using only one GPU.

authors:
Edward H Adelson
Charles H Anderson
James R Bergen
Peter J Burt
Joan M Ogden

title: Pyramid methods in image processing

venue: RCA engineer,

authors:
Martin Arjovsky
Léon Bottou

title: Towards principled methods for training generative adversarial networks

venue: arXiv preprint arXiv:1701.04862,

authors:
Jens Behrmann
Will Grathwohl
Ricky TQ Chen
David Duvenaud
Jörn-Henrik Jacobsen

title: Invertible residual networks

venue: In International Conference on Machine Learning,

authors:
Hugo Berard
Gauthier Gidel
Amjad Almahairi
Pascal Vincent
Simon Lacoste-Julien

title: A closer look at the optimization landscapes of generative adversarial networks

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale GAN training for high fidelity natural image synthesis

venue: In International Conference on Learning Representations,

authors:
Peter Burt
Edward Adelson

title: The laplacian pyramid as a compact image code

venue: IEEE Transactions on communications,

title: Fast filter transform for image processing

venue: Computer graphics and image processing,

authors:
Jianfei Chen
Cheng Lu
Biqi Chenli
Jun Zhu
Tian Tian

title: Vflow: More expressive generative flows with variational data augmentation

authors:
Ricky T.Q. Chen
Yulia Rubanova
Jesse Bettencourt
David Duvenaud

title: Neural ordinary differential equations

venue: Advances in Neural Information Processing Systems,

authors:
Ricky TQ Chen
Jens Behrmann
David K Duvenaud
Jörn-Henrik Jacobsen

title: Residual flows for invertible generative modeling

venue: In Advances in Neural Information Processing Systems,

authors:
Xi Chen
Nikhil Mishra
Mostafa Rohaninejad
Pieter Abbeel

title: Pixelsnail: An improved autoregressive generative model

authors:
Rewon Child
Scott Gray
Alec Radford
Ilya Sutskever

title: Generating long sequences with sparse transformers

venue: arXiv preprint arXiv:1904.10509,

authors:
Hyunsun Choi
Eric Jang
Alexander A Alemi

title: Waic, but why? generative ensembles for robust anomaly detection

venue: arXiv preprint arXiv:1810.01392,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: IEEE conference on computer vision and pattern recognition,

authors:
Emily L Denton
Soumith Chintala
Rob Fergus

title: Deep generative image models using a laplacian pyramid of adversarial networks

venue: In Advances in neural information processing systems,

authors:
Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio

title: Density estimation using real nvp

venue: In International Conference on Learned Representations,

authors:
Conor Durkan
Artur Bekasov
Iain Murray
George Papamakarios

title: Neural spline flows

authors:
Chris Finlay
Jörn-Henrik Jacobsen
Levon Nurbekyan
Adam Oberman

title: How to train your neural ode: the world of jacobian and kinetic regularization

venue: International Conference on Machine Learning,

authors:
Arnab Ghosh
Harkirat Singh Behl
Emilien Dupont
Philip HS Torr
Vinay Namboodiri. Steer

title: Simple temporal regularization for neural odes

authors:
Will Grathwohl
Ricky T.Q. Chen
Jesse Bettencourt
Ilya Sutskever
David Duvenaud

title: Ffjord: Free-form continuous dynamics for scalable reversible generative models

venue: International Conference on Learning Representations,

authors:
Matej Grcić
Ivan Grubišić
Siniša Šegvić

title: Densely connected normalizing flows

venue: arXiv preprint,

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

authors:
Jonathan Ho
Xi Chen
Aravind Srinivas
Yan Duan
Pieter Abbeel

title: Flow++: Improving flow-based generative models with variational dequantization and architecture design

authors:
Jonathan Ho
Nal Kalchbrenner
Dirk Weissenborn
Tim Salimans

title: Axial attention in multidimensional transformers

venue: arXiv preprint arXiv:1912.12180,

authors:
Emiel Hoogeboom
Rianne van den Berg
Max Welling

title: Emerging convolutions for generative normalizing flows

authors:
Emiel Hoogeboom
Jorn Peters
Rianne van den Berg
Max Welling

title: Integer discrete flows and lossless compression

authors:
Emiel Hoogeboom
Victor Garcia Satorras
Jakub Tomczak
Max Welling

title: The convolution exponential and generalized sylvester flows

authors:
Anders Høst-Madsen
Elyas Sabeti
Chad Walton

title: Data discovery and anomaly detection using atypicality: Theory

venue: IEEE Transactions on Information Theory,

authors:
Chin-Wei Huang
Laurent Dinh
Aaron Courville

title: Augmented normalizing flows: Bridging the gap between generative flows and latent variable models

venue: arXiv preprint arXiv:2002.07101,

authors:
Han-Hsien Huang
Mi-Yen Yeh

title: Accelerating continuous normalizing flow with trajectory polynomial regularization

venue: AAAI Conference on Artificial Intelligence,

authors:
Danilo Jimenez Rezende
Shakir Mohamed

title: Variational inference with normalizing flows

authors:
Heewoo Jun
Rewon Child
Mark Chen
John Schulman
Aditya Ramesh
Alec Radford
Ilya Sutskever

title: Distribution augmentation for generative modeling

authors:
Mahdi Karami
Dale Schuurmans
Jascha Sohl-Dickstein
Laurent Dinh
Daniel Duckworth

title: Invertible convolutional flow

authors:
Animesh Karnewar
Oliver Wang

title: Msg-gan: Multi-scale gradients for generative adversarial networks

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

authors:
Tero Karras
Samuli Laine
Miika Aittala
Janne Hellsten
Jaakko Lehtinen
Timo Aila

title: Analyzing and improving the image quality of stylegan

authors:
Jacob Kelly
Jesse Bettencourt
Matthew James Johnson
David Duvenaud

title: Learning differential equations that are easy to solve

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Diederik P. Kingma
Tim Salimans
Rafal Jozefowicz
Xi Chen
Ilya Sutskever
Max Welling

title: Improving variational inference with inverse autoregressive flow. 2016

venue: URL http://arxiv.org/abs/1606

authors:
Durk P Kingma
Prafulla Dhariwal

title: Glow: Generative flow with invertible 1x1 convolutions

authors:
Polina Kirichenko
Pavel Izmailov
Andrew Gordon Wilson

title: Why normalizing flows fail to detect out-of-distribution data

authors:
Ivan Kobyzev
Simon Prince
Marcus Brubaker

title: Normalizing flows: An introduction and review of current methods

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

venue: Technical Report, University of Toronto,

authors:
Kimin Lee
Kibok Lee
Honglak Lee
Jinwoo Shin

title: A simple unified framework for detecting out-ofdistribution samples and adversarial attacks

authors:
Sang-gil Lee
Sungwon Kim
Sungroh Yoon

title: Nanoflow: Scalable normalizing flows with sublinear parameter complexity

authors:
Shiyu Liang
Yixuan Li
Rayadurgam Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

authors:
Zinan Lin
Ashish Khetan
Giulia Fanti
Sewoong Oh

title: Pacgan: The power of two samples in generative adversarial networks

title: Scale-space for discrete signals

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Jonathan Long
Evan Shelhamer
Trevor Darrell

title: Fully convolutional networks for semantic segmentation

authors:
You Lu
Bert Huang

title: Woodbury transformations for deep generative flows

authors:
Xuezhe Ma
Xiang Kong
Shanghang Zhang
Eduard Hovy

title: Macow: Masked convolutional generative flow

title: A theory for multiresolution signal decomposition: the wavelet representation

authors:
Stéphane G. Mallat
Gabriel Peyré

title: A wavelet tour of signal processing: the sparse way

title: Vision: A computational investigation into the human representation and processing of visual information

venue: MIT press,

authors:
Jacob Menick
Nal Kalchbrenner

title: Generating high fidelity images with subscale pixel networks and multidimensional upscaling

authors:
Eric Nalisnick
Akihiro Matsukawa
Yee Whye Teh
Dilan Gorur
Balaji Lakshminarayanan

title: Do deep generative models know what they don’t know

authors:
Eric Nalisnick
Akihiro Matsukawa
Yee Whye Teh
Balaji Lakshminarayanan

title: Detecting out-ofdistribution inputs to deep generative models using a test for typicality

venue: arXiv preprint arXiv:1906.02994,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y. Ng

title: Reading digits in natural images with unsupervised feature learning

venue: NIPS Workshop on Deep Learning and Unsupervised Feature Learning,

authors:
Didrik Nielsen
Ole Winther

title: Closing the dequantization gap: Pixelcnn as a single-layer flow

authors:
Derek Onken
Samy Wu Fung
Xingjian Li
Lars Ruthotto

title: Ot-flow: Fast and accurate continuous normalizing flows via optimal transport

authors:
Aaron van den Oord
Nal Kalchbrenner
Koray Kavukcuoglu

title: Pixel recurrent neural networks

authors:
George Papamakarios
Eric Nalisnick
Danilo Jimenez Rezende
Shakir Mohamed
Balaji Lakshminarayanan

title: Normalizing flows for probabilistic modeling and inference

authors:
Razvan Pascanu
Tomas Mikolov
Yoshua Bengio

title: On the difficulty of training recurrent neural networks

venue: In International conference on machine learning,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

venue: arXiv preprint arXiv:1511.06434,

authors:
Ali Razavi
Aaron van den Oord
Oriol Vinyals

title: Generating diverse high-fidelity images with vq-vae-2

authors:
Scott Reed
Aäron van den Oord
Nal Kalchbrenner
Sergio Gómez Colmenarejo
Ziyu Wang
Dan Belov
Nando De Freitas

title: Parallel multiscale autoregressive density estimation

authors:
Elyas Sabeti
Anders Høst-Madsen

title: Data discovery and anomaly detection using atypicality for real-valued data

authors:
Joan Serrà
David Álvarez
Vicenç Gómez
Olga Slizovskaia
José F Núñez
Jordi Luque

title: Input complexity and out-of-distribution detection with likelihood-based generative models

authors:
Tamar Rott Shaham
Tali Dekel
Tomer Michaeli

title: Singan: Learning a generative model from a single natural image

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Jon Sneyers
Pieter

title: Wuille. Flif: Free lossless image format based on maniac compression

venue: In 2016 IEEE International Conference on Image Processing (ICIP),

authors:
Yang Song
Chenlin Meng
Stefano Ermon

title: Mintnet: Building invertible neural networks with masked convolutions

authors:
Esteban G Tabak
Cristina V Turner

title: A family of nonparametric density estimation algorithms

venue: Communications on Pure and Applied Mathematics,

authors:
Lucas Theis
Aäron van den Oord
Matthias Bethge

title: A note on the evaluation of generative models

authors:
Arash Vahdat
Jan Kautz

title: Nvae: A deep hierarchical variational autoencoder

authors:
Aaron Van den Oord
Nal Kalchbrenner
Lasse Espeholt
Oriol Vinyals
Alex Graves

title: Conditional image generation with pixelcnn decoders

title: On a discrete maximum principle

venue: SIAM Journal on Numerical Analysis,

title: Scale-space filtering

venue: In Readings in Computer Vision, pp. 329–332

authors:
Changyi Xiao
Ligang Liu

title: Generative flows with matrix exponential

authors:
Hanshu Yan
Jiawei Du
Vincent Y.F. Tan
Jiashi Feng

title: On robustness of neural ordinary differential equations

authors:
Jason Yu
Konstantinos Derpanis
Marcus Brubaker

title: Wavelet flow: Fast training of high resolution normalizing flows

heading: 1 INTRODUCTION

text:
Reversible generative models derived through the use of the change of variables technique (Dinh et al., 2017; Kingma & Dhariwal, 2018; Ho et al., 2019a; Yu et al., 2020) are growing in interest as alternatives to generative models based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2016) and Variational Autoencoders (VAEs) (Kingma & Welling, 2013). While GANs and VAEs have been able to produce visually impressive samples of images, they have a number of limitations. A change of variables approach facilitates the transformation of a simple base probability distribution into a more complex model distribution. Reversible generative models using this technique are attractive because they enable efficient density estimation, efficient sampling, and computation of exact likelihoods.
A promising variation of the change-of-variable approach is based on the use of a continuous time variant of normalizing flows (Chen et al., 2018a; Grathwohl et al., 2019; Finlay et al., 2020), which uses an integral over continuous time dynamics to transform a base distribution into the model distribution, called Continuous Normalizing Flows (CNF). This approach uses ordinary differential equations (ODEs) specified by a neural network, or Neural ODEs. CNFs have been shown to be capable of modelling complex distributions such as those associated with images.
While this new paradigm for the generative modelling of images is not as mature as GANs or VAEs in terms of the generated image quality, it is a promising direction of research as it does not have some key shortcomings associated with GANs and VAEs. Specifically, GANs are known to suffer from mode-collapse (Lin et al., 2018), and are notoriously difficult to train (Arjovsky & Bottou, 2017) compared to feed forward networks because their adversarial loss seeks a saddle point instead of a local minimum (Berard et al., 2020). CNFs are trained by mapping images to noise, and their reversible architecture allows images to be generated by going in reverse, from noise to images. This leads to fewer issues related to mode collapse, since any input example in the dataset can be recovered from the flow using the reverse of the transformation learned during training. VAEs only provide a lower bound on the marginal likelihood whereas CNFs provide exact likelihoods. Despite the many advantages of reversible generative models built with CNFs, quantitatively such methods still do not match the widely used Fréchet Inception Distance (FID) scores of GANs or VAEs. However their other advantages motivate us to explore them further.
Furthermore, state-of-the art GANs and VAEs exploit the multi-resolution properties of images, and recent top-performing methods also inject noise at each resolution (Brock et al., 2019; Shaham et al., 2019; Karras et al., 2020; Vahdat & Kautz, 2020). While shaping noise is fundamental to normalizing flows, only recently have normalizing flows exploited the multi-resolution properties of images. For example, WaveletFlow (Yu et al., 2020) splits an image into multiple resolutions using the Discrete Wavelet Transform, and models the average image at each resolution using a normalizing flow. While this method has advantages, it suffers from many issues such as high parameter count and long training time.
In this work, we consider a non-trivial multi-resolution approach to continuous normalizing flows, which fixes many of these issues. A high-level view of our approach is shown in Figure 1. Our main contributions are:
1. We propose a multi-resolution transformation that does not add cost in terms of likelihood.
2. We introduce Multi-Resolution Continuous Normalizing Flows (MRCNF). 3. We achieve comparable Bits-per-dimension (BPD) (negative log likelihood per pixel) on image
datasets using fewer model parameters and significantly less training time with only one GPU.

heading: 2 BACKGROUND

heading: 2.1 NORMALIZING FLOWS

text:
Normalizing flows (Tabak & Turner, 2013; Jimenez Rezende & Mohamed, 2015; Dinh et al., 2017; Papamakarios et al., 2019; Kobyzev et al., 2020) are generative models that map a complex data distribution, such as real images, to a known noise distribution. They are trained by maximizing the log likelihood of their input images. Suppose a normalizing flow g produces output z from an input x i.e. z = g(x). The change-of-variables formula provides the likelihood of the image under this transformation as:
log p(x) = log ∣∣∣∣det dgdx ∣∣∣∣+ log p(z) (1)
The first term on the right (log determinant of the Jacobian) is often intractable, however, previous works on normalizing flows have found ways to estimate this efficiently. The second term, log p(z), is computed as the log probability of z under a known noise distribution, typically the standard Gaussian N (0, I).

heading: 2.2 WAVELET FLOW (YU ET AL., 2020)

text: WaveletFlow decomposes an image using the Discrete Wavelet Transformation, and maps the average image at each resolution to noise using a normalizing flow. The WaveletFlow model builds on the Glow (Kingma & Dhariwal, 2018) architecture, with a high parameter count and a long training time. It uses an orthogonal transformation, which does not preserve the range of the image, and adds a constant term to the log likelihood at each resolution. We aim to fix these issues using our MRCNF.

heading: 2.3 CONTINUOUS NORMALIZING FLOWS

text:
Continuous Normalizing Flows (CNF) (Chen et al., 2018a; Grathwohl et al., 2019; Finlay et al., 2020) are a variant of normalizing flows that operate in the continuous domain. A CNF creates a geometric flow between the input and target (noise) distributions, by assuming that the state transition is governed by an Ordinary Differential Equation (ODE). It further assumes that the differential function is parameterized by a neural network, this model is called a Neural ODE (Chen et al., 2018a). Suppose CNF g transforms its state v(t) using a Neural ODE, with neural network f defining the differential. Here, v(t0) = x is, say, an image, and at the final time step v(t1) = z is a sample from a known noise distribution.
dv(t)
dt = f(v(t), t) =⇒ v(t1) = g(v(t0)) = v(t0) + ∫ t1 t0 f(v(t), t) dt (2)
This integration is typically performed by an ODE solver. Since this integration can be run backwards as well to obtain the same v(t0) from v(t1), a CNF is a reversible model.
Equation 1 can be used to compute the change in log-probability induced by the CNF. However, Chen et al. (2018a) and Grathwohl et al. (2019) proposed a more efficient variant in the context of CNFs, called the instantaneous change-of-variables formula:
∂ log p(v(t))
∂t = −Tr
( ∂f
∂v(t)
) =⇒ ∆ log pv(t0)→v(t1) = − ∫ t1 t0 Tr ( ∂f ∂v(t) ) dt (3)
Hence, the change in log-probability of the state of the Neural ODE i.e. ∆ log pv is expressed as another differential equation. The ODE solver now solves both differential equations eq. (2) and eq. (3) by augmenting the original state with the above. Thus, a CNF provides both the final state v(t1) as well as the change in log probability ∆ log pv(t0)→v(t1) together.
Prior works (Grathwohl et al., 2019; Finlay et al., 2020; Ghosh et al., 2020; Onken et al., 2021; Huang & Yeh, 2021) have trained CNFs as reversible generative models of images, by maximizing the likelihood of the images under the model:
z = g(x) ; log p(x) = ∆ log px→z + log p(z) (4)
where x is an image, z and ∆ log px→z are computed by the CNF using eq. (2) and eq. (3), and log p(z) is the likelihood of the computed z under a known noise distribution, typically the standard Gaussian N (0, I). Novel images are generated by sampling z from the known noise distribution, and running it through the CNF in reverse.

heading: 3 OUR METHOD

text: Our method is a reversible generative model of images that builds on top of CNFs. We introduce the notion of multiple resolutions in images, and connect the different resolutions in an autoregressive fashion. This helps generate images faster, with better likelihood values at higher resolutions, using only one GPU in all our experiments. We call this model Multi-Resolution Continuous Normalizing Flow (MRCNF).

heading: 3.1 MULTI-RESOLUTION IMAGE REPRESENTATION

text:
Multi-resolution representations of images have been explored in computer vision for decades (Burt, 1981; Marr, 2010; Witkin, 1987; Burt & Adelson, 1983; Mallat, 1989; Lindeberg, 1990). This implies that much of the content of an image at a resolution is a composition of low-level information captured at coarser resolutions, and some high-level information not present in the coarser images. We take advantage of this property by first decomposing an image in resolution space i.e. by expressing it as a series of S images at decreasing resolutions: x → (x1,x2, . . . ,xS), where x1 = x is the finest image, xS is the coarsest, and every xs+1 is the average image of xs. This called an image pyramid, or a Gaussian Pyramid if the upsampling-downsampling operations include a Gaussian filter (Burt, 1981; Burt & Adelson, 1983; Adelson et al., 1984; Witkin, 1987; Lindeberg, 1990). In this work, we obtain a coarser image simply by averaging pixels in every 2×2 patch, thereby halving the width and height. However, this representation is redundant since much of the information in x1 is contained in xs>1. Instead, we express x as a series of high-level information ys not present in the immediate coarser images xs+1, and a final coarse image xS :
x→ (y1,x2)→ (y1,y2,x3)→ · · · → (y1,y2, . . . ,yS−1,xS) (5) Our overall method is to map these S terms to S noise samples using S CNFs.
3.2 DEFINING THE HIGH-LEVEL INFORMATION ys
We choose to design a linear transformation with the following properties: 1) invertible i.e. it should be possible to deterministically obtain xs from ys and xs+1, and vice versa ; 2) volume preserving i.e. determinant is 1, change in log-likelihood is 0 ; 3) angle preserving ; and 4) range preserving (under the notion of the maximum principle (Varga, 1966)).
Consider the simplest case of 2 resolutions where x1 is a 2×2 image with pixel values x1, x2, x3, x4, and x2 is a 1×1 image with pixel value x̄ = 14 (x1 + x2 + x3 + x4). We require three values (y1, y2, y3) = y1 that contain information not present in x2, such that x1 is obtained when y1 and x2 are combined.
This could be viewed as a problem of finding a matrix M such that: [x1, x2, x3, x4]> = M [y1, y2, y3, x̄]>. We fix the last column of M as [1, 1, 1, 1]>, since every pixel value in x1 depends on x̄. Finding the rest of the parameters can be viewed as requiring four 3D vectors that are spaced such that they do not degenerate the number of dimensions of their span. These can be considered as the four corners of a tetrahedron in 3D space, under any configuration (rotated in 3D space), and any scaling of the vectors (see Figure 2).
Out of the many possibilities for this tetrahedron, we could choose the matrix that performs the Discrete Haar Wavelet Transform (Mallat, 1989; Mallat & Peyré, 2009):x1x2x3
x4
 =  1 2
1 2
1 2 1
1 2 − 1 2 − 1 2 1
− 12 1 2 − 1 2 1 − 12 − 1 2 1 2 1
 y1y2y3 x̄  ⇐⇒ y1y2y3 x̄  =  1 2
1 2 − 1 2 − 1 2
1 2 − 1 2
1 2 − 1 2
1 2 − 1 2 − 1 2
1 2
1 4
1 4
1 4
1 4
 x1x2x3 x4  (6) However, this has log
∣∣det(M−1)∣∣ = log(1/2) (eq. (6)), and is therefore not volume preserving. Other simple scaling of eq. (6) has been used in the past, for example multiplying the last row of eq. (6) by 2, yielding an orthogonal transformation, such as in WaveletFlow (Yu et al., 2020). However, this transformation neither preserves the volume i.e. the log determinant is not 0, nor the maximum i.e. the range of xs changes.
We wish to find a transformation M where: one of the results is the average of the inputs, x̄; it is unit determinant; the columns are orthogonal; and it preserves the range of x̄. Fortunately such a matrix exists – although we have not seen it discussed in prior literature. It can be seen as a variant of the Discrete Haar Wavelet Transformation matrix that is unimodular, i.e. has a determinant of 1 (and is therefore volume preserving), while also preserving the range of the images for the input and its average:x1x2x3
x4
 = 1 a  c c c ac −c −c a−c c −c a −c −c c a  y1y2y3 x̄ ⇐⇒ y1y2y3 x̄  = c −1 c−1 −c−1 −c−1 c−1 −c−1 c−1 −c−1 c−1 −c−1 −c−1 c−1 a−1 a−1 a−1 a−1  x1x2x3 x4  (7) where c = 22/3, a = 4. Hence, log
∣∣det(M−1)∣∣ = log(1) = 0. This can be scaled up to larger spatial regions by performing the same calculation for each 2×2 patch. Let M be the function that uses matrix M from above and combines every pixel in xs+1 with the three corresponding pixels in ys to make the 2×2 patch at that location in xs using eq. (7): xs = M(ys,xs+1) ⇐⇒ ys,xs+1 = M−1(xs) (8) Equation 1 can be used to compute the change in log likelihood from this transformation xs → (ys,xs+1):
log p(xs) = ∆ log pxs→(ys,xs+1) + log p(ys,xs+1) = log ∣∣det(M−1)∣∣+ log p(ys | xs+1) + log p(xs+1) (9)
where log ∣∣det(M−1)∣∣ = dims(xs+1) log(1/2) in the case of eq. (6), where “dims” is the number of pixels
times the number of channels (typically 3) in the image, and log ∣∣det(M−1)∣∣ = 0 for eq. (7). Although a model could learn to compensate for the additional log-likelihood due to eq. (6), we find that using eq. (7) improves log likelihood values and decreases training time.

heading: 3.3 MULTI-RESOLUTION CONTINUOUS NORMALIZING FLOWS

text:
Using the multi-resolution image representation in eq. (5), we characterize the conditional distribution over the additional information (ys) required to generate a higher resolution image (xs) that is consistent with the average (xs+1) over the pixel space. This framework ensures that one coarse image could generate several potential fine images, but these fine images have the same coarse image as their average. This fact is preserved across resolutions. Note that the 3 additional pixels in ys per pixel in xs+1 are generated conditioned on the entire coarser image xs+1, thus maintaining consistency using the full context.
At each resolution s, we use a CNF to reversibly map between ys (or xS when s=S) and a sample zs from a known noise distribution.
In principle, any generative model could be used to map between the multi-resolution image and noise. Normalizing flows are good candidates for this as they are probabilistic generative models that perform exact likelihood estimates, and can be run in reverse to generate novel data from the model’s distribution. This allows model comparison and measurement of generalization to unseen data. We choose the CNF variant of normalizing flows at each resolution. CNFs have recently been shown to be effective in modeling image distributions using a fraction of the number of parameters typically used in normalizing flows (and non flow-based approaches), and their underlying framework of Neural ODEs have been shown to be more robust than convolutional layers (Yan et al., 2020).
Training: We train an MRCNF by maximizing the average log-likelihood of the images in the training dataset under the model. The log probability of each image log p(x) can be estimated recursively from eq. (9) as:
log p(x) = ∆ log px1→(y1,x2) + log p(y1,x2) = ∆ log px1→(y1,x2) + log p(y1 | x2) + log p(x2)
= S−1∑ s=1 ( ∆ log pxs→(ys,xs+1) + log p(ys | xs+1) ) + log p(xS) (10)
where ∆ log pxs→(ys,xs+1) is log ∣∣det(M−1)∣∣, log p(ys | xs+1) and log p(xS) are given by eq. (4):
zs = gs(ys | xs+1) ; log p(ys | xs+1) = ∆ log p(ys→zs)|xs+1 + log p(zs) (11) zS = gS(xS) ; log p(xS) = ∆ log pxS→zS + log p(zS) (12)
The coarsest resolution S can be chosen such that the last CNF operates on the image distribution at a small enough resolution that is easy to model unconditionally. All other CNFs are conditioned on the immediate coarser image. The conditioning itself is achieved by concatenating the input image of the CNF with the coarser image. This model could be seen as a stack of CNFs connected in an autoregressive fashion.
Typically, likelihood-based generative models are compared using the metric of bits-per-dimension (BPD), i.e. the negative log likelihood per pixel in the image. Hence, we train our MRCNF to minimize the average BPD of the images in the training dataset, computed using eq. (13):
BPD(x) = − log p(x)/dims(x) (13)
We use FFJORD (Grathwohl et al., 2019) as the baseline model for our CNFs. In addition, we use to two regularization terms introduced by RNODE (Finlay et al., 2020) to speed up the training of FFJORD models by stabilizing the learnt dynamics: the kinetic energy of the flow K(θ), and the Jacobian norm B(θ):
K(θ) = ∫ t1 t0 ‖f(v(t), t, θ)‖22 dt; B(θ) = ∫ t1 t0 ‖ >∇zf(v(t), t, θ)‖22 dt, ∼ N (0, I) (14)
Parallel training: Note that although the final log likelihood log p(x) involves sequentially summing over values returned by all S CNFs, the log likelihood term of each CNF is independent of the others. Conditioning is done using ground truth images. Hence, each CNF can be trained independently, in parallel.
Generation: Given an S-resolution model, we first sample zs, s = 1, . . . , S from the latent noise distributions. The CNF gs at resolution s transforms the noise sample zs to high-level information ys conditioned on the immediate coarse image xs+1 (except gS which is unconditioned). ys and xs+1 are then combined to form xs using M from eq. (7). This process is repeated progressively from coarser to finer resolutions, until the finest resolution image x1 is computed (see Figure 1). It is to be noted that the generated image at one resolution is used to condition the CNF at the finer resolution.{
xS = g −1 S (zS) s = S ys = g −1 s (zs | xs+1); xs = M(ys,xs+1) s = S-1→ 1
(15)

heading: 3.4 MULTI-RESOLUTION NOISE

text: We further decompose the noise image as well into its respective coarser components. This means that ultimately we use only one noise image at the finest level, but it is decomposed into multiple resolutions using eq. (7). xs+1 is mapped to noise of a quarter variance, while ys is mapped to noise of c-factored variance (see fig. 1). Although this is optional, it preserves interpretation between the single- and multi-resolution models.

heading: 4 RELATED WORK

text:
Multi-resolution approaches already serve as a key component of state-of-the-art GAN (Denton et al., 2015; Karras et al., 2018; Karnewar & Wang, 2020) and VAE (Razavi et al., 2019; Vahdat & Kautz, 2020) based deep generative models. Deconvolutional CNNs (Long et al., 2015; Radford et al., 2015) use upsampling layers to generate images more effectively. Modern state-of-the-art generative models have also injected noise at different levels to improve sample quality (Brock et al., 2019; Karras et al., 2020; Vahdat & Kautz, 2020).
Several prior works on normalizing flows (Kingma & Dhariwal, 2018; Hoogeboom et al., 2019a;b; Song et al., 2019; Ma et al., 2019; Durkan et al., 2019; Chen et al., 2020; Ho et al., 2019a; Lee et al., 2020; Yu et al., 2020) build on RealNVP (Dinh et al., 2017). Although they achieve great results in terms of BPD and image quality, they nonetheless report results from significantly higher number of parameters (some with 100x!), and several times GPU hours of training. However, many prior works don’t report these metrics.
STEER (Ghosh et al., 2020) introduced temporal regularization to CNFs by making the final time of integration stochastic. However, we found that this increased training time without significant BPD improvement.
Comparison to WaveletFlow: We emphasize that there are important and crucial differences between our MRCNF and WaveletFlow. We generalize the notion of a multi-resolution image representation (section 3.2), and show that Wavelets are one case of this general formulation. WaveletFlow builds on the Glow (Kingma & Dhariwal, 2018) architecture, while ours builds on CNFs (Grathwohl et al., 2019; Finlay et al., 2020). We also make use of the notion of multi-resolution decomposition of the noise, which is optional, but is not taken into account by WaveletFlow. WaveletFlow uses an orthogonal transformation which does not preserve range ; our MRCNF uses eq. (7) which is volume-preserving and range-preserving. Finally, WaveletFlow applies special sampling techniques to obtain better samples from its model. We have so far not used such techniques for generation, but we believe they can potentially help our models as well. By making these important changes, we fix many of the previously discussed issues with WaveletFlow. For a more detailed ablation study, please check subsection 5.1.
“Multiple scales” in prior normalizing flows: Normalizing flows (Dinh et al., 2017; Kingma & Dhariwal, 2018; Grathwohl et al., 2019) try to be “multi-scale” by transforming the input in a smart way (squeezing operation) such that the width of the features progressively reduces in the direction of image to noise, while maintaining the total dimensions. This happens while operating at a single resolution. In contrast, our model stacks normalizing flows at multiple resolutions in an autoregressive fashion by conditioning on the images at coarser resolutions.

heading: 5 EXPERIMENTAL RESULTS

text:
We train MRCNF models on the CIFAR10 (Krizhevsky et al., 2009) dataset at finest resolution of 32x32, and the ImageNet (Deng et al., 2009) dataset at 32x32, 64x64, 128x128. We build on top of the code provided in Finlay et al. (2020)1. Our code is attached in the supplementary material. In all cases, we train using only one NVIDIA RTX 20280 Ti GPU with 11GB.
In Table 1, we compare our results with prior work in terms of (lower is better in all cases) the BPD of the images of the test datasets under the trained models, the number of parameters used by the model, and the number of GPU hours taken to train. The most relevant models for comparison are the 1-resolution FFJORD (Grathwohl et al., 2019) models, and their regularized version RNODE (Finlay et al., 2020), since our model directly converts their architecture into multi-resolution. Other relevant comparisons are previous flow-based methods (Dinh et al., 2017; Kingma & Dhariwal, 2018; Song et al., 2019; Ho et al., 2019a; Yu et al., 2020), however their core architecture (RealNVP (Dinh et al., 2017)) is quite different from FFJORD.
BPD: At lower resolution spaces, we achieve comparable BPDs in lesser time with far fewer parameters than previous normalizing flows (and non flow-based approaches). However, the power of the multi-resolution formulation is more evident at higher resolutions: we achieve better BPD for ImageNet64 and ImageNet128 (Table 2) with significantly fewer parameters and lower time using only one GPU.
It is to be noted that we were not able to reproduce the same BPD as provided by STEER (Ghosh et al., 2020), we report the results of our re-implementation in Table 1. A more complete table can be found in the appendix. We also include analysis of out-of-distribution properties in the appendix.
1https://github.com/cfinlay/ffjord-rnode
Train time: All our experiments used only one GPU, and took significantly less time to train than 1-resolution CNFs, and all prior works including flow-based and non-flow-based models. Since all the CNFs can be trained in parallel, the actual training time in practice could be much lower than reported.
Super-resolution: Our formulation also allows for super-resolution of images (Figure 3) free of cost since our framework is autoregressive in resolution.
Progressive training: We trained an MRCNF model on ImageNet128 by training only the finest resolution (128×128) conditioned on the immediate coarser (64×64) images, and attached it to a 3-resolution model trained on ImageNet64. The resultant 4-resolution Ima-
geNet128 model gives a BPD of 3.31 (Table 2) with just 2.74M parameters in ≈60 GPU hours.

heading: 5.1 ABLATION STUDY

text:
Our MRCNF method differs from WaveletFlow in three respects: 1. we use CNFs, 2. we use eq. (7) instead of eq. (6) as used by WaveletFlow, 3. we use multi-resolution noise. We check the individual effects of these changes in an ablation study in Table 3, and conclude that:
Thus, our MRCNF model is not a trivial replacement of normalizing flows with CNFs in WaveletFlow. We generalize the notion of multi-resolution image representation, in which the Discrete Wavelet Transform is one of many possibilities. We then derived a unimodular transformation that adds no change in likelihood.

heading: 6 CONCLUSION

text:
We have presented a Multi-Resolution approach to Continuous Normalizing Flows (MRCNF). MRCNF models achieve comparable or better performance in significantly less training time, training on a single GPU, with a fraction of the number of parameters of other competitive models. We achieved this by designing a general formulation for a multi-resolution image representation, and derived a unimodular transformation that adds no cost in terms of log-likelihood. Although the likelihood values for 32×32 resolution datasets such as CIFAR10 and ImageNet32 do not improve significantly over the baseline, ImageNet64 and above see a marked improvement. The performance is better for higher resolutions, as seen in the case of ImageNet128.
We also conducted an ablation study to note the effects of each change introduced in the formulation. We found that our unimodular transformation has a significant contribution towards improving BPD and decreasing training time.
ETHICS STATEMENT
In terms of broader social impacts of this work, generative models of images can be used to generate so-called fake images, and this issue has been discussed at length in other works. We have tried to stay away from particularly concerning use of our work. We have not performed experiments on face-specific datasets. Although the ImageNet dataset contains human face images, we have only tested our method on an unconditional model, and have made no effort towards controllable generation. An increasing concern in the machine learning community is that of increased use of computational resources and its impact on the environment. We emphasize lower computational budgets, and achieve comparable performance with far fewer parameters and significantly less training time.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work, we have attached anonymized version of our code repository in the supplementary material. We have mentioned that our code builds on top of a public code repository of the most relevant prior method: https://github.com/cfinlay/ffjord-rnode. We have included the exact mathematical equations governing our method in Section 3.3, which can be verified in our code. We have mentioned other details of our models in the appendix : number of channels, number of blocks, corresponding total number of parameters. We have also mentioned in the appendix that we perform gradient clipping, and added relevant details there. Our code logs and plots the BPD, number of parameters, and training time for every experiment, and saves samples of generated images at regular intervals. It also plots other values of interest such as change in log likelihood, log probability of mapped noise samples, etc. The code also includes options for launching experiments with configurations of prior methods including FFJORD (Grathwohl et al., 2019), RNODE (Finlay et al., 2020), STEER (Ghosh et al., 2020), and WaveletFlow (Yu et al., 2020) using CNFs.

heading: A OUT-OF-DISTRIBUTION PROPERTIES OF (MR)CNF

text:
The derivation of likelihood-based models suggests that the density of an image under the model is an effective measure of its likelihood of being in distribution. However, recent works (Theis et al., 2016; Nalisnick et al., 2019a; Serrà et al., 2020; Nalisnick et al., 2019b) have pointed out that it is possible that images drawn from other distributions have higher model likelihood. Examples have been shown where normalizing flow models (such as Glow) trained on CIFAR10 images assign higher likelihood to SVHN (Netzer et al., 2011) images. This could have serious implications on the practical applicability of these models. Some also note that likelihood-based models do not generate images with good sample quality as they avoid assigning small probability to out-of-distribution (OoD) data points, hence
using model likelihood (-BPD) for detecting OoD data is not effective.
We conduct the same experiments with (MR)CNFs, and find that similar conclusions can be drawn. Figure 4 plots the histogram of log likelihood per dimension (-BPD) of OoD images (SVHN, TinyImageNet) under MRCNF models trained on CIFAR10. It can be observed that the likelihood of the OoD SVHN is higher than CIFAR10 for MRCNF, similar to the findings for Glow, PixelCNN, VAE in earlier works (Nalisnick et al., 2019a; Choi et al., 2018; Serrà et al., 2020; Nalisnick et al., 2019b; Kirichenko et al., 2020).
One possible explanation put forward by Nalisnick et al. (2019b) is that “typical” images are less “likely” than constant images, which is a consequence of the distribution of a Gaussian in high dimensions. Indeed, as our Figure 4 shows, constant images have the highest likelihood under MRCNFs, while randomly generated (uniformly distributed) pixels have the least likelihood (not shown in figure due to space constraints).
Choi et al. (2018); Nalisnick et al. (2019b) suggest using “typicality” as a better measure of OoD. However, Serrà et al. (2020) observe that the complexity of an image plays a significant role in the training of likelihoodbased generative models. They propose a new metric S as an out-of-distribution detector:
S(x) = bpd(x)− L(x) (16)
where L(x) is the complexity of an image x measured as the length of the best compressed version of x (we use FLIF (Sneyers & Wuille, 2016) following Serrà et al. (2020)) normalized by the number of dimensions.
We perform a similar analysis as Serrà et al. (2020) to test how S compares with -bpd for OoD detection. For different MRCNF models trained on CIFAR10, we compute the area under the receiver operating characteristic curve (auROC) using -bpd and S as standard evaluation for the OoD detection task (Hendrycks et al., 2019; Serrà et al., 2020).
Table 4 shows that S does perform better than -bpd in the case of (MR)CNFs, similar to the findings in Serrà et al. (2020) for
Glow and PixelCNN++. It seems that SVHN is easier to detect as OoD for Glow than MRCNFs. However, OoD detection performance is about the same for TinyImageNet. We also observe that MRCNFs are better at OoD than CNFs.
Other OoD methods (Hendrycks & Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Sabeti & Høst-Madsen, 2019; Høst-Madsen et al., 2019; Hendrycks et al., 2019) are not suitable in our case, as identified in Serrà et al. (2020).
A.1 SHUFFLED IN-DISTRIBUTION IMAGES
Kirichenko et al. (2020) conclude that normalizing flows do not represent images based on their semantic contents, but rather directly encode their visual appearance. We verify this for continuous normalizing flows by estimating the density of in-distribution test images, but with patches of pixels randomly shuffled. Figure 5 (a) shows an example of images of shuffled patches of varying size, Figure 5 (b) shows the graph of the their log-likelihoods.
That shuffling pixel patches would render the image semantically meaningless is reflected in the Fréchet Inception Distance (FID) between CIFAR10-Train and these sets of shuffled images — 1x1: 340.42, 2x2: 299.99, 4x4: 235.22, 8x8: 101.36, 16x16: 33.06, 32x32 (i.e. CIFAR10-Test): 3.15. However, we see that images with large pixel patches shuffled are quite close in likelihood to the unshuffled images, suggesting that since their visual content has not changed much they are almost as likely as unshuffled images under MRCNFs.
B FULL TABLE 1

heading: C QUALITATIVE SAMPLES

heading: D SIMPLE EXAMPLE OF DENSITY ESTIMATION

text:
For example, if we use Euler method as our ODE solver, for density estimation Equation 2 reduces to:
v(t1) = v(t0) + (t1 − t0)fs(v(t0), t0 | c) (17)
where fs is a neural network, t0 represents the "time" at which the state is image x, and t1 is when the state is noise z. We start at scale S with an image sample xS , and assume t0 and t1 are 0 and 1 respectively: zS = xS + fS(xS , t0 | xS−1) zS−1 = xS−1 + fS−1(xS−1, t0 | xS−2) ... z1 = x1 + f1(x1, t0 | x0) z0 = x0 + f0(x0, t0) (18)

heading: E SIMPLE EXAMPLE OF GENERATION

text:
For example, if we use Euler method as our ODE solver, for generation Equation 2 reduces to:
v(t0) = v(t1) + (t0 − t1)fs(v(t1), t1 | c) (19)
i.e. the state is integrated backwards from t1 (i.e. zs) to t0 (i.e. xs). We start at scale 0 with a noise sample z0, and assume t0 and t1 are 0 and 1 respectively: x0 = z0 − f0(z0, t1) x1 = z1 − f1(z1, t1 | x0) ... xS−1 = zS−1 − fS−1(zS−1, t1 | xS−2) xS = zS − fS(zS , t1 | xS−1) (20)

heading: F MODELS

text: We used the same neural network architecture as in RNODE (Finlay et al., 2020). The CNF at each resolution consists of a stack of bl blocks of a 4-layer deep convolutional network comprised of 3x3 kernels and softplus activation functions, with 64 hidden dimensions, and time t concatenated to the spatial input. In addition, except at the coarsest resolution, the immediate coarser image is also concatenated with the state. The integration time of each piece is [0, 1]. The number of blocks bl and the corresponding total number of parameters are given in Table 6.

heading: G GRADIENT NORM

text:
In order to avoid exploding gradients, We clipped the norm of the gradients (Pascanu et al., 2013) by a maximum value of 100.0. In case of using adversarial loss, we first clip the gradients provided by the adversarial loss by 50.0, sum up the gradients provided by the log-likelihood loss, and then clip the summed gradients by 100.0.
H 8-BIT TO UNIFORM
The change-of-variables formula gives the change in probability due to the transformation of u to v:
log p(u) = log p(v) + log ∣∣∣∣det dvdu ∣∣∣∣
Specifically, the change of variables from an 8-bit image to an image with pixel values in range [0, 1] is:
b (p) S =
a (p) S
256 =⇒ log p(aS) = log p(bS) + log ∣∣∣∣det dbda ∣∣∣∣ =⇒ log p(aS) = log p(bS) + log ( 1
256 )DS =⇒ log p(aS) = log p(bS)−DS log 256
=⇒ bpd(aS) = − log p(aS) DS log 2
= −(log p(bS)−DS log 256)
DS log 2
= − log p(bS) DS log 2 + log 256 log 2
= bpd(x) + 8
where bpd(x) is given from Equation 13.

heading: I FID V/S TEMPERATURE

text: Table 7 lists the FID values of generated images from MRCNF models trained on CIFAR10, with different temperature settings on the Gaussian.

title: MULTI-RESOLUTION CONTINUOUS NORMALIZING FLOWS