summaries: The paper introduces a novel regularized auto-encoder architecture called the Cramer-Wold AutoEncoders (CWAE). It's objective (Eq. 7) consists of two terms: (i) a standard reconstruction term making sure the the encoder-decoder pair aligns nicely to accurately reconstruct all the training images and (ii) the regularizer, which roughly speaking requires the encoded training distribution to look similar to the standard normal (which is a prior used in the generative model being trained). The main novelty of the paper is in the form of this regularizer. The authors introduce what they call "the Cramer-Wold distance" (for definitions see Theorems 3.1 and 3.2) which is defined between two finite sets of D-dimensional points. The authors provide empirical studies showing that the proposed CWAE method achieves the same quality of samples (measured with FID scores) as the WAE-MMD model [1] previously reported in the literature, while running faster (by up to factor of 2 reduction in the training time, as the authors report).

summaries: This paper proposes the Cramer-Wold autoencoder. The first contribution of the paper is to propose the Cramer-Wold distance between two distributions based on the Cramer-Wold Theorem. More specifically, in order to compute the Cramer-Wold distance, we first find the one dimensional projections of the distributions over random slices, and then compute the average L2 distances of the kernel density estimates of these projections over random slices. The second contribution of the paper is to develop a generative autoencoder which uses the Cramer-Wold distance to match the latent distribution of the data to the prior distribution.

abstractText: Assessing distance between the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE). Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and WAE-MMD (WAE using maximum mean discrepancy based distance function) we construct a new generative model – Cramer-Wold AutoEncoder (CWAE). CWAE cost function is based upon a characteristic kernel we introduce, called CramerWold kernel, and has a a simple closed-form in the case of normal prior. As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD and often improves upon SWAE.

authors:
M. Abramowitz
I.A. Stegun

title: Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume

venue: National Bureau of Standards Applied Mathematics Series. U.S. Government Printing Office,

authors:
S. Axler
P. Bourdon
R. Wade

title: Harmonic function theory, volume 137 of Graduate Texts in Mathematics

authors:
R.W. Barnard
G. Dahlquist
K. Pearce
L. Reichel
K.C. Richards

title: Gram polynomials and the kummer function

venue: J. Approx. Theory,

authors:
H. Cramér
H. Wold

title: Some theorems on distribution functions

venue: J. London Math. Soc.,

title: The Radon Transform and Some of Its Applications

authors:
I. Deshpande
Z. Zhang
A. Schwing

title: Generative modeling using the sliced wasserstein distance

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
I.S. Gradshteyn
I.M. Ryzhik

title: Table of integrals, series, and products

title: Invariant tests for multivariate normality: a critical review

venue: Statist. Papers,

authors:
M. Heusel
H. Ramsauer
T. Unterthiner
B. Nessler
G. Klambauer
S. Hochreiter

title: Gans trained by a two time-scale update rule converge to a nash equilibrium

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

authors:
D.P. Kingma
M. Welling

title: Auto-encoding variational bayes

authors:
S. Kolouri
Ch.E. Martin
G.K. Rohde

title: Sliced-wasserstein autoencoder: an embarrassingly simple generative model

title: Density estimation for statistics and data analysis. Monographs on Statistics and Applied Probability

authors:
F. Tricomi
A. Erdélyi

title: The asymptotic expansion of a ratio of gamma functions

venue: Pacific J. Math.,

heading: 1 INTRODUCTION

text:
One of the crucial aspects in construction of generative models is devising effective method for computing and minimizing distance between the true and the model distribution. Originally in Variational Autencoder (VAE) (Kingma & Welling, 2014) this computation was carried out using variational methods. An important improvement was brought by the introduction of Wasserstein metric (Tolstikhin et al., 2017) and the construction of WAE-GAN and WAE-MMD models, which relax the need for variational methods. WAE-GAN requires a separate optimization problem to be solved to approximate the used divergence measure, while in WAE-MMD the discriminator has the closed-form obtained from a characteristic kernel, i.e. one that is injective on distributions Muandet et al. (2017). A recent contribution to this trend of simplifying the construction of generative models is Sliced-Wasserstein Autoencoder (SWAE, Kolouri et al. (2018)), where a significantly simpler AutoEncoder based model based on Wasserstein distance is proposed.
The main innovation of SWAE was the introduction of the sliced-Wasserstein distance – a fast to estimate metric for comparing two distributions, based on the mean Wasserstein distance of one-dimensional projections. However, even in SWAE there is no close analytic formula that would enable computing the distance of the sample from the standard normal distribution. Consequently in SWAE two types of sampling are needed: (i) sampling from the prior distribution and (ii) sampling over one-dimensional projections.
Our main contribution is introduction of the CramerWold distance between distributions, which has a closed-form for the distance of a sample from standard multivariate normal distribution. Its important feature is that it is given by a characteristic kernel which has a closed-form given by equation 7 for the product of radial Gaussians1. We use it to construct an AutoEncoder based generative model, called Cramer-Wold AutoEncoder (CWAE), in which the cost function, for a normal prior distribution, has a closed analytic formula. Thus
1For more information we refer the reader to Appendix, Sections A and B.
CWAE can be seen as a borderline model between SWAE and WAE-MMD, as we use a characteristic kernel to discriminate distributions, but its formula comes from the application of the sliced approach. An introduction of sliced approach is given in the following section, the comparison of CWAE and WAE-MMD is given in Appendix, Section B. We benchmark CWAE against generative models using a non-parametrized distance functions: WAE-MMD (WAE variant from Tolstikhin et al. (2017) based on the classical MMD distance) and SWAE. Our results show that, while simplifying the optimization procedure by forgoing the need to sample data projections or from the prior distribution we obtain an AutoEncoder based generative model which obtains quantitatively and qualitatively similar results to WAE-MMD and SWAE models, see Section 5.
Let us now briefly describe the outline of the paper. In the following two sections we introduce and theoretically investigate the Cramer-Wold distance. However, the reader interested mainly in the construction of our generative AutoEncoder model can proceed directly to Section 4. Section 5 contains experiments. Conclusions are given in Section 6.

heading: 2 CRAMER-WOLD DISTANCE: CONSTRUCTION

text:
Motivated by the prevalent use of normal distribution as prior in modern generative models, we investigate whether it is possible to simplify optimization of such models. As the first step towards this, in this section we introduce Cramer-Wold distance, which has a simple analytical formula for computing normality of a high-dimensional sample. On a high level our approach uses the traditional L2 distance of kernel-based density estimation, computed across multiple single-dimensional projections of the true data and the output distribution of the model. We base our construction on the following two popular tricks of the trade:
Sliced-based decomposition of a distribution: Following the footsteps of Kolouri et al. (2018); Deshpande et al. (2018), the first idea is to leverage the Cramer-Wold Theorem (Cramér & Wold, 1936) and Radon Transform (Deans, 1983) to reduce computing distance between two distributions to one dimensional calculations. For v in the unit sphere SD ⊂ RD, the projection of the set X ⊂ RD onto the space spanned by v is given by vTX and the projection of N(m,αI) is N(vTm,α). Cramer-Wold theorem states that two multivariate distributions can be uniquely identified by their all one-dimensional projections. For example, to obtain the key component of SWAE model, i.e. the sliced-Wasserstein distance between two samples X,Y ∈ RD, we compute the mean Wasserstein distance between all one-dimensional projections:
dW (X,Y ) = ∫ SD dW (v TX, vTY ) dσD(v), (1)
where SD denotes the unit sphere in RD and σD is the normalized surface measure on SD. This approach is effective since the one-dimensional Wasserstein distance between samples has the closed form, and therefore to estimate (1) one has to sample only over the projections.
Smoothing distributions: Using the sliced-based decomposition requires us to define distance between two sets of samples, in a single dimensional space. To this end we will use a trick-of-trade applied commonly in statistics in order to compare samples or distributions which is to first smoothen (sample) distribution with a Gaussian kernel. For the sample R = (ri)i=1..n ⊂ R by its smoothing with Gaussian kernel N(0, γ) we understand
smγ(R) = 1 n ∑ i N(ri, γ),
where by N(m,S) we denote the one-dimensional normal density with mean m and variance S. This produces a distribution with regular density, and is commonly used in kernel density estimation. If R comes from the normal distribution with standard deviation close to one, the asymptotically optimal choice of γ is given by the Silverman’s rule of thumb γ = ( 43n )
2/5, see Silverman (1986). For continuous density f , its smoothing smγ(f) is given by the convolution with N(0, γ), and in the special case of Gaussians we have smγ(N(m,S)) = N(m,S + γ). While in general kernel density estimations works well only in low-dimensional spaces, this fits the bill for us, as we will only compute distances on single dimensional projections of the data.
Cramer-Wold distance. We are now ready to introduce the Cramer-Wold distance. In a nutshell, we propose to compute the squared distance between two samples by considering the mean squared L2 distance between their smoothed projections over all single dimensional subspaces. By the squared L2 distance between functions f, g : R → R we refer to ‖f − g‖22 = ∫ |f(x) − g(x)|2dx. A key feature of this distance is that it permits a closed-form in the case of normal distribution.
More precisely, the following algorithm fully defines the Cramer-Wold distance between two samples X = (xi)i=1..n, Y = (yj)j=1..k ⊂ RD (for illustration of Steps 1 and 2 see Figure 1):
1. given v in the unit sphere S(0, 1) ⊂ RD consider the projections vTX = (vTxi)i=1..n and vTY = (vT yj)j=1..k,
2. compute the squared L2 distance of the densities smγ(vTX) and smγ(vTX):
‖smγ(vTX)− smγ(vTY )‖22,
3. to obtain squared Cramer-Wold distance average (integrate) the above formula over all possible v ∈ SD.

heading: 3 CRAMER-WOLD DISTANCE: THEORY

text:
The key theoretical outcome of this paper is that the result of the computation of the Cramer-Wold distance from the previous section can be simplified to a closed form solution. Consequently, to compute the distance of two samples there is no need of finding the optimal transport like in WAE or the necessity to sample over the projections like in SWAE. For the case of simplicity we provide in this section the formulas for the distance between two samples and the distance of a sample from the standard normal density. The general definition of Cramer-Wold metric is presented in Appendix, Section A. Theorem 3.1. Let X = (xi)i=1..n, Y = (yj)j=1..n ⊂ RD be given2. We formally define the squared Cramer-Wold distance by the formula
d2cw(X,Y ) := ∫ SD ‖smγ(vTX)− smγ(vTY )‖22 dσD(v).
Then
d2cw(X,Y ) = 1 2n2 √ πγ (∑ ii′ φD( ‖xi−xi′‖ 2 4γ ) + ∑ jj′ φD( ‖yj−yj′‖ 2 4γ )− 2 ∑ ij φD( ‖xi−yj‖2 4γ ) ) , (2)
where φD(s) = 1F1( 12 ; D 2 ;−s) and 1F1 is the Kummer’s confluent hypergeometric function (see, e.g., Barnard et al. (1998)). Moreover, φD(s) has the following asymptotic formula valid for D ≥ 20:
φD(s) ≈ (1 + 4s2D−3 ) −1/2. (3)
To prove the Theorem 3.1 we will need the following crucial technical proposition. Proposition 3.1. Let z ∈ RD and γ > 0 be given. Then∫
SD
N(vT z, γ)(0) dσD(v) = 1√ 2πγ φD ( ‖z‖2 2γ ) . (4)
Proof. By applying orthonormal change of coordinates without loss of generality we may assume that z = (z1, 0, . . . , 0), and then vT z = z1v1 for v = (v1, . . . , vD). Consequently we get∫
SD N(vT z, γ)(0) dσD(v) = ∫ SD N(z1v1, γ)(0) dσD(v).
Making use of the formula for slice integration of functions on spheres (Axler et al., 1992, Corollary A.6) we get:∫
SD f dσD = VD−1 VD ∫ 1 −1(1− x 2)(D−3)/2 ∫ SD−1 f(x, √ 1− x2 ζ) dσD−1(ζ) dx,
2For clarity of presentation we provide here the formula for the case of samples of equal size.
where VK denotes the surface volume of a sphere SK ⊂ RK . Applying the above equality for the function f(v1, . . . , vD) = N(z1v1, γ)(0) and s = z21/(2γ) = ‖z‖2/(2γ) we consequently get that the LHS of (4) simplifies to
VD−1 VD 1√ 2πγ ∫ 1 −1(1− x 2)(D−3)/2 exp(−sx2) dx,
which completes the proof since VK = 2·π K 2
Γ(K2 ) and
∫ 1 −1 exp(−sx
2)(1 − x2)(D−3)/2 dx = √ π
Γ(D−12 ) Γ( D2 )
1F1 ( 1 2 ; D 2 ;−s ) .
Proof of Theorem 3.1. Directly from the definition of smoothing we obtain that
d2cw(X,Y ) = ∫ SD ∥∥ 1 n ∑ i N(vTxi, γ)− 1n ∑ j N(vT yj , γ) ∥∥2 2 dσD(v). (5)
Now applying the one-dimensional formula the the L2-scalar product of two Gaussians:
〈N(r1, γ1), N(r2, γ2)〉2 = N(r1 − r2, γ1 + γ2)(0) and the equality ‖f − g‖22 = 〈f, f〉2 + 〈g, g〉2 − 2〈f, g〉2 (where 〈f, g〉2 = ∫ f(x)g(x)dx), we simplify the squared-L2 norm in the integral of RHS of (5) to
‖ 1n ∑ i N(vTxi, γ)− 1n ∑ i N(vT yj , γ) ∥∥2 2
= 1n2 〈 ∑ i N(vTxi, γ), ∑ i N(vTxi, γ)〉2 + 1n2 〈 ∑ j N(vT yj , γ), ∑ j N(vT yj , γ)〉2
− 2n2 〈 ∑ i N(vTxi, γ), ∑ j N(vT yj , γ)〉2
= 1n2 ∑ ii′ N(vT (xi − xi′), 2γ)(0) + 1n2 ∑ jj′ N(vT (yj − yj′), 2γ)(0)− 2n2 ∑ ij N(vT (xi − yj), 2γ)(0).
Applying directly Proposition 3.1 we obtain formula (2). Proof of the formula for the asymptotics of the function φD is provided in the Appendix.
Thus to estimate the distance of a given sample X to some prior distribution f , one can follow the common approach and take the distance between X and a sample from f . As the main theoretical result of the paper we view the following theorem, which says that in the case of standard Gaussian multivariate prior, we can completely reduce the need for sampling (we omit the proof since it is similar to that of Theorem 3.1). Theorem 3.2. Let X = (xi)i=1..n ⊂ RD be a given sample. We formally define
d2cw(X,N(0, I)) := ∫ SD ‖smγ(vTX)− smγ(N(0, 1))‖22dσD(v).
Then
d2cw(X,N(0, I)) = 1 2n2 √ π ( 1√ γ ∑ i,j φD( ‖xi−xj‖2 4γ ) + n2√ 1+γ − 2n√ γ+ 12 ∑ i φD( ‖xi‖2 2+4γ ) ) . (6)
One can easily obtain the general formula for the distance between mixtures of radial distributions. This follows from the fact that the Cramer-Wold distance is given by a scalar product 〈·, ·〉cw which has a closed-form for the product of two radial Gaussians:
〈N(x, αI), N(y, βI)〉cw = 1√ 2π(α+β+2γ) φD
( ‖x−y‖2
2(α+β+2γ)
) . (7)
The above formula means that Cramer-Wold distance is defined by Cramer-Wold kernel, for more details see Appendix, Section A.

heading: 4 CRAMER-WOLD AUTOENCODER MODEL (CWAE)

text:
This section is devoted to the construction of CWAE. Since we base our construction on the AutoEncoder, to establish notation let us formalize it here.
AutoEncoder. Let X = (xi)i=1..n ⊂ RN be a given data set. The basic aim of AE is to transport the data to a typically, but not necessarily, less dimensional latent space Z = RD with reconstruction error as small as possible. Thus, we search for an encoder E : Rn → Z and decoder D : Z → Rn functions, which minimize the reconstruction error on the data set X:
MSE(X; E ,D) = 1 n n∑ i=1 ‖xi −D(Exi)‖2.
AutoEncoder based generative model. CWAE, similarly to WAE, is a classical AutoEncoder model with modified cost function which forces the model to be generative, i.e. ensures that the data transported to the latent space comes from the (typically Gaussian) prior f . This statement is formalized by the following important remark, see also (Tolstikhin et al., 2017). Remark 4.1. Let X be an N -dimensional random vector from which our data set was drawn, and let Y be a random vector with a density f on latent Z . Suppose that we have constructed functions E : RN → Z and D : Z → RN (representing the encoder and the decoder) such that3
1. D(Ex) = x for x ∈ image(X),
2. random vector EX has the distribution f .
Then by the point 1 we obtain that D(EX) = X, and therefore DY has the same distribution as D(EX) = X.
This means that to produce samples from X we can instead produce samples from Y and map them by the decoder D.
Since an estimator of the image of the random vector X is given by its sample X , we conclude that a generative model is correct if it has small reconstruction error and resembles the prior distribution in the latent. Thus, to construct a generative AutoEncoder model (with Gaussian prior), we add to its cost function a measure of distance of a given sample from normal distribution.
CWAE cost function. Once the crucial ingredient of CWAE is ready, we can describe its cost function. To ensure that the data transported to the latent space Z are distributed according to the standard normal density, we add to the cost function logarithm4 of the Cramer-Wold distance from standard multivariate normal density d2cw(X,N(0, I)):
cost(X; E , D) = log d2cw(EX,N(0, I)) +MSE(X; E ,D). (8) Since the use of special functions involved in the formula for Cramer-Wold distance might be cumbersome, we apply in all experiments (except for the illustrative 2D case) the asymptotic form (12) of function φD:
2 √ πd2cw(X) ≈ 1n2 ∑ ij (γn + ‖xi−xj‖2 2D−3 ) −1/2 + (1 + γn) −1/2 − 2n ∑ i (γn + 1 2 + ‖xi‖2 2D−3 ) −1/2,
where γn = ( 43n ) 2/5 is chosen by the Silverman’s rule of thumb (Silverman, 1986).
Comparison with WAE and SWAE models. Finally, let us briefly recapitulate differences between the introduced CWAE, WAE variants of (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018). In contrast to WAE-MMD and SWAE, CWAE model does not require sampling from normal distribution (as in WAE-MMD) or over slices (as in SWAE) to evaluate its cost function, and in this sense uses a closed formula cost function. In contrast to WAE-GAN, our objective does not require a separately trained neural network to approximate the optimal transport function, thus avoiding pitfalls of adversarial training. In this paper we are interested in WAE-MMD and SWAE models, which do not use parameterized distance functions, e.g. trained adversarially like in WAE-GAN. However, in future work we plan to introduce an adversarial version of CWAE and compare it with WAE-GAN.
3We recall that for function (or in particular random vector) X : Ω→ RD by image(X) we denote the set consisting of all possible values X can attain, i.e. {X(ω) : ω ∈ Ω}.
4We take the logarithm of the Cramer-Wold distance to improve balance between the two terms in the objective function.

heading: 5 EXPERIMENTS

text: In this section we empirically validate the proposed CWAE model on standard benchmarks for generative models: CelebA, Cifar-10 and MNIST. We will compare CWAE model with WAEMMD (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018). As we will see, our results match those of WAE-MMD, and in some cases improve upon SWAE, while using a simpler to optimize cost function (see the previous section for a more detailed discussion). The rest of this section is structured as follows. In Section 5.2 we report results on standard qualitative tests, as well as a visual investigations of the latent space. In Section 5.3 we will turn our attention to quantitative tests using Fréchet Inception Distance and other metrics.

heading: 5.1 EXPERIMENTATION SETUP

text:
In the experiment we have used two basic architecture types. Experiments on MNIST were performed using a feedforward network for both encoder and decoder, and a 20 neuron latent layer, all with ReLU activations. In case of CIFAR-10 and CelebA data sets we used convolution-deconvolution architectures. Please refer to Appendix E for full details.
5.2 QUALITATIVE TESTS
Table 1: CWAE achieves similar FID (lower is better) and sharpness (higher is better) to WAE-MMD on the original WAE architecture (see Appendix E for details).
Algorithm FID Sharpness SWAE 72 0.008 VAE 63 0.003 WAE-MMD 55 0.006 CWAE 54 0.006
WAE-GAN 42 0.006 True data 2 0.020
The quality of a generative model is typically evaluated by examining samples or interpolations. We present such a comparison between CWAE with WAE-MMD in Figure 2. We follow the same procedure as in (Tolstikhin et al., 2017). In particular, we use the same base neural architecture for both CWAE and WAE-MMD. We consider for each model (i) interpolation between two random examples from the test set (leftmost in Figure 2), (ii) reconstruction of a random example from the test set (middle column in Figure 2), and finally a sample reconstructed from a random point sampled from the prior distribution (right column in Figure 2). The experiment shows that there are no perceptual differences between CWAE and WAE-MMD generative distribution.
In the next experiment we qualitatively assess normality of the latent space. This will allow us to ensure that CWAE does not compromise on the normality of its latent distribution, which recall is part of the cost function for all the models except AE. We compare CWAE5 with AE, VAE, WAE and SWAE on the MNIST data with using 2-dimensional latent space and a two dimensional Gaussian prior distribution. Results are reported in Figure 3. As is readily visible, the latent distribution of CWAE is as close, or perhaps even closer, to the normal distribution than that of the other models. Furthermore, the AutoEncoder presented in the second figure is noticeably different from a Gaussian distribution, which is to be expected because it does not optimize for normality in contrast to the other models.
To summarize, both in terms of perceptual quality and satisfying normality objective, CWAE matches WAE-MMD. The next section will provide more quantitative studies.

heading: 5.3 QUANTITATIVE TESTS

text:
In order to quantitatively compare CWAE with other models, in the first experiment we follow the common methodology and use the Fréchet Inception Distance (FID) introduced by Heusel et al. (2017). Further, we evaluate the sharpness of generated samples using the Laplace filter following Tolstikhin et al. (2017). Results for CWAE and WAE are summarized in Tab. 1. In agreement with the qualitative studies, we observe FID and sharpness scores of CWAE to be similar to WAE-MMD.
5Since (3) is valid for dimensions D ≥ 20, to implement CWAE in 2-dimensional latent space we apply equality 1F1(1/2, 1,−s) = e− s 2 I0 ( s 2 ) jointly with the approximate formula (Abramowitz & Stegun, 1964, page 378) for the Bessel function of the first kind I0, for more details see Appendix C.
Next, by comparing training time between CWAE and other models, we found that for batch-sizes up to 1024, which covers the range of batch-sizes used typically for training autoencoders, CWAE is faster (in terms of time spent per batch) than other models. More precisely, CWAE is approximately 2× faster up to 256 batch-size. Details are relegated to the Appendix D. Finally, motivated by Remark 4.1 we propose a novel method for quantitative assessment of the models based on their comparison to standard normal distribution in the latent. To achieve this we have decided to use one of the most popular statistical normality tests, i.e. Mardia tests (Henze, 2002). Mardia’s normality tests are based on verifying whether the skewness b1,D(·) and kurtosis b2,D(·) of a sample X = (xi)i=1..n ⊂ RD:
b1,D(X) = 1 n2 ∑ j,k (xTj xk) 3, and b2,D(X) = 1n ∑ j ‖xj‖4,
are close to that of standard normal density. The expected Mardia’s skewness and kurtosis for standard multivariate normal distribution is 0 and D(D + 2), respectively. To enable easier comparison in experiments we consider also the value of the normalized Mardia’s kurtosis given by b2,D(X) − D(D + 2), which equals zero for the standard normal density.
Results are presented in Figure 4 and Table 2. In Figure 4 we report for CelebA data set the value of FID score, Mardia’s skewness and kurtosis during learning process of AE, VAE, WAE, SWAE and CWAE (measured on the validation data set). WAE, SWAE and CWAE models obtain the best reconstruction error, comparable to AE. VAE model exhibits a sightly worse reconstruction error, but values of kurtosis and skewness indicating their output is closer to normal distribution. As expected, the output of AE is far from normal distribution; its kurtosis and skewness grow during learning. This arguably less standard evaluation, which we hope will find adoption in the community, serves as yet another evidence that CWAE has strong generative capabilities which at least match performance of WAE-MMD. Moreover we observe that VAE model’s output distribution is closest to the normal distribution, at the expense of the reconstruction error, which is reflected by the blurred reconstructions typically associated with VAE model.
Moreover, motivated by the above approach based on normality tests6 we have verified how the Cramer-Wold metric works as a Gaussian goodness of fit, however, the results were not satisfactory. The tests based on Cramer-Wold metric were, in general, in the middle of compared tests (Mardia, Henze-Zirkler and Royston tests).
On the whole, WAE-MMD and CWAE achieve, practically speaking, the same level of performance in terms of FID score, sharpness, and our newly introduced normality test. Additionally, CWAE fares better in many of these metrics than SWAE.
6For more information on the statistical tests based on the kernel approach we refer the reader to (Muandet et al., 2017, Subsection 3.5).

heading: 6 CONCLUSIONS

text:
In the paper we have presented a new autoencoder based generative model CWAE, which matches results of WAE-MMD, while using a cost function given by a simple closed analytic formula. We hope this result will encourage future work in developing simpler to optimize analogs of strong neural models.
Crucial in the construction of CWAE is the use of the developed Cramer-Wold metric between samples and distributions, which can be effectively computed for Gaussian mixtures. As a consequence we obtain a reliable measure of the divergence from normality. Future work could explore use of the Cramer-Wold distance in other settings, in particular in adversarial models.

heading: A CRAMER-WOLD KERNEL

text:
In this section we first formally define the Cramer-Wold metric, and later show that it is given by a characteristic kernel which has closed-form for spherical Gaussians. For more information on the kernels, and in general kernel embedding of distributions we refer the reader to (Muandet et al., 2017).
Let us first introduce the general definition of the cw-metric. To do so we generalize the notion of smoothing for arbitrary measures µ by the formula:
smγ(µ) = µ ∗N(0, γI),
where ∗ denotes the convolution operator for two measures, and we identify the normal density N(0, γI) with the measure it introduces. It is well-known that the resulting measure has the density given by
x→ ∫ N(x, γI)(y)dµ(y).
Clearly smγ(N(0, αI)) = N(0, (α+ γ)I)).
Moreover, by applying the characteristic function one obtains that if the smoothing of two measures coincide, then the measures also coincide:
smγ(µ) = smγ(µ) =⇒ µ = ν. (9)
We also need to define the transport of the density by the projection x → vTx, where v is chosen from the unit sphere SD. The definition is formulated so that if X is a random vector with density f , then fv is the density of the random vector Xv := vTX. Then we have
fv(r) = ∫ y:y−rv⊥v f(z)dD−1(z),
where dD−1 denotes the D − 1-dimensional Lebesgue measure. In general, if µ is a measure on RD, then µv is the measure defined on R by the formula
µv(A) = µ({x : vTx ∈ A}).
Since, if a random vector X has the density N(a, γI), then the random variable Xv has the density N(vTa, α), we directly conclude that
N(a, γI)v = N(v Ta, γ).
It is also worth noticing, that due to the fact that the projection of a Gaussian is a Gaussian, the smoothing and projection operators commute, i.e.:
smγ(µv) = (smγµ)v.
Given fixed γ > 0, the two above notions allow us to formally define the cw-distance of two measures µ and ν by the formula
d2cw(µ, ν) = ∫ SD ‖smγ(µv)− smγ(νv)‖2L2dσD(v). (10)
First observe that this implies that cw-distance is given by the kernel function
k(µ, ν) = ∫ SD 〈smγ(µv), smγ(νv)〉L2dσD(v).
Let us now prove that the function dcw defined by equation 10 is a metric (which, in the kernel function literature means that the kernel is characteristic). Theorem A.1. The function dcw is a metric.
Proof. Since dcw comes from a scalar product, we only need to show that if a distance of two measures is zero, the measures coincide.
So let µ, ν be given measures such that dcw(µ, ν) = 0. This implies that
smγ(µv) = smγ(νv).
By equation 9 this implies that µv = νv. Since this holds for all v ∈ SD, by the Cramer-Wold Theorem we obtain that µ = ν.
Thus we can summarize the above by saying that the Cramer-Wold kernel is a characteristic kernel which has the closed-form the scalar product of two radial Gaussians given by equation 7:
〈N(x, αI), N(y, βI)〉cw = 1√ 2π(α+β+2γ) φD
( ‖x−y‖2
2(α+β+2γ)
) .
Remark A.1. Observe, that except for the Gaussian kernel it is the only kernel which has the closed form for the spherical Gaussians, which as we discuss in the next section is important, as the RBF (Gaussian) kernels cannot be successfully applied in AutoEncoder based generative models. The reason is that the derivative of Gaussian decreases to fast, and therefore it does not enforce the proper learning of the model, see also the comments in (Tolstikhin et al., 2017, Section 4, WAE-GAN and WAE-MMD specifics).

heading: B CWAE VS WAE-MMD

text:
In this section we are going to compare CWAE model to WAE-MMD. In particular we show that CWAE can be seen as the intersection of the sliced-approach together with MMD-based models.
Since both WAE and CWAE use kernels to discriminate between sample and normal density, to compare the models we first describe the WAE model.
WAE cost function for a given characteristic kernel k and sample X = (xi)i=1..n ⊂ RD (in the D-dimensional latent) is given by
WAE cost = MSE + λ · d2k(X,Y ),
where Y = (yi)i=1..n is a sample from the standard normal density N(0, I), and d2k(X,Y ) denotes the kernel-based distance between the probability distributions representingX and Y , that is 1n ∑ i δxi
and 1n ∑ i δyi , where δz denotes the atom Dirac measure at z ∈ RD. The inverse multiquadratic kernel k is chosen as default k(x, y) = C
C + ‖x− y‖22 ,
where in experiments in (Tolstikhin et al., 2017) a value C = 2Dσ2 was used, where σ is the hyper-parameter denoting the size of the normal density. Thus the model has hyper-parameters λ and σ, which were chosen to be λ = 10, σ2 = 1 in MNIST, λ = 100, σ2 = 2 in CelebA. Observe that the hyper-parameters do not depend on the sample size and that, in general, the WAE-MMD model hyper-parameters have to be chosen by hand.
Now let us describe the CWAE model. CWAE cost function for a sample X = (xi)i=1..n ⊂ RD (in the D-dimensional latent) is given by
CWAE cost = MSE + log d2cw(X,N(0, I)),
where distance between the sample and standard normal distribution is taken with respect to the Cramer-Wold kernel with a regularizing hyperparameter γ given by the Silverman’s rule of thumb (the motivation for such a choice of hyper-parameters is explained in Section 2).
Thus, we have the following differences:
• Due to the properties of Cramer-Wold kernel, in the distance we are able to substitute the sample estimation of d2k(X,N(0, I)) given in WAE-MMD by d 2 cw(X,Y ) by its exact
formula. • CWAE, as compared to WAE, has no hyper-parameters:
1. In our preliminary experiments we have observed that in many situations (like in the case of log-likelihood), taking the logarithm of the nonnegative factors of the cost function, which we aim to minimize to zero, improves the learning process. Motivated by this, instead of taking the additional weighting hyper-parameter λ (as in WAEMMD), whose aim is to balance the MSE and divergence terms, we take the logarithm of the divergence. Automatically (independently of dimension) balance those terms in the learning process.
2. The choice of regularization hyper-parameter is given by the Silverman’s rule of thumb, and depends on the sample size (contrary to WAE-MMD, where the hyper-parameters are chosen by hand, and in general do not depend on the sample size).
Summarizing, in CWAE model, contrary to WAE-MMD, we do not have to choose hyper-parameters. Moreover, since we do not have the noise in the learning process given by the random choice of the sample Y from N(0, I), the learning should be more stable. As a consequence, see Figure 7, CWAE in generally learns faster then WAE-MMD, and has smaller standard deviation of the cost-function during the learning process.
C COMPUTATION OF φD
In this section we consider the estimation of values of the function
φD(s) = 1F1( 1 2 ; D 2 ;−s) for s ≥ 0,
which is crucial in the formulation for the Cramer-Wold distance. First we will provide its approximate asymptotic formula valid for dimensions D ≥ 20, and then we shall consider the special case of D = 2 (see Figure 5)
To do so, let us first recall (Abramowitz & Stegun, 1964, Chapter 13) that the Kummer’s confluent hypergeometric function 1F1 (denoted also by M ) has the following integral representation
1F1(a, b, z) = Γ(b)
Γ(a)Γ(b− a) ∫ 1 0 ezuua−1(1− u)b−a−1 du,
valid for a, b > 0 such that b > a. Since we consider that latent is at least of dimension D ≥ 2, it follows that
φD(s) = Γ(D2 )
Γ( 12 )Γ( D 2 − 1 2 ) ∫ 1 0 e−suu−1/2(1− u)D/2−3/2 du.
By making a substitution u = x2, du = 2xdx, we consequently get
φD(s) = 2 · Γ(D/2)Γ(1/2)Γ(D/2−1/2) ∫ 1
0
e−sx 2 (1− x2)(D−3)/2 dx
= Γ(D/2)Γ(1/2)Γ(D/2−1/2) ∫ 1 −1 e−sx 2 (1− x2)(D−3)/2 dx. (11)
Proposition C.1. For large7 D we have
φD(s) ≈ (1 + 4s2D−3 ) −1/2 for all s ≥ 0. (12)
Proof. By (11) we have to estimate asymptotics of
φD(s) = Γ(D2 )
Γ( 12 )Γ( D 2 − 1 2 ) ∫ 1 −1 e−sx 2 (1− x2)(D−3)/2 dx.
Since for large D, for all x ∈ [−1, 1] we have
(1− x2)(D−3)/2e−sx 2 ≈ (1− x2)(D−3)/2 · (1− x2)s = (1− x2)s+(D−3)/2, we get
φD(s) ≈ Γ(D2 ) Γ(D−12 ) √ π · ∫ 1 −1 (1− x2)s+(D−3)/2 dx = Γ(D2 ) Γ(D−12 ) √ π · √ π Γ(s+ D2 − 1 2 ) Γ(s+ D2 ) .
To simplify the above we apply the formula (1) from (Tricomi & Erdélyi, 1951):
Γ(z + α) Γ(z + β) = zα−β(1 + (α− β)(α+ β − 1) 2z +O(|z|−2)),
with α, β fixed so that α+ β = 1 (so only the error term of order O(|z|−2) remains), and get
Γ(D2 )
Γ(D−12 ) =
Γ((D2 − 3 4 ) + 3 4 ) Γ((D2 − 3 4 ) + 1 4 ) ≈ ( D 2 − 3 4 ) 1 2 and Γ(s+ D2 − 1 2 ) Γ(s+ D2 ) ≈ ( s+ D 2 − 3 4 )− 12 . (13)
Summarizing,
φD(s) ≈ (D2 − 3 4 ) 1/2
(s+ D2 − 3 4 )
1/2 = (1 + 4s2D−3 ) −1/2.
In general one can obtain the iterative direct formulas for function φD with the use of erf and modified Bessel functions of the first kind I0 and I1, but for large D they are of little numerical value. We consider here only the special case D = 2 since it is used in the paper for illustrative reasons in the latent for the MNIST data set. Since we have the equality (Gradshteyn & Ryzhik, 2015, (8.406.3) and (9.215.3)):
φ2(s) = 1F1( 1 2 , 1,−s) = e − s2 I0 (s 2 ) ,
to practically implement φ2 we apply the approximation of I0 from (Abramowitz & Stegun, 1964, page 378) given in the following remark. Remark C.1. Let s ≥ 0 be arbitrary and let t = s/7.5. Then φ2(s) ≈ e− s 2 · (1+3.5156229t2+3.0899424t4+1.2067492t6+.2659732t8+.0360768t10+.0045813t12) for s ∈ [0, 7.5],√ 2 s · (.39894228+.01328592t −1+.00225319t−2−.00157565t−3+.0091628t−4−.02057706t−5
+.02635537t−6−.01647633t−7+.00392377t−8) for s ≥ 7.5.

heading: D COMPARISON OF LEARNING TIMES

text:
Figure 6 gives comparison of mean learning time for different most frequently used batch-sizes. Time spent on processing a batch is actually smaller for CWAE for a practical range of batch-sizes [32, 512]. For batch-sizes larger than 1024, CWAE is slower due to its quadratic complexity with respect to the batch-size. However, we note that batch-sizes larger even than 512 are relatively rarely used in practice for training autoencoders.
7In practice we can take D ≥ 20.

heading: E ARCHITECTURE DETAILS

text:
MNIST (28× 28 images) an encoder-decoder feedforward architecture:
encoder three feed-forward ReLU layers, 200 neurons each latent 20-dimensional,
decoder three feed-forward ReLU layers, 200 neurons each.
CelebA (with images centered and cropped to 64×64 with 3 color layers) a convolution-deconvolution network:
encoder four convolution layers with 4× 4 filters and 2× 2 strides (consecutively 32, 32, 64, and 64 output channels), all ReLU activations, two dense layers (1024 and 256 ReLU neurons)
latent 64-dimensional, decoder first two dense layers (256 and 1024 ReLU neurons),
three transposed-convolution layers with 4× 4 filters with 2× 2 strides (consecutively 64, 32, 32 channels) with ReLU activation, transposed-convolution 4× 4 with 2× 2 stride, 3 channels, and sigmoid activation.
CIFAR-10 dataset (32× images with three color layers): a convolution-deconvolution network
encoder four convolution layers with 2 × 2 filters, the second one with 2 × 2 strides, other non-strided (3, 32, 32, and 32 channels) with ReLU activation, 128 ReLU neurons dense layer,
latent with 64 neurons, decoder two dense ReLU layers with 128 and 8192 neurons,
two transposed-convolution layers with 2× 2 filters (32 and 32 channels) and ReLU activation, a transposed convolution layer with 3 × 3 filter and 2 × 2 strides (32 channels) and ReLU activation, a transposed convolution layer with 2× 2 filter (3 channels) and sigmoid activation.
The last layer returns the reconstructed image. The results for all above architectures are given in Table 2. All networks were trained with the Adam optimizer (Kingma & Ba, 2014). The hyperparameters used were learning rate = 0.001, β1 = 0.9, β2 = 0.999, = 1e− 8. MNIST models were trained for 500 epochs, both CIFAR-10 and CelebA for 200.
Additionally, to have a direct comparison to WAE-MMD model on CelebA, an identical architecture was used as that in Tolstikhin et al. (2017) utilized for the WAE-MMD model (WAE-GAN architecture is, naturally, different):
encoder four convolution layers with 5× 5 filters, each layer followed by a batch normalization (consecutively 128, 256, 512, and 1024 channels) and ReLU activation,
latent 64-dimensional, decoder dense 1024 neuron layer,
three transposed-convolution layers with 5 × 5 filters, and each layer followed by a batch normalization with ReLU activation (consecutively 512, 256, and 128 channels), transposed-convolution layer with 5× 5 filter and 3 channels, clipped output value.
The results for this architecture for CWAE compared to VAE and WAE-MMD models are given in Table 1.
Similarly to Tolstikhin et al. (2017), models were trained using Adam with for 55 epochs, with the same optimizer parameters.
F TRAINING RESULTS FOR THE CELEBA DATASET
G TRAINING RESULTS FOR THE CIFAR10 DATASET

heading: H ADDITIONAL FIGURES