summaries: This paper proposes the Random Path Generative Adversarial Network (RP-GAN) to serve as a tool for generative model analysis.  The main idea is to have several different buckets in each block of the generator and then train the generator with random paths. To interpret the features captured by each block, the authors unfreeze one block and show the variance of the generated images via different buckets.

summaries: This paper addresses the issue of interpretability of GAN generation through an alternative approach to the introduction of variability. To seed the generation, instead of providing a random input vector (typically sampled from a standard Gaussian distribution), the authors instead modify the generator architecture so as to allow for randomization in the routing: each layer is replaced by a bucket consisting of several blocks, and in forward propagation only through randomly chosen blocks. In this case, the input vector is chosen to be a constant - the only source of randomization

summaries: This paper presents a variation on the generator of GANs. The authors modify the generator by adding a concept of "blocks" which are randomly activated based on part of the random input vector. It is similar to adding random dropout in the generator, except that the dropout would apply to larger sets of activations instead of single component.

abstractText: In this paper, we introduce Random Path Generative Adversarial Network (RPGAN) — an alternative design of GANs that can serve as a tool for generative model analysis. While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network. As we show, this design allows to understand factors of variation, captured by different generator layers, providing their natural interpretability. With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about the roles that different layers play in the image generation process. Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data.

authors:
Artem Babenko
Anton Slesarev
Alexandr Chigorin
Victor Lempitsky

title: Neural codes for image retrieval

venue: In European conference on computer vision,

authors:
Sebastian Bach
Alexander Binder
Grégoire Montavon
Frederick Klauschen
Klaus-Robert Müller
Wojciech Samek

title: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

venue: PloS one,

authors:
David Bau
Jun-Yan Zhu
Hendrik Strobelt
Zhou Bolei
Joshua B. Tenenbaum
William T. Freeman
Antonio Torralba. Gan

title: dissection: Visualizing and understanding generative adversarial networks

venue: In Proceedings of the International Conference on Learning Representations (ICLR),

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale GAN training for high fidelity natural image synthesis

venue: In International Conference on Learning Representations,

authors:
Tatjana Chavdarova
François Fleuret

title: Sgan: An alternative training of generative adversarial networks

venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Xi Chen
Yan Duan
Rein Houthooft
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets

venue: In Advances in neural information processing systems,

authors:
Alexey Dosovitskiy
Thomas Brox

title: Generating images with perceptual similarity metrics based on deep networks

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron Courville

title: Improved training of wasserstein gans

venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

venue: In Advances in Neural Information Processing Systems,

authors:
Quan Hoang
Tu Dinh Nguyen
Trung Le
Dinh Phung

title: MGAN: Training generative adversarial nets with multiple generators

authors:
Xun Huang
Yixuan Li
Omid Poursaeed
John Hopcroft
Serge Belongie

title: Stacked generative adversarial networks

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A Efros

title: Image-to-image translation with conditional adversarial networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Y Jin
J Zhang
M Li
Y Tian
H Zhu

title: Towards the high-quality anime characters generation with generative adversarial networks

venue: In Proceedings of the Machine Learning for Creativity and Design Workshop at NIPS,

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

title: Learning multiple layers of features from tiny images

venue: Technical report, Citeseer,

authors:
Karol Kurach
Mario Lučić
Xiaohua Zhai
Marcin Michalski
Sylvain Gelly

title: A large-scale study on regularization and normalization in gans

venue: In International Conference on Machine Learning,

title: The mnist database of handwritten digits. http://yann

venue: lecun. com/exdb/mnist/,

authors:
Christian Ledig
Lucas Theis
Ferenc Huszár
Jose Caballero
Andrew Cunningham
Alejandro Acosta
Andrew Aitken
Alykhan Tejani
Johannes Totz
Zehan Wang

title: Photo-realistic single image super-resolution using a generative adversarial network

authors:
Aravindh Mahendran
Andrea Vedaldi

title: Understanding deep image representations by inverting them

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Mehdi S.M. Sajjadi
Olivier Bachem
Mario Lučić
Olivier Bousquet
Sylvain Gelly

title: Assessing Generative Models via Precision and Recall

venue: In Advances in Neural Information Processing Systems (NeurIPS),

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency maps

venue: arXiv preprint arXiv:1312.6034,

authors:
Mukund Sundararajan
Ankur Taly
Qiqi Yan

title: Axiomatic attribution for deep networks

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jon Shlens
Zbigniew Wojna

title: Rethinking the inception architecture for computer vision

authors:
Ting-Chun Wang
Ming-Yu Liu
Jun-Yan Zhu
Guilin Liu
Andrew Tao
Jan Kautz
Bryan Catanzaro

title: Video-to-video synthesis

venue: arXiv preprint arXiv:1808.06601,

authors:
Fisher Yu
Yinda Zhang
Shuran Song
Ari Seff
Jianxiong Xiao

title: Lsun: Construction

venue: Information Processing Systems,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation

venue: European conference on computer vision,

heading: 1 INTRODUCTION

text:
Nowadays, deep generative models are an active research direction in the machine learning community. The dominant methods for generative modeling, such as Generative Adversarial Networks (GANs), are currently able to produce diverse photorealistic images (Brock et al., 2019; Karras et al., 2019). These methods are not only popular among academicians, but are also a crucial component in a wide range of applications, including image editing (Isola et al., 2017; Zhu et al., 2017), super-resolution (Ledig et al., 2017), video generation (Wang et al., 2018) and many others.
Along with practical importance, a key benefit of accurate generative models is a more complete understanding of the internal structure of the data. Insights about the data generation process can result both in the development of new machine learning techniques as well as advances in industrial applications. However, most state-of-the-art generative models employ deep multi-layer architectures, which are difficult to interpret or explain. While many works investigate interpretability of discriminative models (Zeiler & Fergus, 2014; Simonyan et al., 2013; Mahendran & Vedaldi, 2015), only a few (Chen et al., 2016; Bau et al., 2019) address the understanding of generative ones.
In this work, we propose the Random Path GAN (RPGAN) — an alternative design of GANs that allows natural interpretability of the generator network. In traditional GAN generators, the stochastic component that influences individual samples is a noisy input vector, typically sampled from the standard Gaussian distribution. In contrast, RPGAN generators instead use stochastic routing during the forward pass as their source of stochasticity. In a nutshell, the RPGAN generator contains several instances of the corresponding layer. For each sample, only one random instance of each layer is activated during generation. The training of the RPGAN can then be performed in the same adversarial manner as in traditional GANs. In the sections below, we show how RPGAN allows to understand the factors of variation captured by the particular layer and reveals several interesting findings about the image generation process, e.g. that different layers are “responsible for” coloring or objection location. As a practical advantage, RPGANs can be efficiently updated to new data via the simple addition of new instances to the bucket, avoiding re-training the full model from scratch. Finally, we observe that RPGANs allow the construction of generative models without nonlinearities, which can significantly speed up the generation process for fully-connected layers.
In summary, the main contributions of our paper are the following:
• We introduce RPGAN — GAN with an alternative source of stochasticity, based on random routing. While being close to traditional GANs in terms of generation quality, RPGAN allows natural interpretability and efficient model updates with new data.
• With extensive experiments on standard benchmarks we reveal several insights about the image generation process. Many of our insights confirm and extend recent findings from Bau et al. (2019). Note, that our scheme is more general compared to the technique from Bau et al. (2019) as RPGAN does not require labeled datasets or pretrained segmentation models.
• We open-source the PyTorch implementation of RPGAN with common generator architectures1.
The rest of this paper is organized as follows. In Section 2 we review relevant ideas from prior art. The proposed Random Path GAN design is described in Section 3 and experimentally evaluated in Section 4. Section 5 concludes the paper and discusses possible directions for future work.

heading: 2 RELATED WORK

text:
In this section we briefly describe connections of RPGAN to existing ideas from prior works
Generative adversarial networks. GANs are currently one of the main paradigms in generative modelling. Since the seminal paper on GANs by Goodfellow et al. (2014), a plethora of alternative loss functions, architectures, normalizations, and regularization techniques were developed (Kurach et al., 2019). Today, state-of-the-art GANs are able to produce high-fidelity images, often indistinguishable from real ones (Brock et al., 2019; Karras et al., 2019). In essence, GANs consist of two networks – a generator and a discriminator, which are trained jointly in an adversarial manner. In standard GANs, the generation stochasticity is provided by the input noise vector. In RPGANs, we propose an alternative source of stochasticity by using a fixed input but random routes during forward pass in the generator.
Specific GAN architectures. Many prior works investigated different design choices for GANs, but to the best of our knowledge, none of them explicitly aimed to propose an interpretable GAN model. Hoang et al. (2018) proposed the use of several independent generators to address the mode collapse problem. Chavdarova & Fleuret (2018) employ several auxiliary local generators and discriminators to improve mode coverage as well. Huang et al. (2017) use layer-wise generators and discriminators to enforce hidden representations produced by layers of the generator to be similar to the corresponding representations produced by a reversed classification network. Important differences of RPGAN compared to the works described above is that it uses random routes as its latent space and does not enforce to mimic the latent representations of pretrained classifiers.
Interpretability. While interpretability of models based on deep neural networks is an important research direction, most existing work addresses the interpretability of discriminative models. These works typically aim to understand the internal representations of networks (Zeiler & Fergus, 2014; Simonyan et al., 2013; Mahendran & Vedaldi, 2015; Dosovitskiy & Brox, 2016) or explain decisions produced by the network for particular samples (Sundararajan et al., 2017; Bach et al., 2015; Simonyan et al., 2013). However, only a few works address interpretability of generative models. A related work by Bau et al. (2019) develops a technique that allows to identify which parts of the generator are responsible for the generation of different objects. In contrast, we propose GANs with alternative source of stochasticity that allows natural interpretation by design. Some of our findings confirm the results from Bau et al. (2019), which provides stronger evidence about the responsibilities of different layers in the generation process. Note, that the technique (Bau et al., 2019) requires a pretrained segmentation network and cannot be directly applied to several benchmarks, e.g. CIFAR10 or MNIST. In contrast, RPGAN does not require any auxiliary models or supervision and can be applied to any data.
1https://github.com/rpgan-ICLR2020/RPGAN

heading: 3 RANDOM PATH GAN

heading: 3.1 MOTIVATION

text: Before the formal description, we provide an intuition behind the RPGAN model. Several prior works have demonstrated that in discriminative convolutional neural networks different layers are “responsible” for different levels of abstraction (Zeiler & Fergus, 2014; Babenko et al., 2014). For instance, earlier layers aim to detect small texture patterns, while activations in deeper layers typically correspond to semantically meaningful concepts. Similarly, in our paper we aim to understand the roles that different GAN layers play in image generation. Thus, we propose an architecture that provides a direct way to interpret the impact of individual layers. For a given generator architecture, we construct several copies of each layer in its architecture. During the forward pass we randomly choose a layer instance that will be used when generating a particular image. Therefore, we can analyze the role of each RPGAN layer by visualizing how different instances of that layer affect the generated image.

heading: 3.2 MODEL

text:
Here we formally describe the structure of the RPGAN model. The model is highly flexible to the choice of generator and discriminator architectures as well as to the loss function and learning strategy. Similarly to the standard GAN architectures, our model consists of two networks – a generator and a discriminator. The RPGAN discriminator operates exactly like discriminators in common GANs, hence below we focus on the generator description.
Unlike existing GANs, the RPGAN generator always receives a fixed input vector Z during forward pass and aims to produce an image from the real image distribution. The generator consists of several consequent buckets B1, . . . , Bn. Each bucket is a union of independent blocks: Bi={Bi1, . . . , Bimi}, where each block is an arbitrary computational unit and mi=|Bi|. A typical example of a block is a ResNet block (He et al., 2015), a convolutional layer with a nonlinearity or any other (see Figure 1a) layer type. In our experiments, all the units from the same bucket have the same architecture.
For each i=1, . . . , n − 1 a block from the bucket Bi produces an intermediate output tensor that is passed to a block from the next bucket Bi+1. Typically we associate each bucket with a layer (or several layers) in the generator architecture, which we aim to interpret or analyze. A block from the first bucket B1 always receives a fixed input vector Z, which is the same for different forward passes. The stochasticity of the generator arises from a random path that goes from Z to an output image, using only a single block from each bucket. Formally, during each forward pass, we randomly choose indices s1, . . . , sn with 1 ≤ si ≤ mi. The generator output is then computed as Bnsn ◦ · · ·B2s2 ◦ B1s1(Z) (see Figure 1b). Thus, the generator defines a map from the Cartesian product 〈m1〉 × 〈m2〉 × · · · × 〈mn〉 to the image space. Note that we can take an arbitrary existing GAN model, group its generator layers into buckets and replicate them into multiple blocks. In these terms, the original model can be treated as the RPGAN model with a single block in each bucket and
random input noise. Note that during image generation we perform the same number of operations as in the standard GAN generator.
By its design, RPGAN with buckets B1, . . . , Bn and a constant input Z is able to generate at most |B1|×· · ·×|Bn| different samples were |Bk| is the number of blocks in the bucketBk. Nevertheless, this number is typically much larger compared to the training set size. We argue that the probability space of random paths can serve as a latent space to generate high-quality images, as confirmed by the experiments below.
Block diversity loss. To guarantee that blocks in a particular bucket are different, we also add a specific diversity term in the generator loss function. The motivation for this term is to prevent blocksBki, Bkj from learning the same weights. LetW be the set of all parameters of the generator. For each parameter w ∈ W there is a set of its instances {w(1), . . . w(mw)} in the RPGAN model. Then we enforce the instances to be different by the loss term
− ∑
w∈W,i 6=j
MSE ( w(i)
sw , w(j) sw ) Here we also normalize by the standard deviation sw of all parameters from different blocks that correspond to the same layer. This normalization effectively guarantees that all buckets contribute to the diversity term.

heading: 4 EXPERIMENTS

text:
Architecture. In all the experiments in this section, we use ResNet-like generators with spectral normalization and the hinge loss (SN-ResNet) described in Miyato et al. (2018). The blocks in the first bucket are fully-connected layers, the blocks in the last bucket are convolutional layers and blocks in all other buckets are residual blocks with two convolutions and a skip connection. If not stated otherwise, all the buckets have the same number of blocks. Additional experiments with other architectures are provided in Appendix.
Datasets. We performed experiments on CIFAR-10 (Krizhevsky et al., 2009), LSUN-bedroom (Yu et al., 2015) and Anime Faces (Jin et al., 2017) datasets. For different datasets we use different numbers of discriminator steps per one generator step dsteps and different numbers of blocks in a bucket nblocks. We summarize the main parameters used for three datasets in Table 1. In the last column we also report Coverage, which is the ratio of the latent space cardinality (which equals the number of buckets to the power nblocks) to the dataset size. Intuitively, large coverage guarantees that RPGAN has a sufficiently rich latent space of generator routes to capture the reference dataset. In the experiments below, we demonstrate that even moderate coverage is sufficient to generate high-fidelity images (see the LSUN-bedroom dataset with coverage ≈ 3.3).
Training details. We use the Adam optimizer with learning rate equal to 0.25 × 10−3, β1, β2 equal to 0.5, 0.999 and train the model for 45 × 104 generator steps for CIFAR-10 and 25 × 104 generator steps for Anime Faces and LSUN-bedroom datasets. During training we also learn the unique input vector Z. We observed that a learnable Z slightly improves the final generation quality and stabilizes the learning process. During the training step, we pass Z through N independent random paths. Formally, let {x1, . . . , xN} be a batch of samples received from a bucketBk. To pass this batch through the bucket Bk+1 we take random blocks Bki1 , . . . , BkiN and form a new batch {Bki1(x1), . . . , BkiN (xN )}. In all the experiments, we use the same training protocols for both the RPGAN and the standard GAN of the generator architecture. Note, that despite larger number of learnable parameters, RPGAN does not require more data or training time to achieve the same quality, compared to standard GANs.

heading: 4.1 LAYERS INTERPRETATION

text:
In the first series of experiments we investigate the “ responsibility areas” of different generator layers. This can be performed with a technique schematically presented on Figure 2. The goal is to interpret the role of the third bucketB3 in a five-bucket generator. For all other bucketsB1, B2, B4, B5 we fix arbitrary blocks, shown in blue on Figure 2. Then we generate images corresponding to routes that contain all the fixed blocks, with the stochasticity coming only from varying blocks inside the target bucket B3. By inspecting the distribution of the obtained images, we can understand what factors of variation are influenced by B3.
On Figure 3 we plot the samples generated with only one “unfrozen” bucket for one image from the CIFAR-10 dataset. Thus, each row shows how the original generated image could change if different blocks from the corresponding bucket are used. Several observations from Figure 3, Figure 9, Figure 12 and Figure 15 are listed below. The first bucket typically does not influence coloring and mostly affects small objects’ deformations. The intermediate buckets have the largest influence on semantics. The last two buckets are mostly responsible for coloring and do not influence the content shape. In particular, on Figure 3 the fourth layer widely varies color, while the fifth acts as a general tone corrector. Note that these findings are consistent with the insights revealed by Bau et al. (2019).
To confirm the observations quantitatively, we perform the following experiment. We define a metric dimg that evaluates the similarity between two generated images. Different metrics are able to capture different variations (e.g. in terms of semantic, color histogram, etc.) and we describe two particular choices of dimg below. Then we choose a random route in the RPGAN generator and for each bucket
Bl generate four images s (l) 1 , . . . , s (l) 4 , varying blocks in Bl. In other words, rather then taking all possible pairs, we take four random samples from each line of the table in the Figure 3. Then we measure diversity w.r.t. dimg captured by bucket Bl as a ratio
Dl→1,dimg =
∑ i 6=j dimg(s (l) i , s
(l) j )∑ i 6=j dimg(s (1) i , s (1) j )
Intuitively, we compute the relative diversity with respect to the first layer, which typically captures the smallest amount of variations in our experiments. We then average these ratios over 100 independent evaluations. High values of averaged ratio Dl→1,dimg imply higher diversity of the bucket Bl compared to the first bucket in terms of the metric dimg. For dimg we experimented with the following two metrics, capturing semantics and color differences correspondingly.
Inspired by the well-known Fréchet Inception Score concept (Heusel et al., 2017), we consider the Euclidean distance between the outputs of the last layer of the pretrained InceptionV3 network (Szegedy et al., 2016) for semantic distance evaluation. Namely, we define dsemantic(img1, img2) as ‖Iv3(img1)− Iv3(img2)‖2 were Iv3(img) is the InceptionV3 model activations for image img. To measure differences in color, we take the Hellinger distance between the color histograms of generated samples. Namely, for each color channel, we split the range [0, . . . , 255] into 25 equal segments and evaluate the discrete distribution defined by the frequencies the sample’s pixel intensities appear in a given segment. Then the Hellinger distance between two quantified color distributions is
defined as 1√ 2 √ 25∑ i=1 ( √ pi − √ qi)2 . We compute this metric for each color channel independently.
The average values ofDl→1,dimg with the standard deviations are shown on Figure 4. It demonstrates that the semantic diversity is the largest for the intermediate layers. On the contrary, the last buckets, which are closer to the output, do not influence semantics but have higher impact in terms of color. Note that the first layer always shows the smallest variability in terms of both semantics and colors. The last bucket seems to be responsible for color correction and color inversion and has a lower pallet variability impact. Note, that the plots from Figure 4 reaffirm the findings coming from the Figure 3. Note that the similar empirical results also hold for other datasets (see figures 13, 16 in appendix).
Overall, we summarize the main findings common for CIFAR-10, LSUN and Anime Faces datasets as:
• The earlier layers have a smaller variability and seem to be responsible for the viewpoint and the position of the object on the image.
• The semantic details of the image content are mostly determined by the intermediate layers. • The last layers typically affect only coloring scheme and do not affect content semantics or
image geometry.
Note, that these conclusions can differ for other datasets or other generator architectures. For instance, for the four-bucket generator and MNIST (Figure 6, left) or randomly colored MNIST ( Figure 8, left) the semantics are mostly determined by the first two buckets.

heading: 4.2 RPGAN VS STANDARD GAN

text:
In this subsection, we argue that the interpretations of different layers, obtained with RPGAN, are also valid for the standard GAN generator of the same architecture. First, we demonstrate that both standard GAN and RPGAN trained under the same training protocol provide almost the same generation quality. As a standard evaluation measure, we use the Fréchet Inception Distance (FID) introduced in Heusel et al. (2017). We also compute precision-recall curve as defined in Sajjadi et al. (2018). For evaluation on CIFAR-10, we use 50000 generated samples and the whole train dataset. We also take ten independently trained generators and report minimal and average FID values. See Table 2 for FID comparison and Figure 11 in appendix for precision-recall. RPGAN and SN-ResNet perform with the same quality both in terms of FID and precision-recall curves.
To confirm that the layers of the standard GAN generator can be interpreted in the same way as the corresponding layers of its RPGAN counterpart, we perform the following experiment. We take a standard SN-ResNet GAN, consisting of five layers associated with the correspondent buckets in RPGAN, and train it on CIFAR-10. Then for each layer, we add normal noise to its weights. Intuitively, we expect that the noise injection in the particular layer would change generated samples in terms of characteristics, influenced by this layer. For instance, noise in the last two layers is expected to harm coloring scheme, while noise in the intermediate layers is expected to bring maximal semantic damage. Several samples, produced by perturbed layers, are presented on Figure 5. The images support the intuition described above and confirm that RPGAN may serve as an analysis tool for the corresponding generator model. Note, however, that injecting noise per se is not sufficient for interpretability. The perturbed generators produce poor images, which are difficult to analyze. Meanwhile, RPGAN always generates good-looking images, which allows to identify the factors of variation, corresponding to the particular layer. For instance, see Figure 8 for the colored MNIST dataset. Figure 8 (left) shows plausible images, generated by varying RPGAN blocks. In contrast, Figure 8 (right) demonstrates images from generators perturbed with small and large noise. For both noise magnitudes, these images are difficult to interpret. Of course, given the interpretations obtained via RPGAN, one can perceive similar patterns in the noisy generations, but noise injection alone is not sufficient for interpretability.

heading: 4.3 INCREMENTAL LEARNING WITH RPGAN

text:
In the next experiment, we demonstrate that the RPGAN model is also a natural fit for the generative incremental learning task (see e.g., Wu et al. (2018)). Let us assume that the whole train dataset D is split into two disjoint subsets D = D1 ∪ D2. Suppose that initially we have no samples from D2 and train a generative model to approximate a distribution defined by the subset D1. Then, given additional samples from D2, we aim to solve an incremental learning task — to update the model with new data without re-training it from scratch. The RPGAN model allows solving this task naturally. First we train a generator with buckets B1, . . . , Bn to reproduce the subset D1. Once we want to extend the generator with samples fromD2, we add several new blocks to the buckets that are responsible for the features that capture the difference between D1 and D2. Then we optimize the generator to reproduce both D1 and D2 by training only the new blocks. Thus, instead of training a new generator from scratch, we exploit the pretrained blocks that are responsible for features, which
are common for D1 and D2. To illustrate this scenario, we take a partition of the MNIST handwritten digits dataset (LeCun, 1989) into two subsets MNIST0−6 and MNIST7−9 of digits from 0 to 6 and from 7 to 9 correspondingly. As for generator for MNIST0−6 we take a 4-bucket RPGAN model with a number of blocks equal to 20, 20, 20, 8. Note that the last bucket is much thinner than the others, as it turns out to be responsible for variations in writing style, which does not change much across the dataset. Then we train the generator on the subset MNIST0−6 of first 7 digits (see Figure 6, left and center). After that we add five additional blocks to each of the first two layers, obtaining a generator with a number of blocks equal to 25, 25, 20, 8 and pretrained weights in all blocks but five in the first and in the second buckets. Then we train the extended model to fit the whole MNIST by optimizing only the ten new blocks (see Figure 6, right).

heading: 4.4 PURELY LINEAR GENERATOR

text:
As a surprising side effect of our model, we discovered that decent generation quality can be achieved by the RPGAN generator with no nonlinearities, i.e., one can train the RPGAN generator with all blocks consisting of linear transformations only. To demonstrate that, we take an RPGAN with the same ResNet-like generator architecture as in the experiments above. Then we replace all nonlinearities in the generator model by identity operations and train it on the CIFAR-10 dataset. The model demonstrates FID equal to 22.79 that is competitive to the state-of-the-art generative models of comparable sizes (see Figure 23 for generated images examples). Note that this approach fails for a standard GAN generator that maps a Gaussian distribution to an images distribution. Indeed, that generator would be a linear operator from a latent space to the images space with a Gaussian distribution in the images domain.
This purely linear generator architecture allows us to significantly speed up the image generation process for fully-connected layers. We group consequent buckets of fully-connected layers to form a new bucket. Blocks in the new bucket are linear transformations that are products of the blocks from the original buckets. To demonstrate this, we train a fully-connected generator network on the MNIST dataset (see Table 3 in appendix, left column). Then we join the last three buckets into a single one. We form a new bucket by blocks defined as the linear operators B5k ◦ B4j ◦ B3i where
i, j, k are random indices of blocks from the buckets B3, B4, B5 of the original generator. Thus instead of performing three multiplications of features vector from the second layer by matrices of the shapes 256 × 512, 512 × 1024, 1024 × 784 we perform a single multiplication by a 256 × 784 matrix. In our experiments, we achieved ×2.2 speed up. Note, however, that after the compression, the latent space cardinality can decrease if a small subset of tuples (i, j, k) is used to populate the new bucket. Nevertheless, as random products of joining buckets are used, we expect that the generated images would be uniformly distributed in the space of images, produced by the uncompressed generator (see Figure 24 for samples comparison).

heading: 5 CONCLUSION

text: In this paper, we address the interpretability of generative models. In particular, we have introduced RPGAN, an alternative design of generative adversarial networks, which allows natural interpretation of different generator layers via using random routing as a source of stochasticity. With experiments on several datasets, we provide evidence that different layers are responsible for the different factors of variation in generated images, which is consistent with findings from previous work. As a possible direction of future research, one can use the RPGAN analysis to construct efficient models, e.g., via identification of redundant parts of the generator for pruning or inference speedup.

heading: A ADDITIONAL FIGURES AND EXPERIMENTS

text:
A.1 ABLATION ON NUMBER OF BLOCKS
Here we investigate the impact of the number of blocks in each RPGAN bucket on the generation quality. We train RPGAN with the SN-ResNet generator on CIFAR-10 with a different numbers of blocks in each bucket. For simplicity, all buckets have the same number of blocks. The resulting FID values are presented on Figure 7
If the number of blocks is too low, the resulting latent space appears to have insufficient cardinality to cover the dataset. On the other hand, a too high number of blocks results in a difficult training procedure and also fails.
A.2 LSUN BEDROOM DATASET
To generate LSUN-bedroom-like images, we use the 7-bucket RPGAN with ResNet-like generator with five residual blocks, the first fully-connected layer, and the last convolutional layer. Similarly to CIFAR-10 experiments, during the generation, we freeze a random path and vary blocks in a single bucket to investigate its responsibility. See Figure 12 for blocks variations and Figure 13 for buckets responsibility analysis. Note that similarly to CIFAR-10, the central buckets have a maximal semantic impact. Last two buckets are mostly responsible for coloring. The first two buckets are responsible for local geometrical features. Note that here we face mode collapse for the third bucket: mainly, it affects only tiny local features. See Figure 14 for samples generated by the model.
A.3 ANIME FACES DATASET
Though this dataset is not standard for GANs, we use it in the experiments as it nicely reveals the RPGAN analysis tools. Here we use the 6-bucket RPGAN with ResNet-like generator with four residual blocks, the first fully-connected layer, and the last convolutional layer. See Figure 15 for block variations and Figure 16 for bucket responsibility analysis. Again, the content semantics is mostly defined by the intermediate buckets. The last two buckets are mostly responsible for coloring: the fifth bucket has the maximal impact on coloring, and the last bucket varies tones. The first buckets are responsible for small details (one can note the hair on the character’s forehead). See Figure 17 for samples generated by the model.
A.4 WASSERSTEIN GAN
Here we show that the concept of RPGAN works well with different generator architectures and learning strategies. Here we present plots for DCGAN-like generators consisted of consequent convolutional layers without skip connections. All the models were trained with the same parameters as described in Section 4. Despite of Spectral Normalization, we train these models as WGANs with weight penalty (Gulrajani et al., 2017). On the Figure 19 we show plots for a four-bucket generator trained on CIFAR-10. We also train four-bucket generator on colored MNIST, see Figure 8, left. Finaly, we show plots for the five-bucket generator and CelebA-64x64 dataset on Figure 21. See Figure 18, Figure 21, Figure 22 for buckets analysis.

heading: B RPGAN INTERPRETABILITY VS WEIGHTS NOISING

text: In this section we show that injecting noise in the generator weights cannot be used as a stand-alone interpretability method. Namely, we compare images produces by RPGAN and noise injection for models trained on randomly colored MNIST samples. We train both RPGAN and standard generators as a Wasserstein GAN with weights penalty.


summaries: In this paper, CNNs specialized for spherical data are studied. The proposed architecture is a combination of existing frameworks based on the discretization of a sphere as a graph. As a main result, the paper shows a convergence result, which is related to the rotation equivalence on a sphere. The experiments show the proposed model achieves a good tradeoff between the prediction performance and the computational cost.

summaries: The paper studies the problem of designing a convolution for a spherical neural network. The authors use the existing graph CNN formulation and a pooling strategy that exploits hierarchical pixelations of the sphere to learn from the discretized sphere. The main idea is to model the discretized sphere as a graph of connected pixels: the length of the shortest path between two pixels is an approximation of the geodesic distance between them. To show the computational efficiency, sampling flexibility and rotation equivariance, extensive experiments are conducted, including 3D object recognition, cosmological mode classification, climate event segmentation and uneven sampling.

summaries: The paper presents DeepSphere, a method for learning over spherical data via a graphical representation and graph-convolutions. The primary goal is to develop a method that encodes equivariance to rotations, cheaply. The graph is formed by sampling the surface of the sphere and connecting neighbors according to a distance-based similarity measure. The equivariance of the representation is demonstrated empirically and theoretical background on its convergence properties are shown. DeepSphere is then demonstrated on several problems as well as shown how it applies to non-uniform data.

abstractText: Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance. DeepSphere, a method based on a graph representation of the sampled sphere, strikes a controllable balance between these two desiderata. This contribution is twofold. First, we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of vertices and neighbors. Second, we evaluate DeepSphere on relevant problems. Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation. Perhaps surprisingly, comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay. Our code is available at https: //github.com/deepsphere.

authors:
Vincent Vanhoucke
Vijay Vasudevan
Fernanda Viégas
Oriol Vinyals
Pete Warden
Martin Wattenberg
Martin Wicke
Yuan Yu
Xiaoqiang Zheng

title: TensorFlow: Large-scale machine learning on heterogeneous systems

title: Gravitational lensing

venue: Classical and Quantum Gravity,

authors:
John R Baumgardner
Paul O Frederickson

title: Icosahedral discretization of the two-sphere

venue: SIAM Journal on Numerical Analysis,

authors:
Mikhail Belkin
Partha Niyogi

title: Convergence of laplacian eigenmaps

title: Towards a theoretical foundation for laplacian-based manifold methods

venue: Journal of Computer and System Sciences,

authors:
Wouter Boomsma
Jes Frellsen

title: Spherical convolutions and their application in molecular modelling

authors:
Davide Boscaini
Jonathan Masci
Emanuele Rodolà
Michael Bronstein

title: Learning shape correspondence with anisotropic convolutional neural networks

authors:
Joan Bruna
Wojciech Zaremba
Arthur Szlam
Yann LeCun

title: Spectral networks and locally connected networks on graphs

authors:
Angel X Chang
Thomas Funkhouser
Leonidas Guibas
Pat Hanrahan
Qixing Huang
Zimo Li
Silvio Savarese
Manolis Savva
Shuran Song
Hao Su

title: Shapenet: An information-rich 3d model repository

authors:
Taco S Cohen
Mario Geiger
Jonas Koehler
Max Welling

title: Spherical cnns

venue: In International Conference on Learning Representations (ICLR),

authors:
Taco S. Cohen
Maurice Weiler
Berkay Kicanaoglu
Max Welling

title: Gauge equivariant convolutional networks and the icosahedral cnn

venue: In International Conference on Machine Learning (ICML),

authors:
Benjamin Coors
Alexandru Paul Condurache
Andreas Geiger

title: Spherenet: Learning spherical representations for detection and classification in omnidirectional images

venue: In European Conference on Computer Vision,

authors:
Michaël Defferrard
Xavier Bresson
Pierre Vandergheynst

title: Convolutional neural networks on graphs with fast localized spectral filtering

authors:
Michaël Defferrard
Nathanaël Perraudin
Tomasz Kacprzak
Raphael Sgier

title: Deepsphere: towards an equivariant graph-based spherical cnn

venue: In ICLR Workshop on Representation Learning on Graphs and Manifolds,

authors:
J.R. Driscoll
D.M. Healy

title: Computing fourier transforms and convolutions on the 2-sphere

venue: Adv. Appl. Math.,

authors:
Carlos Esteves
Christine Allen-Blanchette
Ameesh Makadia
Kostas Daniilidis

title: Learning so(3) equivariant representations with spherical cnns

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Krzysztof M Gorski
Benjamin D Wandelt
Frode K Hansen
Eric Hivon
Anthony J Banday

title: The healpix primer

venue: arXiv preprint astro-ph/9905275,

authors:
Krzysztof M Gorski
Eric Hivon
AJ Banday
Benjamin D Wandelt
Frode K Hansen
Mstvos Reinecke
Matthia Bartelmann

title: Healpix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere

venue: The Astrophysical Journal,

title: Matplotlib: A 2d graphics environment

venue: Computing in Science & Engineering,

authors:
Chiyu ”Max” Jiang
Jingwei Huang
Karthik Kashinath
Prabhat
Philip Marcus
Matthias Niessner

title: Spherical cnns on unstructured grids

authors:
Renata Khasanova
Pascal Frossard

title: Graph-based classification of omnidirectional images

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Mayur Mudigonda
Sookyung Kim
Ankur Mahesh
Samira Kahou
Karthik Kashinath
Dean Williams
Vincen Michalski
Travis O’Brien
Mr Prabhat

title: Segmenting and tracking extreme climate events using neural networks. In Deep Learning for Physical Sciences (DLPS

authors:
Nathanaël Perraudin
Michaël Defferrard
Tomasz Kacprzak
Raphael Sgier

title: Deepsphere: Efficient spherical convolutional neural network with healpix sampling for cosmological applications

venue: Astronomy and Computing,

authors:
Manolis Savva
Fisher Yu
Hao Su
Asako Kanezaki
Takahiko Furuya
Ryutarou Ohbuchi
Zhichao Zhou
Rui Yu
Song Bai
Xiang Bai

title: Large-scale 3d shape retrieval from shapenet core55: Shrec’17 track

venue: In Eurographics Workshop on 3D Object Retrieval,

authors:
J. Schmelzle
A. Lucchi
T. Kacprzak
A. Amara
R. Sgier
A. Réfrégier
T. Hofmann

title: Cosmological model discrimination with deep learning

authors:
Yu-Chuan Su
Kristen Grauman

title: Learning spherical convolution for fast features from 360 imagery

authors:
Charles R Harris
Anne M. Archibald
Antônio H. Ribeiro
Fabian Pedregosa
Paul van

title: Mulbregt, and SciPy 1. 0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python

venue: Nature Methods,

authors:
Stéfan van der Walt
S Chris Colbert
Gael Varoquaux

title: The numpy array: a structure for efficient numerical computation

authors:
Andrea Zonca
Leo Singer
Daniel Lenz
Martin Reinecke
Cyrille Rosset
Eric Hivon
Krzysztof Gorski

title: healpy: equal area pixelization and spherical harmonics transforms for data on the sphere in python

venue: Journal of Open Source Software,

heading: 1 INTRODUCTION

text:
Spherical data is found in many applications (figure 1). Planetary data (such as meteorological or geological measurements) and brain activity are example of intrinsically spherical data. The observation of the universe, LIDAR scans, and the digitalization of 3D objects are examples of projections due to observation. Labels or variables are often to be inferred from them. Examples are the inference of cosmological parameters from the distribution of mass in the universe (Perraudin et al., 2019), the segmentation of omnidirectional images (Khasanova & Frossard, 2017), and the segmentation of cyclones from Earth observation (Mudigonda et al., 2017).
As neural networks (NNs) have proved to be great tools for inference, variants have been developed to handle spherical data. Exploiting the locally Euclidean property of the sphere, early attempts used standard 2D convolutions on a grid sampling of the sphere (Boomsma & Frellsen, 2017; Su & Grauman, 2017; Coors et al., 2018). While simple and efficient, those convolutions are not equivariant to rotations. On the other side of this tradeoff, Cohen et al. (2018) and Esteves et al. (2018) proposed to perform proper spherical convolutions through the spherical harmonic transform. While equivariant to rotations, those convolutions are expensive (section 2).
As a lack of equivariance can penalize performance (section 4.2) and expensive convolutions prohibit their application to some real-world problems, methods standing between these two extremes are desired. Cohen et al. (2019) proposed to reduce costs by limiting the size of the representation of the symmetry group by projecting the data from the sphere to the icosahedron. The distortions introduced by this projection might however hinder performance (section 4.3).
Another approach is to represent the sampled sphere as a graph connecting pixels according to the distance between them (Bruna et al., 2013; Khasanova & Frossard, 2017; Perraudin et al., 2019). While Laplacian-based graph convolutions are more efficient than spherical convolutions, they are not exactly equivariant (Defferrard et al., 2019). In this work, we argue that graph-based spherical CNNs strike an interesting balance, with a controllable tradeoff between cost and equivariance (which is linked to performance). Experiments on multiple problems of practical interest show the competitiveness and flexibility of this approach.

heading: 2 METHOD

text:
DeepSphere leverages graph convolutions to achieve the following properties: (i) computational efficiency, (ii) sampling flexibility, and (iii) rotation equivariance (section 3). The main idea is to model the sampled sphere as a graph of connected pixels: the length of the shortest path between two pixels is an approximation of the geodesic distance between them. We use the graph CNN formulation introduced in (Defferrard et al., 2016) and a pooling strategy that exploits hierarchical samplings of the sphere.
Sampling. A sampling scheme V = {xi ∈ S2}ni=1 is defined to be the discrete subset of the sphere containing the n points where the values of the signals that we want to analyse are known. For a given continuous signal f , we represent such values in a vector f ∈ Rn. As there is no analogue of uniform sampling on the sphere, many samplings have been proposed with different tradeoffs. In this work, depending on the considered application, we will use the equiangular (Driscoll & Healy, 1994), HEALPix (Gorski et al., 2005), and icosahedral (Baumgardner & Frederickson, 1985) samplings.
Graph. From V , we construct a weighted undirected graph G = (V, w), where the elements of V are the vertices and the weight wij = wji is a similarity measure between vertices xi and xj . The combinatorial graph Laplacian L ∈ Rn×n is defined as L = D − A, where A = (wij) is the weighted adjacency matrix, D = (dii) is the diagonal degree matrix, and dii = ∑ j wij is the weighted degree of vertex xi. Given a sampling V , usually fixed by the application or the available measurements, the freedom in constructing G is in setting w. Section 3 shows how to set w to minimize the equivariance error.
Convolution. On Euclidean domains, convolutions are efficiently implemented by sliding a window in the signal domain. On the sphere however, there is no straightforward way to implement a convolution in the signal domain due to non-uniform samplings. Convolutions are most often performed in the spectral domain through a spherical harmonic transform (SHT). That is the approach taken by Cohen et al. (2018) and Esteves et al. (2018), which has a computational cost of O(n3/2) on isolatitude samplings (such as the HEALPix and equiangular samplings) and O(n2) in general.
1 https://martinos.org/mne/stable/auto_tutorials/plot_visualize_evoked.html 2 https://www.ncdc.noaa.gov/ghcn-daily-description
On the other hand, following Defferrard et al. (2016), graph convolutions can be defined as
h(L)f = ( P∑ i=0 αiL i ) f , (1)
where P is the polynomial order (which corresponds to the filter’s size) and αi are the coefficients to be optimized during training.3 Those convolutions are used by Khasanova & Frossard (2017) and Perraudin et al. (2019) and cost O(n) operations through a recursive application of L.4
Pooling. Down- and up-sampling is natural for hierarchical samplings,5 where each subdivision divides a pixel in (an equal number of) child sub-pixels. To pool (down-sample), the data supported on the sub-pixels is summarized by a permutation invariant function such as the maximum or the average. To unpool (up-sample), the data supported on a pixel is copied to all its sub-pixels.
Architecture. All our NNs are fully convolutional, and employ a global average pooling (GAP) for rotation invariant tasks. Graph convolutional layers are always followed by batch normalization and ReLU activation, except in the last layer. Note that batch normalization and activation act on the elements of f independently, and hence don’t depend on the domain of f .

heading: 3 GRAPH CONVOLUTION AND EQUIVARIANCE

text: While the graph framework offers great flexibility, its ability to faithfully represent the underlying sphere — for graph convolutions to be rotation equivariant — highly depends on the sampling locations and the graph construction.

heading: 3.1 PROBLEM FORMULATION

text:
A continuous function f : C(S2) ⊃ FV → R is sampled as TV(f) = f by the sampling operator TV : C(S2) ⊃ FV → Rn defined as f : fi = f(xi). We require FV to be a suitable subspace of continuous functions such that TV is invertible, i.e., the function f ∈ FV can be unambiguously reconstructed from its sampled values f . The existence of such a subspace depends on the sampling V and its characterization is a common problem in signal processing (Driscoll & Healy, 1994). For most samplings, it is not known if FV exists and hence if TV is invertible. A special case is the equiangular sampling where a sampling theorem holds, and thus a closed-form of T−1V is known. For samplings where no such sampling formula is available, we leverage the discrete SHT to reconstruct f from f = TVf , thus approximating T−1V . For all theoretical considerations, we assume that FV exists and f ∈ FV . By definition, the (spherical) graph convolution is rotation equivariant if and only if it commutes with the rotation operator defined as R(g), g ∈ SO(3): R(g)f(x) = f ( g−1x ) . In the context of this work, graph convolution is performed by recursive applications of the graph Laplacian (1). Hence, if R(g) commutes with L, then, by recursion, it will also commute with the convolution h(L). As a result, h(L) is rotation equivariant if and only if
RV(g)Lf = LRV(g)f , ∀f ∈ FV and ∀g ∈ SO(3), where RV(g) = TVR(g)T−1V . For an empirical evaluation of equivariance, we define the normalized equivariance error for a signal f and a rotation g as
EL(f , g) =
( ‖RV(g)Lf −LRV(g)f‖
‖Lf‖
)2 . (2)
More generally for a class of signals f ∈ C ⊂ FV , the mean equivariance error defined as EL,C = Ef∈C,g∈SO(3) EL(f , g) (3)
represents the overall equivariance error. The expected value is obtained by averaging over a finite number of random functions and random rotations.
3In practice, training with Chebyshev polynomials (instead of monomials) is slightly more stable. We believe it to be due to their orthogonality and uniformity.
4As long as the graph is sparsified such that the number of edges, i.e., the number of non-zeros in A, is proportional to the number of vertices n. This can always be done as most weights are very small.
5The equiangular, HEALPix, and icosahedral samplings are of this kind.
104 105 106 107
n= 12N 2side pixels
10-6
10-5
10-4
10-3
ke rn
el w
id th t k-NN graph, k= 60 k-NN graph, k= 40 k-NN graph, k= 20 k-NN graph, k= 8 Perraudin et al., k= 8
Figure 3: Kernel widths.
Figure 4: 3D object represented as a spherical depth map.
100 101 102 103 spherical harmonic degree 10 3 10 2 10 1
100 101
po we
r s pe
ct ru m SHREC'17 (depth and normal) cosmology (convergence map) climate (16 variables)
Figure 5: Power spectral densities.

heading: 3.2 FINDING THE OPTIMAL WEIGHTING SCHEME

text:
Considering the equiangular sampling and graphs where each vertex is connected to 4 neighbors (north, south, east, west), Khasanova & Frossard (2017) designed a weighting scheme to minimize (3) for longitudinal and latitudinal rotations6. Their solution gives weights inversely proportional to Euclidean distances:
wij = 1
‖xi − xj‖ . (4)
While the resulting convolution is not equivariant to the whole of SO(3) (figure 2), it is enough for omnidirectional imaging because, as gravity consistently orients the sphere, objects only rotate longitudinally or latitudinally.
To achieve equivariance to all rotations, we take inspiration from Belkin & Niyogi (2008). They prove that for a random uniform sampling, the graph Laplacian L built from weights
wij = e − 14t‖xi−xj‖ 2
(5)
converges to the Laplace-Beltrami operator ∆S2 as the number of samples grows to infinity. This result is a good starting point as ∆S2 commutes with rotation, i.e., ∆S2R(g) = R(g)∆S2 . While the weighting scheme is full (i.e., every vertex is connected to every other vertex), most weights are small due to the exponential. We hence make an approximation to limit the cost of the convolution (1) by only considering the k nearest neighbors (k-NN) of each vertex. Given k, the optimal kernel width t is found by searching for the minimizer of (3). Figure 3 shows the optimal kernel widths found for various resolutions of the HEALPix sampling. As predicted by the theory, tn ∝ nβ , β ∈ R. Importantly however, the optimal t also depends on the number of neighbors k.
Considering the HEALPix sampling, Perraudin et al. (2019) connected each vertex to their 8 adjacent vertices in the tiling of the sphere, computed the weights with (5), and heuristically set t to half the average squared Euclidean distance between connected vertices. This heuristic however overestimates t (figure 3) and leads to an increased equivariance error (figure 2).
6Equivariance to longitudinal rotation is essentially given by the equiangular sampling.

heading: 3.3 ANALYSIS OF THE PROPOSED WEIGHTING SCHEME

text:
We analyze the proposed weighting scheme both theoretically and empirically.
Theoretical convergence. We extend the work of (Belkin & Niyogi, 2008) to a sufficiently regular, deterministic sampling. Following their setting, we work with the extended graph Laplacian operator as the linear operator Ltn : L 2(S2)→ L2(S2) such that
Ltnf(y) := 1
n n∑ i=1 e− ‖xi−y‖ 2 4t (f(y)− f(xi)) . (6)
This operator extends the graph Laplacian with the weighting scheme (5) to each point of the sphere (i.e., Ltnf = TVL t nf ). As the radius of the kernel t will be adapted to the number of samples, we scale the operator as
L̂tn := |S2|(4πt2)−1Ltn. Given a sampling V , we define σi to be the patch of the surface of the sphere corresponding to xi, Ai its corresponding area, and di the largest distance between the center xi and any point on the surface σi. Define d(n) := maxi=1,...,n di and A(n) := maxi=1,...,nAi.
Theorem 3.1. For a sampling V of the sphere that is equi-area and such that d(n) ≤ Cnα , α ∈ (0, 1/2], for all f : S2 → R Lipschitz with respect to the Euclidean distance in R3, for all y ∈ S2, there exists a sequence tn = nβ , β ∈ R such that
lim n→∞
L̂tnn f(y) = ∆S2f(y).
This is a major step towards equivariance, as the Laplace-Beltrami operator commutes with rotation. Based on this property, we show the equivariance of the scaled extended graph Laplacian. Theorem 3.2. Under the hypothesis of theorem 3.1, the scaled graph Laplacian commutes with any rotation, in the limit of infinite sampling, i.e.,
∀y ∈ S2 ∣∣∣R(g)L̂tnn f(y)− L̂tnn R(g)f(y)∣∣∣ n→∞−−−−→ 0.
From this theorem, it follows that the discrete graph Laplacian will be equivariant in the limit of n → ∞ as by construction Ltnf = TVLtnf and as the scaling does not affect the equivariance property of Ltn.
Importantly, the proof of Theorem 3.1 (in Appendix A) inspires our construction of the graph Laplacian. In particular, it tells us that t should scale as nβ , which has been empirically verified (figure 3). Nevertheless, it is important to keep in mind the limits of Theorem 3.1 and 3.2. Both theorems present asymptotic results, but in practice we will always work with finite samplings. Furthermore, since this method is based on the capability of the eigenvectors of the graph Laplacian to approximate the spherical harmonics, a stronger type of convergence of the graph Laplacian would be preferable, i.e., spectral convergence (that is proved for a full graph in the case of random sampling for a class of Lipschitz functions in (Belkin & Niyogi, 2007)). Finally, while we do not have a formal proof for it, we strongly believe that the HEALPix sampling does satisfy the hypothesis d(n) ≤ Cnα , α ∈ (0, 1/2], with α very close or equal to 1/2. The empirical results discussed in the next paragraph also points in this direction. This is further discussed in Appendix A.
Empirical convergence. Figure 2 shows the equivariance error (3) for different parameter sets of DeepSphere for the HEALPix sampling as well as for the graph construction of Khasanova & Frossard (2017) for the equiangular sampling. The error is estimated as a function of the sampling resolution and signal frequency. The resolution is controlled by the number of pixels n = 12N2side for HEALPix and n = 4b2 for the equiangular sampling. The frequency is controlled by setting the set C to functions f made of spherical harmonics of a single degree `. To allow for an almost perfect implementation (up to numerical errors) of the operator RV , the degree ` was chosen in the range (0, 3Nside − 1) for HEALPix and (0, b) for the equiangular sampling (Gorski et al., 1999). Using these parameters, the measured error is mostly due to imperfections in the empirical approximation of the Laplace-Beltrami operator and not to the sampling.
Figure 2 shows that the weighting scheme (4) from (Khasanova & Frossard, 2017) does indeed not lead to a convolution that is equivariant to all rotations g ∈ SO(3).7 For k = 8 neighbors, selecting the optimal kernel width t improves on (Perraudin et al., 2019) at no cost, highlighting the importance of this parameter. Increasing the resolution decreases the equivariance error in the high frequencies, an effect most probably due to the sampling. Most importantly, the equivariance error decreases when connecting more neighbors. Hence, the number of neighbors k gives us a precise control of the tradeoff between cost and equivariance.

heading: 4 EXPERIMENTS

heading: 4.1 3D OBJECTS RECOGNITION

text:
The recognition of 3D shapes is a rotation invariant task: rotating an object doesn’t change its nature. While 3D shapes are usually represented as meshes or point clouds, representing them as spherical maps (figure 4) naturally allows a rotation invariant treatment.
The SHREC’17 shape retrieval contest (Savva et al., 2017) contains 51,300 randomly oriented 3D models from ShapeNet (Chang et al., 2015), to be classified in 55 categories (tables, lamps, airplanes, etc.). As in (Cohen et al., 2018), objects are represented by 6 spherical maps. At each pixel, a ray is traced towards the center of the sphere. The distance from the sphere to the object forms a depth map. The cos and sin of the surface angle forms two normal maps. The same is done for the object’s convex hull.8 The maps are sampled by an equiangular sampling with bandwidth b = 64 (n = 4b2 = 16, 384 pixels) or an HEALPix sampling with Nside = 32 (n = 12N2side = 12, 288 pixels).
The equiangular graph is built with (4) and k = 4 neighbors (following Khasanova & Frossard, 2017). The HEALPix graph is built with (5), k = 8, and a kernel width t set to the average of the distances (following Perraudin et al., 2019). The NN is made of 5 graph convolutional layers, each followed by a max pooling layer which down-samples by 4. A GAP and a fully connected layer with softmax follow. The polynomials are all of order P = 3 and the number of channels per layer is 16, 32, 64, 128, 256, respectively. Following Esteves et al. (2018), the cross-entropy plus a triplet loss is optimized with Adam for 30 epochs on the dataset augmented by 3 random translations. The learning rate is 5 · 10−2 and the batch size is 32. Results are shown in table 1. As the network is trained for shape classification rather than retrieval, we report the classification F1 alongside the mAP used in the retrieval contest.10 DeepSphere achieves the same performance as Cohen et al. (2018) and Esteves et al. (2018) at a much lower cost, suggesting that anisotropic filters are an unnecessary price to pay. As the information in those spherical maps resides in the low frequencies (figure 5), reducing the equivariance error didn’t translate into improved performance. For the same reason, using the more uniform HEALPix sampling or lowering the resolution down to Nside = 8 (n = 768 pixels) didn’t impact performance either.
7We however verified that the convolution is equivariant to longitudinal and latitudinal rotations, as intended. 8Albeit we didn’t observe much improvement by using the convex hull. 7As implemented in https://github.com/jonas-koehler/s2cnn.
10We omit the F1 for Cohen et al. (2018) as we didn’t get the mAP reported in the paper when running it.
accuracy time
Perraudin et al. (2019), 2D CNN baseline 54.2 104 ms Perraudin et al. (2019), CNN variant, k = 8 62.1 185 ms Perraudin et al. (2019), FCN variant, k = 8 83.8 185 ms k = 8 neighbors, t from section 3.2 87.1 185 ms k = 20 neighbors, t from section 3.2 91.3 250 ms k = 40 neighbors, t from section 3.2 92.5 363 ms
Table 2: Results on the classification of partial convergence maps. Lower equivariance error translates to higher performance.

heading: 4.2 COSMOLOGICAL MODEL CLASSIFICATION

text:
Given observations, cosmologists estimate the posterior probability of cosmological parameters, such as the matter density Ωm and the normalization of the matter power spectrum σ8. Those parameters are typically estimated by likelihood-free inference, which requires a function to predict the parameters from simulations. As that is complicated to setup, prediction methods are typically benchmarked on the classification of spherical maps instead (Schmelzle et al., 2017). We used the same task, data, and setup as Perraudin et al. (2019): the classification of 720 partial convergence maps made of n ≈ 106 pixels (1/12 ≈ 8% of a sphere at Nside = 1024) from two ΛCDM cosmological models, (Ωm = 0.31, σ8 = 0.82) and (Ωm = 0.26, σ8 = 0.91), at a relative noise level of 3.5 (i.e., the signal is hidden in noise of 3.5 times higher standard deviation). Convergence maps represent the distribution of over- and under-densities of mass in the universe (see Bartelmann, 2010, for a review of gravitational lensing).
Graphs are built with (5), k = 8, 20, 40 neighbors, and the corresponding optimal kernel widths t given in section 3.2. Following Perraudin et al. (2019), the NN is made of 5 graph convolutional layers, each followed by a max pooling layer which down-samples by 4. A GAP and a fully connected layer with softmax follow. The polynomials are all of order P = 4 and the number of channels per layer is 16, 32, 64, 64, 64, respectively. The cross-entropy loss is optimized with Adam for 80 epochs. The learning rate is 2 · 10−4 · 0.999step and the batch size is 8. Unlike on SHREC’17, results (table 2) show that a lower equivariance error on the convolutions translates to higher performance. That is probably due to the high frequency content of those maps (figure 5). There is a clear cost-accuracy tradeoff, controlled by the number of neighbors k (figure 7). This experiment moreover demonstrates DeepSphere’s flexibility (using partial spherical maps) and scalability (competing spherical CNNs were tested on maps of at most 10, 000 pixels).

heading: 4.3 CLIMATE EVENT SEGMENTATION

text:
We evaluate our method on a task proposed by (Mudigonda et al., 2017): the segmentation of extreme climate events, Tropical Cyclones (TC) and Atmospheric Rivers (AR), in global climate simulations (figure 1c). The data was produced by a 20-year run of the Community Atmospheric Model v5 (CAM5) and consists of 16 channels such as temperature, wind, humidity, and pressure at multiple altitudes. We used the pre-processed dataset from (Jiang et al., 2019).11 There is 1,072,805 spherical maps, down-sampled to a level-5 icosahedral sampling (n = 10 · 4l + 2 = 10, 242 pixels). The labels are heavily unbalanced with 0.1% TC, 2.2% AR, and 97.7% background (BG) pixels.
The graph is built with (5), k = 6 neighbors, and a kernel width t set to the average of the distances. Following Jiang et al. (2019), the NN is an encoder-decoder with skip connections. Details in section C.3. The polynomials are all of order P = 3. The cross-entropy loss (weighted or nonweighted) is optimized with Adam for 30 epochs. The learning rate is 1 · 10−3 and the batch size is 64.
Results are shown in table 3 (details in tables 6, 7 and 8). The mean and standard deviation are computed over 5 runs. Note that while Jiang et al. (2019) and Cohen et al. (2019) use a weighted cross-entropy loss, that is a suboptimal proxy for the mAP metric. DeepSphere achieves state-of-
11Available at http://island.me.berkeley.edu/ugscnn/data.
the-art performance, suggesting again that anisotropic filters are unnecessary. Note that results from Mudigonda et al. (2017) cannot be directly compared as they don’t use the same input channels.
Compared to Cohen et al. (2019)’s conclusion, it is surprising that S2R does worse than DeepSphere (which is limited to S2S). Potential explanations are (i) that their icosahedral projection introduces harmful distortions, or (ii) that a larger architecture can compensate for the lack of generality. We indeed observed that more feature maps and depth led to higher performance (section C.3).

heading: 4.4 UNEVEN SAMPLING

text:
To demonstrate the flexibility of modeling the sampled sphere by a graph, we collected historical measurements from n ≈ 10, 000 weather stations scattered across the Earth.12 The spherical data is heavily non-uniformly sampled, with a much higher density of weather stations over North America than the Pacific (figure 1d). For illustration, we devised two artificial tasks. A dense regression: predict the temperature on a given day knowing the temperature on the previous 5 days. A global regression: predict the day (represented as one period of a sine over the year) from temperature or precipitations. Predicting from temperature is much easier as it has a clear yearly pattern.
The graph is built with (5), k = 5 neighbors, and a kernel width t set to the average of the distances. The equivariance property of the resulting graph has not been tested, and we don’t expect it to be good due to the heavily non-uniform sampling. The NN is made of 3 graph convolutional layers. The polynomials are all of order P = 0 or 4 and the number of channels per layer is 50, 100, 100, respectively. For the global regression, a GAP and a fully connected layer follow. For the dense regression, a graph convolutional layer follows instead. The MSE loss is optimized with RMSprop for 250 epochs. The learning rate is 1 · 10−3 and the batch size is 64. Results are shown in table 4. While using a polynomial order P = 0 is like modeling each time series independently with an MLP, orders P > 0 integrate neighborhood information. Results show that using the structure induced by the spherical geometry always yields better performance.

heading: 5 CONCLUSION

text:
This work showed that DeepSphere strikes an interesting, and we think currently optimal, balance between desiderata for a spherical CNN. A single parameter, the number of neighbors k a pixel is connected to in the graph, controls the tradeoff between cost and equivariance (which is linked to performance). As computational cost and memory consumption scales linearly with the number of pixels, DeepSphere scales to spherical maps made of millions of pixels, a required resolution
12https://www.ncdc.noaa.gov/ghcn-daily-description
to faithfully represent cosmological and climate data. Also relevant in scientific applications is the flexibility offered by a graph representation (for partial coverage, missing data, and non-uniform samplings). Finally, the implementation of the graph convolution is straightforward, and the ubiquity of graph neural networks — pushing for their first-class support in DL frameworks — will make implementations even easier and more efficient.
A potential drawback of graph Laplacian-based approaches is the isotropy of graph filters, reducing in principle the expressive power of the NN. Experiments from Cohen et al. (2019) and Boscaini et al. (2016) indeed suggest that more general convolutions achieve better performance. Our experiments on 3D shapes (section 4.1) and climate (section 4.3) however show that DeepSphere’s isotropic filters do not hinder performance. Possible explanations for this discrepancy are that NNs somehow compensate for the lack of anisotropic filters, or that some tasks can be solved with isotropic filters. The distortions induced by the icosahedral projection in (Cohen et al., 2019) or the leakage of curvature information in (Boscaini et al., 2016) might also alter performance.
Developing graph convolutions on irregular samplings that respect the geometry of the sphere is another research direction of importance. Practitioners currently interpolate their measurements (coming from arbitrarily positioned weather stations, satellites or telescopes) to regular samplings. This practice either results in a waste of resolution or computational and storage resources. Our ultimate goal is for practitioners to be able to work directly on their measurements, however distributed.

heading: ACKNOWLEDGMENTS

text: We thank Pierre Vandergheynst for advices, and Taco Cohen for his inputs on the intriguing results of our comparison with Cohen et al. (2019). We thank the anonymous reviewers for their constructive feedback. The following software packages were used for computation and plotting: PyGSP (Defferrard et al.), healpy (Zonca et al., 2019), matplotlib (Hunter, 2007), SciPy (Virtanen et al., 2020), NumPy (Walt et al., 2011), TensorFlow (Abadi et al., 2015).

heading: A PROOF OF THEOREM 3.1

text:
Preliminaries. The proof of theorem 3.1 is inspired from the work of Belkin & Niyogi (2008). As a result, we start by restating some of their results. Given a sampling V = {xi ∈M}ni=1 of a closed, compact and infinitely differentiable manifoldM, a smooth (∈ C∞(M)) function f :M→ R, and defined the vector f of samples of f as follows: TVf = f ∈ Rn, fi = f(xi). The proof is constructed by leveraging 3 different operators:
• The extended graph Laplacian operator, already presented in (6), is a linear operator Ltn : L2(M)→ L2(M) defined as
Ltnf(y) := 1
n n∑ i=1 e− ‖xi−y‖ 2 4t (f(y)− f(xi)) . (7)
Note that we have the following relation Ltnf = TVL t nf .
• The functional approximation to the Laplace-Beltrami operator is a linear operator Lt : L2(M)→ L2(M) defined as
Ltf(y) = ∫ M e− ‖x−y‖2 4t (f(y)− f(x)) dµ(x), (8)
where µ is the uniform probability measure on the manifoldM, and vol(M) is the volume ofM.
• The Laplace-Beltrami operator ∆M is defined as the divergence of the gradient ∆Mf(y) := −div(∇Mf) (9)
of a differentiable function f : M→ R. The gradient ∇f : M→ TpM is a vector field defined on the manifold pointing towards the direction of steepest ascent of f , where TpM is the affine space of all vectors tangent toM at p.
Leveraging these three operators, Belkin & Niyogi (2008; 2007) have build proofs of both pointwise and spectral convergence of the extended graph Laplacian towards the Laplace-Beltrami operator in the general setting of any compact, closed and infinitely differentiable manifold M, where the sampling V is drawn randomly on the manifold. For this reason, their results are all to be interpreted in a probabilistic sense. Their proofs consist in establishing that (6) converges in probability towards (8) as n→∞ and (8) converges towards (9) as t→ 0. In particular, this second step is given by the following: Proposition 1 (Belkin & Niyogi (2008), Proposition 4.4). Let M be a k-dimensional compact smooth manifold embedded in some Euclidean space RN , and fix y ∈M. Let f ∈ C∞(M). Then
1
t
1
(4πt)k/2 Ltf(y) t→0−−→ 1 vol(M) ∆Mf(y). (10)
Building the proof. As the sphere is a compact smooth manifold embedded in R3, we can reuse proposition 1. Thus, our strategy to prove Theorem 3.1 is to (i) show that
lim n→∞
Ltnf(y) = L t(y) (11)
for a particular class of deterministic samplings, and (ii) apply Proposition 1.
We start by proving that for smooth functions, for any fixed t, the extended graph Laplacian Ltn converges towards its continuous counterpart Lt as the sampling increases in size. Proposition 2. For an equal area sampling {xi ∈ S2}ni=1 : Ai = Aj∀i, j of the sphere it is true that for all f : S2 → R Lipschitz with respect to the Euclidean distance ‖·‖ with Lipschitz constant Cf ∣∣∣∣∣ ∫ S2 f(x)dµ(x)− 1 n ∑ i f(xi) ∣∣∣∣∣ ≤ Cfd(n).
Furthermore, for all y ∈ S2 the Heat Kernel Graph Laplacian operator Ltn converges pointwise to the functional approximation of the Laplace Beltrami operator Lt
Ltnf(y) n→∞−−−−→ Ltf(y).
Proof. Assuming f : S2 → R is Lipschitz with Lipschitz constant Cf , we have∣∣∣∣∫ σi f(x)dµ(x)− 1 n f(xi) ∣∣∣∣ ≤ Cfd(n) 1n, where σi ⊂ S2 is the subset of the sphere corresponding to the patch around xi. Remember that the sampling is equal area. Hence, using the triangular inequality and summing all the contributions of the n patches, we obtain∣∣∣∣∣ ∫ S2 f(x)dµ(x)− 1 n ∑ i f(xi) ∣∣∣∣∣ ≤∑ i ∣∣∣∣ 14π2 ∫ σi f(x)dµ(x)− 1 n f(xi)
∣∣∣∣ ≤ nCfd(n) 1n = Cfd(n) A direct application of this result leads to the following pointwise convergences
∀f Lipschitz, ∀y ∈ S2, 1 n ∑ i e− ‖xi−y‖ 2 4t → ∫ e− ‖x−y‖2 4t dµ(x)
∀f Lipschitz, ∀y ∈ S2, 1 n ∑ i e− ||xi−y|| 2 4t f(xi)→ ∫ e− ‖x−y‖2 4t f(x)dµ(x)
Definitions 6 and 8 end the proof.
The last proposition show that for a fixed t, Ltnf(x) → 1/4π2Ltf(x). To utilize Proposition 1 and complete the proof, we need to find a sequence of tn for which this holds as tn → 0. Furthermore this should hold with a faster decay than 14πt2n . Proposition 3. Given a sampling regular enough, i.e., for which we assume Ai = Aj ∀i, j and d(n) ≤ Cnα , α ∈ (0, 1/2], a Lipschitz function f and a point y ∈ S
2 there exists a sequence tn = nβ , β < 0 such that
∀f Lipschitz, ∀x ∈ S2 ∣∣∣∣ 14πt2n (Ltnn f(x)− Ltnf(x)) ∣∣∣∣ n→∞−−−−→ 0. Proof. To ease the notation, we define
Kt(x, y) := e− ‖x−y‖2 4t (12)
φt(x; y) := e− ‖x−y‖2 4t (f(y)− f(x)) . (13)
We start with the following inequality
‖Ltnf − Ltf‖∞ = max y∈S2 ∣∣Ltnf(y)− Ltf(y)∣∣ = max
y∈S2 ∣∣∣∣∣ 1n n∑ i=1 φt(xi; y)− ∫ S2 φt(x; y)dµ(x) ∣∣∣∣∣ ≤ max
y∈S2 n∑ i=1 ∣∣∣∣ 1nφt(xi; y)− ∫ σi φt(x; y)dµ(x) ∣∣∣∣ ≤ d(n) max
y∈S2 Cφty , (14)
whereCφty is the Lipschitz constant of x→ φ t(x, y) and the last inequality follows from Proposition 2. Using the assumption d(n) ≤ C√ n we find
‖Ltnf − Ltf‖∞≤ C√ n max y∈S2 Cφty
We now find the explicit dependence between t and Cφty
Cφty = ‖∂xφ t(·; y)‖∞ = ‖∂x ( Kt(·; y)f ) ‖∞
= ‖∂xKt(·; y)f +Kt(·; y)∂xf ||∞ ≤ ‖∂xKt(·; y)f‖∞+‖Kt(·; y)∂xf‖∞ ≤ ‖∂xKt(·; y)‖∞‖f‖∞+‖Kt(·; y)‖∞‖∂xf‖∞ = ‖∂xKt(·; y)‖∞‖f‖∞+‖∂xf‖∞ = CKty‖f‖∞+‖∂xf‖∞ = CKty‖f‖∞+Cf
where CKty is the Lipschitz constant of the function x→ K t(x; y). We note that this constant does not depend on y:
CKty = ∥∥∥∂xe− x24t ∥∥∥ ∞ = ∥∥∥ x 2t e− x2 4t ∥∥∥ ∞ = x 2t e− x2 4t ∣∣∣ x= √ 2t = (2et)− 1 2 ∝ t− 12 .
Hence we have C√ n max y∈S2 Cφty ≤ C√ n ( (2et)− 1 2 ‖f‖∞ + Cf ) ≤
C ‖f‖∞ nα(2et)1/2 + C nα Cf .
Inculding this result in (14) and rescaling by 1/4πt2, we obtain∥∥∥∥ 14πt2 (Ltnf − Ltf) ∥∥∥∥ ∞ ≤ 1 4πt2 ∥∥(Ltnf − Ltf)∥∥∞
≤ C 4π
[ ‖f‖∞√
2e
1
nαt5/2 + Cf nαt2
] .
In order for C4π [ ‖f‖∞√ 2e 1 nαt5/2 + Cf nαt2 ] n→∞−−−−→ t→0 0, we need { nαt5/2 →∞ nαt2 →∞
It happens if { t(n) = nβ , β ∈ (− 2α5 , 0) t(n) = nβ , β ∈ (−α2 , 0) =⇒ t(n) = nβ , β ∈ (− 2α5 , 0). Indeed, we have nalphat5/2 = n5/2β+α
n→∞−−−−→∞ since 52β + α > 0 ⇐⇒ β > − 2α 5
and nαt2 = n2β+α n→∞−−−−→∞ since 2β + α > 0 ⇐⇒ β > −α2 .
As a result, for t = nβ with β ∈ (− 15 , 0) we have
{ (tn)
n→∞−−−−→ 0∥∥∥ 14πt2nLtnn f − 14πt2nLtnf∥∥∥∞ n→∞−−−−→ 0, which concludes the proof.
Theorem 3.1, is then an immediate consequence of Proposition 3 and 1.
Proof of Theorem 3.1. Thanks to Proposition 3 and Proposition 1 we conclude that ∀y ∈ S2
lim n→∞
1
4πt2n Ltnn f(y) = lim n→∞
1
4πt2n Ltnf(y) =
1
|S2| ∆S2f(y)
In (Belkin & Niyogi, 2008), the sampling is drawn from a uniform random distribution on the sphere, and their proof heavily relies on the uniformity properties of the distribution from which the sampling is drawn. In our case the sampling is deterministic, and this is indeed a problem that we need to overcome by imposing the regularity conditions above.
To conclude, we see that the result obtained is of similar form than the result obtained in (Belkin & Niyogi, 2008). Given the kernel density t(n) = nβ , Belkin & Niyogi (2008) proved convergence in the random case for β ∈ (− 14 , 0) and we proved convergence in the deterministic case for β ∈ (− 2α5 , 0), where α ∈ (0, 1/2] (for the spherical manifold).

heading: B PROOF OF THEOREM 3.2

text:
Proof. Fix x ∈ S2. Since any rotation R(g) is an isometry, and the Laplacian ∆ commutes with all isometries of a Riemanniann manifold, and defining R(g)f =: f ′ for ease of notation, we can write that∣∣∣R(g)L̂tnn f(x)− L̂tnn R(g)f(x)∣∣∣ ≤ ∣∣∣R(g)L̂tnn f(x)−R(g)∆S2f(x)∣∣∣+ ∣∣∣R(g)∆S2f(x)− L̂tnn R(g)f(x)∣∣∣ =
= ∣∣∣R(g)(L̂tnn f −∆S2f)(x)∣∣∣+ ∣∣∣∆S2f ′(x)− L̂tnn f ′(x)∣∣∣ ≤
≤ ∣∣∣(L̂tnn f −∆S2f)(g−1(x))∣∣∣+ ∣∣∣∆S2f ′(x)− L̂tnn f ′(x)∣∣∣
Since g−1(x) ∈ S2 and f ′ still satisfies hypothesis, we can apply theorem 3.1 to say that∣∣∣(L̂tnn f −∆S2f)(g−1(x))∣∣∣ n→∞−−−−→ 0∣∣∣∆S2f ′(x)− L̂tnn f ′(x)∣∣∣ n→∞−−−−→ 0 to conclude that
∀x ∈ S2 ∣∣∣R(g)L̂tnn f(x)− L̂tnn R(g)f(x)∣∣∣ n→∞−−−−→ 0

heading: C EXPERIMENTAL DETAILS

text:
C.1 3D OBJECTS RECOGNITION
Table 5 shows the results obtained from the SHREC’17 competition’s official evaluation script.
(15) [GC16 + BN + ReLU ]nside32 + Pool + [GC32 + BN + ReLU ]nside16 + Pool
+ [GC64 + BN + ReLU ]nside8 + Pool + [GC128 + BN + ReLU ]nside4 + Pool + [GC256 + BN + ReLU ]nside2 + Pool + GAP + FCN + softmax
C.2 COSMOLOGICAL MODEL CLASSIFICATION
(16) [GC16 + BN + ReLU ]nside1024 + Pool + [GC32 + BN + ReLU ]nside512
+ Pool + [GC64 + BN + ReLU ]nside256 + Pool + [GC64 + BN + ReLU ]nside128 + Pool + [GC64 + BN + ReLU ]nside64 + Pool + [GC2]nside32 + GAP + softmax
C.3 CLIMATE EVENT SEGMENTATION
Table 6, 7, and 8 show the accuracy, mAP, and efficiency of all the NNs we ran.
The experiment with the model from Jiang et al. (2019) was rerun in order to obtain the AP metrics, but with a batch size of 64 instead of 256 due to GPU memory limit.
Several experiments were run with different architectures for DeepSphere (DS). Jiang architecture use a similar one as Jiang et al. (2019), with only the convolutional operators replaced. DeepSphere only is the original architecture giving the best results, deeper and with four times more feature maps than Jiang architecture. And the wider architecture is the same as the previous one with two times the number of feature maps.
Regarding the weighted loss, the weights are chosen with scikit-learn function compute class weight on the training set.
DeepSphere with Jiang architecture encoder:
(17) [GC8 + BN + ReLU ]L5 + Pool + [GC16 + BN + ReLU ]L4 + Pool
+ [GC32 + BN + ReLU ]L3 + Pool + [GC64 + BN + ReLU ]L2 + Pool
+ [GC128 + BN + ReLU ]L1 + Pool + [GC128 + BN + ReLU ]L0
decoder:
(18)
Unpool + [GC128 + BN + ReLU ]L1 + concat + [GC128 + BN + ReLU ]L1 + Unpool + [GC64 + BN + ReLU ]L2 + concat
+ [GC64 + BN + ReLU ]L2 + Unpool + [GC32 + BN + ReLU ]L3 + concat + [GC32 + BN + ReLU ]L3 + Unpool
+ [GC16 + BN + ReLU ]L4 + concat + [GC16 + BN + ReLU ]L4 + Unpool
+ [GC8 + BN + ReLU ]L5 + concat + [GC8 + BN + ReLU ]L5 + [GC3]L5
Concat is the operation that concatenate the results of the corresponding encoder layer.
Original DeepSphere architecture with encoder decoder encoder:
(19) [GC32 + BN + ReLU ]L5 + [GC64 + BN + ReLU ]L5
+ Pool + [GC128 + BN + ReLU ]L4 + Pool
+ [GC256 + BN + ReLU ]L3 + Pool + [GC512 + BN + ReLU ]L2 + Pool + [GC512 + BN + ReLU ]L1 + Pool + [GC512]L0
decoder:
(20)
Unpool + [GC512 + BN + ReLU ]L1 + concat + [GC512 + BN + ReLU ]L1 + Unpool + [GC256 + BN + ReLU ]L2 + concat
+ [GC256 + BN + ReLU ]L2 + Unpool + [GC128 + BN + ReLU ]L3 + concat + [GC128 + BN + ReLU ]L3 + Unpool
+ [GC64 + BN + ReLU ]L4 + concat + [GC64 + BN + ReLU ]L4 + Unpool + [GC32 + BN + ReLU ]L5 + [GC3]L5
C.4 UNEVEN SAMPLING
Architecture for dense regression:
(21)[GC50 + BN + ReLU ] + [GC100 + BN + ReLU ] + [GC100 + BN + ReLU ] + [GC1]
Architecture for global regression:
(22)[GC50 + BN + ReLU ] + [GC100 + BN + ReLU ] + [GC100 + BN + ReLU ] + GAP + FCN