summaries: This paper proposes a new trick to improve the stability of GANs. In particular the authors try to tackle the vanishing gradient problem in GANs, when the discriminator becomes to strong and is able to perfectly separate the distribution early in training, resulting in almost zero gradient for the generator. The authors propose to increase the difficulty of the task during training to avoid the discriminator to become too strong.

summaries: Authors argue that the main issue with stability in GANs is due to the discriminator becoming too powerful too quickly. To address this issue they propose to make the task progressively more difficult: Instead of providing only the samples to the discriminator, an additional (processed) bitstring is provided. The idea is that the bitstring in combination with the sample determines whether the sample should be considered true or fake. This in turn requires the decision boundary of the discriminator to become more complicated for increasing lengths of the bitstring. In a limited set of experiments the authors show that the proposed approach can improve the FID scores.

abstractText: Despite recent progress, Generative Adversarial Networks (GANs) still suffer from training instability, requiring careful consideration of architecture design choices and hyper-parameter tuning. The reason for this fragile training behaviour is partially due to the discriminator performing well very quickly; its loss converges to zero, providing no reliable backpropagation signal to the generator. In this work we introduce a new technique progressive augmentation of GANs (PAGAN) that helps to overcome this fundamental limitation and improve the overall stability of GAN training. The key idea is to gradually increase the task difficulty of the discriminator by progressively augmenting its input space, thus enabling continuous learning of the generator. We show that the proposed progressive augmentation preserves the original GAN objective, does not bias the optimality of the discriminator and encourages the healthy competition between the generator and discriminator, leading to a better-performing generator. We experimentally demonstrate the effectiveness of the proposed approach on multiple benchmarks (MNIST, Fashion-MNIST, CIFAR10, CELEBA) for the image generation task.

authors:
Michael Arbel
Dougal J. Sutherland
Mikołaj Bińkowski
Arthur Gretton

title: On gradient regularizers for MMD GANs

authors:
Martin Arjovsky
Léon Bottou

title: Towards principled methods for training generative adversarial networks

venue: In International Conference on Learning Representations (ICLR),

authors:
Martin Arjovsky
Soumith Chintala
Léon Bottou

title: Wasserstein generative adversarial networks

venue: In Advances in Neural Information Processing Systems (NIPS),

authors:
Mikołaj Bińkowski
Dougal J. Sutherland
Michael N. Arbel
Athur Gretton

title: Demystifying MMD GANs

title: Pros and cons of GAN evaluation measures

authors:
Xi Chen
Yan Duan
Rein Houthooft
John Schulman
Ilya Sutskever
Pieter Abbeel

title: InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets

authors:
Ishan P. Durugkar
Ian Gemp
Sridhar Mahadevan

title: Generative multi-adversarial networks

authors:
Gintare Karolina Dziugaite
Daniel M. Roy
Zoubin Ghahramani

title: Training generative neural networks via maximum mean discrepancy optimization

venue: In UAI,

authors:
William Fedus
Mihaela Rosca
Balaji Lakshminarayanan
Andrew M. Dai
Shakir Mohamed
Ian J. Goodfellow

title: Many paths to equilibrium: GANs do not need to decrease a divergence at every step

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron C Courville

title: Improved training of Wasserstein GANs

title: How (not) to train your generative model: Scheduled sampling, likelihood, adversary

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of GANs for improved quality, stability, and variation

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Naveen Kodali
James Hays
Jacob Abernethy
Zsolt Kira

title: On convergence and stability of GANs

title: Learning multiple layers of features from tiny images

venue: Technical report,

authors:
Karol Kurach
Mario Luc̆ić
Xiaohua Zhai
Marcin Michalski
Sylvain Gelly

title: The GAN landscape: Losses, architectures, regularization, and normalization

authors:
Yann LeCun
Corinna Cortes
Christopher J.C. Burges

title: The MNIST database of handwritten digits

authors:
Chun-Liang Li
Wei-Cheng Chang
Yu Cheng
Yiming Yang
Barnabás Póczos

title: MMD GAN: Towards deeper understanding of moment matching network

venue: In International Conference on Computer Vision (ICCV),

authors:
Mario Luc̆ić
Karol Kurach
Marcin Michalski
Sylvain Gelly
Olivier Bousquet

title: Are GANs created equal? A large-scale study

authors:
Xudong Mao
Qing Li
Haoran Xie
Raymond Y.K. Lau
Zhen Wang

title: Multi-class generative adversarial networks with the L2 loss function

authors:
Lars Mescheder
Andreas Geiger
Sebastian Nowozin

title: Which training methods for GANs do actually converge

venue: In International Conference on Machine learning (ICML),

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Sebastian Nowozin
Botond Cseke
Ryota Tomioka

title: f-GAN: Training generative neural samplers using variational divergence minimization

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

authors:
Kevin Roth
Aurelien Lucchi
Sebastian Nowozin
Thomas Hofmann

title: Stabilizing training of generative adversarial networks through regularization

authors:
Mehdi S.M. Sajjadi
Giambattista Parascandolo
Bernhard Schölkopf Arash Mehrjou

title: Tempered adversarial networks

venue: In International Conference on Machine Learning (ICML),

authors:
Tim Salimans
Ian Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training GANs

authors:
Casper Kaae Sønderby
Jose Caballero
Lucas Theis
Wenzhe Shi
Ferenc Huszár

title: Amortised map inference for image super-resolution

authors:
Lucas Theis
Aaron van den Oord
Matthias Bethge

title: A note on the evaluation of generative models

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

authors:
Han Zhang
Ian J. Goodfellow
Dimitris N. Metaxas
Augustus Odena

title: Self-attention generative adversarial networks

authors:
Hongyi Zhang
Moustapha Cisse
Yann N. Dauphin
David Lopez-Paz

title: mixup: Beyond empirical risk minimization

title: SN-DCGAN we employed the same architecture as in (Miyato et al., 2018), which we present in Table A1. For the InfoGAN architecture we followed Luc̆ić et al. (2018) and used the network structure of (Chen et al., 2016), which is described in Table A2. For both experiments with SN-DCGAN and InfoGAN we exploited the implementation provided in https://github.com/google/compare_gan

authors:
GAN For PA - NS
GAN PA - NS

title: across all datasets we start augmentation with level two, i.e. L = 2. The other adopted hyper-parameters for PA - NS GAN are listed in Table A9. It is important to note that they are selected considering the limit on the training epochs specified by Luc̆ić et al

authors: Originally Luc̆ić

heading: 1 INTRODUCTION

text:
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a recent development in the field of deep learning, that have attracted a lot of attention in the research community (Radford et al., 2016; Salimans et al., 2016; Arjovsky et al., 2017; Karras et al., 2018). GANs fall into the category of generative models, i.e. models that allow sampling of new data points from encoded high-dimensional data distributions, such as images. The GAN framework can be formulated as a competing game between the generator and the discriminator. Mathematically, training GANs requires solving a min-max problem. Since both the generator and the discriminator are typically parameterized as deep convolutional neural networks with millions of parameters, optimization is notoriously difficult in practice (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018).
The difficulty lies in maintaining the healthy competition between the generator and discriminator. A commonly occurring problem arises when the discriminator overshoots, leading to escalated gradients and oscillatory GAN behaviour (Mescheder et al., 2018). As a result the generator fails to learn the multimodal structure of the true distribution. Moreover, the supports of the data and model distributions typically lie on low dimensional manifolds and are often disjoint (Arjovsky & Bottou, 2017). Consequently, there exists a nearly trivial discriminator that can perfectly distinguish real data samples from synthetic ones. Once such a discriminator is produced, its loss quickly converges to zero and the gradients used for updating parameters of the generator become useless.
In this work we introduce a new technique to overcome this problem - progressive augmentation of GANs (PA-GAN) - that helps to control the behaviour of the discriminator and thus improve the overall training stability. The key idea is to progressively augment the input of the discriminator network with auxiliary random variables, enlarging the sample space dimensionality, in order to gradually increase the discrimination task difficulty (see Figure 1). In doing so, the discriminator can be prevented from becoming over-confident, enabling continuous learning of the generator. As opposed to standard data augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples, but rather is structurally appended to them. In particular, for the single level augmentation along with the data sample x the discriminator takes also as input the binary random variable s ∈ {0, 1}. The class of the augmented sample (x, s) is then set based on the combination x with s, resulting in real and synthetic samples contained in both true and fake classes. This presents a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the association rule. We can further increase the task difficulty of the discriminator by progressively augmenting its input space and enlarging the dimensionality of s during the course of training.
We show that the proposed PA-GAN preserves the original GAN objective and is an outcome of a systematic derivation. In contrast to prior work (Salimans et al., 2016; Sønderby et al., 2017; Arjovsky & Bottou, 2017), it does not bias the optimality of the discriminator (see Sec. 4). Structurally augmenting the input sample space and mapping it to higher dimensions not only challenges the discrimination task, but, in addition, encourages the generator to explore various paths towards the data distribution, leading to improved variation of the generated samples (see Sec. 5).
Our technique is orthogonal to existing work, it can be successfully employed with other regularizations strategies (Miyato et al., 2018; Gulrajani et al., 2017) and different network architectures (Chen et al., 2016; Radford et al., 2016), which we demonstrate in Sec. 5. We experimentally show the effectiveness of the progressive augmentation of GANs for image generation tasks on multiple benchmarks (MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky, 2009), CELEBA (Liu et al., 2015)) across different evaluation metrics (IS (Theis et al., 2016), FID (Huszár, 2015), KID (Bińkowski et al., 2018)).

heading: 2 RELATED WORK

text:
Many recent works have focused on improving the stability of GAN training and the overall visual quality of generated samples (Radford et al., 2016; Roth et al., 2017; Karras et al., 2018; Gulrajani et al., 2017; Miyato et al., 2018). As reported by Arjovsky & Bottou (2017), the reason for the unstable behaviour of GANs is partly due to a dimensional mismatch or non-overlapping support between the real data and the generative model distributions, resulting in an almost trivial task for the discriminator. Once the performance of the discriminator is maxed out, it provides a non-informative signal to train the generator. To avoid vanishing gradients, the original GAN paper (Goodfellow et al., 2014) proposed to modify the min-max based GAN objective (MM GAN) to a non-saturating loss (NS GAN). However, even with such a re-formulation the generator updates tend to get worse over the course of training and optimization becomes massively unstable (Arjovsky & Bottou, 2017).
Prior approaches tried to mitigate this issue by using heuristics to weaken the discriminator, such as decreasing its learning rate, adding label or input noise. Salimans et al. (2016) proposed a one-sided label smoothing technique to smoothen the classification boundary of the discriminator, thereby preventing it from being overly confident, but at the same time biasing its optimality. The works of Arjovsky & Bottou (2017) and Sønderby et al. (2017) made the job of the discriminator harder by adding Gaussian noise to both generated and real samples. Moving the manifolds of the data and model distributions closer to each other by adding the input noise ensures a meaningful overlap between their supports, which is desired in order for the generator to eventually approach the data distribution. However, adding high-dimensional noise introduces significant variance in the parameter estimation, which slows down the training and requires multiple samples for counteraction (Roth et al., 2017). Similarly, Sajjadi et al. (2018) proposed to blur the input samples and gradually remove the blurring effect during the course of training. Instead of adding noise to the input, Zhang et al. (2018b) created mixup samples by interpolating between synthetic and real ones, which leads to a more stable behaviour of GANs. These techniques, i.e., additive noise, blurring and sample mixup, perform direct modifications on the data samples.
Another line of work resorts to cost function reformulation to improve the stability of GAN training, e.g. by using the Pearson χ2 divergence for least square GANs (LS GANs) (Mao et al., 2016), kernel maximum mean discrepancy (MMD) for MMD-GANs (Li et al., 2017; Dziugaite et al., 2015), or f-divergence for f-GANs (Nowozin et al., 2016). Arjovsky et al. (2017) proposed the Wasserstein GAN (WGAN) with the training objective derived from the Wasserstein distance, aiming to
mitigate the vanishing gradient problem. The drawback of this approach is the weight clipping of the discriminator employed to enforce smoothness, which adversely reduces the capacity of the discriminator. Alternative to weight clipping, Gulrajani et al. (2017) added a soft penalty on the gradient norm which ensures a 1-Lipschitz discriminator. The gradient norm penalty can be seen as a weight regularization technique for the discriminator and was shown to improve the performance with other losses as well (Fedus et al., 2018). Similarly, Roth et al. (2017) proposed to add a penalty on the weighted gradient-norm of the discriminator in the context of f-divergences, showing its equivalence to adding input noise. On the downside, regularizing the discriminator with the gradient penalty depends on the model distribution which changes during training and results in increased runtime due to additional gradient norm computation. Miyato et al. (2018) proposed another way to stabilize the discriminator by normalizing its weights and limiting the spectral norm of each layer to constrain the Lipschitz constant. This normalization technique does not require intensive tuning of hyper-parameters and is computationally light. Most recently, Zhang et al. (2018a) showed that spectral normalization is also beneficial for the generator by preventing the escalation of parameter magnitudes and avoiding unusual gradients.
Several methods have proposed to modify the training methodology of GANs in order to further improve stability, e.g. by considering multiple discriminators with different roles (Durugkar et al., 2017) or growing both the generator and discriminator networks progressively (Karras et al., 2018).
In this work we introduce an orthogonal way to stabilize the GAN training by progressively increasing the discrimination task difficulty. To this end, a novel and structured way of augmenting the discriminator input space is proposed. In contrast to other techniques, our method does not bias the optimality of the discriminator or alter the training samples. Furthermore, the proposed augmentation is complementary to prior work. It can be employed with different GAN architectures, adapted to various divergence measures and combined with other regularization techniques (see Sec. 5).

heading: 3 THEORETICAL BACKGROUND

text: For generative modeling, one common approach is to adopt divergence measures as loss functions for the generator. Our method belongs to this line of work. In contrast to prior work, our primary focus is not on explicitly minimizing the divergence between the data and model distributions defined on the sample space X . Alternatively, we first structurally augment the training samples (both real and synthetic ones) and then minimize the divergence between distributions defined on the augmented sample space. For computing the divergence, we adopt the adversary process introduced by (Goodfellow et al., 2014) (Sec. 3.1), while the proposed augmentation is inspired by the information theory view of Jensen-Shannon (JS) divergence (Sec. 3.2). Both of them are briefly reviewed in this section to lay the theoretical groundwork for our method, which we then discuss in Sec. 4.

heading: 3.1 ADVERSARY PROCESS OF GANS

text:
Let X denote a compact metric space such as the image space [0, 1]d of dimension d. The data distribution Pd and the model distribution Pg are both probability measures defined on X . In the context of GANs, Pg is commonly induced by a function G that maps a random noise vector z, following a given prior distribution Pz, to a synthetic data sample, i.e. xg = G(z) ∈ X . The core idea behind GAN training is to set up a competing game between two players, commonly termed the discriminator D and generator G. Mathematically, their objective can be formulated as
min G max D L(D,G) ∆ = Ex ∼ Pd {log [D(x)]}+ Ex ∼ Pg {log [1−D(x)]} , (1)
with D : X 7→ [0, 1]. The optimal D∗ respectively classifies x ∼ Pd and x ∼ Pg as TRUE and FAKE, i.e. binary classification. Its achieved maximum equals the JS divergence between Pd and Pg, which is then used as the loss function by the generator to optimize G (Goodfellow et al., 2014).

heading: 3.2 INFORMATION THEORY VIEWPOINT

text:
Apart from quantifying distributions’ similarity, the JS divergence has an information theory interpretation that inspires our approach presented in Sec. 4. In accordance with the binary classification task of the discriminator, we introduce a binary random variable s with a uniform distribution Ps.
Associating s = 0 and s = 1 respectively with x ∼ Pd and x ∼ Pg, we obtain a joint distribution
P(x, s) = Ps(s)P(x|s) with P(x|s) ∆ = { Pd(x) if s = 0 Pg(x) if s = 1 . (2)
The marginal distribution with respect to x (a.k.a. the mixture distribution) is equal to
Pm ∆ = Pd + Pg 2 . (3)
Computing the mutual information of the two random variables s and x based on P(x, s), we have I(x; s) = H(x)−H(x|s) (a)= 0.5 ∫ pd(x) log pd(x)dPm(x) + 0.5 ∫ pg(x) log pg(x)dPm(x)
= DJS (Pd‖Pg) , (4) where the equality (a) is the outcome of computing the two entropy terms based on the reference measure Pm. The minimum of the JS divergence DJS (Pd‖Pg) equal to zero is attainable iff Pd = Pg. This condition makes the joint distribution function P(x, s) factorizable, indicating the independence between x and s, and thereby yielding zero mutual information.

heading: 4 PROGRESSIVELY AUGMENTED GAN TRAINING

text:
Relying on the information theory view of the JS divergence given in the previous section, we can cast the optimization objective of the generator as mutual information minimization
min G DJS (Pd‖Pg) ≡ min G I(x; s). (5)
Based on (5), in Sec. 4.1 we will first present an equivalent problem to (5), particularly showing how the auxiliary random bit s leads us to a novel and structured way to augment the sample x ∈ X for training the discriminator. By further identifying a common principle behind the equivalent problems, in Sec. 4.2 we extend the single level augmentation based on one bit s to progressive multi-level augmentation with an arbitrarily long random bit sequence. Progressively increasing the number of augmentation levels equips us with a new mechanism to balance the two-player competing game. In Sec. 4.3 we describe the integration of the proposed augmentation into neural networks and present how to schedule the augmentation progression during training.

heading: 4.1 SINGLE LEVEL AUGMENTATION

text:
Starting from the case of single level augmentation with one bit s, we first note that the following two minimization problems are equivalent
min G DJS (Pd‖Pg) ≡ min G DJS (P(x, s)‖Q(x, s)) (6)
where P(x, s) has been defined in (2) and Q(x, s) is constructed as
Q(x, s) = Ps(s)Q(x|s) with Q(x|s) ∆ = { Pd(x) if s = 1 Pg(x) if s = 0 . (7)
The two joint distribution functions P(x, s) and Q(x, s) differ from each other by the association of the bit s ∈ {0, 1} with the data and synthetic samples, respectively. Their marginals with respect to x are neither the data nor the generative model distribution, but by construction are identical and equal to Pm(x) as given in (3). It is worth noting that the equivalence holds even if the feasible solution set of Pg determined by G does not include the data distribution Pd. This is of practical interest as it is often difficult to guarantee the fulfillment of such premise when modelling G. For the detailed proof of the equivalence in 6 we refer the reader to App. A.1.
Next, we attempt to solve the equivalent problem given in (6), following the adversary learning process behind GANs. Specifically, the objective of the discriminator D converges to estimation of the JS divergence between P(x, s) and Q(x, s)
max D E (x, s) ∼ P {log [D(x, s)]}+ E(x, s) ∼ Q {log [1−D(x, s)]} . (8)
Comparing with the original discrimination task in GANs, i.e. (1), two differences are worth noting. First, the above discriminator takes s in addition to the sample x ∈ X as the input. We, therefore, view s as a single level of augmentation to the sample x. Second, the distributions that form the two classes (i.e. TRUE vs. FAKE) become P(x, s) and Q(x, s), instead of the original data and model distributions. Based on the definitions in (2) and (7), we identify (xd, s = 0) and (xg, s = 1) as belonging to the TRUE class, whereas (xd, s = 1) and (xg, s = 0) to the FAKE class. Thus, the real samples are no longer always in the TRUE class, and the synthetic samples are no longer always in the FAKE class. TRUE and FAKE now depend on the combination of x with s (see Figure 1).
Here, we introduce a simple trick to easily detect the class of a given pair. Namely, let the data and synthetic samples respectively convey one bit of information, with xd encoding bit zero and xg encoding bit one from now on. Then, the checksum of the pair (x, s) determines the respective class, i.e. checksum zero for TRUE and one for FAKE. 1 The checksum computation poses a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the checksum rule. Therefore, such augmentation is usable for preventing early maxing-out of the discriminator. More importantly, it does not compromise the core role of the discriminator in GAN training: informing the generator about the difference between the data and generative model distribution. This statement is confirmed by the equivalence at (6).

heading: 4.2 PROGRESSIVE MULTI-LEVEL AUGMENTATION

text:
We further extend the single level augmentation with one bit s to multi-level augmentation with an arbitrarily long random bit sequence s. Note that the two optimization problems on both sides of (6) rely on the JS divergence to quantify the difference of two distributions. Let us replace the data and generative distributions, i.e. Pd and Pg, respectively with P(x, s) and Q(x, s). Following the same line of argumentation, we can systematically add a new bit. Repeating this procedure L times will give us a bit sequence s with length L plus a series of equivalent problems with the same structure
min G DJS (Pd‖Pg) ≡ min G DJS (P(x, s1)‖Q(x, s1)) ≡ min G DJS (P(x, s1, s2)‖Q(x, s1, s2))
· · · ≡ min G DJS (P(x, s)‖Q(x, s)) . (9)
To estimate the JS divergence in (9), the augmented discriminator as defined in (8) now takes a bit sequence s instead of the single bit s, in addition to x. The combination of the sample x and s yields a multi-level augmentation. Following the analysis of the single bit case, it is not difficult to notice that the checksum mechanism remains. Namely, the discriminator needs to retrieve the one bit information carried by x and then perform a checksum together with the bit sequence s. The task difficulty increases as the length of s grows (see Figure 1). Therefore it makes sense to increase the augmentation level by adding more bits, whenever the discriminator becomes too powerful. More importantly, the consistency of the checksum mechanism across different augmentation levels permits progressive augmentation. The same discriminator can be trained from a lower augmentation level and gradually take more bits into consideration (see Figure 2).

heading: 4.3 IMPLEMENTATION

text:
Network architecture. In this work, we aim to maximally reuse existing neural network architectures tailored for GANs, such as DCGAN with spectral normalization (SN DCGAN) (Miyato et al., 2018) and InfoGAN (Chen et al., 2016). According to the above-introduced approach (and Figure 2), the generator architecture can remain unchanged, while the discriminator network requires an alteration to incorporate the augmentation s. To this end, we only modify the input layer of the discriminator network, yielding minimal changes.
First, the bit sequence s is preprocessed into a form compatible with x. Consider an image sample x with three coordinates, i.e. height, width and color channel (RGB). Each entry of s creates one extra augmentation channel, whereas the bit value is replicated to match the height and width of x.
1By checksum, we mean to conduct the XOR operation over a bit sequence.
It is worth noting that we let each bit take on values {0, 1} as input to the network. This choice of values is mainly due to the progressive augmentation during the course of training. When increasing the augmentation level, the additional bit 0 does not change the checksum and thus the output of D. On the contrary, the additional bit 1 flips the even(odd) checksum to odd(even). An effective change at the discriminator output is necessary to match the discrimination goal, thereby requiring a non-zero input in the first place. Using {0, 1} rather than other pairs of values, e.g. {−1, 1}, helps the discriminator to timely catch the change when progression takes place. Second, we keep the input layer in the network to process x and copy its configuration for processing the reformed s. Its kernel size, stride and padding type remain, but the input channel size is changed to L to process each entry of s. When a new augmentation level is reached, one extra input channel is instantiated to process the bit L+ 1. All the following layers of the discriminator remain unchanged.
Minibatch discrimination. The gradients for updating D are computed from the loss function given in (8), where the augmentation bit s is replaced by the bit sequence s of length L depending on the current augmentation level. The two expectations are approximated by using minibatches. Each minibatch is constructed with the same number of real data samples, synthetic samples and bit sequences. Each bit sequence is randomly sampled and associated with one real and one synthetic sample. By computing the checksums of the formed pairs, we can decide the correct class of each pair (x, s) in the minibatch and feed it into the discriminator to compute the cross-entropy loss. This way of generating (x, s) guarantees a balanced number of TRUE/FAKE samples.
Non-saturating loss (NS). When employing non-saturating loss for G in the experiments, we follow the reformulation introduced by (Goodfellow et al., 2014). Since the two expectation terms in (8) depend on G, this reformulation is applied for both of them, namely
min G −E (x, s) ∼ P {log [1−D(x, s)]} − E(x, s) ∼ Q {log [D(x, s)]} . (10)
Progression scheduling. Bińkowski et al. (2018) introduced the kernel inception distance (KID) to quantify the quality of the synthetic samples and proposed to reduce the learning rate by tracking the reduction of KID over iterations. Here we use KID to decide if the performance of G at the current augmentation level saturates or even starts degrading (typically happens when D starts overfitting or becomes too powerful). Specifically, after t discriminator iterations 2 , we compute the KID between synthetic samples and data samples drawn from the training set. If the current KID score is less than 5% of the average of the two previous ones attained at the same augmentation level, the augmentation is leveled up, i.e. L is increased by one.
Once reaching a new augmentation level, we introduce the following mechanisms to assist the discriminator in quickly picking up the change in the input space. First, the new augmentation bit is generated from a non-uniform distribution, i.e. P(s = 1) = p and P(s = 0) = 1− p with p < 0.5. As mentioned before, critical changes on the discriminator side are required for bit 1. For it to gradually comprehend the new bit, we on purpose create more 0s than 1s and gradually increase p up to 0.5 (the uniform case) after a certain number of iterations. A simple linear model is adopted
p = min{0.5 ∗ (t− tst)/tr, 0.5} (11) where t and tst are the current iteration index and the iteration index when the current augmentation level is started, respectively; and tr controls the rate of increase. It is important to note that p 6= 0.5 does not cause unbalanced TRUE and FAKE classes in the constructed minibatches. It only introduces some bias in the generation of the new augmentation bit.
Finally, it is advisable to slow down the learning rate ofGwhen a new augmentation level is reached. WhenD is not properly adjusted to the new level, its feedback toG can be noisy. For instance, when using the Adam optimizer (Kingma & Ba, 2015), we reset the time step recorded by theG optimizer.

heading: 5 EXPERIMENTS

text:
Datasets. In our experiments we consider MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky, 2009) and CELEBA (Liu et al., 2015) datasets, with the training set sizes equal to 60k, 60k, 50k and 162k respectively.
2Each update of D parameters counts as one discriminator iteration. We assume that the update frequency for D can be as or more frequent than that of G, but not the opposite.
Network architectures. We employ two well established deep convolutional GAN architectures, SN-DCGAN (Miyato et al., 2018) and InfoGAN (Chen et al., 2016) (see Appendix A.2 for detailed configurations). As they respectively employ spectral normalization (SN) and batch normalization (BN), our aim is to explore the compatibility of PA-GAN with these normalization techniques.
Evaluation metrics. We use Fréchet inception distance (FID) (Huszár, 2015) as the primary performance evaluation metric.3 Additionally, we also report inception score (IS) (Theis et al., 2016) and kernel inception distance (KID) (Bińkowski et al., 2018) in Appendix A.4. For quantifying the quality of synthetic samples, all measures are computed based on 10k test data and 10k synthetic samples, following the evaluation framework of Luc̆ić et al. (2018) and Kurach et al. (2018).4
Training details. We use the minibatch size of 64 and the Adam optimizer (Kingma & Ba, 2015) with the default setting: β1 = 0.5, β2 = 0.999 and learning rate 2×10−4 for both the generator and discriminator, which have an equal update rate. The dimension of z is set to 64 and 128 respectively for InfoGAN and SN-DCGAN. The prior distribution Pz is uniform. For scheduling progressive augmentation KID is evaluated every 104 discriminator iterations, using 10k generated samples and 10k samples randomly drawn from the training set, and tr in (11) is set to 5× 103.

heading: 5.1 CIFAR10 WITH SN-DCGAN

text:
In this experiment, we evaluate the progressive augmentation (PA) with NS GAN (GAN with the non-saturating loss) using the SN-DCGAN architecture on CIFAR10. We analyze the benefits of applying PA on top of NS GAN, experiment with starting PA from different augmentation levels and investigate the complementarity of using both PA and the gradient penalty regularization (GP) (Gulrajani et al., 2017). For fair comparison, we follow the experimental setup of (Kurach et al., 2018).
NS GAN with PA. Figure 3 and Table 1 compare NS GAN results with and without applying PA. We are able to closely reproduce the NS GAN results reported in (Kurach et al., 2018, Table 6), after 200k iterations we obtain the median FID value of 26.3 vs. original 26.7. By applying PA on top of NS GAN and starting from the augmentation level 0 (PAL=0 - NS GAN), we achieve superior performance, with the median FID of 25.7 vs. 26.3 of NS GAN. Training for extra 100k iterations boosts the performance of PAL=0 - NS GAN (25.7 vs. 24.6). Starting PA from the level 2 (PAL=2 - NS GAN) further improves the median FID (23.8 vs. 25.7); as CIFAR10 contains diverse images a start from a higher augmentation level is recommended. It is worth noting that at early iterations (up to 50k) PAL=2 - NS GAN has worse performance than NS GAN. Starting at the augmentation level 2 imposes a more challenging task for the discriminator, thereby showing slower improvement at initial iterations but being beneficial in the longer term. The FID value of PAL=2 - NS GAN saturates at a slower pace than NS GAN leading to better overall results.
NS GAN with PA and GP. For GP the interpolates are created analogously to (Gulrajani et al., 2017), i.e. [x̃, s̃] = α[x, s]TRUE + (1 − α)[x, s]FAKE with α ∼ U(0, 1). Note that interpolation takes place in the augmented space [x, s], yielding s̃ ∈ [0, 1]L. In (Kurach et al., 2018), when GP is applied on top of NS GAN a marginal improvement is observed (26.2 vs. 26.7). However, employing PA results in a more noticeable gain for GP (22.5 vs. 23.8). Furthermore, GP helps to accelerate the learning speed of PAL=2 - NS GAN at the initial iterations.
3How to precisely evaluate the performance of GANs is an open question in itself. From the comparison conducted in (Borji, 2018), FID is considered to be the most informative of the measures.
4https://github.com/google/compare_gan
As indicated in Figure 1, in the augmented space (e.g., L ∈ {1, 2}), more paths are created for the generative model to approach the data distribution. GP can help to smoothen the decision boundaries along these paths. As a result, GP and PA jointly improve the performance of NS GAN, being mutually beneficial (22.5 vs. 26.3). In A.3, we additionally report results on CELEBA-HQ (Karras et al., 2018). The enlarged performance gain (18.1 vs. 28.5) reveals a great potential of employing PA for high resolution datasets.

heading: 5.2 COMPARISONS AMONG DATASETS AND GAN-TYPE ALGORITHMS

text:
Luc̆ić et al. (2018) compared various GAN-type algorithms under the InfoGAN architecture and reported their FID scores after a wide range of hyper-parameter searching. Following their experimental setup, we select hyper-parameters within the candidate set considered by Luc̆ić et al. (2018) (see Appendix A.5 for details) and evaluate PA with NS GAN as well as Wasserstein GAN with GP (WGAN - GP). The maximal number of augmentation levels achieved by PA corresponds to seven. The results are reported in Table 2. Overall, we observe significant gains of employing PA with NS GAN and WGAN - GP across different datasets. Note that the numbers provided by Luc̆ić et al. (2018) are outcomes of removing outliers up to 20% among the fifty independent runs. Since the outlier removing ratio is not specified for individual cases in (Luc̆ić et al., 2018), in Table 2 we report the PA results without removing outliers, see A.7 for further discussion.
As shown in Table 2, the NS GAN performance is quite stable across the datasets. By applying PA with NS GAN, we achieve a pronounced improvement, particularly when the dataset (and hence the image generation task) becomes more complicated. Note that the PA - NS GAN performance on MNIST is worse than NS GAN. This is mainly because NS GAN already performs very well on such a simple dataset and PA requires additional iterations to reach similar results or further improve them as in PA - NS GAN (*). In contrast to NS GAN, WGAN - GP is sensitive to the dataset as shown in Table 2 and reported by Mescheder et al. (2018). However, applying PA helps to stabilize the WGAN - GP performance across different benchmarks. Similarly to NS GAN, longer training leads to better results for PA with WGAN - GP. The results in Table 2 indicate that PA generalizes well across different distance measures and is not limited to the JS divergence.
Besides the experiments in Sec. 5.1 and 5.2, we provide a careful investigation of PA itself in Appendix. Ablation studies on the effect of the progression scheduling and the linear model 11 are given in A.6, followed by the analysis of PA training stability in A.7. The increase of the task difficulty of the discriminator with PA is examined in A.8. In addition to GP, we also compare PA with the dropout regularization in A.9.

heading: 6 CONCLUSION

text:
In this work we have proposed a novel method - progressive augmentation (PA) - to improve the stability of GAN training, and showed a way to integrate it into existing GAN architectures with minimal changes. Different to standard data augmentation our approach does not modify the training samples, instead it progressively increases the dimension of the discriminator input space by
augmenting it with auxiliary random variables. Higher sample space dimensionality helps to entangle the discriminator and thus to avoid its early performance saturation. Moreover, in the augmented space the generator can explore more paths to approach the data distribution, improving variation of the generated samples. We experimentally have shown pronounced performance improvements of employing the proposed PA with state-of-the-art GAN methods across multiple benchmarks. We demonstrated that PA generalizes well across different network architectures and loss functions and is complementary to other regularization techniques. For future work, we find a joint optimization of PA with neural architectures an interesting direction, for instance, combining it with progressive growing of GANs (Karras et al., 2018). Apart from generative modeling, our approach can also be exploited for semi-supervised learning, generative latent modeling and transfer learning.

heading: A APPENDIX

heading: A.1 PROOF OF THE EQUIVALENCE IN (6)

text:
In Sec. 3.2, we have provided the connection of the JS divergence to the mutual information between the auxiliary random bit s and the sample x following the joint distribution P(x, s) as given in (2)
DJS (Pd‖Pg) = I(x; s). (12) Interchanging the position of Pd and Pg in constructing P(x, s), we obtain the joint distribution Q(x; s) as given in (7). With respect to Q(x; s), we further compute and denote the mutual information between x and s as Ĩ(x; s). Analogous to 4 in Sec. 3.2, we can show
DJS (Pd‖Pg) = Ĩ(x; s). (13)
The equality (12) and (13) jointly yield
DJS (Pd‖Pg) = 1
2 I(x; s) +
1 2 Ĩ(x; s) (14)
followed by rewriting mutual information as KL divergence
DJS (Pd‖Pg) = 1
2 DKL (P(x, s)‖Pm(x)Ps(s)) +
1 2 DKL (Q(x, s)‖Pm(x)Ps(s)) . (15)
It is noted that the marginals of P(x, s) and Q(x, s) with respect to x are identical and equal to Pm(x) as given in (3), whereas Ps(s) is their marginal with respect to s. With the identification of
Pm(x)Ps(s) = P(x, s) +Q(x, s)
2 (16)
we reach to
DJS (Pd‖Pg) = 1
2 DKL (P(x, s)‖0.5P(x, s) + 0.5Q(x, s))
+ 1
2 DKL (Q(x, s)‖0.5P(x, s) + 0.5Q(x, s)) . (17)
The right-hand side term of the above equality is simply the JS divergence between P(x, s) and Q(x, s). Since the two JS divergences are completely identical, we can use them interchangeably as the objective function while optimizing Pg. With that we conclude the equivalence proof for the two optimization problems in (6).

heading: A.2 NEURAL NETWORK ARCHITECTURES

text:
Following Kurach et al. (2018) for SN-DCGAN we employed the same architecture as in (Miyato et al., 2018), which we present in Table A1. For the InfoGAN architecture we followed Luc̆ić et al. (2018) and used the network structure of (Chen et al., 2016), which is described in Table A2. For both experiments with SN-DCGAN and InfoGAN we exploited the implementation provided in https://github.com/google/compare_gan.
Table A1: SN-DCGAN architecture. (a) Discriminator
Layer Kernel Output shape
Conv, lReLU [3, 3, 1] h× w × 64 Conv, lReLU [4, 4, 2] h2 × w 2 × 128 Conv, lReLU [3, 3, 1] h2 × w 2 × 128 Conv, lReLU [4, 4, 2] h4 × w 4 × 256 Conv, lReLU [3, 3, 1] h4 × w 4 × 256 Conv, lReLU [4, 4, 2] h8 × w 8 × 512 Conv, lReLU [3, 3, 1] h8 × w 8 × 512
Linear − 1
(b) Generator
Layer Kernel Output shape
z − 128 Linear, BN, ReLU − h8 × w 8 × 512 DeConv, BN, ReLU [4, 4, 2] h4 × w 4 × 256 DeConv, BN, ReLU [4, 4, 2] h2 × w 2 × 128
DeConv, BN, ReLU [4, 4, 2] h× w × 64 DeConv, Tanh [3, 3, 1] h× w × 3
Table A2: InfoGAN architecture. (a) Discriminator
Configuration per Layer
4× 4 conv. 64 lReLU, stride 2 4× 4 conv. 128 lReLU, stride 2, BN
Linear, 1024 lReLU, BN
Linear, 1 output
(b) Generator
Configuration per Layer
Linear, 1024 ReLU, BN
Linear, 7× 7× 128 ReLU, BN 4× 4 DeConv, 64 ReLu, stride 2, BN
4× 4 Deconv, 1 or 3 channels
0.5 1 1.5 2 2.5 3 ·105
20
40
60
Iterations
FI D
va lu
e
NS GAN NS GAN - GP PAL=2 - NS GAN PAL=2 - NS GAN - GP
Figure A1: Mean FID values attained over iterations across five independent runs with different NS GAN variants on CELEBA-HQ (128 × 128), using the SN-DCGAN architecture. The maximal number of achieved augmentation levels is 10.
200k 300k Median Best Median Best
NS GAN (Kurach et al., 2018) 31.1 29.1 − − NS GAN (ours) 27.4 26.4 27.4 26.2 NS GAN - GP (ours) 27.5 26.8 27.2 26.8 PAL=2 - NS GAN 23.9 22.2 23.1 21.8 PAL=2 - NS GAN - GP 19.7 19.4 18.8 18.1
Table A3: Median and best FID values attained with different NS GAN variants on CELEBA-HQ (128 × 128), using the SN-DCGAN architecture. Applying PA and GP on top of NS GAN reduces FID by ∼ 32%.

heading: A.3 CELEBA-HQ WITH SN-DCGAN

text: Figure A1 and Table A3 report additional results (FID scores) on CELEBA-HQ (128×128) (Karras et al., 2018) using the SN-DCGAN network architecture, following the same experimental setup as in Sec. 5.1. The results on CELEBA-HQ (128 × 128) are consistent with our observations in Sec. 5.1. By applying PA on top of NS GAN - GP and starting from the augmentation level 2 (PAL=2 - NS GAN - GP), we achieve superior performance, with the median FID of 18.8 vs. 27.3 of NS GAN - GP. Without PA, NS GAN and NS GAN - GP have almost no FID reduction (or very minor) in the last 100k iterations, whereas PA enables further improvement. The achieved best FID value by PAL=2 - NS GAN - GP is even 36% smaller than the best FID score achieved by NS GAN with ResNet19 reported by (Kurach et al., 2018) (18.1 vs. 28.5).

heading: A.4 CIFAR10 WITH SN-DCGAN

text:
Figure A2 and A3 together with Table A4 and Table A5 report the inception scores and KID values that are attained following the same experimental setup as in Sec. 5.1 on CIFAR10. The results on both measures are consistent with our observation drawn from the FID values. In the recent work (Arbel et al., 2018), the reported best KID value for CIFAR10 with SN-DCGAN is 0.015, whereas PAL=2 - NS GAN - GP reduces it to 0.013.
0.5 1 1.5 2 2.5 3 ·105
5
6
7
8
Iterations
In ce
pt io
n sc
or e
NS GAN PAL=0 - NS GAN PAL=2 - NS GAN PAL=2 - NS GAN - GP
Figure A2: Inception scores (IS) attained over iterations with different NS GAN variants averaged across five independent runs on CIFAR10.
200k 300k
NS GAN (ours) 7.54 7.58
PAL=0 - NS GAN 7.63 7.64
PAL=2 - NS GAN 7.64 7.72
PAL=2 - NS GAN - GP 7.59 7.76
Table A4: Median inception scores (IS) attained with different NS GAN variants on CIFAR10.
0.5 1 1.5 2 2.5 3 ·105
10−2
10−1.5
Iterations
K ID
va lu
e
NS GAN PAL=0 - NS GAN PAL=2 - NS GAN PAL=2 - NS GAN - GP
Figure A3: KID values attained over iterations with different NS GAN variants averaged across five independent runs on CIFAR10.
200k 300k
NS GAN (ours) 0.0168 0.0164
PAL=0 - NS GAN 0.0165 0.0153
PAL=2 - NS GAN 0.0149 0.0139
PAL=2 - NS GAN - GP 0.0140 0.0133
Table A5: Median KID values attained with different NS GAN variants on CIFAR10.
Next we present some further remarks on the implementation side. First, to explicitly investigate the combination of GP and PA, we add the GP as an extra regularizer to the discriminator loss function only when the augmentation takes place, and GP is computed with respect to the augmented sample after interpolation, i.e. [x̃, s̃]. Figures A4 - A6 together with Tables A6 - A8 report results evaluated with the FID, IS and KID metrics. With the use of GP, the starting level of the augmentation has negligible influence on the performance after a sufficient number of iterations.
Second, the chosen hyper-parameters are not optimized for PA. In fact, they are dedicatedly chosen by (Kurach et al., 2018) for NS GAN, i.e. β1 = 0.5, β2 = 0.999, λ = 1 (weighting factor for GP) and the learning rate 2 × 10−4. We adopt them for PA for the purpose of fair comparison. Therefore, further optimization on the hyper-parameters for PA may potentially yield better results than the reported ones.
Third, in our implementation all of progressively added augmentation channels take on the values {0, 1} to ease the progression, as described in Sec. 4.3. For those augmentation channels that are present from the start of training and if the pixel values of the image are normalized to [−1, 1], we accordingly experimented with the values {±1}. As a result, in the current experiment (i.e., CIFAR10 with SN-DCGAN), we used {±1} and switch to {0, 1} for subsequent InfoGAN experiment. In general, it is not a critical choice to the performance from our observation.
At last, a set of synthetic images generated by PAL=2 - NS GAN - GP with the FID value of 22.5 on CIFAR10 is shown in Figure A7.

heading: A.5 COMPARISONS AMONG DATASETS AND GAN-TYPE ALGORITHMS

text:
In Sec. 5.2, we apply PA with NS GAN and WGAN - GP and report the achieved FID values in Table 2.
For PA - NS GAN and PA - NS GAN (*) across all datasets we start augmentation with level two, i.e. L = 2. The other adopted hyper-parameters for PA - NS GAN are listed in Table A9. It is important to note that they are selected considering the limit on the training epochs specified by Luc̆ić et al. (2018). Allowing more iterations, the performance can be further improved. For instance,
0.5 1 1.5 2 2.5 3 ·105
20
30
40
Iterations
FI D
va lu
e
PAL=0 - NS GAN - GP PAL=1 - NS GAN - GP PAL=2 - NS GAN - GP
Figure A4: FID attained over iterations with different progression starting levels averaged across five independent runs on CIFAR10.
200k 300k
NS GAN (ours) 26.3 25.7
PAL=0 - NS GAN - GP 23.2 22.8
PAL=1 - NS GAN - GP 23.9 23.0
PAL=2 - NS GAN - GP 23.2 22.5
Table A6: FID values attained with different progression starting levels on CIFAR10.
0.5 1 1.5 2 2.5 3 ·105
5
6
7
8
Iterations
In ce
pt io
n sc
or e
PAL=0 - NS GAN - GP PAL=1 - NS GAN - GP PAL=2 - NS GAN - GP
Figure A5: IS attained over iterations with different progression starting levels averaged across five independent runs on CIFAR10.
200k 300k
NS GAN (ours) 7.54 7.58
PAL=0 - NS GAN - GP 7.63 7.67
PAL=1 - NS GAN - GP 7.64 7.77
PAL=2 - NS GAN - GP 7.59 7.76
Table A7: Median IS attained with different progression starting levels on CIFAR10.
0.5 1 1.5 2 2.5 3 ·105
10−2
10−1.5
Iterations
K ID
va lu
e
PAL=0 - NS GAN - GP PAL=1 - NS GAN - GP PAL=2 - NS GAN - GP
Figure A6: KID values attained over iterations with different progression starting levels averaged across five independent runs on CIFAR10.
200k 300k
NS GAN (ours) 0.0168 0.0164
PAL=0 - NS GAN - GP 0.0141 0.0141
PAL=1 - NS GAN - GP 0.0144 0.0141
PAL=2 - NS GAN - GP 0.0140 0.0133
Table A8: Median KID values attained with different progression starting levels on CIFAR10.
PA - NS GAN (*) in Table 2 shows the achieved performance if we run 40k iterations for MNIST and Fashion-MNIST, and run 140k iterations for CIFAR10 and CELEBA. The gains can be further enlarged if the hyper-parameters are adjusted towards the longer training time. For instance, by reducing the learning rate from 10−3 to 2× 10−4 and performing 140k iterations for MNIST, we can achieve the FID of 4.5± 0.35. Proceeding to the case PA - WGAN - GP, the hyper-parameters are listed in Table A10, the batch normalization for the discriminator is disabled. For both MNIST and Fashion-MNIST, the augmentation level starts from one to ensure that it is in place within the 20 epoch training time. For CELEBA, we start from the augmentation level zero and will reach the level one augmentation within the 40 training epochs. Analogous to the previous case, better performance is achievable by PA with the number of iterations increased beyond the original limit, i.e. 40k for MNIST and Fashion-MNIST and 140k for CELEBA.
Figure A7: Synthetic images generated by PAL=2 - NS GAN - GP with the FID 22.5 on CIFAR10.
Finally, we note that to stabilize progression in WGAN - GP the weighting factor λ for GP requires a careful adjustment. In this experiment, we relax λ following the idea behind (11), namely by linearly increasing λ from zero to its original value within tr iterations (in this case tr = 5× 103). Interestingly, we did not find such adaptation necessary when using GP with NS GAN. One possible reason is that GP is critical to WGAN due to the Lipschitz constraint, but optional to NS GAN. Augmentation changes the input space of the discriminator, and the number of terms involved in the GP also increases with the number of augmentation channels. Further investigation on the weighting factor adjustment to fully exploit the benefit of combining PA with WGAN - GP is a part of our future work. The results reported in this work have confirmed that PA is not limited to the JS divergence.
MNIST Fashion-MNIST CIFAR10 CELEBA
β1 0.5 0 0.5 0.5
β2 0.999 0.999 0.999 0.999
Learning rate 10−3 4× 10−4 4× 10−4 2× 10−4
Table A9: Hyper-parameters for generating our numbers associated to PA - NS GAN in Table 2.
MNIST Fashion-MNIST CIFAR10 CELEBA
β1 0 0 - 0
β2 0.999 0.999 - 0.999
λ 0.1 0.1 - 0.1
Learning rate 10−3 10−3 - 4× 10−4
Table A10: Hyper-parameters for generating our numbers associated to PA - WGAN - GP in Table 2.

heading: A.6 ABLATION STUDIES

text:
Effect of the Progression Scheduling. Here we experiment with InfoGAN on CELEBA, as one of the most challenging cases considered in Table 2, to study the effect of the progression scheduling described in Sec. 4.2 and 4.3. Originally, Luc̆ić et al. (2018) suggested 40 epoches (approximately 100k iterations) for the NS GAN training. As we can observe from Figure A8, further training can only degrade the performance. One important reason for such behaviour is that the discriminator architecture design of InfoGAN (see Table A2) might be suboptimal for CELEBA (e.g., a fully
0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 ·105
40
60
80
100
120
Iterations FI
D va
lu e
NS GAN 6 PAL=2 - NS GAN (no progression) PAL=2 - NS GAN
Figure A8: FID values with InfoGAN on CELEBA, the results are attained after 40 epochs (about 100k iterations). We remove outliers that are outside 4 standard deviation. In all cases, the number of outliers is less than 5% among 50 independent runs. The maximal number of achieved PA levels is 7.
0 1 2 3 4 5 6 7 8 940
50
60
Number of outliers
FI D
va lu
e
NS GAN 6 PAL=2 - NS GAN (no progression) PAL=2 - NS GAN
0 1 2 3 4 5 6 7 8 9 2
3
4
5
Number of outliers
St an
da rd
de vi
at io
n
Figure A9: FID mean and standard deviation values across 50 independent runs with InfoGAN on CELEBA, dependent on the number of removed outliers.
connected rather than a convolutional layer at the end plus batch normalization conducting minibatch discrimination).5 By applying the proposed augmentation strategy, even with one augmentation level and no progression, we can achieve much better FID values at much earlier iterations. Moreover, with the progression scheduling of augmentation (from level 2 to 7) we can dramatically slow down the degradation process for this challenging case. We additionally observe that the standard deviation of FID values across 50 runs is much smaller with PA than without it and the gain of the PA remains consistent, independent of the number of outliers removed, see Figure A9.6 This showcases the positive effect of the proposed PA on the training stability of GANs. Nevertheless, it remains difficult to fully counteract the degradation process caused by the suboptimal network architecture design for the task at hand.
Role of the Linear Model (11) in PA. Here we analyse the role of the linear model 11 in progression scheduling described in Sec. 4.3. Figure A10 depicts D Loss and G Loss over training iterations on CIFAR10 using SN-DCGAN.
Performing PA with a newly augmented bit may lead to the confusion of the discriminator, as EVEN inputs can immediately become ODD ones and vice versa. Empirically, we observe that the loss of the discriminator (i.e., D Loss) may stuck at values close to 1.38 for a considerable number of iterations, because the discriminator is confused about the abrupt change. This is particularly true for regularized networks. Whenever such situation takes place, it is harmful for the generator as well. A confused discriminator can no longer guide it towards the data distribution.
Figure A10 depicts the above mentioned confusions after a new augmentation level is introduced. In this case, there are two plateaus of D Loss for brown curves which do not employ the linear model 11 in PA, occurring between 70k and 120k iterations. Employing the linear model 11 in PA helps to avoid such long lasting plateaus. The blue curves (with 11 in PA) exhibit smooth transitions when
5In fact, this architecture was initially selected by Chen et al. (2016) for MNIST (28× 28). In the released code of (Luc̆ić et al., 2018), such InfoGAN architecture was adopted not only for MNIST and Fashion-MNIST, but also CIFAR10 and CELEBA with higher resolutions.
6We follow (Luc̆ić et al., 2018), which excludes outliers while reporting the FID values.
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 ·105
1.1
1.2
1.3
1.4
1.5
Iterations
D Lo
ss
PAL=2 - NS GAN - GP PAL=2 - NS GAN - GP w. (11)
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 ·105
1.4
1.6
1.8
2
Iterations
G Lo
ss
PAL=2 - NS GAN - GP PAL=2 - NS GAN - GP w. (11)
Figure A10: The behaviour of D Loss and G Loss over iterations on CIFAR10 using SN-DCGAN, with and without the linear model 11 in progression scheduling.
the augmentation level increases. Overall, the linear model 11 in PA helps to speed up the learning process. The main motivation of having it is to take precautions against potential ill adaptation to new augmentation levels.

heading: A.7 EFFECT OF PA ON THE TRAINING STABILITY

text:
Figure A11 compares D Loss and G Loss of NS GAN and NS GAN - GP with and without PA during the training on CIFAR10 using the SN-DCGAN architecture. Without PA the training becomes unstable over time independent of using GP (red curves), while employing PA helps to maintain a healthy competition between the discriminator and the generator. Observing the behaviour of D Loss and G Loss, we conclude that PA can effectively prevent the discriminator from overfitting, alleviating the vanishing gradient issue and thus enabling continuous learning of the generator.
In Table A11 we present mean and standard deviation values of FID across 50 independent runs on four datasets. We follow the work of (Luc̆ić et al., 2018) reporting the numbers while excluding significant outlier runs. Excluding outlier runs mostly influences the standard deviation of the FID values. Table A11 shows that in the same setting as (Luc̆ić et al., 2018) PA not only improves the FID values, but also reduces their standard deviations across multiple runs. This highlights that the training becomes more stable with PA. The improvement of using PA is consistent across four datasets, although the value of the mean FID heavily fluctuates dependent on the dataset.

heading: A.8 ANALYSIS OF THE TASK DIFFICULTY OF THE DISCRIMINATOR WITH PA

text:
With PA we have casted the discrimination task into a checksum operation, involving two steps. The data and synthetic samples are combined with the augmentation bit sequence, resulting in data and
synthetic samples contained in both true and fake classes. So, first the discriminator needs to recover the original sample and the augmentation bit sequence, and to decide if the original sample is the data or synthetic sample (this task is essentially identical to the original discrimination task). Second, the discriminator needs to learn that the recovered sample and the augmentation bit sequence jointly follow the checksum principle.
The difficulties of these two problems strongly depend on the way of feeding the augmentation bit sequence into the discriminator network. Providing the augmentation bit sequence to the input layer or lower layers generally makes the checksum operation more difficult than providing it to the upper layers. As the former has more difficulties decoupling the two tasks and trying to solve them jointly, while the latter can solve them sequentially. One naive design is to combine the bit sequence with the original output of the discriminator. In this case the checksum operation becomes nearly trivial and does not have a major influence on the training. The other way is to concatenate the augmentation bit sequence directly with the data and synthetic samples, ensuring that the task remains non-trivial for the discriminator.
Toy Example: Binary Classification with PA. For an illustration purpose, here we design a simple experiment. We employ discriminator for the classification task, omitting the use of the generator. Namely, two classes of CIFAR10 (cats and dogs) are extracted, conveying the bit 0 and 1. Together with the augmentation bit sequences, they are provided as input to the discriminator, which task is to perform binary classification. Using the SN-DCGAN architecture, we directly concatenate the augmentation bit sequences with the cat and dog images.
Figure A12 depicts the variation of the discriminator loss (D Loss) over the periodic progressive augmentation (every 2k iterations). It is evident that whenever a new augmentation level is reached an abrupt increase of D Loss is observed, showing that the task of the discriminator becomes harder with each level of PA.
Employing Residual/Skip Connections with PA. With the presence of skip connections, the augmentation bit sequence that is concatenated with the inputs will also be directly wired to the
network output. In the next two examples, we examine the task difficulty under such situation. First, we switch to using ResNet CIFAR10 (Kurach et al., 2018) for the toy example of binary classification of cats and dogs, while concatenating the augmentation bits with the inputs of the last residual block. By doing so, the skip connection will directly deliver this information to the network final layer. As we can see from Figure A13, the task remains difficult and PA persistently increases the task difficulty.
Next, we switch to the GAN setting and repeat the experiment of SN-DCGAN CIFAR10 in Sec. 5.1. Instead of only concatenating the augmentation bits with the input images, we also insert a skip connection, additionally concatenating them with the input to the last convolutional layer.7 Depicting the D Loss and G Loss during the course of training, Figure A14 confirms that PA remains effective with the skip connection. As a result, a similar performance is reported in Figure A15 and Table A12. This showcases that the proposed PA generalizes well across different GAN design choices (including networks with residual or skip connections).

heading: A.9 COMPARISON WITH THE DROPOUT REGULARIZATION

text:
In this section we compare the proposed PA with another regularization technique which also aims to weaken the discriminator, such as adding dropout at the first layer of the discriminator. Our
7XOR operation is a non-linear function. Therefore, we do not take the last linear layer.
proposed augmentation scheme is orthogonal to it and thus can be applied along with the dropout regularization.
We adopt dropout at the output of the first convolutional layer of SN-DCGAN and experiment with two configurations. In the first configuration the dropout rate is linearly increased from zero to 0.7, we call it an ascending mode (A). In the second the dropout rate starts from 0.7 and linearly descends to zero during the training, we call it a descending mode (D). From Figure A16 and Table A13 we observe that neither of the two dropout modes improves the performance of NS GAN, whereas PA shows to be more effective and preserves the performance gain even in combination with the dropout.

title: PA-GAN: IMPROVING GAN TRAINING BY PROGRESSIVE AUGMENTATION


summaries: In this interesting study, the authors show that incorporating rotation-equivariant filters  (i.e. enforcing weight sharing across filters with different orientations) in a CNN model of the visual system is a useful prior to predict responses in V1. After fitting this model to data, they find that the RFs of model V1 cells do not resemble the simple Gabor filters of textbooks, and they present other quantitative results about V1 receptive fields. The article is clearly written and the claims are supported by their analyses. It is the first time to my knowledge that a rotation-equivariant CNN is used to model V1 cells.

summaries: The paper analyses the data collected from 6005 neurons in a mouse brain. Visual stimuli are presented and the responses of the neurons recorded. In the next step, a rotational equivariant neural network architecture together with a sparse coding read-out layer is trained to predict the neuron responses from the stimuli. Results show a decent correlation between neuron responses and trained network. Moreover, the rotational equivariant architecture beats a standard CNN with similar number of feature maps. The analysis and discussion of the results is interesting. Overall, the methodological approach is good.

abstractText: Classical models describe primary visual cortex (V1) as a filter bank of orientationselective linear-nonlinear (LN) or energy models, but these models fail to predict neural responses to natural stimuli accurately. Recent work shows that convolutional neural networks (CNNs) can be trained to predict V1 activity more accurately, but it remains unclear which features are extracted by V1 neurons beyond orientation selectivity and phase invariance. Here we work towards systematically studying V1 computations by categorizing neurons into groups that perform similar computations. We present a framework for identifying common features independent of individual neurons’ orientation selectivity by using a rotation-equivariant convolutional neural network, which automatically extracts every feature at multiple different orientations. We fit this rotation-equivariant CNN to responses of a population of 6000 neurons to natural images recorded in mouse primary visual cortex using two-photon imaging. We show that our rotation-equivariant network outperforms a regular CNN with the same number of feature maps and reveals a number of common features, which are shared by many V1 neurons and are pooled sparsely to predict neural activity. Our findings are a first step towards a powerful new tool to study the nonlinear functional organization of visual cortex.

authors:
Vanhoucke
Vijay Vasudevan
Fernanda Viégas
Oriol Vinyals
Pete Warden
Martin Wattenberg
Martin Wicke
Yuan Yu
Xiaoqiang Zheng

title: Tensorflow: Large-scale machine learning on heterogeneous distributed systems

authors:
J. Antolík
S.B. Hofer
J.A. Bednar
T. D

title: Mrsic-flogel. Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes

venue: PLoS Comput Biol,

authors:
Tom Baden
Philipp Berens
Katrin Franke
Miroslav Román Rosón
Matthias Bethge
Thomas Euler

title: The functional diversity of retinal ganglion cells

venue: in the mouse. Nature,

authors:
E. Batty
J. Merel
N. Brackbill
A. Heitman
A. Sher
A. Litke
E.J. Chichilnisky
L. Paninski

title: Multilayer network models of primate retinal ganglion cells

venue: In Advances in Neural Information Processing Systems,

authors:
S.A. Cadena
G.H. Denfield
E.Y. Walker
L.A. Gatys
A.S. Tolias
M. Bethge
A.S. Ecker

title: Deep convolutional models improve predictions of macaque V1 responses to natural images

venue: PLoS Computational Biology,

authors:
Matteo Carandini
David J Heeger
J Anthony Movshon

title: Linearity and normalization in simple cells of the macaque primary visual cortex

venue: Journal of Neuroscience,

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

venue: In International Conference on Machine Learning,

authors:
Sander Dieleman
Jeffrey De Fauw
Koray Kavukcuoglu

title: Exploiting cyclic symmetry in convolutional neural networks

authors:
Thomas Euler
Silke Haverkamp
Timm Schubert
Tom Baden

title: Retinal bipolar cells: elementary building blocks of vision

venue: Nature Reviews Neuroscience,

authors:
Katrin Franke
Philipp Berens
Timm Schubert
Matthias Bethge
Thomas Euler
Tom Baden

title: Inhibition decorrelates visual feature representations in the inner retina

authors:
Martin S Gizzi
Ephraim Katz
Robert A Schumer
J Anthony Movshon

title: Selectivity for orientation and direction of motion of single neurons in cat striate and extrastriate visual cortex

venue: Journal of Neurophysiology,

authors:
Qin Hu
Jonathan D Victor

title: Two-dimensional hermite filters simplify the description of high-order statistics of natural images. Symmetry

authors:
David H Hubel
Torsten N Wiesel

title: Receptive fields and functional architecture of monkey striate cortex

venue: The Journal of Physiology,

title: Matplotlib: A 2d graphics environment

venue: Computing in science & engineering,

authors:
William F Kindel
Elijah D Christensen
Joel Zylberberg

title: Using deep learning to probe the neural code for images in primary visual cortex

venue: Journal of Vision,

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

venue: In International Conference on Learning Representations,

authors:
D. Klindt
A.S. Ecker
T. Euler
M. Bethge

title: Neural system identification for large populations separating “what” and “where

venue: In Advances in Neural Information Processing Systems

authors:
Thomas Kluyver
Benjamin Ragan-Kelley
Fernando Pérez
Brian Granger
Matthias Bussonnier
Jonathan Frederic
Kyle Kelley
Jessica Hamrick
Jason Grout
Sylvain Corlay
Paul Ivanov
Damián Avila
Safia Abdalla
Carol Willing

title: Jupyter notebooks – a publishing format for reproducible computational workflows

venue: Positioning and Power in Academic Publishing: Players, Agents and Agendas,

authors:
Diego Marcos
Michele Volpi
Nikos Komodakis
Devis Tuia

title: Rotation equivariant vector field networks

venue: In Proceedings of the IEEE International Conference on Computer Vision,

title: Docker: Lightweight linux containers for consistent development and deployment

venue: Linux J.,

authors:
Klas Nordberg
Gösta Granlund

title: Equivariance and invariance–an approach based on Lie groups

venue: In International Conference on Image Processing,

authors:
E.A. Pnevmatikakis
Y. Gao
D. Soudry
D. Pfau
C. Lacefield
K. Pskanzer
R. Bruno
R. Yuste
L. Paninski

title: Simultaneous Denoising, Deconvolution, and Demixing of Calcium Imaging Data Eftychios

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein
Alexander C. Berg
Li Fei-Fei

title: ImageNet Large Scale Visual Recognition Challenge

venue: International Journal of Computer Vision (IJCV),

authors:
Joshua R Sanes
Richard H Masland

title: The types of retinal ganglion cells: current status and implications for neuronal classification

venue: Annual Review of Neuroscience,

authors:
Peter H Schiller
Barbara L Finlay
Susan F Volman

title: Quantitative studies of single-cell properties in monkey striate cortex

venue: i. spatiotemporal organization of receptive fields. Journal of Neurophysiology,

authors:
N.J. Sofroniew
D. Flickinger
J. King
K. Svoboda

title: A large field of view two-photon mesoscope with subcellular resolution for in vivo

venue: imaging. eLife,

authors:
Shiming Tang
Tai Sing Lee
Ming Li
Yimeng Zhang
Yue Xu
Fang Liu
Benjamin Teo
Hongfei Jiang

title: Complex Pattern Selectivity in Macaque Primary Visual Cortex Revealed by Large-Scale Two-Photon Imaging

venue: Current Biology,

authors:
Bosiljka Tasic
Zizhen Yao
Lucas T Graybuck
Kimberly A Smith
Thuc Nghi Nguyen
Darren Bertagnolli
Jeff Goldy
Emma Garren
Michael N Economo
Sarada Viswanathan

title: Shared and distinct transcriptomic cell types across neocortical

venue: areas. Nature,

authors:
Jonathan D. Victor
Ferenc Mechler
Michael A. Repucci
Keith P. Purpura
Tatyana Sharpee

title: Responses of V1 Neurons to Two-Dimensional Hermite Functions

authors:
Stéfan van der Walt
S Chris Colbert
Gael Varoquaux

title: The numpy array: a structure for efficient numerical computation

venue: Computing in Science & Engineering,

authors:
Maurice Weiler
Fred A Hamprecht
Martin Storath

title: Learning steerable filters for rotation equivariant cnns

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Daniel E Worrall
Stephan J Garbin
Daniyar Turmukhambetov
Gabriel J Brostow

title: Harmonic networks: Deep translation and rotation equivariance

authors:
D.L.K. Yamins
H. Hong
C.F. Cadieu
E.A. Solomon
D. Seibert
J.J. DiCarlo

title: Performanceoptimized hierarchical models predict neural responses in higher visual cortex

venue: Proceedings of the National Academy of Sciences of the United States of America,

authors:
D. Yatsenko
J. Reimer
A.S. Ecker
E.Y. Walker
F. Sinz
P. Berens
A. Hoenselaar
R.J. Cotton
A.S. Siapas
A.S. Tolias

title: DataJoint: managing big scientific data using MATLAB or Python

heading: 1 INTRODUCTION

text:
The mammalian retina processes image information using a number of distinct parallel channels consisting of functionally, anatomically, and transcriptomically defined distinct cell types. In the mouse, there are 14 types of bipolar cells (Euler et al., 2014), which provide input to 30–50 types of ganglion cells (Baden et al., 2016; Sanes & Masland, 2015). In visual cortex, in contrast, it is currently unknown whether excitatory neurons are similarly organized into functionally distinct cell types. A functional classification of V1 neurons would greatly facilitate understanding its computations just like it has for the retina, because we could focus our efforts on identifying the function of a small number of cell types instead of characterizing thousands of anonymous neurons.
Recent work proposed a framework for learning functional cell types from data in an unsupervised fashion while optimizing predictive performance of a model that employs a common feature space shared among many neurons (Klindt et al., 2017). The key insight in this work is that all neurons that perform the same computation but have their receptive fields at different locations, can be represented by a feature map in a convolutional network. Unfortunately, this approach cannot be applied directly to neocortical areas. Neurons in area V1 extract local oriented features such as edges at different
orientations, and most image features can appear at arbitrary orientations – just like they can appear at arbitrary locations. Thus, to define functional cell types in V1, we would like to treat orientation as a nuisance parameter (like receptive field location) and learn features independent of orientation.
In the present paper, we work towards this goal. While we do not answer the biological question whether there are indeed well-defined clusters of functional cell types in V1, we provide the technical foundation by extending the work of Klindt and colleagues (Klindt et al., 2017) and introducing a rotation-equivariant convolutional neural network model of V1. We train this model directly on the responses of 6000 mouse V1 neurons to learn a shared feature space, whose features are independent of orientation. We show that this model outperforms state-of-the-art CNNs for system identification and allows predicting V1 responses of thousands of neurons with only 16 learned features. Moreover, for most neurons, pooling from only a small number of features is sufficient for accurate predictions.

heading: 2 RELATED WORK

text:
Functional classification of cell types. Characterizing neurons according to some identified, potentially nonlinear response properties has a long history. For instance, researchers have identified simple and complex cells (Hubel & Wiesel, 1968), pattern- and component-selective cells (Gizzi et al., 1990), end-stopping cells (Schiller et al., 1976), gain control mechanisms (Carandini et al., 1997) and many more. However, these properties have been identified using simple stimuli and the models that have been built for them do not apply to natural stimuli in a straightforward way. In addition, most of these studies were done using single-cell electrophysiology, which does not sample cells in an unbiased fashion. Thus, we currently do not know how important certain known features of V1 computation are under natural stimulation conditions and what fraction of neurons express them. Recent work using two-photon imaging in the monkey (Tang et al., 2018) started closing this gap by systematically investigating response properties of V1 neurons to an array of complex stimuli and found, for instance, that about half of the neurons in V1 are selective to complex patterns such as curvature, corners and junctions rather than just oriented line segments.
Recent work in the retina showed that different types of Ganglion cells (Baden et al., 2016) – and to some extend also bipolar cells (Franke et al., 2017) – can be identified based on functional properties. Thus, in the retina there is a relatively clear correspondence of anatomical and genetic cell types and their functional output, and we are getting closer to understanding each retinal cell type’s function.
Learning shared feature spaces for neural populations. Antolík and colleagues pioneered the idea of learning a shared nonlinear feature representation for a large population of neurons (Antolík et al., 2016), which others have also used in both retina and V1 (Batty et al., 2017; Kindel et al., 2019; Cadena et al., 2019). Klindt et al. (2017) proposed a framework to learn functional cell types in an unsupervised fashion as a by-product of performing system identification. They propose a structurally constrained readout layer, which provides a compact characterization of each neuron’s computation. By enforcing this representation to be sparse, they suggest that the learned features may correspond to distinct functional cell types. Our present work builds upon this idea and extends it to be applicable to V1.
Rotation equivariance in CNNs. There is a large body of work on equivariant representations (Nordberg & Granlund, 1996). Here we review only the most closely related approches. Cohen & Welling (2016) introduce group-equivariant CNNs, which include rotation equivariance as a special case and form the basis of our approach. Weiler et al. (2018) use a steerable filter basis to learn rotation-equivariant CNNs. We essentially use the same approach, but with a different set of steerable filters (2d Hermite functions instead of circular harmonics). RotEqNet (Marcos et al., 2017) is related to our work in the sense that it also applies multiple rotated versions of each filter to the input. However, instead of maintaining all resulting feature maps as inputs for the next layer, they apply an orientation pooling operation, which reduces each set of feature maps to a two-dimensional vector field. Harmonic networks (Worrall et al., 2017) are an alternative approach that achieves full 360◦ rotation equivariance by limiting the structure of the convolutional filters. Finally, Dieleman et al. (2016) achieve rotation equivariance by feeding multiple rotated versions of the input image into parallel CNN streams whose weights are tied together.

heading: 3 ROTATION-EQUIVARIANT CNN

text:
Network architecture. Our architecture follows that of Klindt et al. (2017). The image is first processed by a multi-layer convolutional core shared by all neurons (Fig. 1A, blue), which we describe in detail below. The resulting feature representation is then turned into a response prediction for each neuron by applying a sparse linear combination across space, followed by a sparse linear combination across features and a pointwise, soft-thresholding nonlinearity (Fig. 1A, red). The entire model is trained end-to-end directly on predicting neural responses, without employing transfer learning or pre-training on auxiliary tasks (Yamins et al., 2014; Cadena et al., 2019).
This model has a number of desirable properties in terms of data efficiency and interpretability. The separation into convolutional core and readout, together with the strong structural constraints on the readout pushes all the ‘heavy lifting’ into the core while the readout weights (spatial mask and feature weights) provide a relatively low-dimensional characterization of each neuron’s function. Because the core is shared among all neurons (here: thousands), many of which implement similar functions, we can learn complex non-linear functions very accurately.
Equivariance. A function f : X → Y is called equivariant with respect to a group of transformations Π if such transformations of the input lead to predictable transformations of the output. Formally, for every π ∈ Π there is a transformation ψ ∈ Ψ such that for every x ∈ X
ψ[f(x)] = f [π(x)]. (1)
CNNs are shift-equivariant by construction, meaning that every translation of the image leads to a matching translation in the feature maps (i. e. π and ψ are both translation by a number of pixels horizontally and vertically). Shift equivariance is a useful property for neural system identification,
because it allows us to represent many neurons that perform similar computations but in different locations in space by a single convolutional feature map instead of learning each neurons’ nonlinear input-output function ‘from scratch.’
Rotation-equivariant CNN. Neurons in V1 do not only perform similar functions at different locations, but also extract similar features with different orientations. Thus, for modeling populations of V1 neurons, learning a representation that is equivariant to rotation in addition to translation would be desirable. To achieve rotation equivariance, we use group convolutions (Cohen & Welling, 2016). Here we use the group of all rotations by multiples of 45◦. That is, for each convolutional filter in the first layer, we have eight rotated copies, each of which produces one feature map (Fig. 1A, blue). Thus, if we learn 16 different filters in the first layer, it will have a total of 8× 16 = 128 feature maps. Formally, if π is a rotation of the image by, say, 45◦, then ψ is a rotation of the feature maps by also 45◦ combined with a cyclic permutation of the feature maps by one rotation step.
For the second (and all subsequent) layers, the procedure becomes a bit more involved. Sticking with the numbers from above, for every feature in the second layer we now learn 128 filters – one for each input feature map (Fig. 1B, first row). To preserve rotation equivariance, we need to create all rotated copies of these filters and cyclically permute the feature maps such that each rotated filter receives the rotated version of the input (depicted by the colored boxed in Fig. 1B, second and following rows).
To implement weight sharing across filter orientation without aliasing artifacts, we represent the filters in a steerable basis (Weiler et al., 2018). We use the two-dimensional Hermite functions in polar coordinates, which form a steerable, orthonormal basis (Fig. 1C; see also Victor et al., 2006; Hu & Victor, 2016). For filters of size k, we use all 2d Hermite functions up to rank k, which means we have k(k + 1)/2 basis functions. We sample each filter at twice the resolution and then downsample by 2×2 average pooling to reduce aliasing.

heading: 4 EXPERIMENTS

text:
Neural data. We recorded the responses of 6005 excitatory neurons in primary visual cortex (layers 2/3 and 4) from one mouse by taking two consecutive scans with a large-field-of-view twophoton mesoscope (Sofroniew et al., 2016). Activity was measured using the genetically encoded calcium indicator GCaMP6f. V1 was targeted based on anatomical location as verified by numerous previous experiments performing retinotopic mapping using intrinsic imaging. We selected cells based on a classifier for somata on the segmented cell masks and deconvolved their fluorescence traces (Pnevmatikakis et al., 2016). We did not filter cells according to visual responsiveness. The aquisition frame rate was 4.8 Hz in both scans. We monitored pupil position, dilation, and absolute running speed of the animal. However, because eye movements were rare and we are primarily interested in the average response of neurons given the visual stimulus, we did not further take into account eye position and running speed.
Visual stimuli. Stimuli consisted of 5500 images taken from ImageNet (Russakovsky et al., 2015), cropped to fit a 16:9 monitor, and converted to gray-scale. The screen was 55× 31 cm at a distance of 15 cm, covering roughly 120◦ × 90◦. In each scan, we showed 5400 of these images once (training and validation set) and the remaining 100 images 20 times each (test set). Each image was presented for 500ms followed by a blank screen lasting between 500ms and 1s. For each neuron, we extract the accumulated activity between 50ms and 550ms after stimulus onset using a Hamming window.
Preprocessing. We rescaled the images to 64× 36 pixels and standardized them by subtracting the mean over all images in the training set and all pixels from each image and dividing by the standard deviation (also taken over all images in the training set and all pixels). We divided the responses of each neuron by its standard deviation over time. We did not center the neural responses, because they are non-negative after deconvolution and zero has a clear meaning.
Model fitting and evaluation. We initialized all weights randomly from a truncated normal distribution with mean zero and standard deviation 0.01. The biases of the batch normalization layers were initially set to zero. In contrast, we set the biases in each neuron’s readout to a non-zero initial value, since the neural responses are not centered on zero. We initialized these biases such that the initial model prediction was on average half the average response of each neuron. To fit the models, we used
the Adam Optimizer (Kingma & Ba, 2016) with an initial learning rate of 0.002, a single learning rate decay and early stopping. We monitored the validation loss every 50 iterations and decreased the learning rate once by a factor of 10 when the validation loss had not decreased for five validation steps in a row. We then further optimized the model until the same criterion is reached again.
To evaluate the models, we use the following procedure. For each neuron we compute the Pearson correlation coefficient between the model prediction and the average response over the 20 repetitions of the 100 test images. We then average the correlations over all neurons. This approach tells us how well the model predicts the average response of neurons to a given stimulus, ignoring trial-to-trial variability (which is interesting in itself, but not the focus of the present work).
Architecture details. Our architecture consists of three convolutional layers with filter sizes of 13, 5 and 5 pixels, respectively. Thus, the receptive fields of the CNN’s last layer’s units were 21 px, corresponding to ∼60◦ and covering both classical and extra-classical receptive field. We use zero padding such that the feature maps maintain the same size across layers. We use 16 filter sets (i. e. 128 feature maps) in the first two layers and 8 to 48 filter sets in the third layer (number cross-validated, see below). After each layer, we apply batch normalization followed by a learned scale and bias term for each feature map.1 After the first and second layer, but not after the third, we apply a soft-thresholding nonlinearity f(x) = log(1 + exp(x)). The feature maps of the third layer then provide the input for each neuron’s readout, which consists of a linear combination first over space and then over features, followed by an added bias term and a final soft-thresholding nonlinearity. Thus, each neuron implements a cascade of three LN operations.
Regularization. We use the same three regularizers as (Klindt et al., 2017). For smoothness of convolution kernels we set the relative weight of the first layer to twice as strong as in the second and third layer to account for the larger kernels. We apply group sparsity to the convolution kernels of the second and third layer. We regularize the spatial masks and feature weights in the readout to be sparse by applying the L1 penalty on the 3d tensor that results from taking their outer tensor product. The weights of all three regularizers are cross-validated as described in the next paragraph.
Model selection. We cross-validated over the number of filter sets in the third layer, ranging from 8 to 48 (i. e. 64 to 384 feature maps2) and the strength of the regularizers. For each architecture, we fit 32 models with different initial conditions and randomly drawn hyperparameters (smoothness of filters: 0.001–0.03, group sparsity of filters: 0.001–0.1, sparsity of readout: 0.005–0.03) and chose the best one according to its loss on the validation set (i. e. not using the test set). For all baseline and control models (see below), we also cross-validated over 32 randomly sampled sets of hyperparameters drawn from the same range of values.
Baseline and control experiments. As a baseline, we fit a number of regular CNNs without rotation equivariance. These models are completely identical in terms of number of layers, filter sizes, readout layer and fitting procedure, except for the weight sharing constraint across orientation. We fit models with the same number of feature maps as their rotation-equivariant counterparts as well as smaller models with fewer feature maps (see Table 1).
Previous work has enforced the feature weights in the readout to be positive (Klindt et al., 2017). Because we do not enforce such constraint in the present work, we ran a control experiment, in which we enforce the readout weights (both masks and feature weights) to be positive.
Sparsity of the spatial masks is well justified, because we know that receptive fields are localized. However, it is unclear whether sparsity on the feature weights is desirable, since the function each neuron implements may not be well aligned with the coordinate axes spanned by the small number of features we learn. Thus, we also fit a model without the sparsity regularizer for the feature weights.
1In our experiments the network did not implement exact rotation equivariance, because batch normalization was applied to each feature map individually instead of jointly to all rotated versions. We therefore re-ran a subset of experiments where we corrected this issue and verified that model performance was indistinguishable.
2384 feature maps was the maximum we could fit into 16 GB of available GPU memory.
Model Corr. S.D.
Rotation-equivariant CNN — 16x8–16x8–16x8 0.47 0.005 — positive feature weights 0.43 0.006 — no feature sparsity 0.43 0.02
CNN (Klindt et al., 2017) — 32–32–32 0.39 0.009 — 64–64–64 0.39 0.006 — 128–128–128 0.42 0.008 — 128–128–256 0.41 0.009
Table 1: Average correlation on test set of our rotation-equivariant CNN, two controls, and several regular CNNs as baselines. The three numbers for the CNNs are the number of feature maps in each layer; other parameters are identical to the rotation-equivariant CNN. Standard deviations are across the top-five models of each type.
Tools. We performed our model fitting and analyses using DataJoint (Yatsenko et al., 2015), Numpy/Scipy (Walt et al., 2011), Matplotlib (Hunter, 2007), Seaborn (Waskom et al., 2017), Jupyter (Kluyver et al., 2016), Tensorflow (Abadi et al., 2015), and Docker (Merkel, 2014).
Availability of code and models. Code to reproduce all experiments and models as well as pretrained models are available at https://github.com/aecker/cnn-sys-ident.

heading: 5 RESULTS

text:
Architecture search and model selection. We start by evaluating the predictive performance of our rotation-equivariant CNN on a dataset of 6005 neurons from mouse primary visual cortex. First, we performed an initial architecture search, drawing inspiration from earlier work (Klindt et al., 2017) and exploring different numbers of layers and feature maps and different filter sizes. We settled on an architecture with three convolutional layers with filter sizes of 13, 5 and 5 pixels, respectively, and used 16 filter sets (i. e. 128 feature maps) in the first two layers. For the third layer, which provides the nonlinear feature space from which all neurons pool linearly, we cross-validated over the number of features, ranging from 8 to 48 (i. e. 64 to 384 feature maps).
Our model achieved an average correlation on the test set of 0.47. The performance improved with the number of features in the third layer, but the improvement was small beyond 16 features (Fig. 2). For the following analyses we therefore focus on this model with 16 features as a compromise between model complexity and performance.
It is encouraging that a model with only 16 features is sufficient to accurately model a population as large as 6000 neurons. Earlier work modeling V1 responses used a shared feature space of 10–20 (Antolík et al., 2016) or 48 features (Klindt et al., 2017) for populations of 50–100 neurons, which reduced the dimensionality of the feature space by a factor of 2–5 compared to the number of neurons. Here, we reduce the dimensionality by a factor of 375 while achieving a similar level of performance.
Rotation-equivariant CNN outperforms regular CNN. To compare our new model to existing approaches, we also fit a regular CNN with identical architecture except for the weight sharing constraint across ori-
entations. In addition, we fit a number of smaller CNNs with fewer feature maps in each layer, which are more similar in terms of number of parameters, but potentially have less expressive power. Our rotation-equivariant CNN outperforms all baselines (Table 1) and generally requires less data (Fig. 3), showing that enforcing weight sharing across orientations is not only potentially useful for
interpreting the model (as we show below), but also serves as a good regularizer to fit a larger, more expressive model.
Feature space generalizes to unseen neurons. To show that our network learns common features of V1 neurons, we excluded half of the neurons when fitting the network. We then fixed the rotationequivariant convolutional core and trained only the readout (spatial mask and feature weights) for the other half of the neurons. The resulting test correlation for these neurons (0.46) was indistinguishable from that of the full model (0.47), showing that the learned features transfer to neurons not used to train the feature space.
Feature weights are sparse. The intuition behind the sparse, factorized readout layer is that the spatial mask encodes the receptive field location of each neuron while the feature weights parameterize the neuron’s nonlinear computation. We now ask how the neurons are organized within this 16-dimensional function space. On the one extreme of the spectrum, each neuron could pick a random direction, in which case sparsity would not be the right prior and the feature weights should be dense. On the other end of the spectrum, there could be a discrete number of functional cell types. In this case, each cell type would be represented by a single feature and the feature
weights should be maximally sparse, i. e. one-hot encodings of the cell type.
To analyze the feature weights, we first marginalize them over orientation. To do so, we take the sum of squares over all 8 orientations for each of the 16 features and then normalize them such that the energy of the 16 features sums to one. We find that the feature weights are indeed quite sparse
(Fig. 4A): most of the neurons use only 1–5 features (Fig. 4B) and the strongest feature captures more than 50% of the energy for 63% of the neurons.
To ensure that the sparsity of the feature weights is a property of the data and not a trivial result of our L1 penalty, we performed an ablation study, where we fit a model that applies sparsity only on the spatial masks, but uses L2 regularization for the feature weights. This model performed worse than the original model (Table 1) and produced a significantly denser weight distribution (Fig. 4B), suggesting that sparsity is indeed a justified assumption.
Feature weights have consistent sign. There is a second line of evidence that the features learned by our model are meaningful beyond just providing an arbitrary basis of a 16-dimensional feature space in which neurons are placed at random: the weights that different neurons assign to any given feature have remarkably consistent signs (Fig. 5). For 11 of 16 features, more than 90% of the non-zero weights have the same sign.3 Thus, the negation of one feature that drives a large group of neurons is not a feature that drives many neurons.
3Note that the sign itself does not carry any meaning. Because there is no nonlinearity after the last convolution, we can flip the sign of all filters generating this feature and at the same time flip all neurons’ readout weights for this feature, which would leave the model prediction unchanged.
Visualization of the different groups of neurons. Having presented evidence that the features identified by our model do indeed represent a simple and compact, yet accurate description of a large population of neurons, we now ask what these features are. As a first step, we group all neurons into one of 16 groups based on their strongest feature weight. This approach is obviously to some extent arbitrary. By no means to we want to argue that there are exactly 16 types of excitatory neurons in V1 or that excitatory neurons in V1 can even be classified into distinct functional types. Nevertheless, we believe that the 16 groups defined in this way are useful in a practical sense, because they represent features that cover more than 50% of the variance of a large fraction of neurons and therefore yield a very compact description of the most important features of V1 computation across the entire excitatory neuronal population.
To visualize what each of these features compute, we pick the 16 most representative examples4 of each group and use the model to approximate their linear receptive field (RF). To this end, we compute the model gradient at a gray image (Fig. 6). The crucial point of this plot is that it shows that the premise of rotation equivariance holds: cells are clustered according to similar spatial patterns, but independent of their preferred orientation. As expected, most linear RFs resemble Gabor filters, which show differences in symmetry (odd: #2, #11, #15 vs even: #3, #6, #12) as well polarity (#3 vs. #6, #12), while some groups exhibit center-surround structure (#4, #14, #16).

heading: 6 DISCUSSION

text: We developed a rotation-equivariant convolutional neural network model of V1 that allows us to characterize and study V1 computation independent of orientation preference. Although the visual system is not equivariant to rotation – there are known biases in the distribution of preferred orientations –, enforcing weight sharing across orientations allowed us to fit larger, more expressive models given a limited dataset. While our work lays out the technical foundation, we only scratched the surface of the many biological questions that can now be addressed. Future work will have to investigate the learned features in much more detail, test to what extent they generalize across recording sessions and animals, whether they are consistent across changes in the architecture and – most importantly – whether neurons in V1 indeed cluster into distinct, well-defined functional types and this organization finds any resemblance in anatomical or genetic properties (Tasic et al., 2018) of the neurons recorded.

heading: ACKNOWLEDGMENTS

text: Supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) via grant EC 479/1-1 to A.S.E and the Collaborative Research Center (Projektnummer 276693517 – SFB 1233: Robust Vision); the German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ 01IS18039A); the International Max Planck Research School for Intelligent Systems (IMPRS-IS) supporting S.A.C; the Bernstein Center for Computational Neuroscience Tübingen; the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DoI/IBC) contract number D16PC00003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the National Institutes of Health, IARPA, DoI/IBC, or the U.S. Government.

title: A ROTATION-EQUIVARIANT CONVOLUTIONAL NEURAL NETWORK MODEL OF PRIMARY VISUAL CORTEX