summaries: This work proposes a novel analytic framework exploited on a semantic segmentation model to visualize GANs at unit (feature map) level. The authors show that some GAN representations can be interpreted, correlate with the parsing result from the semantic segmentation model but as variables that have a causal effect on the synthesis of semantic objects in the output. This framework could allow to detect and remove the artifacts to improve the quality of the generated images.

summaries: This paper provides a visualization framework to understand the generative neural network in GAN models. To achieve this, they first find a group of interpretable units and then quantify the causal effect of interpretable units. Finally, the contextual relationship between these units and their surrounding is examined by inserting the discovered object concepts into new images. Extensive experiments are presented and a video is provided.

abstractText: Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability. However, they have not been well visualized or understood. How does a GAN represent our visual world internally? What causes the artifacts in GAN results? How do architectural choices affect GAN learning? Answering such questions could enable us to develop new insights and better models. In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level. We first identify a group of interpretable units that are closely related to object concepts using a segmentation-based network dissection method. Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output. We examine the contextual relationship between these units and their surroundings by inserting the discovered object concepts into new images. We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in a scene. We provide open source interpretation tools to help researchers and practitioners better understand their GAN models.

authors:
Sebastian Bach
Alexander Binder
Grégoire Montavon
Frederick Klauschen
Klaus-Robert Müller
Wojciech Samek

title: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

venue: PloS one,

authors:
David Bau
Bolei Zhou
Aditya Khosla
Aude Oliva
Antonio Torralba

title: Network dissection: Quantifying interpretability of deep visual representations

venue: In CVPR, 2017

title: Pros and cons of gan evaluation measures

venue: arXiv preprint arXiv:1802.03446,

authors:
Jeffrey S Bowers
Ivan I Vankov
Markus F Damian
Colin J Davis

title: Why do some neurons in cortex respond to information in a selective manner? insights

authors:
Andrew Brock
Theodore Lim
James M Ritchie
Nick Weston

title: Neural photo editing with introspective adversarial networks

venue: In ICLR,

authors:
Sanjoy Dasgupta
Timothy C Sheehan
Charles F Stevens
Saket Navlakha

title: A neural data structure for novelty detection

venue: Proceedings of the National Academy of Sciences,

authors:
Emily L Denton
Soumith Chintala
Rob Fergus

title: Deep generative image models using a laplacian pyramid of adversarial networks

venue: In NIPS,

authors:
Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio

title: Density estimation using real nvp

authors:
Jeff Donahue
Philipp Krähenbühl
Trevor Darrell

title: Adversarial feature learning

authors:
Alexey Dosovitskiy
Thomas Brox

title: Generating images with perceptual similarity metrics based on deep networks

authors:
Vincent Dumoulin
Ishmael Belghazi
Ben Poole
Alex Lamb
Martin Arjovsky
Olivier Mastropietro
Aaron Courville

title: Adversarially learned inference

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron C Courville

title: Improved training of wasserstein gans

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

authors:
Judy Hoffman
Eric Tzeng
Taesung Park
Jun-Yan Zhu
Phillip Isola
Kate Saenko
Alexei A Efros
Trevor Darrell

title: Cycada: Cycle-consistent adversarial domain adaptation

venue: In ICML,

title: Causal inference, path analysis and recursive structural equations models

venue: ETS Research Report Series,

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A Efros

title: Image-to-image translation with conditional adversarial networks

venue: In CVPR,

authors:
Andrej Karpathy
Justin Johnson
Li Fei-Fei

title: Visualizing and understanding recurrent networks

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

authors:
Been Kim
Justin Gilmer
Fernanda Viegas
Ulfar Erlingsson
Martin Wattenberg

title: Tcav: Relative concept importance testing with linear concept activation vectors

venue: arXiv preprint arXiv:1711.11279,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: ICLR,

authors:
Scott M Lundberg
Su-In Lee

title: A unified approach to interpreting model predictions

authors:
Aravindh Mahendran
Andrea Vedaldi

title: Understanding deep image representations by inverting them

authors:
Michael Mathieu
Camille Couprie
Yann LeCun

title: Deep multi-scale video prediction beyond mean square error

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Ari S Morcos
David GT Barrett
Neil C Rabinowitz
Matthew Botvinick

title: On the importance of single directions for generalization

venue: arXiv preprint arXiv:1803.06959,

authors:
Chris Olah
Arvind Satyanarayan
Ian Johnson
Shan Carter
Ludwig Schubert
Katherine Ye
Alexander Mordvintsev

title: The building blocks of interpretability

title: Concept cells: the building blocks of declarative memory functions

venue: Nature Reviews Neuroscience,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

authors:
Ramprasaath R Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In ICCV,

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency maps

venue: In ICLR Workshop,

authors:
Hendrik Strobelt
Sebastian Gehrmann
Hanspeter Pfister
Alexander M. Rush

title: LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks

venue: IEEE TVCG,

authors:
Mukund Sundararajan
Ankur Taly
Qiqi Yan

title: Axiomatic attribution for deep networks

venue: In PMLR,

authors:
Ting-Chun Wang
Ming-Yu Liu
Jun-Yan Zhu
Guilin Liu
Andrew Tao
Jan Kautz
Bryan Catanzaro

title: Video-to-video synthesis

authors:
Xiaolong Wang
Abhinav Shrivastava
Abhinav Gupta

title: A-fast-rcnn: Hard positive generation via adversary for object detection

authors:
Dedy Rahman Wijaya
Riyanarto Sarno
Enny Zulaika

title: Information quality ratio as a novel metric for mother wavelet selection

venue: Chemometrics and Intelligent Laboratory Systems,

authors:
Tete Xiao
Yingcheng Liu
Bolei Zhou
Yuning Jiang
Jian Sun

title: Unified perceptual parsing for scene understanding

venue: In ECCV,

authors:
Fisher Yu
Ari Seff
Yinda Zhang
Shuran Song
Thomas Funkhouser
Jianxiong Xiao

title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

venue: arXiv preprint arXiv:1506.03365,

title: From the neuron doctrine to neural networks

venue: Nature reviews neuroscience,

authors:
Matthew D Zeiler
Rob Fergus

title: Visualizing and understanding convolutional networks

authors:
Han Zhang
Ian Goodfellow
Dimitris Metaxas
Augustus Odena

title: Self-attention generative adversarial networks

venue: arXiv preprint arXiv:1805.08318,

authors:
Quanshi Zhang
Ying Nian Wu
Song-Chun Zhu

title: Interpretable convolutional neural networks

authors:
Bolei Zhou
Aditya Khosla
Agata Lapedriza
Aude Oliva
Antonio Torralba

title: Object detectors emerge in deep scene cnns

authors:
Bolei Zhou
Hang Zhao
Xavier Puig
Sanja Fidler
Adela Barriuso
Antonio Torralba

title: Scene parsing through ade20k dataset

authors:
Bolei Zhou
David Bau
Aude Oliva
Antonio Torralba

title: Interpreting deep visual representations via network dissection

venue: PAMI, 2018a

authors:
Bolei Zhou
Yiyou Sun
David Bau
Antonio Torralba

title: Interpretable basis decomposition for visual explanation

venue: In ECCV, pp

authors:
Tinghui Zhou
Philipp Krahenbuhl
Mathieu Aubry
Qixing Huang
Alexei A Efros

title: Learning dense correspondence via 3d-guided cycle consistency

authors:
Jun-Yan Zhu
Philipp Krähenbühl
Eli Shechtman
Alexei A. Efros

title: Generative visual manipulation on the natural image manifold

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

heading: 1 INTRODUCTION

text:
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have been able to produce photorealistic images, often indistinguishable from real images. This remarkable ability has powered many real-world applications ranging from visual recognition (Wang et al., 2017), to image manipulation (Isola et al., 2017; Zhu et al., 2017), to video prediction (Mathieu et al., 2016). Since their invention in 2014, many GAN variants have been proposed (Radford et al., 2016; Zhang et al., 2018a), often producing more realistic and diverse samples with better training stability.
Despite this tremendous success, many questions remain to be answered. For example, to produce a church image (Figure 1a), what knowledge does a GAN need to learn? Alternatively, when a GAN sometimes produces terribly unrealistic images (Figure 1f), what causes the mistakes? Why does one GAN variant work better than another? What fundamental differences are encoded in their weights?
In this work, we study the internal representations of GANs. To a human observer, a well-trained GAN appears to have learned facts about the objects in the image: for example, a door can appear on a building but not on a tree. We wish to understand how a GAN represents such structure. Do the objects emerge as pure pixel patterns without any explicit representation of objects such as doors and trees, or does the GAN contain internal variables that correspond to the objects that humans perceive? If the GAN does contain variables for doors and trees, do those variables cause the generation of those objects, or do they merely correlate? How are relationships between objects represented?
We present a general method for visualizing and understanding GANs at different levels of abstraction, from each neuron, to each object, to the contextual relationship between different objects. We first identify a group of interpretable units that are related to object concepts (Figure 1b). These units’ featuremaps closely match the semantic segmentation of a particular object class (e.g., trees). Second, we directly intervene within the network to identify sets of units that cause a type of objects to disappear (Figure 1c) or appear (Figure 1d). We quantify the causal effect of these units using a standard causality metric. Finally, we examine the contextual relationship between these causal object units and the background. We study where we can insert object concepts in new images and how this intervention interacts with other objects in the image (Figure 1d). To our knowledge, our work provides the first systematic analysis for understanding the internal representations of GANs.
Finally, we show several practical applications enabled by this analytic framework, from comparing internal representations across different layers, GAN variants and datasets; to debugging and improving GANs by locating and ablating “artifact” units (Figure 1e); to understanding contextual relationships between objects in scenes; to manipulating images with interactive object-level control.

heading: 2 RELATED WORK

text:
Generative Adversarial Networks. The quality and diversity of results from GANs (Goodfellow et al., 2014) has continued to improve, from generating simple digits and faces (Goodfellow et al., 2014), to synthesizing natural scene images (Radford et al., 2016; Denton et al., 2015), to generating 1k photorealistic portraits (Karras et al., 2018), to producing one thousand object classes (Miyato et al., 2018; Zhang et al., 2018a). GANs have also enabled applications such as visual recognition (Wang et al., 2017; Hoffman et al., 2018), image manipulation (Isola et al., 2017; Zhu et al., 2017), and video generation (Mathieu et al., 2016; Wang et al., 2018). Despite the successes, little work has been done to visualize what GANs have learned. Prior work (Radford et al., 2016; Zhu et al., 2016; Brock et al., 2017) manipulates latent vectors and observes how the results change accordingly.
Visualizing deep neural networks. A CNN can be visualized by reconstructing salient image features (Simonyan et al., 2014; Mahendran & Vedaldi, 2015) or by mining patches that maximize hidden layers’ activations (Zeiler & Fergus, 2014); or we can synthesize input images to invert a feature layer (Dosovitskiy & Brox, 2016). Alternately, we can identify the semantics of each unit (Zhou et al., 2015; Bau et al., 2017; Zhou et al., 2018a) by measuring agreement between unit activations and object segmentation masks, or by training a network to increase interpretability of such units (Zhang et al., 2018b). Visualization of an RNN has also revealed interpretable units that track long-range dependencies (Karpathy et al., 2016; Strobelt et al., 2018). Most previous work on network visualization has focused on networks trained for classification; our work explores deep generative models trained for image generation.
Understanding neural representation in biology. Studies of biological neural networks find evidence of both local representations in which individual neurons are selective for meaningful concepts (Quiroga, 2012), as well as distributed representations in which individual neurons are essentially meaningless (Yuste, 2015). Computational models of biological learning (Bowers et al., 2016; Dasgupta et al., 2018) find sparse and local representations can aid generalization to novel stimuli.
Explaining the decisions of deep neural networks. Individual network decisions can be explained using informative heatmaps (Zhou et al., 2018b; 2016; Selvaraju et al., 2017) or by scoring salience (Simonyan et al., 2014; Bach et al., 2015; Sundararajan et al., 2017; Lundberg & Lee, 2017). Such analyses reveals which inputs contribute most to a categorical prediction by a network. Recent work has also studied the contribution of feature vectors (Kim et al., 2017; Zhou et al., 2018b) or individual channels (Olah et al., 2018) to a final prediction, and Morcos et al. (2018) has examined the effect of individual units by ablating them. Those methods explain discriminative classifiers. Our method aims to explain how an image can be generated by a network, which is much less explored.

heading: 3 METHOD

text:
Our goal is to analyze how objects such as trees are encoded by the internal representations of a GAN generator G : z→ x. Here z ∈ R|z| denotes a latent vector sampled from a low-dimensional distribution, and x ∈ RH×W×3 denotes an H ×W generated image. We use representation to
describe the tensor r output from a particular layer of the generator G, where the generator creates an image x from random z through a composition of layers: r = h(z) and x = f(r) = f(h(z)) = G(z).
Since r has all the data necessary to produce the image x = f(r), r certainly contains the information to deduce the presence of any visible class c in the image. Therefore the question we ask is not whether information about c is present in r — it is — but how such information is encoded in r. In particular, for any class from a universe of concepts c ∈ C, we seek to understand whether r explicitly represents c in some way where it is possible to factor r at locations P into two components
rU,P = (rU,P, rU,P), (1)
where the generation of the object c at locations P depends mainly on the units rU,P, and is insensitive to the other units rU,P. Here we refer to each channel of the featuremap as a unit: U denotes the set of unit indices of interest and U is its complement; we will write U and P to refer to the entire set of units and featuremap pixels in r. We study the structure of r in two phases:
• Dissection: starting with a large dictionary of object classes, we identify the classes that have an explicit representation in r by measuring the agreement between individual units of r and every class c (Figure 1b). • Intervention: for the represented classes identified through dissection, we identify causal
sets of units and measure causal effects between units and object classes by forcing sets of units on and off (Figure 1c,d).

heading: 3.1 CHARACTERIZING UNITS BY DISSECTION

text:
We first focus on individual units of the representation. Recall that ru,P is the one-channel h × w featuremap of unit u in a convolutional generator, where h× w is typically smaller than the image size. We want to know if a specific unit ru,P encodes a semantic class such as a “tree”. For image classification networks, Bau et al. (2017) has observed that many units can approximately locate emergent object classes when the units are upsampled and thresholded. In that spirit, we select a universe of concepts c ∈ C for which we have a semantic segmentation sc(x) for each class. Then we quantify the spatial agreement between the unit u’s thresholded featuremap and a concept c’s segmentation with the following intersection-over-union (IoU) measure:
IoUu,c ≡ Ez ∣∣∣(r↑u,P > tu,c) ∧ sc(x)∣∣∣ Ez ∣∣∣(r↑u,P > tu,c) ∨ sc(x)∣∣∣ ,where tu,c = argmaxt I(r↑u,P > t; sc(x)) H(r↑u,P > t, sc(x)) , (2)
where ∧ and ∨ denote intersection and union operations, and x = G(z) denotes the image generated from z. The one-channel feature map ru,P slices the entire featuremap r = h(z) at unit u. As shown in Figure 2a, we upsample ru,P to the output image resolution as r ↑ u,P. (r ↑ u,P > tu,c) produces a binary mask by thresholding the r↑u,P at a fixed level tu,c. sc(x) is a binary mask where each pixel indicates the presence of class c in the generated image x. The threshold tu,c is chosen to be informative as possible by maximizing the information quality ratio I/H (using a separate validation set), that is, it maximizes the portion of the joint entropy H which is mutual information I (Wijaya et al., 2017).
We can use IoUu,c to rank the concepts related to each unit and label each unit with the concept that matches it best. Figure 3 shows examples of interpretable units with high IoUu,c. They are not the
only units to match tables and sofas: layer3 of the dining room generator has 31 units (of 512) that match tables and table parts, and layer4 of the living room generator has 65 (of 512) sofa units.
Once we have identified an object class that a set of units match closely, we next ask: which units are responsible for triggering the rendering of that object? A unit that correlates highly with an output object might not actually cause that output. Furthermore, any output will jointly depend on several parts of the representation. We need a way to identify combinations of units that cause an object.

heading: 3.2 MEASURING CAUSAL RELATIONSHIPS USING INTERVENTION

text:
To answer the above question about causality, we probe the network using interventions: we test whether a set of units U in r cause the generation of c by forcing the units of U on and off.
Recall that rU,P denotes the featuremap r at units U and locations P. We ablate those units by forcing rU,P = 0. Similarly, we insert those units by forcing rU,P = k, where k is a per-class constant, as described in Section S-6.4. We decompose the featuremap r into two parts (rU,P, rU,P), where rU,P are unforced components of r:
Original image : x = G(z) ≡ f(r) ≡ f(rU,P, rU,P) (3) Image with U ablated at pixels P : xa = f(0, rU,P)
Image with U inserted at pixels P : xi = f(k, rU,P)
An object is caused by U if the object appears in xi and disappears from xa. Figure 1c demonstrates the ablation of units that remove trees, and Figure 1d demonstrates insertion of units at specific locations to make trees appear. This causality can be quantified by comparing the presence of trees in xi and xa and averaging effects over all locations and images. Following prior work (Holland, 1988; Pearl, 2009), we define the average causal effect (ACE) of units U on the generation of on class c as:
δU→c ≡ Ez,P[sc(xi)]− Ez,P[sc(xa)], (4) where sc(x) denotes a segmentation indicating the presence of class c in the image x at P. To permit comparisons of δU→c between classes c which are rare, we normalize our segmentation sc by Ez,P[sc(x)]. While these measures can be applied to a single unit, we have found that objects tend to depend on more than one unit. Thus we wish to identify a set of units U that maximize the average causal effect δU→c for an object class c.
Finding sets of units with high ACE. Given a representation r with d units, exhaustively searching for a fixed-size set U with high δU→c is prohibitive as it has ( d |U| )
subsets. Instead, we optimize a continuous intervention α ∈ [0, 1]d, where each dimension αu indicates the degree of intervention for a unit u. We maximize the following average causal effect formulation δα→c:
Image with partial ablation at pixels P : x′a = f((1−α) rU,P, rU,P) (5) Image with partial insertion at pixels P : x′i = f(α k+ (1−α) rU,P, rU,P) Objective : δα→c = Ez,P [sc(x′i)]− Ez,P [sc(x′a)] ,
where rU,P denotes the all-channel featuremap at locations P, rU,P denotes the all-channel featuremap at other locations P, and applies a per-channel scaling vector α to the featuremap rU,P. We optimize
α over the following loss with an L2 regularization:
α∗ = argmin α (−δα→c + λ||α||2), (6)
where λ controls the relative importance of each term. We add the L2 loss as we seek a minimal set of casual units. We optimize using stochastic gradient descent, sampling over both z and featuremap locations P and clamping the coefficient α within the range [0, 1]d at each step (d is the total number of units). More details of this optimization are discussed in Section S-6.4. Finally, we can rank units by α∗u and achieve a stronger causal effect (i.e., removing trees) when ablating successively larger sets of tree-causing units as shown in Figure 4.

heading: 4 RESULTS

text:
We study three variants of Progressive GANs (Karras et al., 2018) trained on LSUN scene datasets (Yu et al., 2015). To segment the generated images, we use a recent model (Xiao et al., 2018) trained on the ADE20K scene dataset (Zhou et al., 2017). The model can segment the input image into 336 object classes, 29 parts of large objects, and 25 materials. To further identify units that specialize in object parts, we expand each object class c into additional object part classes c-t, c-b, c-l, and c-r, which denote the top, bottom, left, or right half of the bounding box of a connected component.
Below, we use dissection for analyzing and comparing units across datasets, layers, and models (Section 4.1), and locating artifact units (Section 4.2). Then, we start with a set of dominant object classes and use intervention to locate causal units that can remove and insert objects in different images (Section 4.3 and 4.4). In addition, our video demonstrates our interactive tool.

heading: 4.1 COMPARING UNITS ACROSS DATASETS, LAYERS, AND MODELS

text:
Emergence of individual unit object detectors We are particularly interested in any units that are correlated with instances of an object class with diverse visual appearances; these would suggest that GANs generate those objects using similar abstractions as humans. Figure 3 illustrates two such units. In the dining room dataset, a unit emerges to match dining table regions. More interestingly, the matched tables have different colors, materials, geometry, viewpoints, and levels of clutter: the only obvious commonality among these regions is the concept of a table. This unit’s featuremap correlates to the fully supervised segmentation model (Xiao et al., 2018) with a high IoU of 0.34.
Interpretable units for different scene categories The set of all object classes matched by the units of a GAN provides a map of what a GAN has learned about the data. Figure 5 examines units from GANs trained on four LSUN scene categories (Yu et al., 2015). The units that emerge are object classes appropriate to the scene type: for example, when we examine a GAN trained on kitchen scenes, we find units that match stoves, cabinets, and the legs of tall kitchen stools. Another striking phenomenon is that many units represent parts of objects: for example, the conference room GAN contains separate units for the body and head of a person.
Interpretable units for different network layers. In classifier networks, the type of information explicitly represented changes from layer to layer (Zeiler & Fergus, 2014). We find a similar phenomenon in a GAN. Figure 6 compares early, middle, and late layers of a progressive GAN with 14 internal convolutional layers. The output of the first convolutional layer, one step away from the input z, remains entangled: individual units do not correlate well with any object classes except for two units that are biased towards the ceiling of the room. Mid-level layers 4 to 7 have many units that match semantic objects and object parts. Units in layers 10 and beyond match local pixel patterns such as materials, edges and colors. All layers are shown in Section S-6.7.
Interpretable units for different GAN models. Interpretable units can provide insights about how GAN architecture choices affect the structures learned inside a GAN. Figure 7 compares three models from Karras et al. (2018): a baseline Progressive GANs, a modification that introduces minibatch stddev statistics, and a further modification that adds pixelwise normalization. By examining unit semantics, we confirm that providing minibatch stddev statistics to the discriminator increases not only the realism of results, but also the diversity of concepts represented by units: the number of types of objects, parts, and materials matching units increases by more than 40%. The pixelwise normalization increases the number of units that match semantic classes by 19%.

heading: 4.2 DIAGNOSING AND IMPROVING GANS

text:
While our framework can reveal how GANs succeed in producing realistic images, it can also analyze the causes of failures in their results. Figure 8a shows several annotated units that are responsible for typical artifacts consistently appearing across different images. We can identify these units efficiently by human annotation: out of a sample of 1000 images, we visualize the top ten highest activating images for each unit, and we manually identify units with noticeable artifacts in this set. It typically takes 10 minutes to locate 20 artifact-causing units out of 512 units in layer4.
More importantly, we can fix these errors by ablating the above 20 artifact-causing units. Figure 8b shows that artifacts are successfully removed, and the artifact-free pixels stay the same, improving the generated results. In Table 1 we report two standard metrics, comparing our improved images to both the original artifact images and a simple baseline that ablates 20 randomly chosen units. First, we compute the widely used Fréchet Inception Distance (Heusel et al., 2017) between the generated images and real images. We use 50, 000 real images and generate 10, 000 images with high
activations on these units. Second, we score 1, 000 images per method on Amazon MTurk, collecting 20, 000 human annotations regarding whether the modified image looks more realistic compared to the original. Both metrics show significant improvements. Strikingly, this simple manual change to a network beats state-of-the-art GANs models. The manual identification of “artifact” units can be approximated by an automatic scoring of the realism of each unit, as detailed in Section S-6.1.

heading: 4.3 LOCATING CAUSAL UNITS WITH ABLATION

text:
Errors are not the only type of output that can be affected by directly intervening in a GAN. A variety of specific object types can also be removed from GAN output by ablating a set of units in a GAN. In Figure 9 we apply the method in Section 3.2 to identify sets of 20 units that have causal effects on common object classes in conference rooms scenes. We find that, by turning off these small sets of units, most of the output of people, curtains, and windows can be removed from the generated scenes. However, not every object can be erased: tables and chairs cannot be removed. Ablating those units will reduce the size and density of these objects, but will rarely eliminate them.
The ease of object removal depends on the scene type. Figure 10 shows that, while windows can be removed well from conference rooms, they are more difficult to remove from other scenes. In particular, windows are just as difficult to remove from a bedroom as tables and chairs from a conference room. We hypothesize that the difficulty of removal reflects the level of choice that a GAN has learned for a concept: a conference room is defined by the presence of chairs, so they cannot be altered. And modern building codes mandate that all bedrooms must have windows; the GAN seems to have caught on to that pattern.
Table 1: We compare generated images before and after ablating 20 “artifacts” units. We also report a simple baseline that ablates 20 randomly chosen units.
Fréchet Inception Distance (FID)
original images 43.16 “artifacts” units ablated (ours) 27.14
random units ablated 43.17
Human preference score original images
“artifacts” units ablated (ours) 72.4% random units ablated 49.9%
ablate person units ablate curtain units
ablate table unitsablate window units ablate chair units
Figure 9: Measuring the effect of ablating units in a GAN trained on conference room images. Five different sets of units have been ablated related to a specific object class. In each case, 20 (out of 512) units are ablated from the same GAN model. The 20 units are specific to the object class and independent of the image. The average causal effect is reported as the portion of pixels that are removed in 1 000 randomly generated images. We observe that some object classes are easier to remove cleanly than others: a small ablation can erase most pixels for people, curtains, and windows, whereas a similar ablation for tables and chairs only reduces object sizes without deleting them.

heading: 4.4 CHARACTERIZING CONTEXTUAL RELATIONSHIPS VIA INSERTION

text:
We can also learn about the operation of a GAN by forcing units on and inserting these features into specific locations in scenes. Figure 11 shows the effect of inserting 20 layer4 causal door units in church scenes. In this experiment, we insert these units by setting their activation to the fixed mean value for doors (further details in Section S-6.4). Although this intervention is the same in each case, the effects vary widely depending on the objects’ surrounding context. For example, the doors added to the five buildings in Figure 11 appear with a diversity of visual attributes, each with an orientation, size, material, and style that matches the building.
We also observe that doors cannot be added in most locations. The locations where a door can be added are highlighted by a yellow box. The bar chart in Figure 11 shows average causal effects of insertions of door units, conditioned on the background object class at the location of the intervention. We find that the GAN allows doors to be added in buildings, particularly in plausible locations such as where a window is present, or where bricks are present. Conversely, it is not possible to trigger a door in the sky or on trees. Interventions provide insight on how a GAN enforces relationships between objects. Even if we try to add a door in layer4, that choice can be vetoed later if the object is not appropriate for the context. These downstream effects are further explored in Section S-6.5.

heading: 5 DISCUSSION

text:
By carefully examining representation units, we have found that many parts of GAN representations can be interpreted, not only as signals that correlate with object concepts but as variables that have a causal effect on the synthesis of objects in the output. These interpretable effects can be used to compare, debug, modify, and reason about a GAN model. Our method can be potentially applied to other generative models such as VAEs (Kingma & Welling, 2014) and RealNVP (Dinh et al., 2017).
We have focused on the generator rather than the discriminator (as did in Radford et al. (2016)) because the generator must represent all the information necessary to approximate the target distribution, while the discriminator only learns to capture the difference between real and fake images. Alternatively, we
can train an encoder to invert the generator (Donahue et al., 2017; Dumoulin et al., 2017). However, this incurs additional complexity and errors. Many GANs also do not have an encoder.
Our method is not designed to compare the quality of GANs to one another, and it is not intended as a replacement for well-studied GAN metrics such as FID, which estimate realism by measuring the distance between the generated distribution of images and the true distribution (Borji (2018) surveys these methods). Instead, our goal has been to identify the interpretable structure and provide a window into the internal mechanisms of a GAN.
Prior visualization methods (Zeiler & Fergus, 2014; Bau et al., 2017; Karpathy et al., 2016) have brought new insights into CNN and RNN research. Motivated by that, in this work we have taken a small step towards understanding the internal representations of a GAN, and we have uncovered many questions that we cannot yet answer with the current method. For example: why can a door not be inserted in the sky? How does the GAN suppress the signal in the later layers? Further work will be needed to understand the relationships between layers of a GAN. Nevertheless, we hope that our work can help researchers and practitioners better analyze and develop their own GANs.

heading: S-6 SUPPLEMENTARY MATERIAL

heading: S-6.1 AUTOMATIC IDENTIFICATION OF ARTIFACT UNITS

text:
In Section 4.2, we have improved GANs by manually identifying and ablating artifact-causing units. Now we describe an automatic procedure to identify artifact units using unit-specific FID scores.
To compute the FID score (Heusel et al., 2017) for a unit u, we generate 200, 000 images and select the 10, 000 images that maximize the activation of unit u, and this subset of 10, 000 images is compared to the true distribution (50, 000 real images) using FID. Although every such unit-maximizing subset of images represents a skewed distribution, we find that the per-unit FID scores fall in a wide range, with most units scoring well in FID while a few units stand out with bad FID scores: many of them were also manually flagged by humans, as they tend to activate on images with clear visible artifacts.
Figure 12 shows the performance of FID scores as a predictor of manually flagged artifact units. The per-unit FID scores can achieve 50% precision and 50% recall. That is, of the 20 worst-FID units, 10 are also among the 20 units manually judged to have the most noticeable artifacts. Furthermore, repairing the model by ablating the highest-FID units works: qualitative results are shown in Figure 13 and quantitative results are shown in Table 2.

heading: S-6.2 HUMAN EVALUATION OF DISSECTION

text:
As a sanity check, we evaluate the gap between human labeling of object concepts correlated with units and our automatic segmentation-based labeling, for one model, as follows.
For each of 512 units of layer4 of a “living room” Progressive GAN, 5 to 9 human annotations were collected (3728 labels in total). In each case, an AMT worker is asked to provide one or two words describing the highlighted patches in a set of top-activating images for a unit. Of the 512 units, 201 units were described by the same consistent word (such as ”sofa”, ”fireplace” or ”wicker”) in 50% or more of the human labels. These units are interpretable to humans.
Applying our segmentation-based dissection method, 154/201 of these units are also labeled with a confident label with IoU > 0.05 by dissection. In 104/154 cases, the segmentation-based model gave the same label word as the human annotators, and most others are slight shifts in specificity. For example, the segmentation labels “ottoman” or “curtain” or “painting” when a person labels “sofa” or “window” or “picture,” respectively. A second AMT evaluation was done to rate the accuracy of both segmentation-derived and human-derived labels. Human-derived labels scored 100% (of the 201 human-labeled units, all of the labels were rated as consistent by most raters). Of the 154 segmentation-generated labels, 149 (96%) were rated by most AMT raters as accurate as well.
The five failure cases (where the segmentation is confident but rated as inaccurate by humans) arise from situations in which human evaluators saw one concept after observing only 20 top-activating images, while the algorithm, in evaluating 1000 images, counted a different concept as dominant. Figure 14a shows one example: in the top images, mostly sofas are highlighted and few ceilings, whereas in the larger sample, mostly ceilings are triggered.
There are also 47/201 cases where the segmenter is not confident while humans have consensus. Some of these are due to missing concepts in the segmenter. Figure 14b shows a typical example, where a unit is devoted to letterboxing (white stripes at the top and bottom of images), but the segmentation has no confident label to assign to these. We expect that as future semantic segmentation models are developed to be able to identify more concepts such as abstract shapes, more of these units can be automatically identified.

heading: S-6.3 PROTECTING SEGMENTATION MODEL AGAINST UNREALISTIC IMAGES

text:
Our method relies on having a segmentation function sc(x) that identifies pixels of class c in the output x. However, the segmentation model sc can perform poorly in the cases where x does not resemble the original training set of sc. This phenomenon is visible when analyzing earlier GAN models. For example, Figure 15 visualizes two units from a WGAN-GP model (Gulrajani et al., 2017) for LSUN bedrooms (this model was trained by Karras et al. (2018) as a baseline in the original paper). For these two units, the segmentation network seems to be confused by the distorted images.
To protect against such spurious segmentation labels, we can use a technique similar to that described in Section S-6.1: automatically identify units that produce unrealistic images, and omit those “unrealistic” units from semantic segmentation. An appropriate threshold to apply will depend on the distribution being modeled: in Figure 16, we show how applying a filter, ignoring segmentation on units with FID 55 or higher, affects the analysis of this base WGAN model. In general, fewer irrelevant labels are associated with units.

heading: S-6.4 COMPUTING CAUSAL UNITS

text:
In this section we provide more details about the ACE optimization described in Section 3.2.
Specifying the per-class positive intervention constant k. In Eqn. 3, the negative intervention is defined as zeroing the intervened units, and a positive intervention is defined as setting the intervened units to some big class-specific constant k. For interventions for class c, we set k to be mean featuremap activation conditioned on the presence of class c at that location in the output, with each pixel weighted by the portion of the featuremap locations that are covered by the class c. Setting all units at a pixel to k will tend to strongly cause the target class. The goal of the optimization is to find the subset of units that is causal for c.
Sampling c-relevant locations P. When optimizing the causal objective (Eqn. 5), the intervention locations P are sampled from individual featuremap locations. When the class c is rare, most featuremap locations are uninformative: for example, when class c is a door in church scenes, most regions of the sky, grass, and trees are locations where doors will not appear. Therefore, we focus the optimization as follows: during training, minibatches are formed by sampling locations P that are
relevant to class c by including locations where the class c is present in the output (and are therefore candidates for removal by ablating a subset of units), and an equal portion of locations where class c is not present at P, but it would be present if all the units are set to the constant k (candidate locations for insertion with a subset of units). During the evaluation, causal effects are evaluated using uniform samples: the region P is set to the entire image when measuring ablations, and to uniformly sampled pixels P when measuring single-pixel insertions.
Initializing α with IoU. When optimizing causal α for class c, we initialize with
αu = IoUu,c
maxv IoUv,c (7)
That is, we set the initial α so that the largest component corresponds to the unit with the largest IoU for class c, and we normalize the components so that this largest component is 1.
Applying a learned intervention α When applying the interventions, we clip α by keeping only its top n components and zeroing the remainder. To compare the interventions of different classes an different models on an equal basis, we examine interventions where we set n = 20.

heading: S-6.5 TRACING THE EFFECT OF AN INTERVENTION

text:
To investigate the mechanism for suppressing the visible effects of some interventions seen in Section 4.4, in this section we insert 20 door-causal units on a sample of individual featuremap locations at layer4 and measure the changes caused in later layers.
To quantify effects on downstream features, the change in each feature channel is normalized by that channel’s mean L1 magnitude, and we examine the mean change in these normalized featuremaps at each layer. In Figure 17, these effects that propagate to layer14 are visualized as a heatmap: brighter colors indicate a stronger effect on the final feature layer when the door intervention is in the neighborhood of a building instead of trees or sky. Furthermore, we plot the average effect on every layer at right in Figure 17, separating interventions that have a visible effect from those that do not. A small identical intervention at layer4 is amplified to larger changes up to a peak at layer12.

heading: S-6.6 MONITORING GAN UNITS DURING TRAINING

text:
Dissection can also be used to monitor the progress of training by quantifying the emergence, diversity, and quality of interpretable units. For example, in Figure 18 we show dissections of layer4 representations of a Progressive GAN model trained on bedrooms, captured at a sequence of checkpoints during training. As training proceeds, the number of units matching objects increases, as does the number of object classes with matching units, and the quality of object detectors as
cloud 0.01 buildingb 0.10 glass 0.04 floor 0.11
wallb 0.13 floort 0.09 glass 0.14 wood 0.14
ceiling 0.01 bedt 0.10 window 0.18 painting 0.08
lamp 0.05 bedt 0.17 window 0.25 painting 0.11
lamp 0.05 bedt 0.20 window 0.30 painting 0.10
lamp 0.05 bedt 0.19 window 0.32 painting 0.11
lamp 0.05 bedt 0.23 window 0.30 painting 0.12
Iteration 4800 27 object units  25 part units  23 material units  IoU avg 0.045
1
5
10 un its
sky mo un tai n flo ortre e bu ildi ngsky -t clo ud tre e-b bu ildi ng -b tre e-l mo un tai n-t
po lish
ed st
on e pa int ed wo od gla ss
5 ob
je ct
s
6 pa
rts
4 m
at er
ia ls
Iteration 6400 45 object units  56 part units  32 material units  IoU avg 0.067
1
10
19 un its
ce ilin
g ea rthflo ortre e skyfen ce wa ll-t flo orr flo orb flo ort wa ll-b flo orl
bu ildi
ng -b ce ilin g-bwa ll-r bu ildi ng -l bu ildi ng -t ce ilin g-t win do wt ce ilin g-r tre e-rsky -t win do wr ce ilin g-l gla ss pa int ed
po lish
ed st
on e wo od me tal
6 ob
je ct
s
18 p
ar ts
5 m
at er
ia ls
Iteration 8000 65 object units  146 part units  10 material units  IoU avg 0.070
1
14
27 un its
ce ilin
g be d win do w flo or pa int ing bu ildi ng be d-bbe d-t be d-r ce ilin g-bbe d-l win do wt ce ilin g-t flo orb win do wb ce ilin g-r wa ll-b win do wl flo orr flo orl ce ilin g-l pa int ing -b win do wr sky -t fab ric ca rpe t gla ss wo od
6 ob
je ct
s
18 p
ar ts
4 m
at er
ia ls
Iteration 10001 70 object units  166 part units  6 material units  IoU avg 0.077
1
17
34 un its
win do w ce ilin g flo or pa int ing cu rta in pill ow be d-bbe d-t be d-l be d-r wa ll-b ce ilin g-b win do wt ce ilin g-t ce ilin g-l win do wb ce ilin g-r wa ll-l win do wl flo orr flo orb pa int ing -b flo orl win do wr sky -t wa ll-r pill ow -t flo ort ca r-t cu rta inb cu rta inr ca rpe t wo od gla ss
6 ob
je ct
s
25 p
ar ts
3 m
at er
ia ls
Iteration 15009 79 object units  182 part units  11 material units  IoU avg 0.084
1
20
39 un its
ce ilin
g
win do w cu rta in flo or pa int ing pill ow cu sh ion be d-bbe d-t be d-l be d-r win do wt wa ll-b ce ilin g-b win do wb ce ilin g-r ce ilin g-l win do wl ce ilin g-t wa ll-l cu rta int pa int ing -b wa ll-r flo ort win do wr cu rta inb flo orb flo orr bu ildi ng -l pill ow -t flo orl pa int ing -r sky -t pill ow -r pa int ing -t pa int ing -l bu ildi ng -t ca rpe t wo od
7 ob
je ct
s
30 p
ar ts
2 m
at er
ia ls
Iteration 17413 87 object units  184 part units  12 material units  IoU avg 0.086
1
18
36 un its
ce ilin
g
win do
w flo
or cu rta in pill ow pa int ing cu sh ion bu ildi ng be d-bbe d-l be d-t wa ll-bbe d-r win do wt ce ilin g-b win do wb ce ilin g-r ce ilin g-l win do wl cu rta int ce ilin g-t wa ll-l wa ll-r cu rta inb flo orb win do wr flo orr pill ow -r pill ow -t flo orl pa int ing -t cu rta inl pa int ing -r pill ow -l pa int ing -b ca rpe t wo od gla ss
8 ob
je ct
s
27 p
ar ts
3 m
at er
ia ls
Iteration 20000 95 object units  164 part units  12 material units  IoU avg 0.085
1
15
29 un its
win do w ce ilin g flo or pa int ing cu rta in pill ow cu sh ionlam p be d be d-bbe d-t be d-l wa ll-b win do wt be d-r ce ilin g-b ce ilin g-r ce ilin g-l ce ilin g-t win do wl win do wb wa ll-l cu rta int win do wr cu rta inb pa int ing -b wa ll-r flo orr pill ow -t flo orl pa int ing -r cu rta inl pa int ing -l ca rpe t gla ss
9 ob
je ct
s
24 p
ar ts
2 m
at er
ia ls
Figure 18: The evolution of layer4 of a Progressive GAN bedroom generator as training proceeds. The number and quality of interpretable units increases during training. Note that in early iterations, Progressive GAN generates images at a low resolution. The top-activating images for the same four selected units is shown for each iteration, along with the IoU and the matched concept for each unit at that checkpoint.
measured by average IoU over units increases. During this successful training, dissection suggests that the model is gradually learning the structure of a bedroom, as increasingly units converge to meaningful bedroom concepts.

heading: S-6.7 ALL LAYERS OF A GAN

text:
In Section 4.1 we show a small selection of layers of a GAN; in Figure 19 we show a complete listing of all the internal convolutional layers of that model (a Progressive GAN trained on LSUN living room images). As can be seen, the diversity of units matching high-level object concepts peaks at layer4-layer6, then declines in later layers, with the later layers dominated by textures, colors, and shapes.
  Units in layer   Unit class distribution layer1  512 units total
0 object units  2 part units  0 material units

iou=0.10ceilingt layer1 #457 iou=0.07ceilingt layer1 #194
  layer2  512 units total
97 object units  63 part units  5 material units

iou=0.23window layer2 #70 iou=0.25floor layer2 #315
  layer3  512 units total
86 object units  121 part units  10 material units

iou=0.27window layer3 #305 iou=0.24sofat layer3 #55
  layer4  512 units total
86 object units  149 part units  10 material units

iou=0.28sofa layer4 #37 iou=0.15fireplace layer4 #23
  layer5  512 units total
75 object units  153 part units  19 material units

iou=0.29sofat layer5 #190 iou=0.15paintingb layer5 #133
  layer6  512 units total
72 object units  129 part units  16 material units

iou=0.31window layer6 #393 iou=0.04bookcase layer6 #308
  layer7  256 units total
59 object units  48 part units  9 material units

iou=0.23painting layer7 #15 iou=0.07coffee tablet #247
  layer8  256 units total
51 object units  52 part units  12 material units

iou=0.17curtain layer8 #186 iou=0.17foliage layer8 #234
  layer9  128 units total
26 object units  17 part units  9 material units

iou=0.16window layer9 #89 iou=0.23wood layer9 #78
  layer10  128 units total
19 object units  8 part units  11 material units

iou=0.14carpet layer10 #53 iou=0.21glass layer10 #126
  layer11  64 units total
9 object units  1 part units  7 material units

iou=0.06sky layer11 #14 iou=0.11ceiling layer11 #49
  layer12  64 units total
8 object units  1 part units  4 material units

iou=0.23wood layer12 #26 iou=0.04sky layer12 #19
  layer13  32 units total
6 object units  0 part units  3 material units

iou=0.12carpet layer13 #13 iou=0.17wood layer13 #23

1
2
un its
cei ling
-t
1 pa
rt
1
22
43 un its
cei ling win dowsof a floo r sky cur tain
cof fee
tab le
fire pla
ce floo r-b sof a-t cei ling -t sof a-b cei ling -r win dow -b cei ling -l floo r-r floo r-l win dow -r win dow -t fire pla cer sof a-l fire pla cel sky -t car pet
8 ob
jec ts
15 p
ar ts
1 m
at er
ial
1
18
36 un its
cei ling win dowsof a
cof fee
tab le
pai ntin g cur tainfloo r fire pla ce floo r-b sof a-b cei ling -t sof a-t cei ling -b win dow -b wa ll-t floo r-r win dow -t floo r-l sof a-l wa ll-b win dow -l wa ll-l floo r-t fire pla cet cei ling -r win dow -r sof a-r
cof fee
tab le-l cei ling -l
pai ntin
g-bsky -t
cur tain
-t fire pla cel
cof fee
tab le-r
pai ntin
g-r car pet
8 ob
jec ts
27 p
ar ts
1 m
at er
ial
1
12
24 un its
cei ling win dowsof a pai ntin g floo r cur tain
cof fee
tab le
boo kca
se
fire pla
ceshe lf skysof a-t sof a-b floo r-b
win dow
-t
cei ling
-t wa ll-bsof a-l floo r-r
cei ling
-b
win dow
-b floo r-l wa ll-t
cei ling
-r
win dow
-r wa ll-l
cei ling
-l sof a-r wa ll-r
pai ntin
g-b
fire pla
cer
fire pla
cet win dow -l she lf-t cur tain -t pai ntin g-r
cof fee
tab le-r
cof fee
tab le-t
pai ntin
g-l car petgla ss
11 o
bje cts
28 p
ar ts
2 m
at er
ial s
1
15
29 un its
cei lingsof a win dow pai ntin g
boo kca
se cur tainfloo r
fire pla
ce cof fee tab le cha nde lier floo r-b cei ling -t sof a-bsof a-t cei ling -b wa ll-bwa ll-t pai ntin g-b win dow -t pai ntin g-l sof a-rwa ll-l floo r-r floo r-l cei ling -l win dow -r fire pla cel fire pla cet wa ll-r cei ling -r floo r-t win dow -b sof a-lsky -t win dow -l pai ntin g-t fire pla cer cur tain -t cof fee tab le-t pai ntin g-r cof fee tab le-l fire pla ceb car petgla ss lea the r
10 o
bje cts
32 p
ar ts
3 m
at er
ial s
1
10
20 un its
cei ling win dowsof a cur tain pai ntin g fire pla cefloo r boo kca se cus hio n mir ror skysof a-t sof a-b cei ling -t wa ll-b cei ling -b floo r-bwa ll-t win dow -t floo r-t cur tain -t pai ntin g-l pai ntin g-r floo r-r win dow -b floo r-l wa ll-r fire pla cet sof a-r win dow -l
cof fee
tab le-t
cur tain
-b fire pla ceb
cof fee
tab le-r cei ling -r cei ling -l win dow -r sky -t pai ntin g-t
cof fee
tab le-b
fire pla
cel car petgla
ss 11
o bje
cts
30 p
ar ts
2 m
at er
ial s
1
7
14 un its
cei ling win dowsof a floo r cur tain pai ntin g fire pla ce boo kca se sky
cof fee
tab le cha nde lier cus hio n car pet sof a-b cei ling -t floo r-b wa ll-b cei ling -b sof a-t pai ntin g-twa ll-l win dow -t
cof fee
tab le-t floo r-r
win dow
-b floo r-t sof a-l
cei ling
-r
win dow
-r pai ntin g-b
cof fee
tab le-r car petgla ss wo od
13 o
bje cts
18 p
ar ts
3 m
at er
ial s
1
6
11 un its
cei ling win dowsof a cur tain pai ntin g skyfloo r cus hio n lam p wa ll-t sof a-b floo r-b wa ll-b win dow -t cei ling -b cei ling -t pai ntin g-l floo r-t pai ntin g-t pai ntin g-r floo r-l win dow -r cei ling -r sof a-t win dow -b
cof fee
tab le-t
cur tain
-b
cur tain
-t pai ntin g-bsky -t sky -b car petgla ss foli agewo od
9 ob
jec ts
22 p
ar ts
4 m
at er
ial s
1
4
8
un its
cei ling cur tain win dow pai ntin g skysof a pla nt cei ling -t sof a-bfloo r-l win dow -t floo r-t floo r-bsky -t win dow -b cei ling -b cus hio n-b pai ntin g-bwo od car petgla ss
7 ob
jec ts
11 p
ar ts
3 m
at er
ial s
1
3
6
un its
cei ling win dowpla nt sky pai ntin g cur tainfloo r win dow -t floo r-b wa ll-b cei ling -t win dow -b wo od gla ss car pet
7 ob
jec ts
5 pa
rts
3 m
at er
ial s
1
3
5
un its
cei ling win dow cur tainpla nt skywa ll-bwo od gla ss
5 ob
jec ts
1 pa
rt
2 m
at er
ial s
1
2
un its
cei ling win dow cur tain skypla nt cei ling -t wo od gla ss
5 ob
jec ts
1 pa
rt
2 m
at er
ial s
1
2
3
un its
win dowcei ling cur tainwo od car pet
3 ob
jec ts
2 m
at er
ial s
Figure 19: All layers of a Progressive GAN trained to generate LSUN living room images.


summaries: This paper proposes first to measure distances, in a L2 space, between functions computed by neural networks. It then compares those distances with the parameter l2 distances of those networks, and empirically shows that the l2 parameter distance is a poor proxy for distances in the function space. Following those observations, the authors propose to use such constraint to combat catastrophic forgetting, and show some results on the permuted MNIST task. Finally, they propose the Hillber-constrained gradient descent (HCGD), a gradient descent algorithm that constraint movement in the function space, and evaluate it on a CNN (CIFAR10) and an LSTM (permuted MNIST).

summaries: This paper proposes a method for functional regularization for training neural nets, such that the sequence of neural nets during training is stable in function space. Specifically, the authors define a L2 norm (i.e., a Hilbert norm), which can be used to measure distances in this space between two functions. The authors argue that this can aid in preventing catastrophic forgetting, which is demonstrated in a synthetic multi-task variant of MNIST.   The authors also show how to regularize the gradient updates to be conservative in function space in standard stochastic gradient style learning, but with rather inconclusive empirical results.  The authors also draw upon a connection to the natural gradient.

abstractText: To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a L Hilbert space. We examine how typical networks behave in this space, and compare how parameter ` distances compare to function L distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the L/` ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the L distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through L-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.

title: Natural gradient works efficiently in learning

venue: Neural computation,

authors:
Shun-ichi Amari
Andrzej Cichocki
Howard Hua Yang

title: A new learning algorithm for blind signal separation

venue: In Advances in neural information processing systems,

authors:
Shun-Ichi Amari
Hyeyoung Park
Kenji Fukumizu

title: Adaptive method of realizing natural gradient learning for multilayer perceptrons

venue: Neural Computation,

authors:
Koby Crammer
Alex Kulesza
Mark Dredze

title: Adaptive regularization of weight vectors

authors:
Michael S Fine
Kurt A Thoroughman

title: Motor adaptation to single force pulses: sensitive to direction but insensitive to within-movement pulse placement and magnitude

venue: Journal of neurophysiology,

authors:
Raja Giryes
Guillermo Sapiro
Alexander M Bronstein

title: Deep neural networks with random Gaussian weights: a universal classification strategy

venue: IEEE Trans. Signal Processing,

authors:
Moritz Hardt
Benjamin Recht
Yoram Singer

title: Train faster, generalize better: Stability of stochastic gradient descent

venue: arXiv preprint arXiv:1509.01240,

authors:
Antti Honkela
Harri Valpola

title: On-line variational bayesian learning

venue: In 4th International Symposium on Independent Component Analysis and Blind Signal Separation,

authors:
Forrest N Iandola
Song Han
Matthew W Moskewicz
Khalid Ashraf
William J Dally
Kurt Keutzer

title: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size

venue: arXiv preprint arXiv:1602.07360,

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: In International Conference on Machine Learning,

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Yann LeCun
Ido Kanter
Sara A Solla

title: Eigenvalues of covariance matrices: Application to neural-network learning

venue: Physical Review Letters,

title: Gradient episodic memory for continual learning

venue: In Advances in Neural Information Processing Systems,

title: New perspectives on the natural gradient method

venue: arXiv preprint arXiv:1412.1193,

authors:
James Martens
Roger Grosse

title: Optimizing neural networks with kronecker-factored approximate curvature

authors:
Manfred Opper
Ole Winther

title: A bayesian approach to on-line learning

venue: On-line Learning in Neural Networks,

authors:
Razvan Pascanu
Yoshua Bengio

title: Revisiting natural gradient for deep networks

venue: arXiv preprint arXiv:1301.3584,

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

authors:
Gabriel Pereyra
George Tucker
Jan Chorowski
Łukasz Kaiser
Geoffrey Hinton

title: Regularizing neural networks by penalizing confident output distributions

venue: arXiv preprint arXiv:1701.06548,

authors:
Tapani Raiko
Harri Valpola
Yann LeCun

title: Deep learning made easier by linear transformations in perceptrons

venue: In Artificial Intelligence and Statistics,

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Georg Sperl
Christoph H Lampert

title: icarl: Incremental classifier and representation learning

venue: In Proc. CVPR,

authors:
Hippolyt Ritter
Aleksandar Botev
David Barber

title: Online structured laplace approximations for overcoming catastrophic forgetting

venue: arXiv preprint arXiv:1805.07810,

authors:
Tim Salimans
Diederik P Kingma

title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks

title: Accelerated gradient descent by factor-centering decomposition

authors:
Nicol N Schraudolph
Terrence J Sejnowski

title: Tempering backpropagation networks: Not all weights are created equal

authors:
Patrice Simard
Yann LeCun
John Denker
Bernard Victorri

title: Transformation invariance in pattern recognitiontangent distance and tangent propagation

venue: Neural networks: tricks of the trade,

authors:
Ilya Sutskever
James Martens
George Dahl
Geoffrey Hinton

title: On the importance of initialization and momentum in deep learning

venue: In International conference on machine learning,

authors:
Chong Wang
Xi Chen
Alexander J Smola
Eric P Xing

title: Variance reduction for stochastic gradient optimization

authors:
Friedemann Zenke
Ben Poole
Surya Ganguli

title: Continual learning through synaptic intelligence

venue: arXiv preprint arXiv:1703.04200,

heading: 1 INTRODUCTION

text:
A neural network’s parameters collectively encode a function that maps inputs to outputs. The goal of learning is to converge upon a good input/output function. In analysis, then, a researcher should ideally consider how a network’s input/output function changes relative to the space of possible functions. However, since this space is not often considered tractable, most techniques and analyses consider the parameters of neural networks. Most regularization techniques, for example, act directly on the parameters (e.g. weight decay, or the implicit constraints stochastic gradient descent (SGD) places upon movement). These techniques are valuable to the extent that parameter space can be taken as a proxy for function space. Since the two might not always be easily related, and since we ultimately care most about the input/output function, it is important to develop metrics that are directly applicable in function space.
In this work we show that it is relatively straightforward to measure the distance between two networks in function space, at least if one chooses the right space. Here we examine L2-space, which is a Hilbert space. Distance in L2 space is simply the expected `2 distance between the outputs of two functions when given the same inputs. This computation relies only on function inference.
Using this idea of function space, we first focus on characterizing how networks move in function space during optimization with SGD. Do random initializations track similar trajectories? What happens in the overfitting regime? We are particularly interested in the relationship between trajectories in function space and parameter space. If the two are tightly coupled, then parameter change can be taken as a proxy for function change. This common assumption (e.g. Lipschitz bounds) might not always be the case.
∗aarrii@seas.upenn.edu
Next, we demonstrate two possibilities as to how a function space metric could assist optimization. In the first setting we consider multitask learning, and the phenomenon of catastrophic forgetting that makes it difficult. Many well-known methods prevent forgetting by regularizing how much the parameters are allowed to shift due to retraining (usually scaled by a precision matrix calculated on previous tasks). We show that one can instead directly regularize changes in the input/output function of early tasks. Though this requires a ”working memory” of earlier examples, this scheme turns out to be quite data-efficient (and more so than actually retraining on examples from old tasks).
In the second setting we propose a learning rule for supervised learning that constrains how much a network’s function can change any one update. This rule, which we call Hilbert-constrained gradient descent (HCGD), penalizes each step of SGD to reduce the magnitude of the resulting step in L2-space. This learning rule thus changes the course of learning to track a shorter path in function space. If SGD generalizes in part because large changes to the function are prohibited, then this rule will have advantages over SGD. Interestingly, HCGD is conceptually related to the natural gradient. As we derive in §3.2.1, the natural gradient can be viewed as resulting from constrains changes in a function space measured by the Kullbeck-Leibler divergence.

heading: 2 EXAMINING NETWORKS IN FUNCTION SPACE

text:
We propose to examine the trajectories of networks in the space of functions defined by the inner product 〈f, g〉 = ∫ X f(x)g(x)dµ(x), which yields the following norm:
‖f‖2 = ∫ X |f |2dµ.
Here µ is a measure and corresponds to the probability density of the input distribution X. Note that this norm is over an empirical distribution of data and not over the uniform distribution of all possible inputs. The | · |2 operator refers to the 2-norm and can apply to vector-valued functions. While we refer to this space as a Hilbert space, we make no use of an inner product and can also speak of this as any normed vector space, e.g. a Banach space. This norm leads to a notion of distance between two functions f and g given by
‖f − g‖2 = ∫ X |f − g|2dµ.
Since µ is a density, ∫ X dµ = 1, and we can write
‖f − g‖2 = EX[|f(x)− g(x)|2].
The expectation can be approximated as an empirical expectation over a batch of examples drawn from the input distribution:
‖f − g‖2 ≈ 1 N N∑ i=0 |f(xi)− g(xi)|2.
The quality of the empirical distance, of course, will depend on the shape and variance of the distribution of data as well as the form of f and g. In section 2.3, we empirically investigate the quality of this estimator for reasonably sample sizes N .
2.1 DIVERGENCE OF NETWORKS IN L2-SPACE DURING TRAINING
We wish to compare at high level how networks move through parameter and function space. Our first approach is to compare a low-dimensional embedding of the trajectories through these spaces. In Figure 1, we take a convolutional neural network and train three random initializations on a 5000-image subset of CIFAR-10. By saving the parameters of the network at each epoch as well as the output on a single large validation batch, we can later compute the `2 parameter distance and the
L2 function distance between the snapshots of network at each epoch. The resulting distance matrix is then visualized as a two-dimensional embedding.
In parameter space, the networks are initialized at very different points and proceed to diverge yet further from these points. Despite this divergence, each trajectory yields a network that has learned the training data perfectly and generalizes with ∼ 50% accuracy to a test set. This illustrates the wide range of parameter settings that can be used to represent a given neural network function. The behavior of the same initializations in function space is quite different. First, note that all three initializations begin at approximately the same point in function space. This is an intriguing property of random initializations that, rather than encoding entirely random functions, random sets of parameters lead on average to the same function (for related work, see e.g. Giryes et al. (2016)). The initializations then largely follow an identical path for the initial stage of learning. Different initializations thus learn in similar manners, even if the distance between their parameters diverges. During late-stage optimization, random initializations turn from a shared trajectory and begin to diverge in L2 space. These differences underlie the general principle that L2 distances behave differently than `2 distances, and that functional regularization could assist training and reduce overfitting.

heading: 2.2 COMPARING L2 FUNCTION DISTANCE WITH `2 PARAMETER DISTANCE

text:
How well do parameter distances reflect function distances? The answer to this question is relevant for any method that directly considers the behavior of parameters. Certain theoretical analyses, furthermore, desire bounds on function distances but instead find bounds on parameter distances and relate the two with a Lipschitz constant (e.g. Hardt et al. (2015)). Thus, for theoretical analyses and optimization methods alike, it is important to empirically evaluate how well parameter distances correspond to function distances in typical situations.
We can compare these two situations by plotting a change in parameters ‖∆θ‖ against the corresponding change in the function ‖fθ − fθ+∆θ‖. In Figure 2 we display this relation for several relevant distance during the optimization of a CNN on CIFAR-10. There are three scales: the distance between individual updates, the distance between epochs, and the distance from initialization.
Note, first, that networks continue to move in function space as well as in parameter space after test error converges, which is around epoch 60. (The test error can be seen in Appendix A, along with identical plots colored by test error instead of epoch.) Their movement relative to initialization slows, but there is still large movement relative to previous iterations and previous epochs.
What changes strikingly throughout optimization is the relationship between parameter and function space. There is a qualitative difference in the ratio of parameter distances to function distances that visible at all three distance scales. Early epochs generally see larger changes in L2 space for a given change in parameters. Intriguingly, the ratio of the two distances appears to converge to a single value at late optimization, after test error saturates. This is not because the network ceases to move, as noted above. Rather, the loss landscape shifts such that this ratio become constant.
It is also clear from these plots that there is not a consistent positive correlation between the parameter and function distances between any two points on the optimization trajectory. For example, the parameter distance between successive epochs is negatively correlated with the L2 distance for most of optimization (Fig. 2b). The distance from initialization shows a clean and positive relationship, but the relationship changes during optimization. Between successive batches, L2 distance correlates with parameter distance at late epochs, but less so early in optimization when learning is quickest. Thus, at different stages of optimization, the L2/`2 ratio is often quite different.
The usage of Batch Normalization (BN) and weight decay in this analysis somewhat affects the trends in the L2/`2 ratio. In Appendix A we reproduce these plots for networks trained without BN and without weight decay. The overall message that the L2/`2 ratio changes during optimization is unchanged. However, these methods both change the scale of updates, and appear to do so differently throughout optimization, and thus some trends are different. In Appendix B, we also isolate the effect of training data, by reproducing these plots for a CNN trained on MNIST and find similar trends. Overall, the correspondence between parameter and function distances depends strongly on the context.

heading: 2.3 CONVERGENCE OF THE EMPIRICAL ESTIMATOR

text: It might be worried that since function space is of infinite dimension, one would require prohibitively many examples to estimate a function distance. However, we find that one can compute a distance between two functions with a relatively small amount of examples. Figure 3 shows how the estimated L2 distance converges with an increasing number examples. In general, we find that only a few hundred examples are necessary to converge to an estimation within a few percent.

heading: 3 APPLICATIONS

heading: 3.1 COMBATTING CATASTROPHIC FORGETTING IN AN ONLINE LEARNING TASK (WITH WORKING MEMORY)

text:
If, after having been trained on a task, a neural network is retrained on a new task, it often forgets the first task. This phenomenon is termed ’catastrophic forgetting’. It is the central difficulty of multitask training as well as applications requiring that learning be done online (especially in non-IID situations). Essentially, new information must be encoded in the network, but the the information pertinent to the previous task must not be overwritten.
Most efforts to combat catastrophic forgetting rely on restricting how much parameters can change between tasks. Elastic Weight Consolidation (EWC; Kirkpatrick et al. (2017)), for example, adds a penalty to the loss on a new task B that is the distance from the weights after learning on an earlier
task A, multiplied by the diagonal of the Fisher information matrix F (calculated on task A):
LEWC(θ) = LB(θ) + λ
2 ∑ i Fi(θi − θi,A)2
This idea is closely related to well-studied approaches to Bayesian online learning, if F is interpreted as a precision matrix (Honkela & Valpola (2003), Opper & Winther (1998)). Other similar approaches include that of Ritter et al. (2018), who use a more accurate approximation of the Fisher, and Synaptic Intelligence (SI; Zenke et al. (2017)), which discounts parameter change via a diagonal matrix in which each entry reflects the sum contribution of that parameter to the loss. Each of these method discourages catastrophic forgetting by restricting movement in parameter space between tasks, scaled by a (perhaps diagonal) precision matrix calculated on previous tasks.
Using a function space metric, it is not hard to ensure that the network’s output function on previous tasks does not change during learning. In this case, the loss for a new task B is modified to be:
L(θ) = LB(θ) + λ
2 ‖fθA − fθB‖
The regularization term is the L2 distance between the current function fθB and the function after training on task A, fθA . Since our function space metric is defined over a domain of examples, we will store a small set of previously seen examples in a working memory, as well as the output on those examples. This memory set will be used to calculate the L2 distance between the current iteration and the snapshot after training. This is a simple scheme, but novel, and we are not aware of direct precedence in the literature.
A working memory approach is employed in related work (Lopez-Paz et al. (2017); Rebuffi et al. (2017)). Note, however, that storing old examples violates the rules of strict online learning. Nevertheless, for large networks it will be more memory-efficient. EWC, for example, requires storing a snapshot of each parameter at the end of the previous task, as well as a diagonal precision matrix with as many entries as parameters. For the 2 hidden layer network with 400 nodes each that was used in the MNIST task in Kirkpatrick et al. (2017), this is 1,148,820 new parameters, or as many pixels as 1,465 MNIST images. When each layer has as many as 2,000 nodes, as in Fig. 3B of Kirkpatrick
et al. (2017), the extra stored parameters are comparable to 15,489 MNIST images. The working memory approach that is required to regularize function change from old tasks is thus comparable or cheaper in memory.

heading: 3.1.1 EMPIRICAL RESULTS

text:
We compared the performance of our approach at the benchmark task of permuted MNIST. This task requires a single MLP to learn to classify a sequence of MNIST tasks in which the pixels have been randomly permuted differently on each task. We trained an MLP with 2 hidden layers, 400 nodes each, for 10 epochs on each of 8 such permuted datasets. In Figure 4, we display how the test accuracy on the first of 8 tasks degrades with subsequent learning.
To build the working memory, we keep 1024 examples from previous tasks, making sure that the number of examples from each task is equal. We also remember the predictions on those examples at the end of training on their originating tasks. To calculate the L2 distance, we simply re-infer on the examples in working memory, and regularize the distance from the current outputs to the remembered outputs. We chose λ = 1.3 as the regularizing hyperparameter from a logarithmic grid search.
In Figure 4, we compare this method to four comparison methods. The ”ADAM” method is ADAM with a learning rate of 0.001, which nearly forgets the first task completely at the end of the 8 tasks. The ”ADAM+retrain” method is augmented with a working memory of 1024 examples that are stored from previous tasks. Every n iterations (we found n = 10 to be best), a step is taken to decrease the loss on the memory cache. This method serves as a control for the working memory concept. We also include EWC and SI as comparisons, using the hyperparameters used in their publications (λ = 500, = c = 0.1). Overall, we found that regularizing the L2 distance on a working memory cache was more successful than simply retraining on the same cache. It also outperformed EWC, but not SI. Note that these methods store diagonal matrices and the old parameters, and in this circumstance these were larger in memory than the memory cache.

heading: 3.2 CONSTRAINING CHANGES IN L2 DURING LEARNING

text:
In this section we propose that the L2 distance can be used for regularization in a single supervised task. In the space of parameters, SGD is a strongly local update rule and large jumps are generally prohibited. SGD is thus more likely to find solutions that are close to the initialization, and furthermore to trace a path of limited length. This discourages the sampling a large volume of parameter space during optimization. If the mapping between parameter and function space is not already very tight, and locality is important for generalization, then additionally constricting changes in function space should help.
On the basis of this logic, we propose a learning rule that directly constrains the path length of the optimization trajectory L2 space. If a network would have been trained to adjust the parameters θ to minimize some cost C0, we will instead minimize at each step t a new cost given by:
C = C0 + λ‖fθt − fθt+∆θ‖ (1) Like all regularization terms, this can also be viewed as a Langrangian that satisfies a constraint. Here, this constraint ensures that the change in L2-space does not exceed some constant value. To evaluate Equation 1, we can approximate the norm with an empirical expectation over X:
C = C0 + λ ( 1 N N∑ i=0 |fθt(xi)− fθt+∆θ(xi)|2 )1/2 .
This cost function imposes a penalty upon the difference between the output of the current network at time t and the proposed network at t+ 1. The data xi may derive from some validation batch but must pull from the same distribution X. It would also be possible to use unlabeled data.
We can write an update rule to minimize Equation 1 that is a modification of gradient descent. We call the rule Hilbert-constrained gradient descent (HCGD). It minimizes C in Equation 1 via an inner loop of gradient descent. To optimize C via gradient descent, we first replace C0 with its first order approximation JT∆θ, where J is the Jacobian. Thus we seek to converge to a ∆θ′ at each update step, where
∆θ′ = argmin ∆θ
( JT∆θ + λ
N N∑ i=0 |fθt(xi)− fθt+∆θ(xi)|2 )
(2)
Minimization of the proper ∆θ can be performed in an inner loop by a first order method. We first propose some ∆θ0 = − J = − ∇θC0 (for learning rate ) and then iteratively correct this proposal by gradient descent towards ∆θ′. If only one correction is performed, we simply add the derivative of the Hilbert-constraining term after ∆θ0 has been proposed. We found empirically that a single correction was often sufficient. In Appendix C, we demonstrate that this algorithm does actually decrease the distance traveled in function space, as expected. This algorithm is shown in Algorithm 1.
Algorithm 1: Hilbert-constrained gradient descent. Implements Equation 2.
Require: . Overall learning rate Require: η . Learning rate for corrective step
1: procedure 2: θ ← θ0 . Initialize parameters 3: while θt not converged do 4: draw X ∼ Px . Draw training batch 5: J ← ∇θC0(X) . Calculate gradients 6: ∆θ0 ← − J . Proposed update
7: drawXV ∼ Px . Draw validation batch 8: gL2 ← ∇∆θ (λ2 N N∑ xi∈XV |fθt(xi)− fθt+∆θ(xi)|2 )1/2 . Calculate correction
9: ∆θ′ ← ∆θ0 − η(gL2) 10: θt ← θt−1 + ∆θ′ 11: return θt
Note that the ”proposed update” is presented as an SGD step, but could be a step of another optimizer (e.g. ADAM). In the Appendix, we display an extended version of this algorithm. This version allows for multiple corrective iterations in each step. It also allows for a form of momentum. In standard momentum for SGD, one follows a “velocity” term v which is adjusted at each step with the rule v ← βv + J (e.g. see Sutskever et al. (2013)). For HCGD, we also keep a velocity term but update it with the final Hilbert-constrained update ∆θ rather than J . The velocity is used to propose the initial ∆θ0 in the next update step. We found that this modification of momentum both quickened optimization and lowered generalization error.

heading: 3.2.1 RELATION TO THE NATURAL GRADIENT

text:
The natural gradient turns out to carry a similar interpretation as HCGD, in that the natural gradient also regularizes the change in functions’ output distributions. Specifically, the natural gradient can be derived from a penalty upon the change in a network’s output distribution as measured by the Kullbeck-Leibler divergence (rather than the L2 distance).
To show this, we start with a similar goal of function regularization and will come upon the natural gradient. Let us seek to regularize the change in a network’s output distribution Pθ throughout optimization of the parameters θ, choosing the Kullbeck-Leibler (KL) divergence as a measure of similarity between any two distributions. To ensure the output distribution changes little throughout optimization, we define a new cost function
C = C0 + λDKL(Pθt+1‖Pθt) (3) where C0 is the original cost function and λ is a hyperparameter that controls the importance of this regularization term. Optimization would be performed with respect to the proposed update θt+1.
Evaluating the KL divergence directly is problematic because it is infeasible to define the output density Pθ everywhere. One can obtain a more calculable form by expanding DKL(Pθt+1‖Pθt) around θt to second order with respect to θ. The Hessian of the KL divergence is the Fisher information metric F . With ∆θ ≡ (θt+1 − θt), we can rewrite our regularized cost function as
C ≈ C0 + λ
2 ∆θTF∆θ (4)
To optimize C via gradient descent we first replace C0 with its first order approximation.
C ≈ JT∆θ + λ 2 ∆θTF∆θ (5)
At each evaluation, J is evaluated before any step is made, and we seek the value of ∆θ that minimizes Equation 5. By setting the derivative with respect to ∆θ to be zero, we can see that this value is
∆θ = 1
λ F−1J (6)
When λ = 1 this update is equal to the natural gradient. Thus, the natural gradient emerges as the optimal update when one regularizes the change in the output distribution during learning.
In Appendix E, we show how one can approximate the natural gradient with an inner first-order optimization loop, like in HCGD. We note that HCGD is computationally cheaper than the exact natural gradient. It does not require any matrix inversions, nor the calculation of separate per-example gradients. When the validation batch XV is drawn anew for each of n corrective iterations (step 8 in Algorithm 1), HCGD requires an additional two forward passes and one backwards pass for each correction, for a total of 2 + 3n passes each outer step.

heading: 3.2.2 THE NATURAL GRADIENT IN THE LITERATURE

text:
In addition to being seen as a regularizer of functional change, it in an interesting aside to note that variants of the natural gradient have appeared with many justifications. These include data efficiency, minimizing a regret bound during learning, speeding optimization, and the benefits of whitened gradients.
Amari originally developed the natural gradient in the light of information geometry and efficiency (Amari et al. (1996); Amari (1998)). If some directions in parameter space are more informative of the network’s outputs than others, then updates should be scaled by each dimension’s informativeness. Equivalently, if not all examples carry equal information about a distribution, then the update step should be modified to make use of highly informative examples. That is, we wish to find a Fisherefficient algorithm (see Amari et al. (2000)). The natural gradient uses the Fisher information matrix to scale the update by parameters’ informativeness.
There is also a connection between the natural gradient (and thus HCGD) and techniques that normalize and whiten gradients. The term F−1J , after all, simply ensures that steps are made in a parameter space that is whitened by the covariance of the gradients. Whitening the gradients thus has the effect that SGD becomes more similar to the natural gradient. It appears that many approaches to normalize and whiten activations or gradients have been forwarded in the literature (Raiko et al. (2012);Simard et al. (1998); Schraudolph & Sejnowski (1996); Crammer et al. (2009); Wang et al. (2013); LeCun et al. (1991); Schraudolph (1998); Salimans & Kingma (2016)). A similar effect is able to be learned with Batch Normalization, as well (Ioffe & Szegedy (2015)). By normalizing and whitening the gradients, or by proxy, the activations, these various methods ensure that parameter space is a better proxy for function space.

heading: 3.3 EMPIRICAL COMPARISON OF HCGD

text:
We compared HCGD and SGD on feedforward and recurrent architectures. If it is important that SGD limits changes in function space, and parameter and function space are loosely coupled, then HCGD should improve upon SGD. In all tests, we used a tuned learning rate for SGD, and then used the same learning rate for HCGD. We use values of λ = 0.5 and η = 0.02, generally about 10 times less than the principal learning rate . (For the n = 1 version, λ can be folded into the inner learning rate η. Values were chosen so that λη = 0.01.) We chose the batch size for the “validation” batch to be 256. While the examples in each “validation” batch were different than the training batch, they were also drawn from the train set. All models were implemented in PyTorch (Paszke et al. (2017)).
We tested HCGD as applied to the CIFAR-10 image classification problem. For reproducibility, we trained a Squeezenet v1.1, a convolutional neural network model with batch normalization optimized for parameter efficiency (Iandola et al. (2016)). Overall HCGD does not outperform SGD in the final learning stage when trained with the same learning rate as SGD (initial = 0.1), though it does perform better in the early stage while the learning rate is high (Figure 5). When we increase the initial learning rate to = 0.3 (red trace), the training accuracy decreases but the test accuracy is still marginally higher than SGD. Given the difference in relative performance between the high and low learning rate stages, it is possible that HCGD requires a different learning rate schedule to achieve the same level of gradient noise. HCGD thus decreases the test error at a given learning rate, but needs to be trained at a higher learning rate to achieve the same level of gradient noise.
We next tested the performance of HCGD on a recurrent task. We trained an LSTM on the sequential MNIST task, in which pixels are input one at a time. The order of the pixels was permuted to further complicate the task. We found that HCGD outperformed SGD (Figure 6. We used 1 correction step, as before, but found that using more correction steps yielded even better performance. However, HCGD underperformed ADAM. While not the ideal optimizer for this task, the fact that SGD can be improved indicates that SGD does not move as locally in function space as it should. Parameter space thus a poor proxy for function space in recurrent networks.
HCGD first proposes an update by SGD, and then corrects it, but the first update step can also be other optimizers. Since Adam worked well for the sequential MNIST task, we tested if Adam could also be improved by taking a step to penalize the change in function space. We found that this is indeed the case, and show the results as well in Figure 6. To differentiate the SGD- and Adam-based methods, we refer to in the figure as SGD+HC and Adam+HC. This combination of Adam and L2 functional regularization could help to achieve state-of-the-art performance on recurrent tasks.

heading: 4 DISCUSSION

text:
Neural networks encode functions, and it is important that analyses discuss the empirical relationship between function space and the more direct parameter space. Here, we argued that the L2 Hilbert space defined over an input distribution is a tractable and useful space for analysis. We found that networks traverse this function space qualitatively differently than they do parameter space. Depending on the situation, a distance of parameters cannot be taken to represent a proportional distance between functions.
We proposed two possibilities for how the L2 distance could be used directly in applications. The first addresses multitask learning. By remembering enough examples in a working memory to accurately
Figure 6: Results of a singlelayer LSTM with 128 hidden units trained on the sequential MNIST task with permuted pixels. Shown are the traces for SGD and Adam (both with learning rate 0.01). We then take variants of the HCGD algorithm in which the first proposed step is taken to be an SGD step (SGD+HC) or an Adam step (Adam+HC). For SGD+HC we also show the effect of introducing more iterations n in the SGD+HC step.
estimate an L2 distance, we can ensure that the function (as defined on old tasks) does not change as a new task is learned. This regularization term is agnostic to the architecture or parameterization of the network. We found that this scheme outperforms simply retraining on the same number of stored examples. For large networks with millions of parameters, this approach may be more appealing than comparable methods like EWC and SI, which require storing large diagonal matrices.
We also proposed a learning rule that reduces movement in function space during single-task optimization. Hilbert-constrained gradient descent (HCGD) constrains the change in L2 space between successive updates. This approach limits the movement of the encoded function in a similar way as gradient descent limits movement of the parameters. It also carries a similar intuition as the forgetting application: to learn from current examples only in ways that will not affect what has already been learned from other examples. HCGD can increase test performance at image classification in recurrent situations, indicating both that the locality of function movement is important to SGD and that it can be improved upon. However, HCGD did not always improve results, indicating either that SGD is stable in those regimes or that other principles are more important to generalization. This is by no means the only possibility for using an L2 norm to improve optimization. It may be possible, for example, to use the norm to regularize the confidence of the output function (e.g. Pereyra et al. (2017)). We are particularly interested in exploring if more implicit, architectural methods, like normalization layers, could be designed with the L2 norm in mind.
It interesting to ask if there is support in neuroscience for learning rules that diminish the size of changes when that change would have a large effect on other tasks. One otherwise perplexing finding is that behavioral learning rates in motor tasks are dependent on the direction of an error but independent of the magnitude of that error (Fine & Thoroughman, 2006). This result is not expected by most models of gradient descent, but would be expected if the size of the change in the output distribution (i.e. behavior) were regulated to be constant. Regularization upon behavioral change (rather than synaptic change) would predict that neurons central to many actions, like neurons in motor pools of the spinal cord, would learn very slowly after early development, despite the fact that their gradient to the error on any one task (if indeed it is calculated) is likely to be quite large. Given our general resistance to overfitting during learning, and the great variety of roles of neurons, it is likely that some type of regularization of behavioral and perceptual change is at play.
CODE AVAILABILITY
A Pytorch implementation of the HCGD optimizer can be found at https://github.com/KordingLab/hilbert-constrained-gradient-descent.

heading: ACKNOWLEDGMENTS

text: The authors would like to thank Roozbeh Farhoodi for helpful conversations, Mohammad Pezeshki for the suggestion to use the Adam optimizer to produce the proposed step within HCGD, and NIH grant number MH103910.

heading: A CIFAR-10 L2 AND `2 COMPARISON

heading: B COMPARING FUNCTION AND PARAMETER SPACES DURING MNIST OPTIMIZATION

heading: C HCGD DECREASES THE DISTANCE TRAVELED IN L2 SPACE

heading: D DETAILED HCGD ALGORITHM

text:
This version of the algorithm includes momentum. It also allows for multiple corrections.
Algorithm 2: Hilbert-constrained gradient descent. Implements Equation 7.
Require: n ≥ 1 . Number of corrective steps. May be 1. Require: . Overall learning rate Require: η . Learning rate for corrective step Require: β . Momentum
1: procedure 2: θ ← θ0 . Initialize parameters 3: v ← 0 . Initialize momentum buffer 4: while θt not converged do 5: reset dropout mask, if using 6: draw X ∼ Px . Draw training batch 7: J ← ∇θC0(X) . Calculate gradients 8: v ← βv + J 9: ∆θ0 ← −v . Proposed update
10: drawXV ∼ Px . Draw validation batch 11: gL2 ← ∇∆θ (λ2 N N∑ i=0 |fθt(xi)− fθt+∆θ(xi)|2 )1/2 12: ∆θ1 ← ∆θ0 − η(gL2) . First correction 13: v ← v + η(gL2) . Update buffer 14: for 1 < j < n do . Optional additional correc-
tions 15: gL2 ← J +∇∆θ ( λ2
N N∑ i=0 |fθt(xi)−
16: fθt+∆θj−1(xi)|2 )1/2

heading: 17: ∆θj ← ∆θj−1 − η(gL2)

heading: 18: v ← v + η(gL2)

text: 19: θt ← θt−1 + ∆θ 20: return θt

heading: E NATURAL GRADIENT BY GRADIENT DESCENT

text:
In order to better compare the natural gradient to the Hilbert-constrained gradient, we propose a natural gradient algorithm of a similar style.
Previous work on the natural gradient has aimed to approximate F−1 as best and as cheaply as possible. This is equivalent to minimizing Equation 2 (i.e. J∆θ+ λ2 ∆θ
TF∆θ) with a single iteration of a second-order optimizer. For very large neural networks, however, it is much cheaper to calculate matrix-vector products than to approximately invert a large matrix. It is possible that the natural gradient may be more accessible via an inner gradient descent, which would be performed during each update step as an inner loop.
We describe this idea at high level in Algorithm 2. After an update step is proposed by a standard optimizer, the algorithm iteratively corrects this update step towards the natural gradient. To start with a good initial proposed update, it is better to use a fast diagonal approximation of the natural gradient (such as Adagrad or RMSprop) as the main optimizer. Each additional correction requires just one matrix-vector product after the gradients are calculated. Depending on the quality of the proposed update, the number of iterations required is likely to be small, and even a small number of iterations will improve the update.
Algorithm 3: Natural gradient by gradient descent. This algorithm can be paired with any optimizer to increase its similarity to the natural gradient. Require: n . Number of corrective steps. May be 1. Require: η . Learning rate for corrective step
1: procedure 2: θ ← θ0 . Initialize parameters 3: while θt not converged do 4: ∆θ0 ← RMSprop(θt) . Use any optimizer to get proposed update 5: for i < n do . Corrective loop 6: ∆θi+1 = ∆θi − η(J + λF∆θi) . Step towards 1λF −1J 7: θ ← θ + ∆θ 8: return θt
Since the Fisher matrix F can be calculated from the covariance of gradients, it never needs to be fully stored. Instead, for an array of gradients G of size (# parameters, # examples), we can write
F∆θ = (GGT )∆θ = G(GT∆θ) (7)
The choice of G is an important one. It cannot be a vector of aggregated gradients (i.e. J), as that would destroy covariance structure and would result in a rank-1 Fisher matrix. Thus, we must calculate the gradients on a per-example basis. To compute G efficiently it is required that a deep learning framework implement forward-mode differentiation, which is currently not supported in popular frameworks.
If we choose G to be the array of per-example gradients on the minibatch, F is known as the ’empirical Fisher’. As explained in Martens (2014) and in Pascanu & Bengio (2013), the proper method is to calculate G from the predictive (output) distribution of the network, Pθ(y|x). This can be done as in Martens & Grosse (2015) by sampling randomly from the output distribution and re-running backpropagation on these fictitious targets, using (by necessity) the activations from the minibatch. Alternatively, as done in Pascanu & Bengio (2013), one may also use unlabeled or validation data to calculate G on each batch.

title: MEASURING AND REGULARIZING NETWORKS IN FUNCTION SPACE