summaries: This article extends to a kernel setting the idea of Elesedy and Zaidi to leverage a decomposition of L^2 into invariant and null-orbit subspaces. The authors claim to prove the benefit for generalisation of projecting the solution of KRR on the invariant subspace. The article mainly consists of the lemmas and proofs to reach the main theorem on generalisation bounds.

summaries: This paper proposes a theoretical analysis of the generalization benefit obtained for some classes of problems which exhibit invariance with respect to a group action. Especially, the paper focuses on kernel ridge regression in the statistical model $Y = f^*(X) + \epsilon$ and shows that using orbit-averaging on the learnt model provide strictly positive benefit whenever $f^\star$ is invariant with respect to the group action used in avering. Some decomposition of the involved RKHSs in terms of invariance is provided, and the generalization benefit is characterized in terms of the number of samples and typical quantities related to the kernel.

summaries: The paper aims at quantifying the possible benefits of incorporating invariances to kernel methods. The approach takes the hypothesis returned by kernel ridge regression and enforces invariance by orbit averaging relative to actions of a compact group, i.e., the test prediction at a point $x$ is the average of predictions over the orbit set $\\{ gx\ |\ g \in G \\}$, with $G$ denoting the group defining an invariance principle. The main contribution is a theoretical result showing that for target hypotheses which are G-invariant, i.e., $f(gx)=f(x)$ for all $g \in G$ and $x \in X$, the excess risk is strictly positive (relative to kernel ridge regression hypothesis) and the sample complexity when enforcing invariance can be much lower compared to pursuing standard kernel ridge regression learning (Theorem 5). The main idea is to observe that orbit averaging is a linear operator which defines a subspace of the original RKHS and by exploiting this orthogonal decomposition one gets a lower bound on the excess risk, similar to treatments in low-rank approximations via sub-space restricted inner products.

summaries: The authors present a theoretical generalization gap based on the invariance incorporated by the kernel. An orbit-averaged functional is defined with a set of invariant transformations G on data to produce an operator which maps a function to another function that averages the functions of the transformed data. The functional is used as the projection operator to produce the set of orthogonal functions to the invariant functions, and the generalization gap has been derived for kernel ridge regression using the effective dimensionality of those orthogonal functions.

summaries: This paper proves a benefit, in terms of expected risk, of orbit-averaging the solution of kernel ridge regression (with respect to actions by a given compact group) compared to the original solution of kernel ridge regression. The extent of this benefit depends on the "effective dimension" of the space orthogonal to the space invariant to group action, as well as what the authors call "differences in bias". Finally, the authors demonstrate a simple example in which the input domain is the unit sphere in d-dimensions, and the kernel is linear. The contributions are entirely theoretical.

abstractText: It is a commonly held belief that enforcing invariance improves generalisation. Although this approach enjoys widespread popularity, it is only very recently that a rigorous theoretical demonstration of this benefit has been established. In this work we build on the function space perspective of Elesedy and Zaidi [8] to derive a strictly non-zero generalisation benefit of incorporating invariance in kernel ridge regression when the target is invariant to the action of a compact group. We study invariance enforced by feature averaging and find that generalisation is governed by a notion of effective dimension that arises from the interplay between the kernel and the group. In building towards this result, we find that the action of the group induces an orthogonal decomposition of both the reproducing kernel Hilbert space and its kernel, which may be of interest in its own right.

title: Hints and the VC dimension

venue: Neural Computation

title: Unsupervised Learning of Invariant Representations in Hierarchical Architectures

title: Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

venue: Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

authors:
Taco S Cohen
Mario Geiger
Maurice Weiler

title: A general theory of equivariant cnns on homogeneous spaces

venue: Advances in Neural Information Processing Systems

title: Spherical cnns

venue: arXiv preprint arXiv:1801.10130

authors:
Bryn Elesedy
Sheheryar Zaidi

title: Provably Strict Generalisation Benefit for Equivariant Models

venue: Proceedings of the 38th International Conference on Machine Learning,

authors:
Adam Foster
Rattana Pukdee
Tom Rainforth

title: Improving Transformation Invariance in Contrastive Representation Learning

venue: arXiv preprint arXiv:2010.09515

authors:
B. Haasdonk
A. Vossen
H. Burkhardt

title: Invariance in Kernel Methods by Haar Integration Kernels

venue: SCIA

authors:
Arthur Jacot
Franck Gabriel
Clément Hongler

title: Neural tangent kernel: Convergence and generalization in neural networks

venue: Advances in neural information processing systems

authors:
Risi Kondor
Shubhendu Trivedi

title: On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups

title: Wide neural networks of any depth evolve as linear models under gradient descent

title: Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

authors:
Clare Lyle
Marta Kwiatkowksa
Yarin Gal

title: An analysis of the effect of invariance on generalization in neural networks

venue: In: International conference on machine learning Workshop on Understanding and Improving Generalization in Deep Learning

title: On the Benefits of Invariance in Neural Networks

venue: [cs.LG] (pages

authors:
Jonathan H Manton
Pierre-Olivier Amblard

title: A primer on reproducing kernel hilbert spaces

venue: arXiv preprint arXiv:1408.0952

title: On the Universality of Invariant Networks

authors:
Song Mei
Theodor Misiakiewicz
Andrea Montanari

title: Learning with invariances in random features and kernel models

venue: arXiv preprint arXiv:2102.13219

authors:
Youssef Mroueh
Stephen Voinea
Tomaso A Poggio

title: Learning with Group Invariant Features: A Kernel Perspective.

title: Local group invariant representations via orbit embeddings

venue: Artificial Intelligence and Statistics. PMLR

authors:
Siamak Ravanbakhsh
Jeff Schneider
Barnabas Poczos

title: Equivariance through parametersharing

venue: In: International Conference on Machine Learning. PMLR

authors:
Akiyoshi Sannai
Masaaki Imaizumi

title: Improved Generalization Bound of Group Invariant / Equivariant Deep Networks via Quotient

venue: Feature Space

authors:
Bernhard Schölkopf
Chris Burges
Vladimir Vapnik

title: Incorporating Invariances in Support Vector Learning Machines

title: Constructing invariant features by averaging techniques

venue: Proceedings of the 12th IAPR International Conference on Pattern Recognition,

title: On the existence of complete invariant feature spaces in pattern recognition

venue: In: International Conference On Pattern Recognition. Citeseer

title: Generalization error of invariant classifiers

venue: Artificial Intelligence and Statistics

title: Better, Faster Fermionic Neural Networks

venue: arXiv preprint arXiv:2011.07125

authors:
Ingo Steinwart
Andreas Christmann

title: Support Vector Machines

venue: Information science and statistics. Springer,

authors:
Ingo Steinwart
Clint Scovel

title: Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs

venue: Constructive Approximation

authors:
Marysia Winkels
Taco S Cohen

title: 3D G-CNNs for pulmonary nodule detection

authors:
Jeffrey Wood
John Shawe-Taylor

title: Representation theory and invariant neural networks

venue: Discrete applied mathematics

authors:
Huan Xu
Shie Mannor

title: Robustness and generalization

venue: Machine learning

title: Universal approximations of invariant maps by neural networks. 2018

heading: 1 Introduction

text:
Recently, there has been significant interest in models that are invariant to the action of a group on their inputs. It is believed that engineering models in this way improves sample efficiency and generalisation. Intuitively, if a task has an invariance, then a model that is constructed to be invariant ahead of time should require fewer examples to generalise than one that must learn to be invariant. Indeed, there are many application domains, such as fundamental physics or medical imaging, in which the invariance is known a priori [29, 32]. Although this intuition is certainly not new (e.g. [33]), it has inspired much recent work (for instance, see [36, 15]).
However, while implementations and practical applications abound, until very recently a rigorous theoretical justification for invariance was missing. As pointed out in [8], many prior works such as [28, 24] provide only worst-case guarantees on the performance of invariant algorithms. It follows that these results do not rule out the possibility of modern training algorithms automatically favouring invariant models, irrespective of the choice of architecture. Steps towards a more concrete theory of the benefit of invariance have been taken by [8, 20] and our work is a continuation along the path set by [8].
In this work we provide a precise characterisation of the generalisation benefit of invariance in kernel ridge regression. In contrast to [28, 24], this proves a provably strict generalisation benefit for invariant, feature-averaged models. In deriving this result, we provide insights into the structure of reproducing kernel Hilbert spaces in relation to invariant functions that we believe will be useful for analysing invariance in other kernel algorithms.
The use of feature averaging to produce invariant predictors enjoys both theoretical and practical success [17, 9]. For the purposes of this work, feature averaging is defined as training a model as normal (according to any algorithm) and then transforming the learned model to be invariant. This transformation is done by orbit-averaging, which means projecting the model on the space of invariant functions using the operator O introduced in Section 2.3. Kernel methods have a long been a mainstay of machine learning (see [30, Section 4.7] for a brief historical overview). Kernels can be viewed as mapping the input data into a potentially infinite
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
dimensional feature space, which allows for analytically tractable inference with non-linear predictors. While modern machine learning practice is dominated by neural networks, kernels remain at the core of much of modern theory. The most notable instance of this is the theory surrounding the neural tangent kernel [11], which states that the functions realised by an infinitely wide neural network belong to a reproducing kernel Hilbert space (RKHS) with a kernel determined by the network architecture. This relation has led to many results on the theory of optimisation and generalisation of wide neural networks (e.g. [14, 3]). In the same vein, via the NTK, we believe the results of this paper can be extended to study wide, invariant neural networks.

heading: 1.1 Summary of Contributions

text:
This paper builds towards a precise characterisation of the benefit of incorporating invariance in kernel ridge regression by feature averaging.
Lemma 3, given in Section 3, forms the basis of our work, showing that the action of the group G on the input space induces an orthogonal decomposition of the RKHS H as
H = H⊕H⊥ where each term is an RKHS and H consists of all of the invariant functions in H. We stress that, while the main results of this paper concern kernel ridge regression, Lemma 3 holds regardless of training algorithm and could be used to explore invariance in other kernel methods.
Our main results are given in Section 4 and we outline them here. We define the generalisation gap ∆(f, f ′) for two predictors f, f ′ as the difference in their test errors. If ∆(f, f ′) > 0 then f ′ has strictly better test performance than f . Theorem 5 describes ∆(f, f ′) for f being the solution to kernel ridge regression and f ′ its invariant (feature averaged) version and shows that it is positive when the target is invariant.
More specifically, let X ∼ µ where µ is G-invariant and Y = f∗(X) + ξ with f∗ G-invariant and E[ξ] = 0, E[ξ2] = σ2 < ∞. Let f be the solution to kernel ridge regression with kernel k and regularisation parameter ρ > 0 on n i.i.d. training examples {(Xi, Yi) ∼ (X,Y ) : i = 1, . . . , n} and let f ′ be its feature averaged version. Our main result, Theorem 5, says that
E[∆(f, f ′)] ≥ σ 2 dimeff(H⊥) + E
( √ nMk + ρ/ √ n)2
where Mk = supx k(x, x) < ∞, E ≥ 0 describes the approximation errors and dimeff(H⊥) is the effective dimension of the RKHS H⊥. For an RKHS H with kernel k the effective dimension is defined by
dimeff(H) = ∫ X k(x, y)2 dµ(x) dµ(y).
where X = suppµ. We return to this quantity at various points in the paper. It is important to note that the use of the feature averaged predictor f ′ as a comparator is without loss of generality. Any other predictor f ′′ that has test risk not larger than f ′ would satisfy the above bound, simply because this means ∆(f ′, f ′′) ≥ 0 so ∆(f, f ′′) = ∆(f, f ′)+∆(f ′, f ′′) ≥ ∆(f, f ′).1
Finally, for intuition, in Theorem 7 we specialise Theorem 5 to the linear setting and compute the bound exactly. Assumptions and technical conditions are given in Section 2 along with an outline of the ideas of Elesedy and Zaidi [8] on which we build. Related works are discussed in Section 5.

heading: 2 Background and Preliminaries

text:
In this section we provide a brief introduction to reproducing kernel Hilbert spaces (RKHS) and the ideas we borrow from Elesedy and Zaidi [8]. Throughout this paper, H with be an RKHS with kernel k. In Section 2.2 we state some topological and measurability assumptions that are needed for our proofs. These conditions are benign and the reader not interested in technicalities need take from Section 2.2 only that µ is G-invariant and that the kernel k is bounded and satisfies Eq. (1). We defer proofs to the Supplementary Material.
1To be completely clear: if, for instance, it so happens that projecting the RKHS onto a space of invariant predictors before doing KRR gives lower test risk than projecting afterwards (what we are calling feature averaging), then our result applies in that case too.

heading: 2.1 RKHS Basics

text: A Hilbert space is an inner product space that is complete with respect to the norm topology induced by the inner product. A reproducing kernel Hilbert space (RKHS) H is Hilbert space of real functions f : X → R on which the evaluation functional δx : H → R with δx[f ] = f(x) is continuous ∀x ∈ X , or, equivalently is a bounded operator. The Riesz Representation Theorem tells us that there is a unique function kx ∈ H such that δx[f ] = ⟨kx, f⟩H for any f ∈ H, where ⟨·, ·⟩H : H×H → R is the inner product on H. We identify the function k : X ×X → R with k(x, y) = ⟨kx, ky⟩H as the reproducing kernel of H. Using the inner product representation, one can see that k is positive-definite and symmetric. Conversely, the Moore-Aronszajn Theorem shows that for any positive-definite and symmetric function k, there is a unique RKHS with reproducing kernel k. In addition, any Hilbert space admitting a reproducing kernel is an RKHS. Finally, another characterisation of H is as the completion of the set of linear combinations of the form fc(x) = ∑n i=1 cik(x, xi) for c1, . . . , cn ∈ R and x1, . . . , xn ∈ X . For (many) more details, see [30, Chapter 4].

heading: 2.2 Technical Setup and Assumptions

text:
Input Space, Group and Measure Let G be a compact2, second countable, Hausdorff topological group with Haar measure λ (see [12, Theorem 2.27]). Let X be a non-empty Polish space admitting a finite, G-invariant Borel measure µ, with suppµ = X . We normalise µ(X ) = λ(G) = 1, the latter is possible because λ is a Radon measure. We assume that G has a measurable action on X that we will write as gx for g ∈ G, x ∈ X . A measurable action is one such that the map g : G × X → X is (λ ⊗ µ)-measurable. A function f : X → R is G-invariant if f(gx) = f(x) ∀x ∈ X ∀g ∈ G. Similarly, a measure µ on X is G-invariant if ∀g ∈ G and any µ-measurable B ⊂ X the pushforward of µ by the action of G equals µ, i.e. (g∗µ)(B) = µ(B). This means that if X ∼ µ then gX ∼ µ ∀g ∈ G. We will make use of the fact that the Haar measure is G-invariant when G acts on itself by either left or right multiplication, the latter holding because G is compact. Up to normalisation, λ is the unique measure on G with this property.
The Kernel and the RKHS Let k : X × X → R be a measurable kernel with RKHS H such that k(·, x) : X → R is continuous for any x ∈ X . Assume that supx∈X k(x, x) = Mk < ∞ and note that this implies that k is bounded since
k(x, x′) = ⟨kx, kx′⟩H ≤ ∥kx∥H∥kx′∥H = √ k(x, x) √ k(x′, x′) ≤ Mk.
Every f ∈ H is µ-measurable, bounded and continuous by [30, Lemmas 4.24 and 4.28] and in addition H is separable using [30, Lemma 4.33]. These conditions allow the application of [30, Theorem 4.26] to relate H to L2(X , µ) in the proofs building towards Lemma 3, given in the Supplementary Material. We assume that the kernel satisfies, for all x, y ∈ X ,∫
G k(gx, y) dλ(g) = ∫ G k(x, gy) dλ(g). (1)
Equation (1) is our main assumption and we will make frequent use of it. For Eq. (1) to hold, it is sufficient to have k(gx, y) equal to k(x, gy) or k(x, g−1y), where the latter uses compactness (hence unimodularity) of G to change variables g ↔ g−1. Highlighting two special cases: any inner product kernel k(x, x′) = κ(⟨x, x′⟩) such that the action of G is unitary with respect to ⟨·, ·⟩ satisfies Eq. (1), as does any stationary kernel k(x, x′) = κ(∥x− x′∥) with norm that is preserved by G in the sense that ∥gx− gx′∥ = ∥x− x′∥ for any g ∈ G, x, x′ ∈ X . If the norm/inner product is Euclidean, then any orthogonal representation of G will have this property.3

heading: 2.3 Invariance from a Function Space Perspective

text:
Given a function f : X → R we can define a corresponding orbit-averaged function Of : X → R with values
Of(x) = ∫ G f(gx) dλ(g).
2The set of compact groups covers almost all invariances in machine learning, including all finite groups (such as permutations or reflections), many continuous groups such as rotations or translations on a bounded domain (e.g. an image) and combinations thereof.
3An orthogonal representation of G on Rd is an action of G via orthogonal matrices, i.e. a homomorpishm G → O(d).
Of will exist whenever f is µ-measurable. Note that O is a linear operator and, from the invariance of λ, Of is always G-invariant. Interestingly, f is G-invariant only if f = Of . Elesedy and Zaidi [8] use these observations to characterise invariant functions and study their generalisation properties. In short, this work extends these insights to kernel methods. Along the way, we will make frequent use of the following (well known) facts about O. Lemma 1 ([8, Propositions 24 and 25]). A function f is G-invariant if and only if Of = f . This implies that O is a projection operator, so can have only two eigenvalues 0 and 1. Lemma 2 ([8, Lemma 1]). O : L2(X , µ) → L2(X , µ) is well-defined and self-adjoint. Hence, L2(X , µ) has the orthogonal decomposition
L2(X , µ) = S ⊕A
where S = {f ∈ L2(X , µ) : f is G invariant} and A = {f ∈ L2(X , µ) : Of = 0}. The meaning of Lemma 2 is that any f ∈ L2(X , µ) has a (unique) decomposition f = f̄ + f⊥ where f̄ = Of is G-invariant and Of⊥ = 0. A noteworthy consequence of this setup, as discussed in [8], is a provably non-negative generalisation benefit for feature averaging. In particular, for any predictor f ∈ L2(X , µ), if the target f∗ ∈ L2(X , µ) is G-invariant then the test error R(f) = EX∼µ[(f(X)− f∗(X))2] satisfies
R(f)−R(f̄) = ∥f⊥∥2L2(X ,µ) ≥ 0.
The same holds if the target is corrupted by independent, zero mean (additive) noise. 4

heading: 3 Induced Structure of H

text:
In this section we present Lemma 3, which is an analog of Lemma 2 for RKHSs. Lemma 3 shows that for any compact group G and RKHS H, if the kernel for H satisfies the assumptions in Section 2.2, then H can be viewed as being built from two orthogonal RKHSs, one consisting of invariant functions and another of those that vanish when averaged over G. Later in the paper, this decomposition will allow us to analyse the generalisation benefit of invariant predictors.
It may seem at first glance that Lemma 3 should follow immediately from Lemma 2, but this is not the case. First, it is not obvious that for any f ∈ H, its orbit averaged version Of is also in H. Moreover, in contrast with L2(X , µ), an explicit form for the inner product on H is not immediate, which means that some work is needed to check that O is self-adjoint on H. These are important requirements for the proofs of both Lemmas 2 and 3 and we establish them, along with O being continuous on H, in the Supplementary Material. The assumption that the kernel satisfies Eq. (1) plays a central role.
Lemma 3. H admits the orthogonal decomposition
H = H⊕H⊥ where H = {f ∈ H : f is G-invariant} and H⊥ = {f ∈ H : Of = 0}. Moreover, H is an RKHS with kernel
k̄(x, y) = ∫ G k(x, gy) dλ(g)
and H⊥ is an RKHS with kernel
k⊥(x, y) = k(x, y)− k̄(x, y).
Finally, k̄ is G-invariant in both arguments. As stated earlier, the perspective provided by Lemma 3 will support our analysis of generalisation. Just as with Lemma 2, Lemma 3 says that any f ∈ H can be written as f = f̄ + f⊥ where f̄ is G-invariant and Of⊥ = 0 with ⟨f̄ , f⊥⟩H = 0. As an aside, k̄ happens to qualify as a Haar Integration Kernel, a concept introduced by Haasdonk, Vossen, and Burkhardt [10]. We will see that a notion of effective dimension of the RKHS H⊥ with kernel k⊥ governs the generalisation gap between an arbitrary predictor f and its invariant version Of . This effective dimension arises from the spectral theory of an integral operator related to k, which we develop in the next section.
4The result [8, Lemma 1] is given for equivariance, of which invariance is a special case.

heading: 3.1 Spectral Representation and Effective Dimension

text:
In this section we consider the spectrum of an integral operator related to the kernel k. This analysis will ultimately allow us to define a notion of effective dimension of H⊥ that we will later see is important to the generalisation of invariant predictors. While the integral operator setup is standard, the use of this technique to identify an effective dimension of H⊥ is novel. Define the integral operator Sk : L2(X , µ) → H by
Skf(x) = ∫ X k(x, x′)f(x′) dµ(x′).
One way of viewing things is that Sk assigns to every element in L2(X , µ) a function in H. On the other hand, every f ∈ H is bounded so has ∥f∥L2(X ,µ) < ∞ and belongs to some element of L2(X , µ). We write ι : H → L2(X , µ) for the inclusion map that sends f to the element of L2(X , µ) that contains f . In the Supplementary Material we show that ι is injective, so any element of L2(X , µ) contains at most one f ∈ H. One can define Tk : L2(X , µ) → L2(X , µ) by Tk = ι ◦ Sk, and [30, Theorem 4.27] says that Tk is compact, positive, self-adjoint and trace-class. In addition, L2(X , µ) is separable by [7, Proposition 3.4.5], because X is Polish and µ is a Borel measure, so has a countable orthonormal basis. Hence, by the Spectral Theorem, there exists a countable orthonormal basis {ẽi} for L2(X , µ) such that Tkẽi = λiẽi where λ1 ≥ λ2 ≥ · · · ≥ 0 are the eigenvalues of Tk. Moreover, since ι is injective, for each of the ẽi for which λi > 0 there is a unique ei ∈ H such that ιei = ẽi and Skẽi = λiei. Now, since ιkx ∈ L2(X , µ) we have
ιkx = ∑ i ⟨ιkx, ẽi⟩L2(X ,µ)ẽi = ∑ i (Skẽi)(x)ẽi = ∑ i λiei(x)ẽi. (2)
From now on we permit ourself to drop the ι to reduce clutter. We use the above to define
j(x, y) = ⟨kx, ky⟩L2(X ,µ), j̄(x, y) = ⟨k̄x, k̄y⟩L2(X ,µ) and j ⊥(x, y) = ⟨k⊥x , k⊥y ⟩L2(X ,µ).
These quantities will appear again in our analysis of the generalisation of invariant kernel methods. Indeed, we will see later in this section that E[j⊥(X,X)] is a type of effective dimension of H⊥. Following Eq. (2), one finds the series representations given below in Lemma 4.
The reader may have noticed that our setup is very similar to the one provided by Mercer’s theorem. However, we do not assume compactness of X and so the classical form of Mercer’s Theorem does not apply. This aspect of our work is a feature, rather than a bug: the loosening of the compactness condition allows application to common settings such as X = Rn. For generalisations of Mercer’s Theorem see [31] and references therein.
Lemma 4. We have j = j̄ + j⊥.
Furthermore, let ēi = Oei and e⊥i = ei − ēi then j(x, y) = ∑ i λ2i ei(x)ei(y), j̄(x, y) = ∑ i λ2i ēi(x)ēi(y), and j ⊥(x, y) = ∑ i λ2i e ⊥ i (x)e ⊥ i (y).
Finally, the function ∑
i λ 2 i ēi ⊗ e⊥i : X × X → R with values (x, y) 7→ ∑ i λ 2 i ēi(x)e ⊥ i (y) vanishes
everywhere.
Before turning to generalisation, we describe how the above quantities can be used to define a measure effective dimension. We define dimeff(H) = E[j(X,X)] where X ∼ µ. Applying Fubini’s theorem, we find
dimeff(H) = ∑ i λ2i E[ei(X)2] = ∑ i λ2i ∥ẽi∥2L2(X ,µ) = ∑ i λ2i . (3)
The series converges by the comparison test because λi ≥ 0 and ∑
i λi = Tr(Tk) < ∞ (using Lidskii’s theorem) because Tk is trace-class. We have dimeff(H) = Tr(T 2k ) and we can think of this
(very informally) as taking L2(X , µ), pushing it through H twice using Tk and then measuring its size. Now because j = j̄ + j⊥ we get
dimeff(H) = dimeff(H) + dimeff(H⊥)
with dimeff(H⊥) = ∑ i λ2i ∥ẽ⊥i ∥2L2(X ,µ) = Tr(T 2 k )− Tr((OTk)2)
where ẽ⊥i = ιe ⊥ i . Again, very informally, this can be thought of as pushing L2(X , µ) through H⊥ twice and measuring the size of the output. In the next section we will consider the generalisation of kernel ridge regression and find that dimeff(H⊥) plays a critical role.

heading: 4 Generalisation

text: In this section we apply the theory developed in Section 3 to study the impact of invariance on kernel ridge regression with an invariant target. We analyse the generalisation benefit of feature averaging, finding a strict benefit when the target is G-invariant.

heading: 4.1 Kernel Ridge Regression

text:
Given input/output pairs {(xi, yi) : i = 1, . . . , n} where xi ∈ X and yi ∈ R, kernel ridge regression (KRR) returns a predictor that solves the optimisation problem
argmin f∈H
C(f) where C(f) = n∑
i=1
(f(xi)− yi)2 + ρ∥f∥2H (4)
and ρ > 0 is the regularisation parameter. KRR can be thought of as performing ridge regression in a possibly infinite dimensional feature space H. The representer theorem tells us that the solution to this problem is of the form f(x) = ∑n i=1 αikxi(x) where α ∈ Rn solves
argmin α∈Rn
{ ∥Y −Kα∥22 + ρα⊤Kα } , (5)
Y ∈ Rn is the standard row-stacking of the training outputs with Yi = yi and K is the kernel Gram matrix with Kij = k(xi, xj). We consider solutions of the form5 α = (K + ρI)−1Y which results in the predictor
f(x) = kx(X) ⊤(K + ρI)−1Y
where kx(X) ∈ Rn is the vector with components kx(X)i = kx(xi). We will compare the generalisation performance of this predictor with that of its averaged version
f̄ = k̄x(X) ⊤(K + ρI)−1Y ∈ H.
To do this we look at the generalisation gap.

heading: 4.2 Generalisation Gap

text:
The generalisation gap is a quantity that compares the expected test performances of two predictors on a given task. Given a probability distribution P, data (X,Y ) ∼ P and loss function l defining a supervised learning task, we define the generalisation gap between two predictors f and f ′ to be
∆(f, f ′) = E[l(f(X), Y )]− E[l(f ′(X), Y )]
where the expectations are conditional on the given realisations of f, f ′ if the predictors are random. In this paper we consider l(a, b) = (a−b)2 the squared-error loss and we will assume Y = f∗(X)+ξ for some target function f∗ where ξ is has mean 0, finite variance and is independent of X . In this case, the generalisation gap reduces to
∆(f, f ′) = E[(f(X)− f∗(X))2]− E[(f ′(X)− f∗(X))2].
Clearly, if ∆(f, f ′) > 0 then we expect strictly better test performance from f ′ than f .
5When K is a positive definite matrix this will be the only solution. If K is singular then ∃c ∈ Rn with∑ ij Kijcicj = ∥ ∑ i cikxi∥ 2 H = 0 so ∑ i cikxi is identically 0 and ∀f ∈ H we get ∑ i cif(xi) = 0 (see [18, Section 4.6.2]). Clearly, this can’t happen if H is sufficiently expressive. In any case, the chosen α is the minimum in Euclidean norm of all possible solutions.

heading: 4.3 Generalisation Benefit of Feature Averaging

text:
We are now in a position to give our main result, which is a characterisation of the generalisation benefit of invariance in kernel methods. This is in some sense a generalisation of [8, Theorem 6] and we will return to this comparison later. We emphasise that Theorem 5 holds under quite general conditions that cover many practical applications.
Theorem 5. Let the training data be {(Xi, Yi) : i = 1, . . . , n} i.i.d. with Yi = f∗(Xi) + ξi where Xi ∼ µ, f∗ ∈ L2(X , µ) is G-invariant and bounded, and {ξi : i = 1, . . . , n} are independent of each other and the {Xi}, with E[ξi] = 0 and E[ξ2i ] = σ2 < ∞. Let f = argminf∈H C(f) be the solution to Eq. (4) and let f̄ = Of ∈ H be the result of applying feature averaging to f , then the generalisation gap with the squared-error loss satisfies
E[∆(f, f̄)] ≥ E[f ∗(X)2j⊥(X,X)] + σ2 dimeff(H⊥)
( √ nMk + ρ/ √ n)2
where each term is non-negative and dimeff(H⊥) := Tr(T 2k )− Tr((OTk)2) = E[j⊥(X,X)] = ∑ α λ2α∥ẽ⊥α∥2L2(X ,µ) ≥ 0
is the effective dimension of H⊥.
Theorem 5 shows that feature averaging is provably beneficial in terms of generalisation if the mean of the target distribution is invariant. If H contains any functions that are not G-invariant then the lower bound is strictly positive. One might think that, given enough training examples, the solution f to Eq. (4) would learn to be G-invariant. Theorem 5 shows that this cannot happen unless the number of examples dominates the effective dimension of H⊥.
Recall the subspace A in Lemma 2. The role of dimeff(H⊥) mirrors that of dimA in [8, Theorem 6] and in the context of the theorem (linear models) A can be thought of as H⊥ when k is the linear kernel. In this sense Theorem 5 is a generalisation of [8, Theorem 6]. It is for this reason that we believe that, although the constant Mk in the denominator is likely not optimal, the O(1/n) rate that matches [8] is tight. We leave a more precise analysis of the constants to future work.
The second term in the numerator can be interpreted as quantifying the differences in bias. One has by the definition of j⊥, that
E[f∗(X)2j⊥(X,X)] = ∫ X f∗(y)2k⊥(x, y)2 dµ(x) dµ(y) (6)
using j⊥(x, y) = ∫ X k ⊥(t, x)k⊥(t, y) dµ(t). We also have the following proposition.
Proposition 6.∫ X f∗(y)2k⊥(x, y)2 dµ(x) dµ(y) = ∫ X f∗(y)2 ( k(x, y)2 − k̄(x, y)2 ) dµ(x) dµ(y)
For intuition, we present a simple special case of Theorem 5. In particular, the next result shows that Eq. (6) reduces to an approximation error that is reminiscent of the one in [8, Theorem 6] in a linear setting. For the rest of this section we find it helpful to refer to the action ϕ of G explicitly, writing ϕ(g)x instead of gx.
Theorem 7. Assume the setting and notation of Theorem 5. In addition, let X = Sd−1 be the unit d− 1 sphere and let µ = Unif(X ). Let G act via an orthogonal representation ϕ on X and define the matrix Φ = ∫ G ϕ(g) dλ(g). Let k(x, y) = x
⊤y be the linear kernel and suppose f∗(x) = θ⊤x for some θ ∈ Rd. Then the bound in Theorem 5 becomes
E[∆(f, f̄)] ≥ 1 ( √ n+ ρ/ √ n)2
( d− ∥Φ∥2F
d2 + (d− ∥Φ∥2F)∥θ∥22 d2(d+ 2) ) where ∥·∥F is the Frobenius norm. The first term in the parentheses is exactly dimeff(H⊥) and the second term is exactly E[f∗(X)2j⊥(X,X)].
One can confirm that the generalisation gap cannot be negative in Theorem 7 using Jensen’s inequality ∥Φ∥2F = ∥∥∥∥∫
G ϕ(g) dλ(g) ∥∥∥∥2 F ≤ ∫ G ∥ϕ(g)∥2F dλ(g) = ∫ G Tr(ϕ(g)⊤ϕ(g)) dλ(g) = Tr(I) = d
because the representation ϕ is orthgonal.
The matrix Φ in Theorem 7 can be computed analytically for various G and in the linear setting describes the importance of the symmetry to the task. For instance, in the simple case that G = Sd the permutation group on d elements and ϕ is the natural representation in terms of permutation matrices, we have Φ = 1d11
⊤ where 1 ∈ Rd is the vector of all 1s. In this case, since the target is assumed to be G-invariant, we must have θ = t1 for some t ∈ R. Specifically, Theorem 7 then asserts
E[∆(f, f̄)] ≥ (d− 1)(dt 2 + d+ 2)
d2(d+ 2)( √ n+ ρ/ √ n)2 .

heading: 5 Related Work

text:
Incorporating invariance into machine learning models is not a new idea. The majority of modern applications concern neural networks, but previous works have used kernels [10, 22], support vector machines [25] and polynomial feature spaces [26, 27]. Indeed, early work also considered invariant neural networks [33], using methods that seem to have been rediscovered in [23]. Modern implementations include invariant/equivariant convolutional architectures [4, 6] that are inspired by concepts from mathematical physics and harmonic analysis [13, 5]. Some of these models even enjoy universal approximation properties [19, 35].
The earliest attempt at theoretical justification for invariance of which we are aware is [1], which roughly states that enforcing invariance cannot increase the VC dimension of a model. Anselmi et al. [2] and Mroueh, Voinea, and Poggio [21] propose heuristic arguments for improved sample complexity of invariant models. Sokolic et al. [28] build on the work of Xu and Mannor [34] to obtain a generalisation bound for certain types of classifiers that are invariant to a finite set of transformations, while Sannai and Imaizumi [24] obtain a bound for models that are invariant to finite permutation groups. The PAC Bayes formulation is considered in [16, 17].
The above works guarantee only a worst-case improvement and it was not until very recently that Elesedy and Zaidi [8] derived a strict benefit for invariant/equivariant models. Our work is similar to [8] in that we provide a provably strict benefit, but differs in its application to kernels and RKHSs as opposed to linear models. We are careful to state that our setting does not directly reduce to that of [8, Theorem 6] for two reasons. First, [8, Theorem 6] considers G invariant linear models without regularisation. This may turn out to be accessible by a ρ → 0+ limit (the so called ridgeless limit) of Theorem 5. More importantly, linear regression is equivalent to kernel regression with the linear kernel. However, the linear kernel can be unbounded (e.g. on R), so does not meet our technical conditions in Section 2.2. We conjecture that the boundedness assumption on k can be removed, or at least with mild care weakened to hold µ-almost-surely.
Also very recently, Mei, Misiakiewicz, and Montanari [20] analyse the generalisation benefit of invariance in kernels and random feature models. Our results differ from [20] in some key aspects. First, Mei, Misiakiewicz, and Montanari [20] focus on kernel ridge regression with an invariant inner product kernel whereas we study symmetrised predictors from more general kernels. Second, they obtain an expression for the generalisation error that is conditional on the training data and in terms of the projection of the predictor onto a space of high degree polynomials, while we are able to integrate against the training data and express the generalisation benefit directly in terms of properties of the kernel and the group.

heading: 6 Discussion

text:
We have demonstrated a provably strict generalisation benefit for feature averaging in kernel ridge regression. In doing this we have leveraged an observation on the structure of RKHSs under the action of compact groups. We believe that this observation is applicable to other kernel methods too.
There are many possibilities for future work. As we remarked in the introduction, there is an established connection between kernels and wide neural networks via the neural tangent kernel. Using this connection, generalisation properties of wide, invariant neural networks might be accessible
through the techniques of this paper. Another natural extension of this paper is to equivariant (sometimes called steerable) matrix valued kernels. Approximate invariance may be handled by adding an approximation term to the bound in our main result. Finally, the ideas of this paper should also be applicable to Gaussian processes.
Acknowledgments and Disclosure of Funding We thank Sheheryar Zaidi for many helpful discussions in the early stages of this project and Yee Whye Teh for suggesting the application of [8] to kernels. Additionally, we thank Varun Kanade and Yee Whye Teh for advice and support throughout this and other projects. This work was supported in part by the UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant reference EP/L015897/1).

title: Provably Strict Generalisation Benefit for Invariance in Kernel Methods


summaries: This paper presents an approach to simultaneously train a prediction network along with a generative model to map the neural network activations back to inputs. The use cases of doing so include (1) interpreting the information encoded in an activation, (2) composing the network with itself and other pre-trained networks, which in turn provides a means to get uncertainties and to regularize the network, and (3) to assert that the network satisfies some constraints on what it can and cannot learn in the intermediate layers. The paper is evaluated on several image benchmarks and speech benchmarks and showcases all the above benefits.

summaries: This paper proposes to jointly train a generative model that maps activations back to the input space. The work demonstrates several benefits of this approach.  * Probing: It provides a way to decode the activations which enables the users to make sense out of it since the decoded activations are in the input space.  * Composing networks: The decoded activations allow to compose a DecNN model with another model with the same input type.  * Data Augmentation: The DecNN model can be composed with itself. This can be viewed as data augmentation since the decoded activations are of the same "type" as the input and retain the features that are relevant to the prediction task. * Ensemble network. The DecNN model can be recursively composed with itself to get the “Recursively Decodable Neural Network” (ReDecNN) model. This model can be viewed as an ensemble network. The paper shows the usefulness of ReDecNN for the tasks of measuring uncertainty, OOD detection, and calibration. * Constrain the activations: The decodable activations makes it possible to specify the kind of information an activation is not supposed to capture. This provides assertion-like capability to the neural network.

summaries: This work proposes DecNN, and the main goal is to allow people to interpret the intermediate neural network activations. The base model is an 8-layer MLP. The core idea is to train another generative model that takes the MLP activation as the input, and the generative model aims to decode an image in the model input space. During training, the loss of the generative model aims to reconstruct the input image given the activations of all intermediate layers. They further design ReDecNN, which recursively composes DecNN by taking the output of the generative model as the input of DecNN. They demonstrate that the generative model produces interestingly interpretable outputs to explain why the classifier makes correct or wrong predictions. They also show some promising results for out-of-distribution detection, calibration, and fairness.

summaries: This paper argues for inverting intermediate latent representations of convolutional classifiers -- predicting the original input -- as a tool for improving robustness and interpretability. The ability of the model to learn invertible intermediate hidden representations acts as a regularizer during training.  This basic premise is used in several ways. For example, a recursive version of this idea (using the reconstructed input to classify, and the intermediate hiddens of that to re-reconstruct the input, adding that reconstruction loss into the overall loss calculation) leads to an interesting form of drop out: choose which reconstructions (from which layer, and which recursive depth) to use in the final output. That can be used for enhanced forms of Monte Carlo Dropout, improving uncertainty computations over regular MC Dropout.  Another use is in placing constraints on intermediate hidden layers. For a fairness goal, such a constraint might be that, given a good classifier for some protected attribute, that classifier has poor performance on the reconstructed inputs from the original model's hidden layers. In a sense, this constrains the training of the model to lose the information that would allow the external classifier to predict the protected feature.

summaries: This paper explores jointly training generative model probes alongside a neural architecture to imbue the model with capabilities analogous to "debugging" in traditional software. By training the generative model probes to reproduce the model inputs, every intermediate activation can be decoded to input space (images). The authors use this (1) to interpret the partial progress of the model made at intermediate layers, (2) to restrict the model from using sensitive information from the inputs, and (3) as an interface to compose the model with itself and with other pretrained image models. The authors further use self-composition to produce an ensemble from a tree of connected classifiers, which in turn gives uncertainty measurements that enable (4) out-of-distribution detection and (5) calibration. Experimental evidence backs each of these contributions.

abstractText: In traditional software programs, it is easy to trace program logic from variables back to input, apply assertion statements to block erroneous behavior, and compose programs together. Although deep learning programs have demonstrated strong performance on novel applications, they sacrifice many of the functionalities of traditional software programs. With this as motivation, we take a modest first step towards improving deep learning programs by jointly training a generative model to constrain neural network activations to “decode” back to inputs. We call this design a Decodable Neural Network, or DecNN. Doing so enables a form of compositionality in neural networks, where one can recursively compose DecNN with itself to create an ensemble-like model with uncertainty. In our experiments, we demonstrate applications of this uncertainty to out-of-distribution detection, adversarial example detection, and calibration — while matching standard neural networks in accuracy. We further explore this compositionality by combining DecNN with pretrained models, where we show promising results that neural networks can be regularized from using protected features.

authors:
Guillaume Alain
Yoshua Bengio

title: Understanding intermediate layers using linear classifier probes

venue: arXiv preprint arXiv:1610.01644,

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: arXiv preprint arXiv:1409.0473,

authors:
Sören Becker
Marcel Ackermann
Sebastian Lapuschkin
Klaus-Robert Müller
Wojciech Samek

title: Interpreting and explaining deep neural networks for classification of audio

venue: signals. CoRR,

authors:
Charles Blundell
Julien Cornebise
Koray Kavukcuoglu
Daan Wierstra

title: Weight uncertainty in neural network

venue: In International Conference on Machine Learning,

authors:
Prafulla Dhariwal
Alex Nichol

title: Diffusion models beat gans on image synthesis

venue: arXiv preprint arXiv:2105.05233,

authors:
Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio

title: Density estimation using real nvp

venue: arXiv preprint arXiv:1605.08803,

title: Blitz - bayesian layers in torch zoo (a bayesian deep learing library for torch)

authors:
William Falcon
Kyunghyun Cho

title: A framework for contrastive self-supervised learning and designing a new approach

venue: arXiv preprint arXiv:2009.00104,

authors:
Reuben Feinman
Ryan R Curtin
Saurabh Shintre
Andrew B Gardner

title: Detecting adversarial samples from artifacts

venue: arXiv preprint arXiv:1703.00410,

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning

venue: In international conference on machine learning,

authors:
Yonatan Geifman
Ran El-Yaniv

title: Selective classification for deep neural networks

venue: arXiv preprint arXiv:1705.08500,

title: Selectivenet: A deep neural network with an integrated reject option

authors:
Ian J Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial networks

venue: arXiv preprint arXiv:1406.2661,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: arXiv preprint arXiv:1412.6572,

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q Weinberger

title: On calibration of modern neural networks

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: arXiv preprint arXiv:1610.02136,

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

venue: arXiv preprint arXiv:1812.04606,

authors:
Dan Hendrycks
Mantas Mazeika
Saurav Kadavath
Dawn Song

title: Using self-supervised learning can improve model robustness and uncertainty

venue: In Advances in Neural Information Processing Systems,

authors:
Jonathan Ho
Ajay Jain
Pieter Abbeel

title: Denoising diffusion probabilistic models

venue: arXiv preprint arXiv:2006.11239,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

venue: Advances in neural information processing systems,

authors:
Kimin Lee
Kibok Lee
Honglak Lee
Jinwoo Shin

title: A simple unified framework for detecting out-of-distribution samples and adversarial attacks

authors:
Shiyu Liang
Yixuan Li
R Srikant

title: Enhancing the reliability of out-of-distribution image detection in neural networks

venue: In International Conference on Learning Representations,

authors:
Shiyu Liang
Yixuan Li
Rayadurgam Srikant

title: Enhancing the reliability of out-ofdistribution image detection in neural networks

venue: arXiv preprint arXiv:1706.02690,

authors:
Loren Lugosch
Mirco Ravanelli
Patrick Ignoto
Vikrant Singh Tomar
Yoshua Bengio

title: Speech model pre-training for end-to-end spoken language understanding

authors:
Aravindh Mahendran
Andrea Vedaldi

title: Understanding deep image representations by inverting them

authors:
Jan Hendrik Metzen
Tim Genewein
Volker Fischer
Bastian Bischoff

title: On detecting adversarial perturbations

venue: arXiv preprint arXiv:1702.04267,

authors:
Norman Mu
Justin Gilmer

title: Mnist-c: A robustness benchmark for computer vision

venue: arXiv preprint arXiv:1906.02337,

authors:
Mahdi Pakdaman Naeini
Gregory Cooper
Milos Hauskrecht

title: Obtaining well calibrated probabilities using bayesian binning

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Alexandru Niculescu-Mizil
Rich Caruana

title: Predicting good probabilities with supervised learning

venue: In Proceedings of the 22nd international conference on Machine learning,

authors:
Tianyu Pang
Chao Du
Yinpeng Dong
Jun Zhu

title: Towards robust detection of adversarial examples

venue: In NeurIPS,

authors:
George Papamakarios
Eric Nalisnick
Danilo Jimenez Rezende
Shakir Mohamed
Balaji Lakshminarayanan

title: Normalizing flows for probabilistic modeling and inference

venue: arXiv preprint arXiv:1912.02762,

authors:
George Papamakarios
Theo Pavlakou
Iain Murray

title: Masked autoregressive flow for density estimation

venue: arXiv preprint arXiv:1705.07057,

authors:
Guim Perarnau
Joost Van De Weijer
Bogdan Raducanu
Jose M Álvarez

title: Invertible conditional gans for image editing

venue: arXiv preprint arXiv:1611.06355,

title: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

venue: Advances in large margin classifiers,

authors:
Robin Rombach
Patrick Esser
Bjorn Ommer

title: Network-to-network translation with conditional invertible neural networks

venue: Advances in Neural Information Processing Systems,

authors:
Chandramouli Shama Sastry
Sageev Oore

title: Detecting out-of-distribution examples with in-distribution examples and gram matrices

venue: arXiv preprint arXiv:1912.12510,

title: Deep learning in neural networks: An overview

venue: Neural networks,

authors:
Jiaming Song
Pratyusha Kalluri
Aditya Grover
Shengjia Zhao
Stefano Ermon

title: Learning controllable fair representations

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

authors:
Jiaming Song
Chenlin Meng
Stefano Ermon

title: Denoising diffusion implicit models

venue: arXiv preprint arXiv:2010.02502,

authors:
Yang Song
Stefano Ermon

title: Generative modeling by estimating gradients of the data distribution

venue: arXiv preprint arXiv:1907.05600,

authors:
Yang Song
Jascha Sohl-Dickstein
Diederik P Kingma
Abhishek Kumar
Stefano Ermon
Ben Poole

title: Score-based generative modeling through stochastic differential equations

venue: arXiv preprint arXiv:2011.13456,

authors:
Ilya Sutskever
Oriol Vinyals
Quoc V Le

title: Sequence to sequence learning with neural networks

venue: arXiv preprint arXiv:1409.3215,

authors:
Alex Tamkin
Mike Wu
Noah Goodman

title: Viewmaker networks: Learning views for unsupervised representation learning

venue: arXiv preprint arXiv:2010.07432,

title: Deep neural network ensembles

venue: In International Conference on Machine Learning, Optimization, and Data Science,

authors:
Weilin Xu
David Evans
Yanjun Qi

title: Feature squeezing: Detecting adversarial examples in deep neural networks

venue: arXiv preprint arXiv:1704.01155,

authors:
Bianca Zadrozny
Charles Elkan

title: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers

venue: In Icml,

title: Transforming classifier scores into accurate multiclass probability estimates

venue: In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Ev Zisselman
Aviv Tamar

title: Deep residual flow for out of distribution detection

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

heading: 1 Introduction

text:
Traditional hand-written computer programs are comprised of a computational graph of typed variables with associated semantic meaning. This structure enables practitioners to interact with programs in powerful ways (even if they are not the author) — such as debug code by tracing variables back to inputs, apply assertions to block errors, and compose programs together for more complex functionality. However, traditional software has its limitations: it is difficult to hand-write programs to classify images or extract sentiment from natural language. For these functionalities, deep learning and neural networks [41] have become the dominant approach [23, 2, 46].
While neural networks have made impressive progress on complex tasks, they come at a sacrifice of many of the desirable properties of traditional software. Specifically, the closest approximation to a “variable” in a neural network is an activation. Yet it is difficult to understand a neural network’s computation from an activation value, and there is little to associate an activation with semantic meaning. A practitioner cannot write assertion statements to constrain valid neural network logic: checking the values that an activation takes is usually not enough to gauge correctness nor meaning. Moreover, given multiple neural networks, composing them together requires retraining from scratch.
In this paper, we take a modest first step towards bridging the expressivity of deep learning with the engineering practicality of traditional software. We aim to uncover new ways for practitioners to build and use neural networks by leveraging compositionality. Specifically, we propose to train a neural network classifier jointly with a generative model whose role is to map the classifier’s activations back to the input, approximating invertibility. For any input, a neural network’s computation can
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
be represented by a sequence of activations (from each layer), each of which can now be mapped back to the input space. It is this insight that enables a special form of compositionality for neural networks, as inputs derived from activations can be fed back into other models.
In our experiments, we study this compositionality by (1) recursively composing a neural network with itself, creating an ensemble-like model with a measure of uncertainty that is useful for out-ofdistribution detection, adversarial example detection, and model calibration; (2) composing neural networks with pretrained models as forms of regularization and distillation; and (3) using decodable activations to discourage a neural network from using protected attributes. Throughout, we show that decodability comes at low cost as we find equivalent accuracy to a standard neural network.

heading: 2 Background

text:
Neural Networks We will focus on supervised neural networks for classification, although it is simple to extend to regression tasks. We denote a neural network by fθ where θ represents its trainable parameters. A neural network fθ maps an example x to predicted probabilities over K classes. Typically, a neural network is composed of L blocks. For example, if fθ is a multi-layer perception, each block is a linear layer followed by a non-linearity. If fθ is a ResNet [17], each layer is a residual block. In computing its prediction, a neural network fθ produces activations {h1, . . . , hL} where hl is the output of the l-th block. To train a neural network, we solve an optimization function of the form
Lnn(x; θ) = log p(y|fθ(x)) + βΩ(x; θ) (1)
using stochastic gradient descent. In Equation 1, the notation x denotes an input example and y its label. The function Ω represents an auxiliary objective such as a regularizer. The hyperparameter β > 0 is used to scale the auxiliary loss. For classification, log p(y|·) is cross entropy. Generative Models Broadly, we are given a latent variable H and an observed variable X . A generative model gφ captures the distribution p(x|h), mapping samples of the latent code h to samples
no hat
Reconstruction loss per layer (total of 4 here)
has hat
Prediction
Input
Decoded activations through reconstruction. Information is discarded (removal of hat) as activations interpolate between the input and (incorrect) prediction.
Hidden layers of the neural network,
True label
Cross entropy loss
h2 h3
fθ
gφ
h4h1
(a) Decodable Neural Networks
gφ
(b) Ensembling through Self-Composition
Continue to recurse
Parameter Sharing
Parameter SharingTreat decoded activation as input
Has hat
Uncertainty
No hat
has hat
has hat
no hat
fθ
fθ
fθ
of the observed variable x. For a deep generative model, gφ is parameterized by a neural network. In the simplest case where gφ is an autoencoder, we are also given an encoder q that maps observed samples x to latent codes h. Then, we optimize the objective
Lae(x;φ) = log p(x|gφ(q(x))) (2) where for continuous x, the expression log p(x|h) is the log Gaussian probability with mean h and fixed variance (or equivalently, mean squared error). More complex generative models might impose a prior distribution on h [22], an adversarial discriminator [29, 14], or invertibility of gφ [35].

heading: 3 Neural Networks with Decodable Activations

text:
In Section 1, we motivated our desiderata for activations that can be mapped back to the input space. We now propose to do this by “inverting” a neural network at a particular activation. Concretely, given an input x, a neural network fθ, we can compute an activation h by computing the forward pass fθ(x). Then, suppose we could construct the set:
S(h) = {x̂ : x̂ ∼ pD, fθ(x̂) = h} (3) where pD represents the empirical distribution that x is sampled from. Now, any input x̂ ∈ S(h) is a valid mapping of h back to the input space. (Trivially, x ∈ S(h).) We call this x̂ a decoding of the activation h. Unfortunately, in practice we do not know how to construct such a set S(h).
But, we do know how to (approximately) sample from S(h). We can train a generative model gφ to reconstruct the original input x from neural network activations h (recall x ∈ S(h)). Then, we can sample x ∼ gφ(h), which we know how to do, to sample from S(h). If we jointly train this generative model gφ with the neural network fθ, we can optimize for activations that both solve the classification task and decode back to inputs at once.
So far, we have only considered a single activation despite our neural network having L such activations. Although one could train a separate generative model per layer, we found it efficient and performant to share weights φ across all layers. Our joint objective is:
L(x; θ, φ) = log p(y|fθ(x)) + β 1
L L∑ l=1 log p(x|gφ(hl)), h1, ..., hL come from fθ(x) (4)
where the first term is cross entropy as in Equation 1 and the second term is a reconstruction error (Equation 2) averaged over layers. Comparing Equation 4 to Equation 1, we can interpret the generative loss as a regularization term Ω. Intuitively, in a standard neural network, the activations h1, . . . , hL are unconstrained, chosen only to minimize the loss. However, in Equation 4, the network fθ is constrained to produce activations that the generative model gφ can invert. In practice, gφ is a ResNet-18 decoder with upsampling layers (see Appendix A.1). We call this setup a Decodable Neural Network, or DecNN. Figure 1 depicts an illustration of the DecNN architecture.
We highlight that although Equation 4 requires good reconstruction, it does not enforce that reconstructed inputs f(gφ(h)) must map to the same label y as f(x). We show a few examples of this by visualizing decoded activations throughout a network in Appendix A.3.

heading: 4 Composing Decodable Neural Networks

text:
In traditional software, we can easily compose two compatible functions f1 and f2 together by f2(f1(x)) for some input x. With this as loose inspiration, we wish to similarly compose neural networks together. As motivation, we may want to do this to faciliate interaction between networks, such as for regularization or distillation. But given two image classifiers, it is not clear how to tie outputs from one neural network to inputs for the other? Both networks expect images as input.
The workaround comes from decodable activations. Since they are images of the same shape as inputs, we are free to treat them as inputs for another model. So, we can compose neural networks together using decoded activations as an intermediary: given two models f1 and f2, and an input x, we can take an activation h coming from f1(x), decode it, then provide it to the other model, f2(gφ(h)). In the following subsections, we explore two instances of this compositionality: one by composing a model with itself (i.e., f1 = f2), and the other by composing a model f1 with a pretrained one, f2. See Figure 1b,c for a graphical depiction of both compositions.

heading: 4.1 Recursive Self-Composition

text:
Our first composition will be to compose a DecNN model with itself. But if we can do that, we can also recursively compose a DecNN model with itself infinitely:
Given a decoded activation gφ(h (0) l ) for any layer l, we can feed it as input back into the classifier, fθ(gφ(h (0) l )) and maximize agreement with the true label y. Intuitively, if gφ(hl) is a good reconstruction of the input x (i.e. gφ(hl) and x belong to the same set S(hl)), then a good classifier fθ should predict the correct label with it as input. Now, we also observe that the computation fθ(gφ(h (0) l )) produces L additional activations, h(1)1 , . . . , h (1) L where we use the superscript to represent the number of compositions. We can repeat this exact process with the same intuition, decoding h(1)l and passing it to fθ to produce h (2) l . Computational limits aside, this can be repeated ad infinitum.
One interpretation of this procedure is as data augmentation: we saw in Section A.3 that decoded activations share features relevant to the prediction task (e.g. generated celebrities wear hats) but vary other irrelevant features (e.g. face shape, hair color, etc.). Notably, these generated images are not in the original dataset. Thus, training a model using decoded activations as inputs is similar to augmenting the initial dataset. The only difference is that gradients are propagated back through recursion to the inputs, which is also a form of regularization.
Admittedly, what we have described so far is intractable. Besides recursing infinitely, each activation produces L more activations, creating a tree (with root node x) with a large branching factor L. In practice, we make two adjustments — First, we limit the recursion to a max depth D. Second, instead of constructing the full graph of activations, which would have O(LD) nodes, we randomly sample “paths” in the graph. Specifically, starting at the root, we compute {h(0)1 , . . . , h (0) L } through fθ(x) and randomly choose an integer l0 ∈ [1, L]. Then, we compute {h(1)1 , . . . , h (1) L } through fθ(gφ(h (0) l0
)) i.e. we classify the l0-th decoded activation from depth 0. Again, we randomly pick l1 to compute {h(2)1 , . . . , h (2) L } through fθ(gφ(h (1) l1 )) and repeat until depth D.
Building on Equation 4, we can write this “recursive” objective as:
L̂(x; θ, φ) = log p(y|fθ(x))+ D∑ d=1 αd log p(y|fθ(gφ(h(d−1)ld−1 )))+β D∑ d=1 αd log p(x|gφ(h(d−1)ld−1 )) (5)
where the sequence of integers l0, . . . lD−1 are each sampled uniformly from [1, L]. We introduce a new hyperparameter α ∈ [0, 1] that geometrically downweights later depths (if α = 0, Equation 5 reduces to maximum likelihood). The first term in Equation 5 is the usual classification objective. The second term is a sum of cross entropy terms that encourage the decoded activation at depth d to predict the correct label through fθ. The third and final term is a sum of reconstruction losses that encourage the decoded activation at depth d to approximate the original input x.
We call this model a “Recursively Decodable Neural Network”, abbreviated ReDecNN. Note that ReDecNN is a special case of an ensemble network. That is, every path sampled in the activation tree by picking l0, . . . lD−1 builds a legitimate classifier. There are O(LD) such classifiers implicit in the ReDecNN. We can interpret optimizing Equation 5 as training individual classifiers that are Monte Carlo sampled from the tree every gradient step. See Figure 1b for an illustration.
Our goal in the next few paragraphs is to utilize this ensemble to measure uncertainty.
Method MNIST Fashion CelebA
Standard 97.1 (0.11) 87.7 (0.19) 90.8 (<0.1) DecNN 97.8 (0.17) 88.8 (0.19) 90.8 (<0.1) ReDecNN 97.5 (0.20) 88.1 (0.16) 90.8 (0.16) Dropout 95.9 (0.13) 80.7 (0.24) 88.5 (0.12) MC-Dropout 91.3 (0.14) 78.6 (0.23) 87.8 (0.19) BayesNN 96.3 (0.51) 87.0 (0.36) 88.2 (0.22) Ensemble 97.6 (0.11) 88.4 (0.12) 90.9 (0.11)
(a) Classification performance on test set.
Method MNIST Fashion CelebA
ReDecNN 0.869 0.795 0.692 MC-Dropout 0.878 0.818 0.657 BayesNN 0.537 0.574 0.587 Ensemble 0.531 0.559 0.605
(b) ROC-AUC of separating correctly classified and misclassified examples using Eq. 6 as a score. The stdev. for all entries over 3 runs are <0.01.
Measuring Uncertainty As a first step, we compare ReDecNN to several baselines: a standard neural network, a neural network with dropout, two kinds of Bayesian neural networks — Monte
Carlo dropout [11], abbreviated MC-Dropout, and weight uncertainty [4], abbreviated BayesNN — and a naive ensemble network where we train D copies of the standard neural network with different initializations. For all experiments, see Appendix A.1 and A.2 for training and task details. Table 1a shows accuracies over a held-out test set, averaged over three runs. While we observe lower accuracy from Bayesian NNs, we observe (perhaps surprisingly) equal performance of DecNN and ReDecNN to a standard neural network. Typically, adding uncertainty to neural networks comes at a cost to performance (as is true for other baselines), but this seems to not be the case here.
The next step is to compare the quality of model uncertainty. For the ReDecNN, for a given input x, we measure uncertainty as follows: sample N different classifiers from the activation tree, and make predictions with each classifier on x. The uncertainty is the entropy of the predictions:
Uncertainty(x) = − K∑ c=1 p̂(y = c) log p̂(y = c) (6)
where p̂(y = c) is the empirical probability (e.g. normalized count) of predicting class c out of N classifiers. A higher uncertainty metric would represent more disagreement between classifiers. We can compute the same metric for a naive ensemble, as well as for MC-Dropout (where repeated calls drop different nodes) and BayesNN (by sampling weights from the learned posterior).
We want to test if the model’s uncertainty is useful. One way is to correlate the uncertainty with when the model makes prediction errors (we call this “misclassification”). It would be useful if the model was less uncertain on correct predictions and more uncertain on incorrect ones. A practitioner could then use uncertainty to anticipate when a model might make a mistake. Table 1b reports the area under the ROC curve, or ROC-AUC of separating correctly and incorrectly classified examples in the test set (using uncertainty as a score). A higher ROC-AUC, closer to 1, represents a better separation. We find that ReDecNN is competitive with MC-Dropout, even out performing it on CelebA. In Section A.5, we include examples of images with low and high uncertainty.
Out-of-Distribution Detection A second application of uncertainty is detecting out-of-distribution (OOD) inputs. Given an input that is far from the training distribution, a model is likely to make a mistake. Instead, the model should recognize its uncertainty, and refuse to make a prediction. There is a rich literature on detecting OOD inputs using downstream computation on a trained model [18, 25, 24, 52, 19]. Unlike those works, we study OOD detection using uncertainty alone.
We explore three kinds of OOD examples: (1) adversarial examples crafted for a single classifier using FGSM [15], (2) examples from a different dataset (e.g. train on MNIST and use FashionMNIST as OOD) as done in [25], and (3) corrupted examples by 14 image transformations (e.g. adding pixel noise), borrowed from [31]. See Appendix A.4 for expanded corruption results.
Table 2 reports the ROC-AUC of separating inlier examples taken from the test split of the dataset the models were trained on, and outlier examples. From Table 2, we observe ReDecNN is just under MC-Dropout, but while MC-Dropout achieves these results at the cost of classification accuracy, ReDecNN does not (Table 1a). Furthermore, we find ReDecNN generalizes better to CelebA, a more complex image dataset, where it outperforms MC-Dropout (and other baselines).
Focusing on CelebA, we study a domain-specific OOD challenge by holding out all images with positive annotations for a single attribute as out-of-distribution. Here, inlier and outlier examples are very similar in distribution. We report ROC-AUCs in Table 3 for three held-out attributes (randomly chosen): is the celebrity in the image “wearing a hat”, has “blonde hair”, or “bald”? As this is a more challenging problem (since inlier and outlier examples are from the same dataset versus different datasets), the performance is lower than in Table 2. But while MC-Dropout, BayesNN, and a naive ensemble all perform near chance, ReDecNN is consistently near 0.6.
Calibration A third application of uncertainty we explore is calibration of predicted probabilities. Standard neural networks are notoriously over-confident and are incentived in training to predict with extreme probabilities (e.g. 0.99). A model with useful uncertainty would make predictions with proper probabilities. Although many calibration algorithms exist [33, 16], we want to measure how calibrated each model is out-of-the-box to compare the quality of model uncertainties.
Table 4a reports the expected calibration error, or ECE [32], which approximates the difference in expectation between confidence and accuracy using discrete bins. A lower number (closer to 0) represents a more calibrated model. We observe that while a standard neural network has a relatively high ECE, many of the approaches (dropout, MC-Dropout, BayesNN, and ReDecNN) reduce ECE. On the other hand, naive ensembles increase the calibration error, as they are more prone to overfitting. In two of three datasets, ReDecNN achieves the lowest ECE whereas MC-Dropout achieves the lowest in the third. These results should be viewed in parallel to Table 1a, which measures “sharpness”. Otherwise, we can trivially reduce ECE to zero by ignoring the input.
Method MNIST Fashion CelebA
Standard 1.18 (0.10) 0.56 (0.14) 2.75 (0.13) ReDecNN 0.33 (0.05) 0.21 (0.03) 1.92 (0.13) Dropout 0.63 (0.09) 0.48 (0.09) 2.51 (0.13) MC-Dropout 0.41 (0.06) 0.13 (0.03) 2.99 (0.15) BayesNN 0.91 (0.20) 0.18 (0.03) 2.41 (0.19) Ensemble 1.80 (0.12) 0.89 (0.07) 3.12 (0.12)
(a) We compare the expected calibration error (ECE) of ReDecNN to a standard neural network with various regularization and ensembling.
Depth Acc. OOD (Fashion) OOD (CelebA)
2 97.7 0.665 0.752 4 97.6 0.788 0.804 6 97.7 0.805 0.868 8 97.5 0.812 0.893
(b) Effect of recursion depth D on classification accuracy and OOD detection for a ReDecNN model (8-layer MLP) trained on MNIST.
Effect of Depth Finally, we study the effect of depth on the efficacy of ReDecNN. For the experiments above, we chose the recursive depth to be equal to the number of layers in the MLP (D = L = 8). In Table 4b, we vary the depth from 2 to 8 (while keeping the number of layers fixed to 8) and study the effect on classification accuracy and OOD detection. We find increasing OOD performance as D increases, although with marginally decreasing gains. Moreover, we observe constant accuracy, matching standard neural networks regardless of the choice of D.

heading: 4.2 Composing with Pretrained Models

text:
Apart from self-composition, we can also compose our neural networks with off-the-shelf pretrained models as a form of regularization or distillation. Suppose we have a pretrained model m that maps an input x to a class prediction. We can modify the ReDecNN objective as follows:
L̃(x; θ, φ) = L̂(x; θ, φ) + D∑ d=1 αd log p(fθ(gφ(h (d−1) ld−1 ))|m(gφ(h(d−1)ld−1 ))) (7)
where L̂ is as defined in Equation 5. The additional term acts as a divergence, bringing the classifer’s predictions close to those of the pretrained model. A similar edit can be made to the DecNN. We revisit Section 4 experiments now using two pretrained models — a linear classifier or a ResNet-18 classifier. For each, we optimize Equation 7 and compute performance on OOD detection.
Table 5 compares the results of composing with either (1) a linear model, (2) a residual network, or (3) no composition (None). Using a linear model, we observe a slight drop in test accuracy but an increase across all OOD experiments and calibration. In fact, these new results rival or surpass MC-Dropout from Table 2. Conversely, with ResNet-18, we observe a 1 point increase in test
accuracy but notable drops in OOD and calibration results. These polarizing outcomes show two use cases of pretrained models: using simpler models allow for regularization that trades off accuracy for robustness while more complex models encourage distillation, making the opposing tradeoff.
Method Eq. 7 Direct Reg.
Accuracy 97.17 96.72 Misclass. 0.904 0.828 Adversarial 0.805 0.707 OOD (Fashion) 0.845 0.699 OOD (CelebA) 0.928 0.782 Corruption (Mean) 0.790 0.667 Calibration (ECE) 0.296 0.280
as well. Intuitively, constraining the model to be similar to what a pretrained model thinks of the reconstruction (which is a dynamic value that varies with model weights) is a stronger learning signal than what the pretrained model thinks of the original input (which is static). In other words, regularizing fθ(gφ(hk)) to be like m(x) at all layers l is too strict of a constraint.

heading: 5 Constraining Decodable Computation

text:
A milestone for deep learning would be the ability to specify what information an activation should not capture. This is especially important in cases when we have protected attributes that should not be abused in optimizing an objective. While this milestone remains out of grasp, we take a small step towards it by optimizing a neural network to “be indifferent” with respect to a protected attribute through composition with pretrained models.
Suppose we have access to a pretrained classifierm that predicts the assignment for aK-way protected feature. Then, building on Section 4.2, we can optimize our classifier to “ignore” information about the protected feature through m. To do this, we define the objective:
L̃(x; θ, φ) = L̂(x; θ, φ) + D∑ d=1 αd log p(m(gφ(h (d−1) ld−1 ))| 1 K 1K) (8)
where 1K is a K-dimensional vector of ones. That is, Equation 8 encourages the decoded activation gφ(h (d−1) ld−1
), when passed through the protected classifier m, to predict chance probabilities i.e. have no information about the protected attribute. A high performing classifier trained in this manner must have solved the prediction task without abusing the protected attribute(s).
To test this, we extract two attributes from CelebA: “beardedness” and “baldness”. Suppose we wish to design a neural network to predict beardedness. If we were to naively do so, there could be a discrepancy in performance between groups of individuals — for example, between bald individuals and individuals with hair, the latter group is more common in the CelebA dataset. Indeed, Table 7 shows a 10 point difference in F1 and a 5 point difference in average precision (AP) in classifying beardedness between the two groups. The second row of Table 7 provides a baseline ReDecNN
optimized without knowledge of the protected attribute. Unsurprisingly, we find a similar discrepancy between groups, as with a standard neural network. In the third row, we evaluate a ReDecNN that was optimized to ignore the “baldness” attribute using a pretrained classifier for baldness. Critically, we find much more balanced (and higher) F1, AP, and ECE across the two groups.
Method F1
Standard 0.383 ReDecNN (Eq. 5) 0.371 ReDecNN (Eq. 8) 0.108

heading: 6 Generalizing to other Modalities

text: While we have focused on image classification, the proposed approach is more general and can be extended to other modalities. We apply the same decodable representations to speech, in particular utterance and action recognition. We compare DecNN and ReDecNN to the same baselines, measuring accuracy and uncertainty through similar experiments as we did for images.

heading: 7 Related Work

text:
Autoencoders Although DecNNs are reminiscent of autoencoders, there are differences: while autoencoders only reconstruct the final hidden layer, DecNN has a reconstruction for every single hidden layer. This, and because DecNNs have a supervised objective (which autoencoders do not have) critically changes what information is captured in the hidden layers. Whereas the autoencoder seeks perfect reconstruction, the DecNN does not, as evidenced by loss of information for reconstructions later in the network (see Appendix A.3). Finally, we point out that the applications of autoencoders are very different from the DecNN. The former is not used for calibration or composition.
Invertible and Ensemble Networks The DecNN is akin to invertible generative models [6, 36] as DecNN is training a generative model to “invert” the classifier up to a hidden layer. However, unlike invertible flows, DecNN does not impose an explicit prior on the activations (i.e. latent variables), and further, DecNN has a supervised objective. Our proposed recursive network, ReDecNN, has similarities to Bayesian neural networks [11, 4] and ensemble networks [48] — two baselines we compare against for evaluating model uncertainty. However, we find ReDecNN to be easier to train than doing inference over weights, and cheaper in parameter count than naive ensembles.
Probing Neural Networks A related line of work seeks to probe neural network post-training to understand the underlying computation [1]. Most relevant to our work is an approach that inverts CNN activations through reconstruction [28]. Unlike this approach, we are not proposing any additional computation post-hoc. Rather, we are interested in optimizing neural networks such that their representations are more easily mapped back to the input domain.
Robustness and Protected Attributes Out-of-distribution detection [26, 24, 52, 40, 20, 19], selective classification [12, 13], adversarial perturbation detection [49, 10, 30, 34], and neural network calibration [16, 51, 50, 32, 38] each have a rich subfield of algorithms. While task-specific methods likely outperform our results, we are excited about representations that enable uncertainty without task information or additional algorithms. Our approach also shares similarities to [42] where the authors minimize the mutual information between learned representations and a protected attribute. We view Equation 8 as an approximation of this where we treat the pretrained classifier for the protected attribute as a proxy for the mutual information.

heading: 8 Limitations and Future Work

text:
Method MNIST Fashion CelebA
Standard 13.6 (0.3) 13.5 (0.2) 82.6 (6.2) DecNN 80.5 (6.8) 99.2 (9.6) 486.2 (12.9) ReDecNN 96.7 (9.7) 78.6 (6.6) 521.6 (9.2) Dropout 13.9 (0.6) 14.0 (0.1) 79.5 (5.7) MC-Dropout 13.9 (0.6) 14.0 (0.1) 79.5 (5.7) Ensemble 109.6 (7.1) 132.6 (8.5) 659.4 (8.2)
mizing recursive networks, although parameter efficient, costs more compute as backpropagation is more expensive. See Table 10 for timings of 1 epoch in seconds. The proposed method does have significant computational overhead, averaging about 6x cost, while baseline methods like Dropout and MC-Dropout impose little to no overhead. On the other hand, ReDecNN has comparable cost to DecNN despite requiring recursive gradients. Also, both DecNN and ReDecNN are both cheaper than naive ensembles, significantly in the CelebA case. Future work could explore weight sharing across layers (not just depth) to reduce compute.
In summary, we explored building neural networks with decodable representations, and leveraged this new property to compose models together. By re-purposing decoded activations as novel inputs, we were able to join neural networks together in useful ways for out-of-distribution detection and calibration, as well as influence what information a network retains in optimization. We are optimistic about promising results and look to future research for more applications of decodability.
Acknowledgments and Disclosure of Funding
We thank the Ermon group for their comments and suggestions. We thank the reviewers for their many iterations of feedback that helped this paper improve significantly. This research was supported in part by NSF (#1651565, #1522054, #1733686), ONR (N000141912145), ONR MURI (N000141612007), AFOSR (FA95501910024), ARO (W911NF-21-1-0125), and Sloan Fellowship. MW is supported by the Stanford Interdisciplinary Graduate Fellowship as the Karr Family Fellow.

title: Improving Compositionality of Neural Networks by Decoding Representations to Inputs