## EQUIVARIANT NEURAL NETWORKS AND PIECEWISE LINEAR REPRESENTATION THEORY

JOEL GIBSON, DANIEL TUBBENHAUER AND GEORDIE WILLIAMSON

Abstract. Equivariant neural networks are neural networks with symmetry. Motivated by the theory of group representations, we decompose the layers of an equivariant neural network into simple representations. The nonlinear activation functions lead to interesting nonlinear equivariant maps between simple representations. For example, the rectified linear unit (ReLU) gives rise to piecewise linear maps. We show that these considerations lead to a filtration of equivariant neural networks, generalizing Fourier series. This observation might provide a useful tool for interpreting equivariant neural networks.

## Contents

| 1.         | Introduction                                                     |   1 |
|------------|------------------------------------------------------------------|-----|
| 2.         | Equivariant neural networks and piecewise linear representations |   3 |
| 3.         | Example: cyclic groups                                           |  17 |
| References | References                                                       |  23 |

## 1. Introduction

Neural networks provide flexible and powerful means of approximating a function. In many applications, one wants to learn a function that is invariant or equivariant with respect to some symmetries. A prototypical example is image recognition, where problems are often invariant under translation. Equivariant neural networks provide a flexible framework for learning such invariant or equivariant functions.

Equivariant neural networks can be studied using the mathematical theory of representation theory. (The mathematical concept of representation is different from the typical meaning of 'representation' in machine learning. In this paper we exclusively use the term in the mathematical sense.) In representation theory, simple representations provide the irreducible atoms of the theory. A main strategy in representation theory is to take a problem, decompose it into simple representations, and study the problem on these basic pieces separately. As we will see, this doesn't quite work for equivariant neural networks: their nonlinear nature allows for interaction between simple representations, which is impossible in the linear world.

However, we will argue in this paper that decomposing the layers of an equivariant neural network into simple representations is still a very interesting thing to do. We are led naturally to the study of piecewise linear maps between simple representations and piecewise linear representation theory . In concrete terms, the decomposition into simple representations leads to a new basis of the layers of a neural network, generalizing the Fourier transform. We hope that this new basis provides a useful tool to understand and interpret equivariant neural networks.

1A. What we prove. Before diving into the main results of our paper, we give a simple but nontrivial example of our main observations.

Consider the small vanilla neural network (we often omit labels):

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-c9732073-280e-4f88-8da6-f87e3e9d88d9.jpg)

As usual, each node represents a copy of R , each arrow is labeled by a weight w , and the result of each linear map between layers is composed with a nonlinear activation function f before proceeding to the next layer.

The key motivation for building equivariant neural networks is to replace R and w by more complicated objects with more symmetry. For example, consider the replacement

R ⇝ a suitable space Fun of functions on R ,

2020

Mathematics Subject Classification.

Key words and phrases.

Primary: 20C05; Secondary: 05E10, 68T07.

Equivariant neural networks, representation theory, piecewise linear maps.

↦

w ⇝ a convolution operator c ∗ : Fun → Fun , f ⇝ an activation function Fun → Fun with γ → f ◦ γ.

We can depict this as:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-84931cc0-bb06-4bd7-b798-daacf0620472.jpg)

The actual implementation of this structure on a computer would be impossible, but bear with us! Now assume for a moment that our function is periodic of period 2 π . It is very natural to ask what happens to our neural network (1A.1) when we expand it in terms of Fourier series. A fundamental result in Fourier theory is that convolution operators become diagonal in the Fourier basis. Hence, in order to understand how signals flow through the neural network in (1A.1), it remains to understand how the activation function acts on the fundamental frequencies.

A basic, but key, observation is that the Fourier series ∑ N ∈ Z c N e i 2 π N/P x ( ) of f (sin( x )) only involves terms of higher resonant frequency :

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-0bff9073-0d88-41c9-9ab5-fe228ca2f085.jpg)

↦

Here 'Order N ' means the function x → c -N e i 2 π ( -N/P x ) + c N e i 2 π N/P x ( ) .

This shows the first few Fourier series terms (small, red) of f (sin( x )) (large, blue) when f is the rectifier linear unit ReLU. What one observes is, while the Fourier series of sin( x ) is trivial, the one of f (sin( x )) has infinitely many nonzero c -N , c N corresponding to summands of higher and higher frequency as N →∞ . This is very similar to what happens when we pluck a string on a guitar: one has a fundamental frequency corresponding to the note played, as well as higher frequencies ('overtones', similar to the bottom three pictures above) which combine to give the distinctive timbre of the guitar.

We show in general that in equivariant neural networks one has a flow of lower frequency to higher resonant frequency, but not conversely :

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-d22e32bf-12a8-49a2-b505-e7c8a70e0673.jpg)

This has two concrete consequences for equivariant neural networks:

- (i) The majority of the complexity of equivariant neural networks occurs at high frequency.
- (ii) If one would like to learn a low frequency function, then one can ignore a large part of the neural network corresponding to high frequency.

For example, a prototypical flow-type diagram, that we will call an interaction graph , for equivariant neural networks built on C 8 ∼ = Z / 8 Z , the cyclic group of order 8, is (the colors are for readability only)

,

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-ee9a86cc-05ce-42db-bf29-e4aee412108e.jpg)

where the nodes are the simple representations of C 8 , and the values in the nodes indicates the action of a generator. In this picture, 'lower frequency' simple representations are depicted at the top, and again one has a flow of information from low to high frequency, but not conversely. Thus, the same principal as in (2G.1) implies that high frequency will dominate the picture in large networks.

## 1B. Main contributions. Our main results are:

- (i) We point out that it is meaningful and interesting to decompose an equivariant neural network into simple representations.
- (ii) We argue that equivariant neural networks must be built from permutation representations. See Theorem 2E.7.
- (iii) We prove that the existence of equivariant maps that are piecewise linear (but not linear) is controlled by normal subgroups in fashion similar to Galois theory. See Theorem 2H.3.
- (iv) We compute several examples, showing the richness of theory even in 'easy' examples like cyclic groups. See e.g. Section 3.

Remark 1B.1 . All the code that was used for this project can be found following the links on [ GTW23 ]. Moreover, some of the examples in Section 3 can be run in a browser, see the second link in [ GTW23 ]. That page also contains additional examples such as dihedral and symmetric groups and their piecewise linear representation theory. We encourage the reader to explore these examples alongside the reading of this paper.

- 1C. Related work. Various forms of equivariant neural networks have become basic tools in machine learning. Standard references in the machine learning literature include [ CW16 ], [ BBCV21 , §5.2] and [ GAC 23 + ]. We will not survey the extensive literature here, but instead refer the reader to [ BBCV21 , §7] and [ GAC 23 + , §1.6] for historical discussion. For excellent short introductions to equivariant neural networks intended for mathematicians, see [ Est20 ] and [ LN23 ].

Many of the results of this paper have already appeared in some form in the machine learning literature. In particular, our discussion of the basic structure of equivariant neural networks, and the fact that all equivariant maps are convolutions (Lemma 2E.3) follows the existing literature closely. Our intention is to give a motivated introduction for mathematicians. To the best of our knowledge, our explanation of the importance of permutation representations (Theorem 2E.7), our general approach of decomposing into simple representations and considering piecewise linear maps, and our classification of such maps in terms of normal subgroups (Theorem 2H.3) are new.

Acknowledgments. We are thankful to Anne Dranowski, Andrew Dudzik, Georg Gottwald, Henry Kvinge, Sébastien Racanière, and Petar Veličković for feedback and enlightening conversations. We thank the reviewer for a careful reading and valuable feedback. D.T. was sponsored by the ARC Future Fellowship FT230100489, and they thank rotating pizza slices for inspiration.

## 2. Equivariant neural networks and piecewise linear representations

We now give a (partially informal) account of what we understand by equivariant neural networks and by piecewise linear representation theory. However, before we get to these topics we need to review a few basics from the mathematical theory of group representations.

2A. Representation theory crash course. Representation theory is the study of linear symmetries. The emphasis here is on symmetries, e.g. spaces with a group action. A thorough introduction can be found in many textbooks such as [ Ben98 ], [ FH91 ], [ CR62 ] or [ Ser77 ].

Notation 2A.1. Unless otherwise stated, we use the real numbers R as our ground field, all R -vector spaces are finite dimensional, and the groups that we consider are finite.

A finite dimensional representation ϕ of a finite group G is a way of writing it down as matrices, i.e. a homomorphism ϕ : G → GL n ( R ) . Equivalently, such a representation is an R -vector space V with a linear action · of G such that g · ( h · v ) = gh · v holds for all g, h ∈ G and v ∈ V . A subrepresentation is a subspace W ⊂ V which is G -stable in the sense that g · w ∈ W for all g ∈ G and w ∈ W .

A G -equivariant linear map ϕ : V → W between two G -representations is a linear map satisfying ϕ g ( · v ) = g ϕ v · ( ) for all g ∈ G and v ∈ V . Such a map is an isomorphism if it is a bijection, and two G -representations are equivalent if these is a G -equivariant linear isomorphism between them. A G -representation V is simple if the only G -subrepresentations of V are 0 and V itself. These are the elements or primes of the theory. Indeed, by the Jordan-Hölder theorem and Maschke's theorem: every G -representation is isomorphic to a direct sum of simple G -representations, and the summands that appear are unique up to permutation and isomorphism. Finally, Schur's lemma says that all G -equivariant linear maps between two simple real G -representations are either invertible or zero.

Representation theory is a powerful tool to study linear algebra problems with symmetry. For example, representation theorists run the following procedure:

- (i) Assume you have a G -equivariant linear map ϕ : V → W . This map might be given by some complicated matrix M ϕ .
- (ii) Now decompose V and W into simple G -representations. This induces a base-change on M ϕ and one gets the matrix N ϕ = PM P ϕ -1 .
- (iii) Schur's lemma then implies that N ϕ is typically much simpler (e.g. it is a diagonal matrix if every simple representation occurs only once). Now calculations might simplify drastically.

This is illustrated in (2G.1).

The following is our main source of G -representations (the reason will become clear in Theorem 2E.7 below).

Definition 2A.2. Let X be a (finite) left G -set. The R -vector space Fun ( X, R ) of R -valued maps on X has a G -action: given any function f , we obtain a new function g · f via

$$( g \cdot f ) ( x ) = f ( g ^ { - 1 } \cdot x ).$$

(The inverse is there to ensure that this is a left action.) We will call representations Fun ( X, R ) arising in this way permutation representations .

Permutation representations give a large class of G -representations.

Example 2A.3. Taking X = G in Definition 2A.2 gives the regular G -representation R = Fun ( G, R ) . ✸ 2B. Neural networks. We assume the reader is somewhat familiar with the classical theory of neural networks as e.g. in [ GBC16 ] or [ Nie15 ]. In the simplest incarnation, the picture is:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-e2fa83ab-5938-4b24-ae34-09a293f26f84.jpg)

.

The nodes represent basis vectors v i and the arrows represent linear maps. (For simplicity of exposition we ignore the 'bias' term as they are typically absent in equivariant neural networks.) For example, the piece

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-1eb01359-ae64-49fe-99ed-a7fb55e3acb2.jpg)

represents a linear map R 4 → R 3 . Labeling each arrow with a real number w j called the weight expresses the image of a basis vector as a linear combination, using the weights as coefficients, of the basis vectors in the next layer. This is not the complete story however. To bring nonlinearity into the neural network, one applies a nonlinear map between each linear map, the so-called activation map f , so that the picture at each layer really is:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-ce3b6824-6bee-42b4-85b5-e5bb6fb8dad4.jpg)

.

In formulas, the output of such a neural network is

$$A _ { m } f \left ( A _ { m - 1 } f \left ( \dots f ( A _ { 1 } v ) \right ) \right ),$$

↦

where v is the input vector and the A i are the matrices of weights. As an example, a popular choice for the activation map is f = ReLU : x → max( x, 0) .

One can think of neural networks as providing a large space of interesting functions, parameterized by the space of all possible weights. One then trains the neural network to approximate a target function via variants of gradient descent over the space of weights, see for example [ GBC16 Nie15 , ].

- 2C. Equivariant neural networks: first example. The starting point of this paper is that one often is interested in learning a map that is equivariant with respect to some symmetry . For example:
- (i) Image recognition is often invariant under translation; for example, whether ice cream is present in an image should not depend on the position of the ice cream in the image.
- (ii) Text-to-speech translation is often invariant under shift of time; for example, 'ice cream' should be labeled as such regardless of when it appears in an audio recording.
- (iii) Many problems in engineering and applied mathematics involve the analysis of point clouds. Here one is often interested in qualitative assessments of the collection of points, independent of ordering. In other words, such problems are invariant under permuting the points. Thus our learning problem is invariant under the symmetric group.

In order to explain how to build equivariant neural networks we start with a simplistic toy example of a convolutional neural networks of periodic images . For us such a periodic image consists of a choice of a real number for each point on an n -byn grid. Interpolating the real numbers as grayscale values we get a picture (for n = 10 ):

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-8dbcac06-8d7e-497c-b537-09ba003b4678.jpg)

We identify bottom and top, as well as left and right, which makes the picture periodic, so the picture lives on a torus. Let C n = Z /n Z be the cyclic group of order n and C 2 n = C n × C n . Said in a mathematical way, a periodic image is an element of the R -vector space

$$V = \text{Fun} ( C _ { n } ^ { 2 }, \mathbb { R } )$$

of maps from the group C 2 n to R . In this model of periodic images, V is a C 2 n -representation as in Section 2A. Indeed, given ( a, b ) ∈ C 2 n and f ∈ V we obtain a new periodic image by shifting coordinates:

$$\left ( ( a, b ) \cdot f \right ) ( x, y ) = f ( x + a, y + b ).$$

In other words, translation of a periodic images products a new periodic image, e.g.:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-01eb38c3-73a8-467f-81a9-e66ba644994a.jpg)

A first crucial observation leading to equivariant neural networks is: whereas the R -vector space of all linear maps from V to V is of dimension n 4 , the R -vector space of all C 2 n -equivariant linear maps is of dimension n 2 . We now consider a typical example of a C 2 n -equivariant map. For c = ∑ g ∈ C 2 n s g g ∈ R = Fun ( G, R ) one gets an C 2 -equivariant map V → V by a convolution-type formula:

n

↦

$$c \colon V \to V, f \mapsto \left ( x \mapsto \sum _ { g \in C _ { n } ^ { 2 } } s _ { g } f ( x + g ) \right ).$$

↦

For example, take c = 1 4 ( (1 , 0) + (0 , 1) + ( -1 0) + (0 , , -1) ) . Then c · f is the periodic image whose value at pixel ( a, b ) is the average of the values of its neighbors ( a +1 , b ) , ( a, b +1) ( , a -1 , b ) and ( a, b -1) . Explicitly (filling the squares with the gray scale values for readability):

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-a7af73c5-8872-4ba5-a622-8adaacfd0aa1.jpg)

More generally, convolution for different c provides examples of maps used widely in image processing, see for example [ Jäh05 , Section I.2].

Now we can define a C 2 n -equivariant neural network in this case. Its structure is as follows:

Input layer

Hidden layer 1

Hidden layer 2

Output layer

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-e65cfa84-c836-4948-934b-8f0188f92dcd.jpg)

.

Each arrow is a convolution c k ij with k denoting the layer and i and j being the usual matrix indexes. Moreover, W is typically R or V . The above picture is a (simplified) picture of a convolutional neural network , which are of great importance in the machine learning literature, see for example [ GBC16 , Section II.9]. The main conceptual points to note with respect to this construction are:

- (i) The structure of the neural network forces the resulting map V → W to be equivariant.
- (ii) The space of all weights is much smaller than for traditional ('fully connected') neural networks. In practice this means that equivariant neural networks can handle much larger examples than 'vanilla'

neural networks. (This phenomenon is often termed weight sharing in the machine learning literature.)

We stress that the picture above implicitly includes the activation map, with our favorite choice being ReLU. This means that the compositions in neural networks are actually piecewise linear maps. Thus, in order to apply our second main observation to equivariant neural networks - reduction of problems by decomposition into simples - the study of piecewise linear representation theory comes up naturally. Before going there we will recall piecewise linear maps next.

- 2D. Piecewise linear and affine maps. Recall that a piecewise linear map f : R → R is a map whose graph is composed of straight-line segments. The generalization of this to arbitrary R -vector spaces needs a little preparation.

Definition 2D.1. Let V be an R -vector space. A subset A ⊂ V is convex polyhedral if it is the intersection of finitely many closed half-spaces in V , and has open interior. A polyhedral covering A of a subset P ⊂ V is a finite collection of convex polyhedral subsets ( A , 1 ... , A m ) such that ⋃ m i =1 A i = P .

Recall that a map f : A → R n for A ⊂ R n is affine linear if it is of the form f ( x ) = M x ( ) + b for some linear map M : R m → R n and bias b ∈ R n .

Definition 2D.2. A map f : V → W between R -vector spaces V and W is called piecewise linear if there exists a polyhedral covering A = ( A , 1 ... , A m ) of V such that the restriction of each f to A i is affine linear.

Piecewise linear maps are the main players in this paper.

Lemma 2D.3. We have the following.

- (a) Piecewise linear maps are continuous.
- (b) If f : U → V , g : V → W and h : X → Y are piecewise linear, then so are g ◦ f and f ⊕ h .
- (c) The inverse of an invertible piecewise linear map is piecewise linear.

Proof. Easy and omitted.

□

It follows that linear maps are piecewise linear, and composition preserves the property of being piecewise linear, see Lemma 2D.3. Here are more examples:

Example 2D.4. Key examples for this paper are the ReLU (rectified linear unit) and the absolute value maps given by

ReLU

:

R

→

R

,

Abs

:

R

→

R

,

x

↦

→

max(

x,

0)

↭

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-87394089-1d0c-4e2e-b1b3-3db814f53202.jpg)

,

x

↦

→

max(

x,

-

x

)

↭

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-f09b27ca-f905-4bf5-b949-81d94d70544e.jpg)

.

We extend these piecewise linear maps componentwise to maps R n → R n , and Lemma 2D.3 shows that these extension are still piecewise linear. ✸

Example 2D.5. In order to provide more intuition on piecewise linear maps, we give a two-dimensional example. Let f : R 2 → R be the map f ( x, y ) = | sgn( x x ) +sgn( ) y y | , where sgn is the sign of a real number. Plotting this map in R 3 or as a contour (that is, using level sets) in R 2 gives

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-9a09a5bb-21b6-4f5e-9ead-47d272304083.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-d7b5108d-ba8e-42b0-bfc8-28757a79193d.jpg)

Another example is g : R 2 → R given by

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-0cfa6bed-e48e-4a83-a1e3-b5388661a32d.jpg)

.

-

-

-

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-c16ac13c-6c04-4bca-a5f5-b7aa5aad784b.jpg)

-

3

-

2

-

1

0

1

2

3

.

-

-

-

Suppose V is n -dimensional. Then any affine linear map on V is determined by its values on any n + 1 points which do not lie on a hyperplane. This implies the following:

Lemma 2D.6. A piecewise linear map f : V → W with polyhedral covering A = ( A , 1 ... , A m ) is determined on each A i by evaluating f on any ( n +1 )-vectors in A i which do not lie in a hyperplane.

Notation 2D.7. We write PL : R → R to denote any nonlinear piecewise linear map, for example, any of the maps in Example 2D.4. In contrast, we use f for any piecewise linear map, including the linear ones.

We will often consider based G -representations ( V, B ) , that is, G -representations V together with a choice of basis B . We often simply write V instead of ( V, B ) if the choice of basis is clear. This is motivated by the following. Given any vector space V with fixed basis and PL as above, we get a piecewise linear endomorphism of V by evaluating PL componentwise:

Definition 2D.8. Fix a basis B = { b 1 , ... , b n } ⊆ V of the R -vector space V . Define the piecewise linear map PL B : V → V by PL B ( ∑ i λ b i i ) := ∑ i PL ( λ i ) b i . (We often omit the subscript B if no confusion can arise.)

The map ReLU B is insensitive to the order of the basis, and the scaling of each basis element by a positive number. Conversely, given only the map ReLU B : V → V , we can determine B , up to permutation and scaling by positive numbers, by looking at the convex polyhedron on which PL B restricts to the identity map.

- 2E. Equivariant neural networks. In this section we will define what we mean by equivariant neural networks. The definition is modeled following the example given in Section 2C. As discussed in the introduction, equivariant neural networks are standard in machine learning. Standard references include [ CW16 ], [ BBCV21 , §5.2] and [ GAC 23 + ].

Let G be a finite group. Recall that we consider permutation representations Fun ( X, R ) of a finite group G as recalled in Definition 2A.2. As a based G -representation we fix the set of indicator functions as a basis.

Definition 2E.1. An equivariant neural network is a neural network in which each layer is a direct sum of permutation representations, and all linear maps are G -equivariant. In pictures:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-59eafc8f-7c1f-4362-8653-ab442749a589.jpg)

(Here, the green, blue and red dots indicate the input, hidden and output layers respectively, and 'perm' indicates a permutation representation, which are not assumed to be equal. As in vanilla neural networks, we assume a fixed activation function throughout, which is applied componentwise in each hidden layer.)

Remark 2E.2 . We stay with finite groups in this paper, but Definition 2E.1 is also interesting for other objects as well, e.g. for Lie groups, see [ BBCV21 ].

When G is the trivial group, the notion of an equivariant neural network reduces to that of a vanilla neural network (without biases). In that setting, the weights emerge as classifying linear maps

$$\mathbb { R } \rightarrow \mathbb { R }.$$

In the setting of equivariant neural networks it is essential to know the space of G -equivariant maps

$$\ F u n ( X, \mathbb { R } ) \to \ F u n ( Y, \mathbb { R } )$$

where X and Y are permutation representations. A useful description of these ' G -equivariant weights' is provided by the following lemma (which is well-known, both in the representation theory and machine learning literature, see e.g. [ CR62 , §10, Exercise 2]):

Lemma 2E.3. For finite G -sets X,Y we have

$$\text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ), \text{Fun} ( Y, \mathbb { R } ) ) \cong \text{Fun} _ { G } ( X \times Y, \mathbb { R } )$$

where Fun G ( X × Y, R ) denotes G -invariant functions X × Y → R .

Before we prove the lemma, we want to explain why it is useful. Given any G -orbit on X × Y , its indicator function provides an element of Fun G ( X × Y, R ) which is manifestly G -invariant. Moreover, one checks easily that such indicator functions provide a basis of Fun G ( X × Y, R ) . Thus, G -equivariant weights are classified by something very concrete: G -orbits on X × Y .

We now give a proof of Lemma 2E.3:

Proof. We define maps and

$$\ F u n _ { G } ( X \times Y, \mathbb { R } ) \ni \beta \mapsto \hat { \beta } \in \text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ), \text{Fun} ( Y, \mathbb { R } ) )$$

$$\text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ), \text{Fun} ( Y, \mathbb { R } ) ) \ni \alpha \mapsto \check { \alpha } \in \text{Fun} _ { G } ( X \times Y, \mathbb { R } )$$

↦

↦

and check that they are mutually inverse.

↦

The map β → ˆ β : Given β ∈ Fun G ( X × Y, R ) consider the 'convolution operator'

$$\ F u n _ { G } ( X \times Y, \mathbb { R } ) \to \text{Hom} ( \text{Fun} ( X, \mathbb { R } ), \text{Fun} ( Y, \mathbb { R } ) ),$$

↦

$$\beta \mapsto ( f \mapsto \hat { \beta } ( f ) ) \text{ where } \hat { \beta } ( f ) ( y ) = \sum _ { x \in X } f ( x ) \beta ( x, y ).$$

↦

One checks easily that ˆ ( β g · f ) = g · ˆ ( β f ) , indeed

$$\hat { \beta } ( g \cdot f ) ( y ) = \sum _ { x \in X } ( g f ) ( x ) \beta ( x, y ) = \sum _ { x \in X } f ( g ^ { - 1 } x ) \beta ( x, y ) = \sum _ { x \in X } f ( x ) \beta ( g x, y ) = \sum _ { x \in X } f ( x ) \beta ( x, g ^ { - 1 } y ) = g \cdot ( \hat { \beta } ( f ) ) ( y ).$$

Thus the image of ˆ β lies in Hom G ( Fun ( X, R ) , Fun ( Y, R ) ) .

↦

The map α → ˇ α : For any x ∈ X , let δ x denote the corresponding δ -function. Note that g · δ x = δ gx , as follows immediately from Definition 2A.2. Now given α ∈ Hom G ( Fun ( X, R ) , Fun ( Y, R ) ) consider the function ˇ α on X × Y defined by

$$\check { \alpha } ( x, y ) = \alpha ( \delta _ { x } ) ( y ).$$

One checks easily that ˇ( α gx, gy ) = ˇ( α x, y ) , indeed

$$\check { \alpha } ( g x, g y ) = \alpha ( \delta _ { g x } ) ( g y ) = \alpha ( g \cdot \delta _ { x } ) ( g y ) = ( g \cdot \alpha ( \delta _ { x } ) ) ( g y ) = \alpha ( \delta _ { x } ) ( g ^ { - 1 } g y ) = \alpha ( \delta _ { x } ) ( y ) = \check { \alpha } ( x, y ).$$

Thus ˇ α lies in Fun G ( X × Y, R ) .

↦

Finally, we check that α → ˇ α and β → ˆ β are mutually inverse bijections. In order to show that ˆ = ˇ α α it is enough to that both sides are equal on delta functions, and this is true:

↦

$$\hat { \dot { \alpha } } ( \delta _ { x } ) ( y ) = \sum _ { z \in X } \delta _ { x } ( z ) \check { \alpha } ( x, y ) = \check { \alpha } ( x, y ) = \alpha ( \delta _ { x } ) ( y ).$$

Finally, we check that β = ˇ ˆ β by checking equality on every point ( x, y ) ∈ X × Y :

$$\check { \hat { \beta } } ( x, y ) & = \hat { \beta } ( \delta _ { x } ) ( y ) = \sum _ { z \in X } \delta _ { x } ( z ) \beta ( z, y ) = \beta ( x, y ). & \square$$

Remark 2E.4 . With a little more theory, one can prove Lemma 2E.3 slickly, using the fact that Fun ( X, R ) is self-dual, for any G -set X :

$$\text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ), \text{Fun} ( Y, \mathbb { R } ) ) \cong \text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ), \mathbb { R } \otimes \text{Fun} ( Y, \mathbb { R } ) ) \cong \text{Hom} _ { G } ( \text{Fun} ( X, \mathbb { R } ) \otimes \text{Fun} ( Y, \mathbb { R } ) ^ { * }, \mathbb { R } )$$

$$\cong \hom _ { G } ( \text{Fun} ( X, \mathbb { R } ) \otimes \text{Fun} ( Y, \mathbb { R } ), \mathbb { R } ) \cong \text{Hom} _ { G } ( \text{Fun} ( X \times Y, \mathbb { R } ), \mathbb { R } ) \cong \text{Fun} _ { G } ( X \times Y, \mathbb { R } ).$$

We prefer the more complicated proof that we have given above, as it gives a concrete form of the bijection.

Example 2E.5. A blueprint example of an equivariant neural network is given by point clouds, i.e. collections of n indistinguishable points in R d for some n, d ∈ Z ≥ 0 . In this case our group G is S n , the symmetric group on n letters, and our input layer is given by ( R d ) n = ( R n d ) which we can view as d copies of the permutation module Fun ( { 1 , ... , n } , R ) . Writing n = Fun ( { 1 , ... , n } , R ) , a typical equivariant neural network looks like

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-f75f8a6c-e172-4dc1-a149-46f32cf610ac.jpg)

(Here d = 3 and our network has 2 hidden layers.) As explained above, the linear maps above should be S n -equivariant and we can use Lemma 2E.3 to quickly determine the possible maps. Indeed, by the lemma,

$$\text{Hom} _ { G } ( \text{Fun} ( \{ 1, \dots, n \}, \mathbb { R } ), \text{Fun} ( \{ 1, \dots, n \}, \mathbb { R } ) ) = \text{Fun} _ { G } ( \{ 1, \dots, n \} \times \{ 1, \dots, n \}, \mathbb { R } )$$

and G = S n has two orbits on { 1 , ... , n } × { 1 , ... , n } given by the diagonal and its complement. Thus there is a 2-dimensional space of equivariant maps n → n , and this holds independently of n .

(Equivariant neural networks for S n of this form go by the name deep sets in the machine learning literature [ ZKR 17 + ], and are widely used. The reader is referred to [ GRBK23 ] for a recent interesting application of the partition algebra to understanding permutation equivariant layers.) ✸

Example 2E.6. Recall that in Section 2C we had C 2 n acting on 'gray-scale images' Fun ( C , 2 n R ) by translation. In this case the number of C 2 n -orbits on C 2 n × C 2 n is n 2 . For example, for n = 2 the C 2 n -orbit that includes ( (0 , 0) (0 , , 0) ) is

$$\left \{ ( ( 0, 0 ), ( 0, 0 ) ), ( ( 1, 0 ), ( 1, 0 ) ), ( ( 0, 1 ), ( 0, 1 ) ), ( ( 1, 1 ), ( 1, 1 ) ) \right \}$$

Similarly for all other elements of C 2 n × C 2 n . Thus, dim R End C 2 n ( Fun ( C , 2 n R )) = n 2 by Lemma 2E.3. ✸

The reader might wonder why we use permutation representations in Definition 2E.1. We will now see that there is essentially no other choice . Recall that a function f : R → R is odd if f ( -x ) = -f ( x ) for all x ∈ R .

Theorem 2E.7 ( Why permutation representations? . ) Let V be a G -representation and assume that PL is not odd. Then there exists a basis B = { b 1 , ... , b n } such that PL B : V → V is G -equivariant if and only if V is equivalent to a permutation representation.

It is easy to see that PL B is equivariant if V is permutation. The content of the theorem is the other implication. We give two proofs, one when PL is assumed to be ReLU , and one in the general case.

Proof (when our activation function is ReLU ). Consider the simplicial cone

$$C = \{ \sum _ { i = 1 } ^ { n } s _ { i } b _ { i } \ | \ s _ { i } \geq 0 \} = \{ v \in V \ | \text{ReLU} _ { B } ( v ) = v \}.$$

By assumption, ReLU B is G -equivariant and we get g · v ∈ C whenever v ∈ C .

Hence,

G

acts on

C

and in particular permutes the rays

{

R

≥

0

b

i

}

.

In other words, all

g

∈

G

act in the basis

B

by a permutation matrix times a diagonal matrix.

It remains to argue that the diagonal matrices can be chosen to be trivial. By classical representation theory, V admits a positive definite, symmetric and G -invariant form ⟨- -⟩ , (see e.g. [ Ser77 , §1.3, Remark]. By rescaling each b i if necessary, we can assume that each b i is of unit length, i.e. ⟨ b , b i i ⟩ = 1 . Now, ⟨ gb , gb i i ⟩ = ⟨ b , b i i ⟩ = 1 and hence G permutes the set { b 1 , ... , b n } which is what we wanted to show. □

Remark 2E.8 . The assumption in Theorem 2E.7 that G is finite cannot be dropped. For example, fix λ ∈ R and let G = Z act on R so that m ∈ Z acts by λ m . Then ReLU is Z -equivariant if and only if λ &gt; 0 .

Proof (for a general piecewise linear activation function). Consider an element g ∈ G , and a fixed but arbitrary element of our basis b . Let us write the action of g on b as

$$g \cdot b & = \supset \nmid \gamma _ { i } b _ { i }.$$

$$g \cdot b = \sum _ { i } \gamma _ { i } b _ { i }.$$

Our invariance condition reads PL B ( g · v ) = g · PL B ( v ) for any v ∈ V . In particular, for v = λb , we deduce g · PL B ( λb ) = PL B ( g · ( λb ))

which we expand to obtain

$$\sum _ { i } P L ( \lambda ) \gamma _ { i } b _ { i } = \sum _ { i } P L ( \lambda \gamma _ { i } ) b _ { i }.$$

Comparing coefficients yields

(2E.10)

PL ( λγ i ) = PL ( λ γ ) i for all λ and i .

̸

If γ i = 0 then this is easily seen to be equivalent to

$$( 2 E. 1 1 )$$

$$P L ( \lambda \gamma _ { i } ^ { - 1 } ) = P L ( \lambda ) \gamma _ { i } ^ { - 1 } \text{ \ for all } \lambda \text{ and } i, \text{ if } \gamma _ { i } \neq 0.$$

̸

̸

We claim that if γ i = 0 1 , , -1 then (2E.10) and (2E.10) force PL to be almost linear. We illustrate this first under the assumption that γ i &gt; 1 . Because PL may be decomposed into finitely many pieces, all of which are affine linear, we may find m such that PL is given by

$$\text{PL} ( x ) = a _ { > } x + b \text{ \ for all } x \in [ \gamma _ { i } ^ { m }, \gamma _ { i } ^ { m + 1 } ].$$

Expanding PL ( γ m +1 i ) = PL ( γ m i ) γ i yields b = 0 , i.e. that PL is linear on [ γ m i , γ m +1 i ] . Now for any x &gt; 0 we may find ℓ so that xγ ℓ i ∈ [ γ m i , γ m +1 i ] and expanding axγ l i = PL ( xγ ℓ i ) = PL ( x γ ) ℓ i yields that

$$\text{PL} ( x ) = a _ { > } x \text{ \ for all } x > 0.$$

Similar considerations show that PL ( x ) = a &lt; x for all x &lt; 0 . Thus, by continuity,

$$\text{PL} ( x ) = \begin{cases} a _ { > } x & \text{for $x\geq 0$}, \\ a _ { < } x & \text{for $x\leq 0$}. \end{cases}$$

$$\begin{matrix} 2 \\ 3 \end{matrix}$$

The same conclusion holds when 0 &lt; γ i &lt; 1 by using (2E.11) in place of (2E.10).

̸

If γ i &lt; 0 and γ i = -1 then, by using PL ( λγ 2 i ) = PL ( λ γ ) 2 i we can also conclude that PL has the form (2E.12). However now something stronger is true. For λ &gt; 0 we have

$$a _ { < } \lambda \gamma _ { i } = P L ( \lambda \gamma _ { i } ) = P L ( \lambda ) \gamma _ { i } = a _ { > } \lambda \gamma _ { i }$$

is odd, which again contradicts our assumptions.

and hence a &lt; 0 = a &gt; 0 . That is, PL is linear, which contradicts our assumptions on PL . Finally, when γ i = -1 then (2E.10) says that PL The upshot is that all γ i ≥ 0 in (2E.9). In other words, G preserves the cone

$$C = \{ \sum _ { i = 1 } ^ { n } s _ { i } b _ { i } \ | \ s _ { i } \geq 0 \} \\ \text{--} \cdots$$

and we may conclude as in the case of ReLU .

□

Remark 2E.13 . We were a little surprised to see the appearance of the assumption that PL is not odd pop up in Theorem 2E.7. In hindsight it is not surprising: the first example of a representation which is not permutation is the sign representation of G = Z / 2 Z and PL is equivariant if and only if PL is odd.

Remark 2E.14 . The reader may easily adapt the proofs above to the case of common non-piecewise linear activation functions like the sigmoid function 1 / (1 + e -x ) or GELU [ HG16 ]. There one comes to the stronger conclusion that all γ i in the proof are 1. (Note that this conclusion does not hold for tanh , another common activation function, as it is odd.)

2F. Piecewise linear representation theory - first example. Representation theory is intrinsically about linear maps: actions are given by matrices and representations are related by linear maps. For us piecewise linear representation theory keeps the linear actions, but allows piecewise linear maps between representations. In particular, the maps appearing in equivariant neural networks fall under the umbrella of piecewise linear representation theory.

In this paper we start a systematical study of piecewise linear representation theory. Hereby we focus on the piecewise linear Schur lemma . As we will see, the piecewise linear version of Schur's lemma is very different from the linear version.

For starters, consider the group C 4 = Z / 4 Z acting by 90 degree rotation on R 2 . In other words, C 4 acts on a pizza by rotating four equally sized pizza slices (the reader might want to consult Section 3D for a more thorough description):

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-bf7d7a46-403a-4983-af72-72751d862ff6.jpg)

Consider a piecewise linear R -valued function on one of the pizza slices above. By restriction, it induces a linear function on each edge of the slice. Consider a function which induces the same value along each edge. Such a function glues together to give a piecewise linear map R 2 → R which is C 4 -equivariant. Here R denotes the trivial C 4 -representation. Thus, Schur's lemma is false in the piecewise linear world: there exist interesting piecewise linear maps between different (nonisomorphic) simple representations.

This is not an isolated example but actually prototypical. Explicitly, doing the same, but swapping signs with each rotation defines a C 4 -equivariant map with respect to the rotation C 4 -action on R 2 and the sign C 4 -action on R .

The graphs of explicit piecewise linear maps from the simple representation R 2 to the trivial and sign representation are given as follows:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-4bd3309e-1bb6-4b3a-8ae9-eb2a89fc8916.jpg)

We have illustrated the rotation orbit of one point: on the left-hand side the orbit stays at the same height while it changes sign with each turn on the right-hand side.

Note that this construction gives uncountable many maps from the two dimensional rotation C 4 -representation to the trivial and the sign C 4 -representation. We will see that there are no nonzero maps back. The piecewise linear Schur lemma is essentially a hierarchy statement on G -representations.

2G. Why is this useful? In order to explain why this is useful for equivariant neural networks let us first assume that all maps in such a network are G -equivariant and linear . In this case Schur's lemma implies we can decompose the problem into smaller bits. To be explicit, say we are interested in computing ... ◦ f 2 ◦ f 1 for f i : V i → V i +1 being the G -equivariant linear map from the i th to the ( i +1) th layer. Then Schur's lemma allows us to decompose V i into the simple building blocks and thus the f i into matrix blocks. Explicitly, say

V i ∼ = L 0 ⊕ L 1 ⊕ L 1 ⊕ L 2 and V i +1 = V i +2 ∼ = L 0 ⊕ L 1 for nonequivalent simple real G -representations L i . Then we get the picture

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-1e0d011c-04a1-4e57-981e-0912e239f261.jpg)

Here every f j i is a scalar times the identity matrix, and the computation of f 2 ◦ f 1 simplifies drastically. For example, if one is only interested in the evaluation of the composition at L 0 , then one only needs to compute a scalar.

Since neural networks do not involve only linear maps, it is a crucial task to study Schur's lemma in the context of piecewise linear maps. For starters, consider the following problem. Let f : V → W be G -equivariant and piecewise linear. Take two simple real G -representations L and K that appear as direct summands of V and W , respectively. Let L ↪ → V and W ↠ K be an inclusion and projection. Schur's lemma for f is then the question whether one can describe the G -equivariant piecewise linear map

$$L \hookrightarrow V \xrightarrow { f } W \twoheadrightarrow K.$$

As we will see and in contrast to the linear case, the answer crucially depends on f . A naturally first candidate for f is ReLU and we will discuss Schur's lemma for ReLU in this paper.

2H. Piecewise linear representation theory. The notion of G -representations (are always on R -vector spaces if not stated otherwise) stays the same as the classical one. However, the spaces of homomorphisms we consider have elements being G -equivariant piecewise linear maps, denoted by Hom pl G ( V, W ) . As usual, we denote by End pl G ( V ) = Hom pl G ( V, V ) . We therefore have two different categories of G -representations: R ep R ( G ) , with the usual G -equivariant linear maps, and R ep pl R ( G ) , with G -equivariant piecewise linear maps.

The following two examples should be compared with Theorem 2E.7.

↦

Example 2H.1. Let G = Z / 2 Z = ⟨ a a | 2 = 1 ⟩ acting by a →-1 on R . Since 1 = ReLU (1) = ReLU ( a · -1) = a · ReLU ( -1) = 0 , we see that ReLU is not G -equivariant on this G -representation. On R 2 we have the G -action given by permuting coordinates. Then ReLU is G -equivariant with respect to this G -representation. ✸

̸

Example 2H.2. As another example, let G = Z / 3 Z = ⟨ a a | 3 = 1 ⟩ act on R 2 by rotation by 2 π/ 3 , so that the action matrix of a is 1 / 2 ( ( -1 , -√ 3) , ( √ 3 1) , ) in row notation. Let ReLU : R 2 → R 2 be the map ReLU defined componentwise. This map is not G -equivariant since ReLU ( a · (1 , 0) ) = ReLU ( ( -1 / 2 , √ 3 / 2) ) = (0 , √ 3 / 2) but a · ReLU ( (1 , 0) ) = a · (1 0) = ( , -1 / 2 , √ 3 / 2) .

On the other hand, let G act on R 3 = R { 1 , a, a 2 } (with this fixed basis) by left multiplication, i.e. the action matrix of a in row notation is ( (0 0 1) (1 0 0) (0 1 0) , , , , , , , , ) where we use { 1 &lt; a &lt; a 2 } . Then ReLU : R 3 → R 3 is G -equivariant. ✸

The following is a crucial difference between linear and piecewise linear representation theory: Let L and K be two simple real G -representations. The collection Hom pl G ( L, K ) of all G -equivariant piecewise linear maps L → K is an R -vector space for which Schur's lemma fails in general by Example 2H.8.

Actually, Schur's lemma in the setting of piecewise linear representation theory is 'maximally false':

Theorem 2H.3 ( Control by normal subgroups . ) Let L and K be two simple real G -representations with actions maps ρ L and ρ K . Then:

̸

$$\text{Hom} _ { G } ^ { p l } ( L, K ) \neq 0 \Longleftrightarrow \ker ( \rho _ { L } ) \subset \ker ( \rho _ { K } ).$$

In particular, a faithful simple real G -representation admits nonzero G -equivariant piecewise linear maps to any other G -representation.

Proof. ⇒ . Assume that f ∈ Hom pl G ( L, K ) is nonzero. Let H = ker( ρ L ) ⊂ G . By equivariance, and the fact that H acts trivially on L we have Im( f ) ⊂ K H .

Clifford theory (see e.g. [ Ben98 , Section 3.13]) shows that the restriction of K to H is semisimple, and isomorphic to a direct sum of G -conjugates of the same representation of H . Hence, either K H = 0 or K H = K . The first case gives a contradiction since then f is zero, the second case gives H = ker( ρ L ) ⊂ ker( ρ K ) , as desired.

⇐ . By replacing G with G/ ker( ρ L ) we can, and will, assume that ρ L is faithful. The following well-known fact will be useful:

Claim 1. There exists v ∈ V with trivial stabilizer under the G -action.

̸

̸

Proof of Claim 1. The set of fixed points V g is a proper subset of V for any g ∈ G with g = 1 since V g = V implies g ∈ ker( ρ L ) contradicting faithfulness. Hence, V \ ⋃ g =1 V g = ∅ . Now take v ∈ V \ ⋃ g =1 V g and we are done. □

̸

̸

Consider the set of all G -equivariant piecewise linear maps V → V denoted as before by End pl G ( V ) .

̸

G

Suppose Hom pl G ( L, K ) = { 0 } and take some nonzero f ∈ Hom pl G ( L, K ) . After choosing a basis of K , write it as f = ( f , 1 ... , f m ) where m = dim R K . Note that f thus gives us a piecewise linear maps on L which transforms according to ρ K . In other words, we would have found a G -equivariant embedding K ⊂ End pl G ( L ) . On the other hand, if we can find a G -equivariant embedding K ⊂ End pl G ( L ) , then we can produce a nonzero map f ∈ Hom pl ( L, K ) .

It thus remains to find such an embedding. Because every simple G -representation embeds into the regular representation, it is enough to find a G -equivariant embedding R ⊂ End pl G ( L ) . To this end, let us use v from Claim 1.

The points in { g · v | g ∈ G } are all distinct and we can find a piecewise linear function f which is 1 at v and 0 at { g · v | g ∈ G } \ { v } .

↦

Claim 2. g → g · f extends to a G -equivariant embedding R ⊂ End pl G ( L ) .

Proof of Claim 2. The map is G -equivariant, by construction. To check that this is an embedding suppose ∑ s g · f = 0 . Evaluating at g · v gives s = 0 , so linear independence follows. □

g ∈ G g g The proof completes. □

Remark 2H.4 . The proof of Theorem 2H.3 is an adaption of a familiar fact in representation theory: the socalled Burnside-Brauer-Steinberg theorem , see e.g. [ Bra64 ], [ Ste62 ] or [ Ste14 ], which says that every simple G -representation appears in some tensor power of a faithful G -representation.

Proposition 2H.5. For G -representations V , W we have Hom pl G ( V, W ) = 0 or dim R Hom pl G ( V, W ) &gt; | Z ≥ 0 | .

Proof sketch by example. Instead of giving a proof here, which is left to the reader, we give an illustrative and prototypical example from which a general proof can be derived.

Consider C 6 = Z / 6 Z = ⟨ a a | 6 = 1 ⟩ , i.e. the cyclic group of order six. More on piecewise linear representation of cyclic groups will follow in Section 3; for now we fix V to be the two dimensional C 6 -representation where a acts by counterclockwise rotation by 2 π/ 6 . We think of V as being the x y --plane. Let W be the trivial C 6 -representation.

We will explain how to first construct a nonlinear and nonzero map f ∈ Hom pl G ( V, W ) , and then we explain how to massage f to construct uncountable many such maps.

We first fix a fundamental cone filling 1 / 6 of the x y --plane. Illustrated as a pizza slice:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-cd22faf8-0d22-40f1-9bcc-f64765d1c96e.jpg)

We have also illustrated the rotation action of a . Now, fix a triangular spike on this fundamental cone, that is, a function that is zero at the boundary on the cone and a triangular spike in between:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-925a10f0-25a8-4e75-a036-d622e8cee43d.jpg)

This gives a nonlinear, but piecewise linear, nonzero map on the fundamental cone. Now rotating this map by 2 πk/ 6 for k ∈ { 1 , ... , 5 } gives six such maps that we can glue together to obtain one map:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-cf1c7dcb-76dc-4403-8bdd-e7a69a43fe35.jpg)

This map is the desired map f . Note moreover that we could have use any piecewise linear map instead of the triangular spike as long as the boundary hits zero. Thus, we get uncountable many, and linearly independent, such maps and the example completes. □

Lemma 2H.6. Let f : V → V be a G -equivariant piecewise linear endomorphism of a real G -representation V . Fix projections p K : V → K and their respective inclusions i L : L → V for all simple real G -representations L K , that appear in V . Then p K ◦ f ◦ i L : L → K is G -equivariant piecewise linear.

Proof. The property of being G -equivariant piecewise linear is preserved by composition, cf. Lemma 2D.3 for part of the statement. □

We will often use Theorem 2E.7 and Lemma 2H.6 silently, e.g. the examples sections.

Notation 2H.7. Write f K L = p K ◦ f ◦ i L : L → K . We also write f j i for p L j ◦ f ◦ i L i for short.

The maps from Theorem 2E.7 combined with Lemma 2H.6 are elements in Hom pl G ( L, K ) , e.g. ReLU K L ∈ Hom pl G ( L, K ) . It is crucial to determine whether they are zero or not.

Example 2H.8. Let R ∼ = R 3 denote the regular ( G = Z / 3 Z )-representation as in Example 2H.2. R ∼ = R 3 decomposes into a one dimensional trivial G -representation L 0 and the two dimensional simple real G -representation L 1 from the first paragraph of Example 2H.2. The change of basis matrix that realizes this decomposition is

$$\sum _ { \substack { Q = \left ( \begin{smallmatrix} 1 - \frac { \sqrt { 3 } } { 2 } - \frac { 1 } { 2 } } \\ 1 \frac { \sqrt { 3 } } { 2 } - \frac { 1 } { 2 } \\ 1 \ 0 \ 1 \end{smallmatrix} \right ), \quad Q ^ { - 1 } = \left ( \begin{smallmatrix} \frac { 1 } { 3 } \frac { \frac { 1 } { 3 } } { 3 } \frac { \frac { 1 } { 3 } } { 3 } \\ - \frac { 1 } { 3 } - \frac { 1 } { 3 } \frac { 2 } { 3 } \end{smallmatrix} \right ) \\ \sum _ { \substack { \sim \\ \ \ } } \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \end{smallmatrix}.$$

with the indicated inverse. Then, for example, the G -equivariant piecewise linear map L 1 → L 0 obtained as ReLU 0 1 is determined by

↦

$$( a, b ) \mapsto ( 1, 0, 0 ) ^ { T } ( \mathfrak { q } ^ { - 1 } \circ \text{ReLU} \circ \mathfrak { q } ) ( 0, a, b ), \quad \nolimits _ { \alpha k }$$

.

-

-

-

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-5d8fe422-11e2-41f1-a1c4-6b9e44498b67.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-b1b27864-53bd-4a3c-9583-4c2d8ee7c45a.jpg)

3

2

1

0

1

2

3

As in Example 2D.5, we have plotted the corresponding map.

The other three possibilities can be computed similarly and are as follows. The G -equivariant piecewise linear map ReLU 0 0 is ReLU itself, while ReLU 1 0 is zero. The final map is ReLU 1 1 is a map R 2 → R 2 is illustrated on [ GTW23 ]. ✸

2I. Interaction graphs. Suppose we have a G -equivariant piecewise linear map f ∈ Hom pl G ( V, W ) between real G -representations V and W . If f is linear, then we can decompose V ∼ = L i 1 ⊕ ... ⊕ L i k and W ∼ = L i k +1 ⊕ ... ⊕ L i r and f can be computed blockwise.

For a general G -equivariant piecewise linear map this is no longer true, cf. Example 2H.8, and we use the following measurement of complexity of how difficult simple real G -representations are under f .

We will stay with Fun ( X, R ) = V = W being a permutation G -representation for simplicity.

̸

Definition 2I.1. Let Fun ( X, R ) be a permutation G -representation. For f ∈ End pl G ( Fun ( X, R ) ) the interaction graph Γ Fun ( X, R ) f is the graph with vertices indexed by the simple real constituents L i of V , counted with multiplicities, and an edge from L i to L j if f j i = 0 .

The isotypic interaction graph i Γ Fun ( X, R ) f is obtained from Γ Fun ( X, R ) f by using isotypic components, i.e. by identifying vertices up to isomorphism of real G -representations (and removing duplicate edges).

The interaction graph depends on the choices of inclusion and projection maps for the simple real G -representation into Fun ( X, R ) , see e.g. [ GTW23 ] for an explicit discussion.

Lemma 2I.2. The isotypic interaction graph is independent of the choice of inclusion and projection. The interaction graph is independent of the choice of inclusion and projection if and only if the decomposition of Fun ( X, R ) into simple real G -representations is multiplicity free.

Proof. In the cases of the lemma the respective idempotents in the group algebra are unique.

□

To simplify notation, we write Γ f = Γ R f and i Γ f = Γ i R f .

Remark 2I.3 . For all examples in this paper we have Γ f = Γ i f , and we will strategically ignore the difference. But the distinction is necessary in general, see [ GTW23 ].

̸

̸

By Theorem 2H.3, the interaction graphs need not to be symmetric (i.e. f j i = 0 does not necessarily imply f i j = 0 ). Explicitly:

Example 2I.4. For fixed vertex L i let γ i (1) denote the number of paths of length one ending in the fixed vertex. With respect to Example 2H.8 we get

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-d865a26e-ca22-45dc-89ae-a56df55a89a8.jpg)

The map ReLU : R → R will be

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-71289ebf-e8be-47f2-99d3-35d41af7de1d.jpg)

under the decomposition R ∼ = L 0 ⊕ L 1 . Thus, γ ReLU ( ) i corresponds to the number of calculations needed to evaluate at L i after one step. ✸

More general than the one step complexity

Proposition 2I.5. Let γ i ( k ) be the number of path of length k that end in the vertex L i for Γ Fun ( X, R ) f . Then γ i ( k ) equals the number of maps needed to calculate f evaluated at L i .

Proof. Let i 1 → ... → i k → i be a path of length k . Then f i i k ◦ f i k i k -1 ◦ ... ◦ f 2 1 is needed for the evaluation of f at L i , and all paths give different maps. By the pictures in Example 2I.4, it is clear that every map that we need comes from a path. □

Remark 2I.6 . Motivated by Proposition 2I.5 one can ask for which choices of inclusions and projections Γ Fun ( X, R ) f is minimal in the sense that the number of edges is minimal. We do not have a complete answer but we address this in the special case of the symmetric group on { 1 2 3 , , } in [ GTW23 ].

Definition 2I.7. A sink in Γ Fun ( X, R ) f is a sink in the graph-theoretical sense but ignoring loops. Similarly, a source in Γ Fun ( X, R ) f is a source, again ignoring loops.

By Proposition 2I.5, sinks in Γ Fun ( X, R ) f are difficult to evaluate at, while sources are easy.

Proposition 2I.8. The trivial G -representation L triv is a sink in Γ Fun ( X, R ) f , and every other vertex has an edge towards it, i.e. γ triv (1) equals the number of vertices.

↦

̸

Proof. For any Fun ( X, R ) the projector to the trivial G -representation L triv is f → c | X | ∑ x ∈ X f ( x ) , where c : X → R is the constant map with value 1 . For any nonzero G -subrepresentation V ⊆ Fun ( X, R ) (not necessarily simple) there exists f ∈ L and x ∈ X with f ( x ) = 0 . Now the inclusion V → Fun ( X, R ) followed by PL X and the projector to L triv is nonzero. So every G -representation has an arrow to L triv . The fact that there are no arrows out of the trivial follows from Proposition 2I.9. □

Piecewise-linear maps can be complicated in general, but we can classify G -equivariant piecewise linear maps from a one dimensional representation.

Proposition 2I.9. Let χ : G → {± } 1 be a one dimensional character of G , and R χ the associated one dimensional representation. Suppose that f : R χ → V is a nonzero piecewise linear G -equivariant map. If χ is nontrivial, then precisely one of the three cases holds:

- (a) f is linear, and Im f ∼ = R χ .
- (b) Im f is a ray, R Im f is a trivial representation, and f ( x ) = | x f | (1) .
- (c) Im f consists of two distinct rays, and R Im f is isomorphic to the permutation representation on { 1 , - } 1 defined by χ .

If χ is trivial, then the linear span R Im f is always a trivial G -subrepresentation, but beyond that there are no further conditions.

In particular, if χ is nontrivial and f : R χ → V is a nonzero G -equivariant piecewise-linear map to a simple G -representation, then only the first two cases can occur. For interaction graphs, this implies that there may be arrows from R χ to the trivial representation, but not to any other simple representations.

Proof. Let v + = f (1) and v -= f ( -1) , and note by equivariance that whenever χ g ( ) = -1 , g swaps v + and v -. If χ is trivial, then G fixes v + and v -pointwise, therefore Im f is contained in a trivial representation. Assume from now that χ is nontrivial, with g ∈ G satisfying χ g ( ) = -1 .

- (a) If Im f is a line, then it is a subrepresentation since it is G -stable. Since v + and v -must lie on opposite sides of the origin and are interchanged by g , we have v -= -v + and hence f is linear. It is plain to see that Im f is isomorphic to R χ .
- (b) If Im f is a ray, then v -is a positive multiple of v + , but since they are interchanged by g they must be equal. Hence each is fixed by G , and R Im f is a trivial representation. Furthermore, f ( x ) = | x v | + .
- (c) If Im f is two linearly independent rays, each interchanged by G , then R Im f is a two dimensional subrepresentation, where G permutes the two basis vectors v + and v -according to the character χ , as claimed.

The proof completes.

□

We remark that there is another way to interpret Proposition 2I.9. Assume that χ is nontrivial, and that R χ and V are equipped with G -invariant inner products such that f : R χ → V is length-preserving (this can always be done, by rescaling f by a positive scalar). Then f either keeps R χ as it is (linear), folds it in half on top of itself (a ray), or folds it halfway to 90 degrees (two rays).

- 2J. Product groups and these maps. We will discuss the case of cyclic groups in Section 3, and the discussion below then implies that we understand the case of general abelian groups as well.

Let G ∼ = H × I . For the following lemma recall how one can tensor representations of H and I to get a G -representation, see e.g. [ Ser77 , Section 3.2], and the Frobenius-Schur indicator ν which takes values in { 0 +1 , , - } 1 (these cases are called complex, real and quaternionic, respectively), see e.g. [ FH91 , Exercise 3.38]. We will write L C H and L C I denote simple complex H - respectively I -representations and we drop the subscript for the real versions obtained by restricting scalars (such real versions exist if the complex representation is R -valued on characters), and write an overline to indicate composition of actions with complex conjugation.

## Lemma 2J.1. Let G ∼ = H × I .

- (a) We have the following.
- (i) If at most one of L C H and L C I is R -valued on characters, then ( L C H ⊗ L C I ) ⊕ ( L C H ⊗ L C I ) can be given the structure of a simple real G -representation.
- (ii) If both, L C H and L C I , are R -valued on characters (note that thus ν cannot be zero), then we have two cases:
- (A) For ν L ( C H ) ν L ( C I ) = 1 we get that L H ⊗ L I can be given the structure of a simple real G -representation.
- (B) For ν L ( C H ) ν L ( C I ) = -1 we get that ( L H ⊗ L I ) ⊕ 2 can be given the structure of a simple real G -representation.
- (b) Every simple real G -representation arises in this way.
- (c) In the notation of (a), let e L ( H ) and e L ( I ) be the associated primitive orthogonal idempotents in the regular H and I -representation, respectively. Then e L ( H ) ⊗ e L ( I ) + e L ( H ) ⊗ e L ( I ) , and e L ( H ) ⊗ e L ( I ) or 2 ( e L ( H ) ⊗ e L ( I ) ) is the primitive orthogonal idempotent in the regular G -representation associated to L H ⊗ R L I (in the order as above).

Proof. The lemma is well-known, but we still give a (dense) proof here.

For a general group G (not necessarily a product group), recall that the endomorphism algebra of a simple real G -representation V is either R C , or H . In the first case V ⊗ C is simple over C . In the second case, V ⊗ C is the direct sum of two nonisomorphic simple complex G -representations which must afford mutually conjugate simple characters, so be dual to each other. In the last case, V ⊗ C is a direct sum of two isomorphic complex simple G -representations.

The lemma then follows easily from a case-by-case analysis using the above description. For example, for the case in (a).(i) one replaces where i is the usual choice of √ -1 .

$$a + b i \twoheadrightarrow \left ( \begin{smallmatrix} a & - b \\ b & a \end{smallmatrix} \right )$$

□

Example 2J.2. Let ζ n,k = exp(2 πk/n ) . Recall that Z / 3 Z has three simple complex Z / 3 Z -representations L C i for i ∈ { 0 1 2 , , } , all of dimension one, where ζ i 3 1 , is the image of the generator [1] ∈ Z / 3 Z . Similarly, Z / 5 Z has five simple complex Z / 5 Z -representations L C j for j ∈ { 0 1 2 3 4 , , , , } . The Frobenius-Schur indicators are zero unless i = 0 or j = 0 where they take value 1 . Note that e.g. L C j ∼ = L C 5 -j for j ∈ { 1 2 3 4 , , , } .

Thus, Lemma 2J.1 and getting rid of redundancy implies that the simple real ( Z / 3 Z × Z / 5 Z ) -representations are of the form L 0 ⊗ L 0 , as well as L C 1 ⊕ L C 1 and L C j ⊕ L C j for j ∈ { 1 2 , } , and finally ( L C 1 ⊗ L C j ) ⊕ ( L C 1 × L C j ) , again for j ∈ { 1 2 , } . The dimensions of these ( Z / 3 Z × Z / 5 Z ) -representations are 1 , 2 , 2 , 2 , 4 and 4 . ✸

Proposition 2J.3. The intersection graph of G ∼ = H × I is the product of the ones of H and I .

Proof. By Lemma 2J.1 and construction.

Remark 2J.4 . Every finite abelian group is a product of cyclic groups. Thus, Lemma 2J.1, Proposition 2J.3 and the discussion in Section 3 determine the piecewise linear maps on and the intersection graphs of finite abelian groups.

Example 2J.5. Recall that we have seen the intersection graph for Z / 3 Z in Example 2I.4. The intersection graph for Z / 5 Z can be found in Example 3B.2 below. We get:

,

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-477be64e-9860-4825-b1a8-6c5c4566e2db.jpg)

Here L ij is a shorthand notation for what one gets using Lemma 2J.1 from L i of Z / 3 Z and L j of Z / 5 Z . ✸

$$3. \text{ EXAMPLE: CYCLIC GROUPS}$$

Let C n = Z /n Z = ⟨ a a | n = 1 ⟩ be the cyclic group of order n ∈ Z ≥ 1 . This group acts on the n dimensional R -vector space R = R { x , 0 ... , x n -1 } by cyclic permutation of the coordinates, i.e.:

$$a \cdot x _ { i } = x _ { i + 1 },$$

reading indexes modulo n . This (based) C n -representation R = ( R,B = { x , 0 ... , x n -1 } ) is the regular C n -representation . The action matrix M a of a on R on the basis { x , 0 ... , x n -1 } is

$$\mathbf M _ { a } = \begin{pmatrix} 0 & 0 & \cdots & 0 & 1 \\ 1 & 0 & \cdots & 0 & 0 \\ 0 & \ddots & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 & 0 \\ 0 & \cdots & 0 & 1 & 0 \end{pmatrix}.$$

3A. Simple representations, projections and inclusions. We now compute (or rather recall) the simple real C n -representations.

Notation 3A.1. Let m be determined by n = 2 m or n = 2 m + 1 , depending on parity. Moreover, the rotation angle θ = 2 π/n will play a crucial role, and we further let ζ k = exp(2 πik/n ) be the respective complex roots of unity where i is the usual choice of √ -1 .

For n even, the trivial L 0 = R { v } and sign L m = R { v } C n -representations are

$$L _ { 0 } \colon a \cdot v = v, \ \ L _ { m } \colon a \cdot v = - v.$$

↦

Let N 0 denote the 1 -by1 matrix (1) , and let N m denote the 1 -by1 matrix ( -1) . In terms of matrices we have a → N 0 or m ∈ End ( R L 0 or m ) . If n is odd, then we only define L 0 .

We let k ∈ { 1 , ... , m - } 1 , for n even, and k ∈ { 1 , ... , m } , for n odd. For such k endow the two dimensional R -vector space L k = R { v k , v -k } with the C n -action given by

$$a \cdot L _ { k } = \cos ( k \theta ) \cdot v _ { k } + \sin ( k \theta ) \cdot v _ { - k }, \ \ a \cdot L _ { - k } = - \sin ( k \theta ) \cdot v _ { k } + \cos ( k \theta ) \cdot v _ { - k }.$$

Or in terms of matrices:

↦

$$a \mapsto \mathbb { N } _ { k } = \begin{pmatrix} \cos ( k \theta ) & - \sin ( k \theta ) \\ \sin ( k \theta ) & \cos ( k \theta ) \end{pmatrix} \in \text{End} _ { \mathbb { R } } ( L _ { k } ).$$

The representations L k are rotation C n -representations .

Lemma 3A.3. As real C n -representations we have

$$R & \cong L _ { 0 } \oplus L _ { 1 } \oplus \dots \oplus L _ { m - 1 } \oplus L _ { m }.$$

(Note that dim R L m ∈ { 1 2 , } , depending on parity.) All of the appearing real C n -representations are simple and pairwise nonisomorphic. All simple real C n -representations appear in this way.

Proof. We will use the complex case that can, for example, be found in [ Ser77 , Section 5.1].

This well-known fact follows from the splitting of R ⊗ R C over C into one dimensional C n -representations L ζ k given by the eigenvalues ζ k .

If ζ k ∈ R , then we can regard L ζ k as a real C n -representation, which is simple since it is one dimensional. This happens only if ζ k = ± 1 , so either k ∈ { 0 , m } or k = 0 depending on parity, and we get the trivial and sign C n -representations.

If ζ k / ∈ R , then we pair ζ k and its complex conjugate ζ -k . It follows that the complex C n -representation L ζ k ⊕ L ζ -k can be regarded as a real C n -representation via

$$\begin{pmatrix} \zeta _ { k } & 0 \\ 0 & \zeta _ { - k } \end{pmatrix} \sim _ { \mathbb { C } } \mathbb { N } _ { k } = \begin{pmatrix} \cos ( k \theta ) & - \sin ( k \theta ) \\ \sin ( k \theta ) & \cos ( k \theta ) \end{pmatrix}.$$

Thus, we get the C n -representations L k .

One easily checks that all of the above are nonisomorphic simple real C n -representations, and counting dimensions implies that there are no others since dim R R = n = ∑ m i =1 dim R L i . □

We now describe the projections and inclusions p L : R ↠ L and i L : L ↪ → R for L a simple real C n -representation.

To this end, we observe that it is enough to give the base change realizing (3A.4). For n even this works as follows. The matrices for a acting on the left and the right-hand side in (3A.4) are M determined by (3.1) and the matrix N = diag(1 , N 1 , ... , N m -1 , -1) with diagonal blocks given by the trivial one-by-one matrix, the matrices in (3A.2) and the negative of the trivial one-by-one matrix.

We now define a change-of-basis matrix Q ∈ End ( R R ) between M and N , meaning that N = Q -1 MQ . This matrix can be defined as follows. For L 0 we use the length n vector w 0 = (1 , ... , 1) , for L m we use w m = ( -1 1 , , -1 , ... , -1 1) , . For L k we take

$$w _ { k } = \left ( - \sin ( j k \theta ) \right ) _ { j \in \{ 1, \dots, n \} }, \ \ w _ { - k } = \left ( \cos ( j k \theta ) \right ) _ { j \in \{ 1, \dots, n \} }.$$

For n odd the construction of these vectors is similar.

Lemma 3A.5. Let n be even, and let Q = ( w , w , w 0 1 -1 , ... , w m -1 , w -( m -1) , w m ) .

- (a) Let D = diag(1 /n, 2 /n, ... , 2 /n, 1 /n ) be a diagonal matrix with the indicated diagonal entries. The matrix Q is invertible with Q -1 = DQ T .
- (b) The matrix Q is C n -equivariant and satisfies N = Q -1 MQ .

Similarly for Q = ( w , w , w 0 1 -1 , ... , w m , w -m ) when n is odd.

Proof. A direct calculation. As a hint how the matrix Q appears, use Jordan decomposition for M and N , say J = C -1 MC = E -1 NE for a choice of a Jordan matrix J , and let Q = CE -1 . Alternatively, the columns of Q consists of linearly independent columns of the projectors p L , one column if dim R L = 1 , and two columns for dim R L = 2 . These projectors can be easily computed by general theory. □

## Example 3A.6. For n = 5 we get

$$\mathcal { U } \mathcal { H } \text{ } \text{ for } n = 5 \text{ we get} \\ M = \left ( \begin{array} { c c c c c } 0 \, 0 \, 0 \, 0 \, 1 \\ 1 \, 0 \, 0 \, 0 \, 0 \\ 0 \, 0 \, 1 \, 0 \, 0 \\ 0 \, 0 \, 0 \, 1 \, 0 \end{array} \right ), & N = \left ( \begin{array} { c c c c c } 1 & 0 & 0 & 0 & 0 \\ 0 \, 0 \, 0 \, 1 \, 0 \\ 0 \, 0 \, 0 \, 1 \, 0 \end{array} \right ), & N = \left ( \begin{array} { c c c c c } 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 \, 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 \\ 0 & 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 | 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 } 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 { 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 0 \\ 0 \\ 0 \\ 0 \\ 0 0$$

The inverse Q -1 is obtained by transposing Q and then multiplying the the first row by 1 / 5 and the other four by 2 / 5 . One easily checks that N = Q -1 MQ . ✸

3B. ReLU and cyclic groups. We now want to calculate

$$\text{(3B.1)} & & \text{ReLU} _ { L, K } = L \xrightarrow { i _ { L } } R \xrightarrow { \text{ReLU} } R \xrightarrow { p _ { K } } K \in \text{Hom} _ { G } ^ { p l } ( L, K )$$

for simple real C n -representations L and K . Below we will determine whether ReLU L,K is zero or not, and even better, we will explain what these piecewise linear C n -equivariant maps are.

To this end, we will actually calculate

$$R \xrightarrow { q ^ { - 1 } } R \xrightarrow { \text{ReLU} } R \xrightarrow { q } R$$

for the matrix Q from Section 3A. This will tell us the nonlinear interactions between the various simple real C n -representations. We denote the composition Q -1 · ReLU · Q as ReLU Q .

Example 3B.2. Let n = 5 and let e i be the standard vector in R 5 . We calculate that

$$L _ { 0 } & \text{component} \colon \text{ReLU} _ { q } ( e _ { 1 } ) = ( 1, 0, 0, 0, 0 ), \\ L _ { 1 } & \text{component} \colon \text{ReLU} _ { q } ( e _ { 2 } ) \approx ( 0. 3 1, 0. 5, - 0. 0 7, 0, - 0. 2 4 ), \quad \text{ReLU} _ { q } ( e _ { 3 } ) \approx ( 0. 3 2, 0, 0. 4 8, 0, 0. 2 ), \\ L _ { 2 } & \text{component} \colon \text{ReLU} _ { q } ( e _ { 4 } ) \approx ( 0. 3 1, 0, - 0. 2 4, 0. 5, - 0. 0 7 ), \quad \text{ReLU} _ { q } ( e _ { 5 } ) \approx ( 0. 3 2, 0, 0. 2, 0, 0. 4 8 ).$$

̸

Note that ReLU Q is not linear, for example, ReLU Q ( e 2 + e 3 ) = ReLU ( e 2 )+ ReLU ( e 3 ) . Hence, the above calculation does not determine the whole map. However, one can guess that the interaction graph for ReLU is

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-f974f046-6ea2-46a0-8a0a-13a91bcd697d.jpg)

We will see momentarily that this is indeed the case.

✸

Note that every simple real C n -representation L has one or two associated complex roots of unity and if there are two such roots, then they are conjugates. In particular, we can define:

Definition 3B.3. The order ord( L ) of L to be the order of any of the associated roots of unity. We use this to define

$$\text{ord} ^ { \prime } ( L ) = \begin{cases} \text{ord} ( L ) & \text{if ord} ( L ) \text{ is odd}, \\ \text{ord} ( L ) / 2 & \text{if ord} ( L ) \text{ is even}. \end{cases}$$

In other words, ord( L ) is the order of the action matrix of a . For the two dimensional simple real C n -representations L i the order is just the order of rotation, and ord ( ′ L ) appears due to a parity issue.

̸

Theorem 3B.4. Consider the interaction graph Γ ReLU . Every vertex has a loop. Moreover, for L ∼ = K there is an edge from L to K if and only if ord( K ) divides ord ( ′ L ) .

The isotypic interaction graph i Γ ReLU is the same as Γ ReLU .

Proof. We give an proof of Theorem 3B.4 later in Section 3D where we explicitly construct C n -equivariant piecewise linear maps.

For now we obtain Theorem 3B.4 directly from Theorem 2H.3 since the order is just an explicit description of the normal subgroup structure of C n . □

Example 3B.5. We have the following regarding Γ ReLU .

̸

- (a) If n is an odd prime, then Γ ReLU has a sink at L 0 and a fully connected graph on the L i for i = 0 . For example, for n = 7 :

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-62c4c650-4b39-4e87-81eb-39d2a44bb19e.jpg)

- (b) If n is composite, then the story is more interesting. Let us do n = 8 . In this case we get:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-55eb7d04-1f31-4a4d-85d3-a6ee068b71e4.jpg)

.

- (c) The composite n = 6 is also interesting as we will see how ord ( ′ L ) enters the game, i.e. we have:

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-3fa4362a-2ab5-471d-a351-6c0a47dc0a6e.jpg)

Note that ord( L 1 ) = 6 and ord( L 3 ) = 2 , however they are not connected by an edge since ord ( ′ L 1 ) = 3 . Interactive examples can be found online following th link in [ GTW23 ]. ✸

3C. Absolute value and cyclic groups. Similarly as in Section 3B we now discuss

$$R \stackrel { q ^ { - 1 } } { \longrightarrow } R \stackrel { \text{Abs} } { \longrightarrow } R \stackrel { q } { \rightarrow } R$$

for the absolute value map Abs . We denote the composition Q -1 · Abs · Q as Abs Q .

Example 3C.1. We do a similar calculation as in Example 3B.2 but for n = 6 . We get:

L 0 component : Abs Q ( e 1 ) = (1 , 0 0 0 0 0) , , , , , L 1 component : Abs Q ( e 2 + e 3 ) ≈ (0 91 . , 0 0 0 58 0 089 0) , , . , . , , L 2 component : Abs Q ( e 4 + e 5 ) ≈ (0 91 . , 0 0 , , -0 58 0 089 0) . , . , , L 3 component : Abs Q ( e 6 ) = (1 , 0 0 0 0 0) , , , , .

We get the following graph:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-5fdb8dab-a3d1-4378-b1fe-88b2e4689000.jpg)

Note that we have the vertex for L 1 has no loop, so the graph is different from the one for ReLU . ✸

Recall ord ( ′ L ) as in Section 3B. The proof of the next theorem will also be given in tandem with the proof of Theorem 3B.4, or follows directly from Theorem 2H.3.

Theorem 3C.2. There is an edge from L to K in Γ Abs if and only if ord( K ) divides ord ( ′ L ) . The isotypic interaction graph i Γ Abs is the same as Γ Abs .

In contrast to Theorem 3B.4 we do not have a separate loop condition in Theorem 3C.2.

Example 3C.3. If n is an odd prime, then Γ ReLU and Γ Abs are the same. In contrast, if n = 2 j for some j ∈ Z ≥ 1 , then Γ ReLU and Γ Abs are quite different and Γ Abs is only a subgraph of Γ ReLU . For example, for n = 8 we get

.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-126e43b1-286c-42dc-89f0-6d1a08a7b380.jpg)

For comparison, the graph Γ ReLU is given in Example 3B.5.

✸

3D. The piecewise linear maps for the cyclic group. For fixed n ∈ Z ≥ 1 , we consider all the n th roots of unity ζ k in the R -vector space C ∼ = R 2 . Here k ∈ { 0 , ... , n - } 1 .

Definition 3D.1. For k ∈ { 0 , ... , n -1 } let H pa ζ k and H pe ζ k be the hyperplane in C that is parallel and perpendicular to the line spanned by ζ k . The hyperplane arrangment associated to L i are as follows. For dim R L i = 2 we call H i = ⋃ j ∈{ 1 ,ζ i ,ζ 2 i , ... , } H pa j , and for dim R L i = 1 we use H i = ⋃ j ∈{ 1 ,ζ i ,ζ 2 i , ... , } H pe j .

Example 3D.2. Let n = 6 and recall from Lemma 3A.3 that R ∼ = L 0 ⊕ L 1 ⊕ L 2 ⊕ L 3 of dimensions 1 , 2 , 2 and 1 . The associated hyperplane arrangments are

iζ

iζ

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-1f3e1dc3-a3cd-4617-b892-153f3ebd9316.jpg)

3

We additionally indicated the roots of unity ζ 0 and ζ 3 in the left illustration.

✸

Example 3D.3. Starting at n = 3 , the illustrations

show the first few hyperplane arrangements associated to L 1 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-816d66f0-0e86-4b1f-bb7a-0e9aac9ec948.jpg)

✸

Remark 3D.4 . As in Example 3D.2, note that the one dimensional simple real C n -representations always have the same associated hyperplane arrangment. The two dimensional simple real C n -representations can have the same hyperplane arrangment, as in Example 3D.2, but need not to. The smallest example where the hyperplane arrangements differ is n = 8 . In this case the hyperplane arrangements for L 1 and L 3 are as in Example 3D.3 for n = 8 , while L 2 has the picture for n = 4 associated to it.

Recall the inclusion i L from (3B.1). The pullback i ∗ L : L → R defines a set of hyperplanes H L ( ) in L .

Lemma 3D.5. We have H L ( i ) = H i , and ReLU ◦ i L and Abs ◦ i L are linear on the complement A i = L \ H i . Hence, A = {A 1 , ... , A } k is a polyhedral covering for ReLU ◦ i L and Abs ◦ i L .

Proof. Assume dim R L i = 2 and n even, and let us focus on ReLU . The lemma follows from the explicit form of Q -1 which was computed in Lemma 3A.5. That is, the respective square submatrix of Q consist of the coordinates of the n th roots of unity that appear in the definition of the C n -action on L i , see (3A.2). The map ReLU is linear unless one hits the input zero. Taking both observations together, we see that ReLU ◦ i L is linear on the alcove between H pa ζ 0 and H pa ζ 1 . Using that C n -equivariance, see Lemma 2H.6, the claim follows. The remaining cases are similar but easier and omitted. □

## Lemma 3D.6.

- (a) Let dim R L = 1 . Then A is of size two and has one C n -orbit.
- (b) Let dim R L = 2 . If n is even, then A is of size n and has one C n -orbit. If n is even, then A is of size 2 n and has two C n -orbits.

Proof. This follows from Lemma 3D.5. In more details, let us just do L = L 1 for n = 5 and n = 6 . Fixing A = A 1 for n = 5 and n = 6 , and additionally B = A 2 for n = 5 , we get:

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-03ac6853-8bc0-4727-80ae-fd89a63748bc.jpg)

The picture generalizes without problems to general n .

The following is useful for calculations:

Lemma 3D.7. If f as in Lemma 2D.6 is G -equivariant, then it is determined on the polyhedral subsets by evaluate it on one set of vectors per G -orbit.

Proof.

$$P r o o f. \text{ This follows directly from Lemma 2D.6.}$$

## Lemma 3D.8.

- (a) Let dim R L = 1 . Then ReLU ◦ i L and Abs ◦ i L are determined by evaluating them on (1) .
- (b) Let dim R L = 2 . If n is even, then ReLU ◦ i L and Abs ◦ i L are determined by evaluating them on (0 1) , . If n is odd, then ReLU ◦ i L and Abs ◦ i L are determined by evaluating them on (0 , 1) and (0 , -1) .

Proof. This follows from Lemma 3D.7, Lemma 3D.5 and Lemma 3D.6.

□

Proof of Theorem 3B.4 and Theorem 3C.2 (via constructing maps). To prove the two named theorems, by Lemma 3D.8, we first need to evaluate the matrix Q from Section 3A on the images of the vectors from Lemma 3D.8 in R n upon the evident embedding. We then need to compose the result with ReLU and Abs , respectively, and then apply Q -1 . We do this for ReLU since Abs can be analyzed completely analogously.

Since Q -1 is Q T up to a diagonal matrix, see Lemma 3A.5, the calculation boils down to computing ( Q T · ReLU · Q ) e k where e k is the k th unit vector. To match the conventions we use, we will write ( f , f 0 1 , f -1 , ... ) = ( e , e 1 2 , e 3 , ... ) and let x ± k = ( Q T · ReLU · Q ) f ± k .

We get x 0 = n · f 0 for the component of the trivial C n -representation, and x m = n/ 2 · f 0 + n/ 2 · f m for the component of the sign C n -representation (in case n is even). For the two dimensional simple real C n -representations we get Q f k = ( -sin( k 1 ) θ , ... , -sin( knθ ) ) or Q f -k = ( cos( k 1 ) θ , ... , cos( knθ ) ) . The entries of these vectors that are negative are completely determined by the order ord( L ) , and an easy analysis then gives the claimed result. For example, consider the regular n gon that contains the point (1 0) , and is centered at (0 0) , . Then Q f -k consists of all the x -values of this n gon, while Q f k has its entries being the negatives of the y -values. Thus, we immediately get the which values of ( ReLU ◦ Q ) f ± k are nonzero. Except the rows for the trivial and sign C n -representation, the rows of Q T are of the form ( -sin( k 1 ) θ , ... , -sin( knθ ) ) and ( cos( k 1 ) θ , ... , cos( knθ ) ) so applying Q T to ( ReLU ◦ Q ) f ± k implies the result. The case for general k is similar and omitted.

Finally, we clearly have Γ ReLU = Γ i ReLU and the proof completes.

□

One can now further analyze the precise form of the various piecewise linear maps. An example of ReLU 1 1 : L 1 → L 1 is displayed in Example 2H.8, and the general picture is similar. The map ReLU 0 0 is just ReLU itself, while, for n is even, the map ReLU 0 m is the absolute value.

Let us now analyze the piecewise linear maps in the case of f = ReLU 0 1 : L 1 → L 0 and g = ReLU m 1 : L 1 → L m for n even or n ≡ 0 mod 4 , respectively, which we can display as a map R 2 → R using level sets.

By the above, f (0 1) , ∈ R &gt; 0 , more precisely, f (0 1) , is equal to 1 /n times the sum over all positive numbers in ( cos( iθ ) ) n -1 i =0 . For n even this information is enough to determine the whole map since there is only one C n -orbit. In other words, in this case the level sets are regular n -gons.

For example, for n = 4 one gets (the level sets are the squares)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00949v2-figure-9eafcf66-37eb-47f2-95b3-299488540761.jpg)

and the map f is the one in Example 2D.5.

For g something similar happens, but now with a sign swap after each rotation. For n = 4 one gets the map in Example 2D.5.

Remark 3D.9 . We have also worked out Theorem 3B.4 and Theorem 3C.2 for the dihedral group and partially for the symmetric group. An additional document containing details can be found following the links in [ GTW23 ].

## References

D.J. Benson. Representations and cohomology. I , volume 30 of Cambridge Studies in Advanced Mathematics . Cambridge University Press, Cambridge, second edition, 1998. Basic representation theory of finite groups and associative

[Ben98] algebras.

[Bra64]

R. Brauer. A note on theorems of Burnside and Blichfeldt. Proc. Amer. Math. Soc. , 15:31-34, 1964. doi:10.2307/ 2034344 .

[BBCV21] M.M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021. URL: https://arxiv.org/abs/2104.13478 .

[CW16] T. Cohen and M. Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016. URL: https://arxiv.org/abs/1602.07576 .

[CR62]

[Est20]

C.W. Curtis and I. Reiner. Representation theory of finite groups and associative algebras , volume Vol. XI of Pure and Applied Mathematics . Interscience Publishers (a division of John Wiley &amp; Sons, Inc.), New York-London, 1962. C. Esteves. Theoretical aspects of group equivariant neural networks. 2020. URL: https://arxiv.org/abs/2004.05154 .

[FH91] W. Fulton and J. Harris. Representation theory , volume 129 of Graduate Texts in Mathematics . Springer-Verlag, New York, 1991. A first course, Readings in Mathematics. doi:10.1007/978-1-4612-0979-9 .

[GAC + 23] J.E. Gerken, J. Aronsson, O. Carlsson, H. Linander, F. Ohlsson, C. Petersson, and D. Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review , 56(12):14605-14662, 2023. URL: https:

//arxiv.org/abs/2105.13926 .

[GTW23]

J. Gibson, D. Tubbenhauer, and G. Williamson. Code related to the paper 'Equivariant neural networks and piecewise linear representation theory'. 2023. https://github.com/dtubbenhauer/pl-reps and https://www.dtubbenhauer. com/pl-reps/site/index.html .

[GRBK23] C. Godfrey, M.G. Rawson, D. Brown, and H. Kvinge. Fast computation of permutation equivariant layers with the partition algebra. 2023. URL: https://arxiv.org/abs/2303.06208 .

[GBC16]

I. Goodfellow, Y. Bengio, and A. Courville. Deep learning . Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2016.

[HG16]

[Jäh05]

[LN23]

[Nie15]

[Ser77]

[Ste62]

D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). 2016. URL: https://arxiv.org/abs/1606.08415 . B. Jähne. Digital Image Processing . Springer Berlin, Heidelberg, 2005. doi:10.1007/3-540-27563-0 .

L.-H. Lim and B.J. Nelson. What is . . . an equivariant neural network? Notices Amer. Math. Soc. , 70(4):619-625, 2023. URL: https://arxiv.org/abs/2205.07362 , doi:10.1090/noti2666 .

M.A. Nielsen. Neural Networks and Deep Learning . Determination Press, 2015. URL: http:// neuralnetworksanddeeplearning.com/ .

J.-P. Serre. Linear representations of finite groups . Graduate Texts in Mathematics, Vol. 42. Springer-Verlag, New York-Heidelberg, 1977. Translated from the second French edition by Leonard L. Scott.

R. Steinberg. Complete sets of representations of algebras. Proc. Amer. Math. Soc. , 13:746-747, 1962. doi:10.2307/ 2034167 .

[Ste14]

B. Steinberg. On the Burnside-Brauer-Steinberg theorem. 2014. URL: https://arxiv.org/abs/1409.7632

.

[ZKR + 17] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R.R. Salakhutdinov, and A.J. Smola. Deep sets. Advances in neural information processing systems , 30, 2017. URL: https://arxiv.org/abs/1703.06114 .

J.G.: The University of Sydney, School of Mathematics and Statistics F07, Office Carslaw 827, NSW 2006, Australia, www.jgibson.id.au

Email address : joel@jgibson.id.au

D.T.: The University of Sydney, School of Mathematics and Statistics F07, Office Carslaw 827, NSW 2006, Australia, www.dtubbenhauer.com, https://orcid.org/0000-0001-7265-5047

Email address : daniel.tubbenhauer@sydney.edu.au

G.W.: The University of Sydney, School of Mathematics and Statistics F07, Office L4.41 Quadrangle A14, NSW 2006, Australia, www.maths.usyd.edu.au/u/geordie/, https://orcid.org/0000-0003-3672-5284

Email address : g.williamson@sydney.edu.au
Joel Gibson Daniel Tubbenhauer Geordie Williamson
Equivariant neural networks and piecewise linear representation theory
Equivariant neural networks are neural networks with symmetry. Motivated by
the theory of group representations, we decompose the layers of an equivariant
neural network into simple representations. The nonlinear activation functions
lead to interesting nonlinear equivariant maps between simple representations.
For example, the rectified linear unit (ReLU) gives rise to piecewise linear
maps. We show that these considerations lead to a filtration of equivariant
neural networks, generalizing Fourier series. This observation might provide a
useful tool for interpreting equivariant neural networks.