summaries: This paper proposes to make the geometric neurons of Melnyk et al.'21 to be steerable so that objects undergoing arbitrary rotations can be classified with higher accuracy. This is done in multiple stages. First, the neurons of Melnyk et al. are trained to convergence with a hyperspherical output layer. Then, the frozen weights are transformed such that the input of a *steerable neuron* can be written as a linear combinations of the rotated versions of itself. The experiments act as a sanity check while demonstrating the validity of the algorithm on a simple human pose dataset. The paper proposes a method for constructing steerable spherical neurons, building on the recent Geometric Neurons developed in Melnyk et al 2021. The main technical result in the paper is a steerability constraint for a geometric neuron, as given by eq. (13) in the paper. This constraint is used in an implementation whereby a steerable model is constructed and is then use for two tasks. The first task involves the use of steerable spherical neutrons for the classification of 3D Tetris objects seen under rotations, and the second is a similar experiment applied to 3D skeleton data. The results demonstrate a very large performance boost when using the steerable versions. The second experiment builds on this idea to construct a version that adapts a possibly imperfect initial rotation estimate, using the representation. This second experiment is in the spirit of demonstrating equivariance under 3D rotations with perturbations. ## Summary and contributions. Authors propose 3D "spherical neurons" leading to rotationally equivariant layers. They do so by building on the spherical and geometric neurons introduced in Melnyk et al. (2021), which leverages the conformal space (for $\mathbb{R}^n$) to perform operations. The authors then solve for the steerability constraint for this neuron and empirically show that the proposed approach overperforms Melnyk et al. (2021) on rotated 3D data.  The paper aims to derive a steerability constraint for spherical neurons (3D point classifiers with spherical decision boundary). The steerability constraint enables test-time optimization of a pre-trained classifier to make predictions equivariant to 3D rotation perturbations applied to the input. When input rotation perturbations are unknown, the authors propose a method to recover the unknown rotations and therefore make rotation-invariant predictions. The experiments on a few small scale datasets verifies some of the claims.
abstractText: Emerging from low-level vision theory, steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations. In our work, we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds. Focusing on 3D geometry, we derive a 3D steerability constraint for hypersphere neurons, which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets. Exploiting the rotational equivariance, we show how our model parameters are fully steerable at inference time. We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and, after online optimization, invariant class predictions for known point sets in unknown orientations.
references.authors: Brandon Anderson Truong Son Hy Risi Kondor
references.title: Cormorant: Covariant molecular neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Vladimir Banarer Christian Perwass Gerald Sommer
references.title: Design of a multilayered feed-forward neural network using hypersphere neurons
references.venue: In International Conference on Computer Analysis of Images and Patterns,
references.authors: Vladimir Banarer Christian Perwass Gerald Sommer
references.title: The hypersphere neuron
references.venue: In ESANN, pp
references.authors: William T Freeman Edward H Adelson
references.title: The design and use of steerable filters
references.venue: IEEE Transactions on Pattern analysis and machine intelligence,
references.authors: Fabian Fuchs Daniel Worrall Volker Fischer Max Welling
references.title: Se(3)-transformers: 3d rototranslation equivariant attention networks
references.venue: Advances in Neural Information Processing Systems,
references.authors: Eckhard Hitzer
references.title: Geometric operations implemented by conformal geometric algebra neural nodes
references.venue: Proc. SICE Symposium on Systems and Information 2008,
references.authors: Roger Alan Horn Charles Royal Johnson
references.title: Topics in Matrix Analysis
references.authors: Bowen Jing Stephan Eismann Patricia Suriana Raphael JL Townshend Ron Dror
references.title: Learning from protein structure with geometric vector perceptrons
references.venue: arXiv preprint arXiv:2009.01411,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In Proceedings of the International Conference on Learning Representations (ICLR),
references.authors: Hans Knutsson Leif Haglund Håkan Bårman Gösta H. Granlund
references.title: A framework for anisotropic adaptive filtering and analysis of image sequences and volumes
references.venue: In [Proceedings]
references.authors: Risi Kondor Zhen Lin Shubhendu Trivedi
references.title: Clebsch–Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
references.venue: Advances in Neural Information Processing Systems,
references.authors: Hongbo Li David Hestenes Alyn Rockwood
references.title: Generalized homogeneous coordinates for computational geometry
references.venue: In Geometric Computing with Clifford Algebras,
references.authors: Hongbo Li David Hestenes Alyn Rockwood
references.title: A universal model for conformal geometries of Euclidean, spherical and double-hyperbolic spaces. In Geometric computing with Clifford algebras, pp. 77–104
references.authors: Diego Marcos Michele Volpi Nikos Komodakis Devis Tuia
references.title: Rotation Equivariant Vector Field Networks
references.venue: In Proceedings of the IEEE International Conference on Computer Vision (ICCV),
references.authors: Pavlo Melnyk Michael Felsberg Mårten Wadenbäck
references.title: Embed Me if You Can: A Geometric Perceptron
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Pietro Perona
references.title: Deformable kernels for early vision
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Christian Perwass Vladimir Banarer Gerald Sommer
references.title: Spherical decision surfaces using conformal modelling
references.venue: In Joint Pattern Recognition Symposium,
references.authors: Marco Reisert
references.title: Group integration techniques in pattern analysis - a kernel view
references.venue: PhD thesis, University of Freiburg, Freiburg im Breisgau,
references.authors: Eero P. Simoncelli William T. Freeman
references.title: The steerable pyramid: a flexible architecture for multiscale derivative computation
references.venue: In Proceedings., International Conference on Image Processing,
references.authors: Eero P. Simoncelli William T. Freeman Edward H. Adelson David J. Heeger
references.title: Shiftable multiscale transforms
references.venue: IEEE Transactions on Information Theory,
references.authors: Patrick C. Teo Yacov Hel-Or
references.title: Lie generators for computing steerable functions
references.venue: Pattern Recognition Letters,
references.authors: Nathaniel Thomas Tess Smidt Steven Kearnes Lusann Yang Li Li Kai Kohlhoff Patrick Riley
references.title: Tensor field networks: Rotation-and translation-equivariant neural networks for 3D point clouds
references.venue: arXiv preprint arXiv:1802.08219,
references.authors: Luc Van Gool Theo Moons Eric Pauwels André Oosterlinck
references.title: Vision and Lie’s approach to invariance
references.venue: Image and Vision Computing,
references.authors: Maurice Weiler Gabriele Cesa
references.title: General E(2)-Equivariant Steerable CNNs
references.venue: arXiv preprint arXiv:1911.08251,
references.authors: Maurice Weiler Mario Geiger Max Welling Wouter Boomsma Taco S Cohen
references.title: 3D steerable CNNs: Learning rotationally equivariant features in volumetric data
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Maurice Weiler Fred A Hamprecht Martin Storath
references.title: Learning steerable filters for rotation equivariant CNNs
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Lu Xia Chia-Chih Chen Jagdishkumar K. Aggarwal
references.title: View invariant human action recognition using histograms of 3d joints
references.venue: In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Yongheng Zhao Tolga Birdal Jan Eric Lenssen Emanuele Menegatti Leonidas Guibas Federico Tombari
references.title: Quaternion equivariant capsule networks for 3d point clouds
references.venue: In European Conference on Computer Vision,
sections.text: Emerging from low-level vision theory, steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations. In our work, we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds. Focusing on 3D geometry, we derive a 3D steerability constraint for hypersphere neurons, which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets. Exploiting the rotational equivariance, we show how our model parameters are fully steerable at inference time. We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and, after online optimization, invariant class predictions for known point sets in unknown orientations.
sections.heading: 1 INTRODUCTION
sections.text: We present a novel feed-forward model consisting of steerable 3D neurons for point cloud classification, an important and challenging problem with many applications such as autonomous driving, human-robot interaction, and mixed-reality installations. We achieve the steerability by using geometric neurons (Melnyk et al., 2021), leading to a rotation equivariant network. Besides becoming a geometrically explainable approach, further benefits of the proposed method include rotation invariant classification and, therefore, lowered data augmentation requirements during learning.
  We make use of a conformal embedding to obtain higher-order decision surfaces. Following the motivation in the recent work of Melnyk et al. (2021), we focus on 3D geometry and spherical decision surfaces, arguing for their natural suitability for problems in Euclidean space. We show how a spherical neuron (classifier), i.e., the hypersphere neuron (Banarer et al., 2003b) or its generalization for 3D input point sets — the geometric neuron (Melnyk et al., 2021) — can be turned into a steerable neuron. We prove that the aforementioned spherical neurons in any dimension require only up to first-degree spherical harmonics to accommodate the effect of rotation. This allows us to derive a 3D steerability constraint for the spherical neurons and to describe a recipe for creating a steerable model from a pretrained spherical classifier.
  Using the synthetic Tetris dataset (Thomas et al., 2018) and the skeleton data from the UTKinectAction3D dataset (Xia et al., 2012), we first verify the derived constraint and check its stability with respect to perturbations in the input. We further conduct an experiment in a realistic setting, where we initialize the steerable model parameters using a noisy or imperfect prediction of the transformation applied to the input and optimize the parameters in an unsupervised way.
  The contributions of our work are as follows: (a) We prove that the activation of spherical neurons on rotated input only varies by up to first-degree spherical harmonics (Section 4.1.1). (b) Based on a minimal set of four spherical neurons that are rotated to the corresponding vertices of a regular tetrahedron (Section 4.1.2), we derive the main result of our paper (Section 4.1.3) – a 3D steerability constraint for spherical neurons (13). (c) We propose a method to turn a trained spherical classifier into a fully steerable model (Section 5.2) that produces predictions invariant to 3D rotations, even in the presence of noise (Section 5.3), which we verify experimentally using both synthetic and real 3D data.
sections.heading: 2 RELATED WORK
sections.heading: 2.1 STEERABILITY AND EQUIVARIANCE
sections.text: Steerability is a powerful concept from early vision and image processing (Freeman et al., 1991; Knutsson et al., 1992; Simoncelli et al., 1992; Perona, 1995; Simoncelli & Freeman, 1995; Teo & Hel-Or, 1998) that resonates in the era of deep learning. The utility of steerable filters and the main theorems for their construction are presented in the seminal work of Freeman et al. (1991). Geometric equivariance in the context of computer vision has been an object of extensive research over the last decades (Van Gool et al., 1995). Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator. For this reason, Lie theory can be used to to study steerable filters for equivariance and invariance (Reisert, 2008).
  Nowadays, equivariance-related research extends into deep learning, e.g., the SE(3)-equivariant models of Fuchs et al. (2020), Thomas et al. (2018), and Zhao et al. (2020), and the SO(3)-equivariant network of Anderson et al. (2019), as well as Marcos et al. (2017) and Kondor et al. (2018). Also steerable filter concepts are increasingly used in current works, e.g., Cohen & Welling (2016) considered image data and proposed a CNN architecture to produce equivariant representations with steerable features, which involves fewer parameters than traditional CNNs. They considered the dihedral group (discrete rotation, translation, and reflection), and the steerable representations in their work are proposed as formation of elementary feature types. One limitation of their approach is that rotations are restricted to four orientations, i.e., by π/2. More recently, Weiler et al. (2018b) utilized group convolutions and introduced steerable filter convolutional neural networks (SFCNNs) operating on images to jointly attain equivariance under translations and discrete rotations. In their work, the filter banks are learned rather than fixed. Further, the work of Weiler & Cesa (2019) proposed a unified framework for E(2)-equivariant steerable CNNs and presented their general theory.
  The steerable CNNs for 3D data proposed by Weiler et al. (2018a) are closely related to our work. The authors employed a combination of scalar, vector and tensor fields as features transformed by SO(3) representations and presented a model that is equivariant to SE(3) transformations. They also considered different types of nonlinearities suitable for nonscalar components of the feature space. The novel SE(3)-equivariant approach by Fuchs et al. (2020) introduced a self-attention mechanism that is invariant to global rotations and translations of its input and solves the limitation of angularly constrained filters in other equivariant methods, e.g., Thomas et al. (2018). Noteworthy, the work of Jing et al. (2020) proposed the geometric vector perceptron (GVP) consisting of two linear transformations for the input scalar and vector features, followed by nonlinearities. The GVP scalar and vector outputs are invariant and equivariant, respectively, with respect to an arbitrary composition of rotations and reflections in the 3D Euclidean space.
  The key point distinguishing our approach from other equivariant networks is that we use conformal modeling (Li et al., 2001a; Hitzer, 2008), with which the scalar product of two points yields their Euclidean distance (allowing us to construct spherical decision surfaces) and nonlinear conformal transformations on Rn can be linearized (allowing us to steer the decision spheres). Besides, we do not constrain the space of learnable parameters (as opposed to, e.g., Thomas et al. (2018), Weiler et al. (2018a), and Fuchs et al. (2020)), but construct our steerable model from a freely trained base network, as we discuss in detail in Section 4.
sections.heading: 2.2 CONFORMAL MODELING AND THE HYPERSPHERE NEURON
sections.text: The utility of conformal embedding for Euclidean geometry and the close connection to Minkowski spaces are thoroughly discussed by Li et al. (2001a). An important result is that one can construct hyperspherical decision surfaces using representations in the conformal space (Li et al., 2001b), as done in the work of Perwass et al. (2003). The hypersphere neuron proposed by Banarer et al. (2003b) is such a spherical classifier. Remarkably, since a hypersphere can be seen as a generalization of a hyperplane, the standard neuron can be considered as a special case of the hypersphere neuron. Stacking multiple hypersphere neurons in a feed-forward network results in a multilayer hypersphere perceptron (MLHP), which was shown by Banarer et al. (2003a) to outperform the standard MLP for some classification tasks. However, its application to point sets was not discussed. This motivated the work of Melnyk et al. (2021) on the geometric neuron, where the learned parameters were shown to represent a combination of spherical decision surfaces. Moreover, the geometric neuron activations
  were proved to be isometric in the 3D Euclidean space. In our work, we use the latter observation as the necessary condition for deriving the steerability constraint. Making spherical classifiers steerable adds to the practical value of the prior work by Melnyk et al. (2021).
sections.heading: 3 BACKGROUND
sections.heading: 3.1 STEERABILITY
sections.text: As per Freeman et al. (1991), a 2D function f(x, y) is said to steer if it can be written as a linear combination of rotated versions of itself, i.e., when it satisfies the constraint
  fθ(x, y) =
  M∑
  j=1
  vj(θ)f θj (x, y) , (1)
  where vj(θ) are the interpolation functions, θj are the basis function orientations, and M is the number of basis function terms required to steer the function. An alternative formulation can be found in the work of Knutsson et al. (1992). In 3D, the steering equation becomes
  fR(x, y, z) =
  M∑
  j=1
  vj(R)fRj (x, y, z) , (2)
  where fR(x, y, z) is f(x, y, z) rotated by R ∈ SO(3), and each Rj ∈ SO(3) orients the corresponding jth basis function.
  Theorems 1, 2 and 4 in Freeman et al. (1991) describe the conditions under which the steerability constraints (1) and (2) hold, and how to determine the minimum number of basis functions for the 2D and 3D case, respectively.
sections.heading: 3.2 CONFORMAL EMBEDDING
sections.text: We refer the reader to Section 3 in the work of Melnyk et al. (2021) for more details, and only briefly introduce important notation in this section. The conformal space for the Euclidean Rn counterpart can be formed as MEn ≡ Rn+1,1 = Rn ⊕ R1,1, where R1,1 is the Minkowski plane (Li et al., 2001a) with orthonormal basis defined as {e+, e−} and null basis {e0, e∞} representing the origin e0 = 1 2 (e− − e+) and point at infinity e∞ = e− + e+. Thus, a Euclidean vector x ∈ Rn can be embedded in the conformal space MEn as
  X = C(x) = x + 1 2 ∥x∥2 e∞ + e0 , (3)
  where X ∈ MEn is called normalized. The conformal embedding (3) represents the stereographic projection of x onto a projection sphere in MEn and is homogeneous, i.e., all embedding vectors in the equivalence class [X] = { Z ∈ Rn+1,1 : Z = γX, γ ∈ R \ {0} } are taken to represent the same vector x. Importantly, given the conformal embedding X and some Y = y + 12∥y∥2 e∞ + e0, their scalar product in the conformal space corresponds to their Euclidean distance, X · Y = − 12∥x − y∥2. This interpretation of the scalar product in the conformal space is the main motivation in constructing spherical classifiers.
sections.heading: 3.3 SPHERICAL CLASSIFIERS
sections.text: By spherical classifiers, we collectively refer to hypersphere (Banarer et al., 2003a) and geometric (Melnyk et al., 2021) neurons, which have spherical decision surfaces.
  As discussed by Banarer et al. (2003a), by embedding both a data vector x ∈ Rn and a hypersphere S ∈ MEn in Rn+2 as
  X = ( x1, . . . , xn,−1,− 1
  2 ∥x∥2
  ) ∈ Rn+2, S = ( c1, . . . , cn, 1
  2 (∥c∥2 − r2), 1
  ) ∈ Rn+2 , (4)
  where c = (c1, . . . , cn) ∈ Rn is the hypersphere center and r ∈ R is the radius, their scalar product X · S in the conformal space MEn can be computed equivalently in Rn+2 as X⊤S:
  X · S = X⊤S = −1 2 ∥x − c∥2 + 1 2 r2 . (5)
  This result enables the implementation of a hypersphere neuron in MEn using the standard dot product in Rn+2. The hypersphere vector components are treated as independent learnable parameters during training. Thus, a spherical classifier effectively learns non-normalized hyperspheres of the form S̃ = (s1, . . . , sn+2) ∈ Rn+2. Due to the homogeneity of the representation, both normalized and non-normalized hyperspheres represent the same decision surface. More details can be found in Section 3.2 in the work of Melnyk et al. (2021).
  ∑
  z
  S̃1
  S̃2 S̃k
  x1 −1 `1 x2 −1 `2 xk −1 `k
  Furthermore, Melnyk et al. (2021) demonstrated that the geometric neuron activations are isometric in 3D. That is, rotating the input is equivalent to rotating the decision spheres. This result is a necessary condition to consider rotation and translation equivariance of models constructed with geometric neurons and forms the basis for our methodology.
  In the following sections, we use the same notation for a 3D rotation R represented in the Euclidean space R3, the homogeneous (projective) space P(R3), and ME3 ∼= R5, depending on the context. This is possible since we can add the required number of ones to the diagonal of the original rotation matrix without changing the transformation representation.
sections.heading: 4 METHOD
sections.text: To build a 3D steerable model, we perform the following steps: We first train an MLGP model (Melnyk et al., 2021), which consists of spherical neurons. After optimizing the model parameters, we freeze them and transform according to the 3D steerability constraint we derive. Finally, by combining the resulting parameters in filter banks and adding interpolation coefficients as free parameters, we create a steerable model.
sections.heading: 4.1 FULLY STEERABLE 3D SPHERICAL NEURONS
sections.text: In this section, we identify the conditions under which a geometric neuron as a function of its 3D input can be steered. In other words, we derive an expression that gives us the response of a hypothetical geometric neuron for some input, using rotated versions of the learned geometric neuron parameters. We start by considering the steerability conditions for a single sphere classifying the corresponding input point X, i.e., f(X) = X⊤S, where X and S are embedded in ME3 ∼= R5 according to (4). Since the geometric neuron output is a linear combination of these functions, as per (6), we will use the identified conditions to build a steerable feed-forward network by reassembling modified spheres.
sections.heading: 4.1.1 BASIS CONSTRUCTION
sections.text: To formulate a steerability constraint for a spherical neuron (sphere), first, we need to determine the minimum number of basis functions, i.e., the number of terms M in (2). This number only depends on the degree of the spherical harmonics that are required to compute the steered result (Freeman et al., 1991). Thus, we need to determine the required degrees. Theorem. Let S ∈ Rn+2 be an nD classifier with center c ∈ Rn (R := ∥c∥) and radius r, and x ∈ Rn be a point represented by X ∈ Rn+2, see (4). Let further S′ be the classifier that is obtained by rotating S in nD space, then X⊤S′ and X⊤S are related by spherical harmonics up to first degree.
  Proof. Without loss of generality, the rotation is defined by the plane of rotation π by the angle θ. Denote the projection of a vector v ∈ Rn onto π by vπ and define v⊥π = v− vπ . From (5) we obtain
  2X⊤S = r2 −∥x − c∥2 = r2 − ∥∥(x − c)⊥π ∥∥2 − ∥∥(x − c)π ∥∥2 . A rotation in π only affects the rightmost term above and there exists a ϕ ∈ [0, 2π) such that
  ∥∥(x − c)π ∥∥2 =∥xπ − cπ∥2 =∥xπ∥2 +∥cπ∥2 − 2∥xπ∥ ∥cπ∥ cosϕ .
  With a similar argument, we obtain
  2X⊤S′ = r2 − ∥∥(x − c)⊥π ∥∥2 −∥xπ∥2 −∥cπ∥2 + 2∥xπ∥ ∥cπ∥ cos(ϕ+ θ) .
  This result is valid in any dimension, but we are primarily interested in n = 2 and n = 3, as illustrated in Figure 2. Following the result of Theorem 4 in Freeman et al. (1991) and using N = 1, we have that M = (N + 1)2 = 4 basis functions suffice in the 3D case (2).
sections.heading: 4.1.2 SPHERICAL FILTER BANKS IN 3D
sections.text: In 3D, we thus select four rotated versions of the function f(X), as the basis functions. The rotations {Rj}4j=1 must be chosen to satisfy the condition (b) in Theorem 4 (Freeman et al., 1991). Therefore, we transform f(X) such that the resulting four spheres are spaced in three dimensions equally, i.e., form a regular tetrahedron with the vertices (1, 1, 1), (1,−1,−1), (−1, 1,−1), and (−1,−1, 1), as shown in Figure 3 b). We stack the homogeneous coordinates of the tetrahedron vertices mj in a matrix column-wise (scaled by 1/2) to get the orthogonal matrix
  M = [ m1 m2 m3 m4 ] = 1
  2
   
  1 1 −1 −1 1 −1 1 −1 1 −1 −1 1 1 1 1 1
    . (7)
  We will use this matrix operator M to compute the linear coefficients in the vector space generated by the vertices of the regular tetrahedron (Granlund & Knutsson, 1995). This will be necessary to find the appropriate interpolation functions and formulate the steerability constraint in Section 4.1.3.
  The four rotated versions of the function f(X) will constitute the basis functions that we call a spherical filter bank. To construct this filter bank, we choose the following convention. The originally learned spherical classifier f(X) = X⊤S is first rotated to (1, 1, 1) (see Figure 3 b)) with the corresponding transformation denoted as RO. Next, we rotate the transformed sphere into the other three vertices of the regular tetrahedron and transform back to the original coordinate system (see Figure 3 a) for the case of (1,−1,−1)). The resulting filter bank for one spherical classifier is thus composed as the following 20× 1 matrix:
  B(S) = [ R⊤O RTi RO S ] , (8)
  where each of {RTi}3i=0 is the rotation isomorphism in R5 corresponding to a 3D rotation from (1, 1, 1) to the vertex i+ 1 of the regular tetrahedron. Therefore, RT0 = I5.
sections.heading: 4.1.3 3D STEERABILITY CONSTRAINT
sections.text: The steerability constraint can be formulated as follows. For an arbitrary orientation R applied to the input of the function f(X), we want the output of the spherical filter bank B(S) in (8) to be interpolated with vj(R) such that the response is equal to the original function output, i.e.,
  f(X) = fR(RX) = M∑
  j=1
  vj(R) fRj (RX) = (v(R)⊗ RX)⊤B(S), (9)
  where X ∈ R5 is a single, appropriately embedded, 3D point, v(R) ∈ R4 is a vector of the interpolation coefficients, and ⊗ denotes the Kronecker product (Horn & Johnson, 1991). The coefficients v(R) should conform to the basis function construction (condition (b) in Theorem 4 in Freeman et al. (1991)), which is why they are computed with M defined in (7). Given X ∈ R5 as input and an unknown rotation R acting on it, the steering equation (9) implies
  (v(R)⊗ RX)⊤B(S) = (v(I)⊗ X)⊤B(S) . (10) Given a tetrahedron rotation, e.g., RT1, as shown in the diagram in Figure 3 a), we can define the unknown rotation accordingly as R = R⊤ORT1RO. In this case, it is easy to see that to satisfy the constraint (10), v(R) must be (0, 1, 0, 0), i.e., the second filter in the filter bank B(S) must be chosen. This can be achieved by transforming a constant vector m1 = ( 12 , 1 2 , 1 2 , 1 2 ) by rotation RT1 and multiplying it by the basis matrix M as follows:
  v(R) = M⊤(RT1 m1) = M⊤ (1 2 ,−1 2 ,−1 2 , 1 2 ) = (0, 1, 0, 0) . (11)
  Note that, in general, a geometric neuron (6) takes a set of embedded points as input. Therefore, with the setup above, R will be different for each input shape point k if the same v is used for all k, which contradicts that the shape is transformed by a rigid body motion, i.e., the same RB for all k. Thus, we need to consider a suitable vector vk for each input shape point k, such that the resulting RB is the same for all k. This can be achieved recalling how we construct the basis functions in the spherical filter bank (8): we need to consider the respective initial rotation RkO. The desired interpolation coefficients vk are thus computed as
  vk(RB) = M⊤ (RkORBR k⊤ O m1) . (12)
  The resulting vk(RB) interpolate the responses of the tetrahedron-copies of the originally learned sphere Sk to replace the rotated sphere.
  By plugging (9) into (6), we can now establish the main result of our paper – the steerability constraint for a geometric neuron that takes a set of N embedded points {Xk}k as input:
  fR(RX) = N∑
  k=1
  γk f k R(RXk) =
  N∑
  k=1
  γk (v k(R)⊗ RXk)⊤B(Sk) . (13)
sections.heading: 5 EXPERIMENTS
sections.text: We use a single GTX 1050 Ti GPU and conduct two types of experiments to confirm our findings presented in Section 4.
  original
  derotated
sections.heading: 5.1 DATASETS
sections.text: 3D Tetris Following the experiments reported by Thomas et al. (2018), Weiler et al. (2018a), and Melnyk et al. (2021), we use the synthetic point set of eight 3D Tetris shapes consisting of four points each (see, e.g., Figure 3 in Melnyk et al. (2021) and the numerical description in Appendix A).
  3D skeleton data We also perform experiments on real-world data to substantiate the validity of our theoretical results. We use the UTKinect-Action3D dataset introduced by Xia et al. (2012), in particular, the 3D skeletal joint locations extracted from Kinect depth maps. For each action sequence and from each frame, we extract the skeleton consisting of twenty points and assign it to the class of actions (ten categories) this frame is part of. Therefore, we formulate the task as shape recognition (i.e., a kind of action recognition from a static point cloud), where each shape is of size 20 × 3. Since the orientations of the shapes vary significantly across the sequences, we perform the following standardization: We first center each shape at the origin, and then compute the orientation from its three hip joints and derotate the shape in the xy-plane (viewer coordinate system) accordingly. We illustrate the effect of derotation in Figure 4. From each action sequence, we randomly select 50% of the skeletons for the test set and 20% of the remainder as validation data. The resulting data split is as follows: 2295 training shapes, 670 shapes for validation, and 3062 test shapes.
sections.heading: 5.2 STEERABLE MODEL CONSTRUCTION
sections.text: To construct and test steerable models, we perform the same steps for both datasets. The minor differences are the choice of training hyperparameters and the presence of validation and test subsets in the 3D skeleton dataset.
  We first train a two-layer MLGP model (Melnyk et al., 2021), where the hidden layer consists of geometric neurons and the output layer of hypersphere neurons, to classify the shapes. Since the architecture choice is not the objective of the experiments, when building the MLGP, we use only one configuration with five hidden units for the Tetris data and twelve hidden units (determined by using the validation data) for the 3D skeleton dataset throughout the experiments. Similar to Melnyk et al. (2021), we do not use any activation function in the hidden layer due to the nonlinearity of the conformal embedding. We implement both MLGP models in PyTorch (Paszke et al., 2019) and keep the default parameter initialization for the linear layers. We train both models by minimizing the cross-entropy loss function and use the Adam optimizer (Kingma & Ba, 2015) with the default hyperparameters (the learning rate is 0.001). The Tetris MLGP learns to classify the eight shapes in the canonical orientation perfectly after 2000 epochs, whereas the Skeleton model trained for 10000 epochs achieves a test set accuracy of 92.9%. We refer to both trained models as ancestor MLGP.
  For both, we then freeze the trained parameters and construct a steerable model. Note that we form steerable units only in the hidden layer and keep the output, i.e., classification, layer hypersphere neurons as they are. The steerability is not required for the subsequent layers as the output of the first layer becomes rotation-independent. The steerable units are formed from the corresponding frozen parameters as the (fixed) filter banks according to (8).
  The only free parameters of this constructed steerable model are interpolation coefficients vk(RB) ∈ R4 defined in (12), where k indexes the learned hidden layer parameters (spheres) in the ancestor MLGP model. But since the only variables in (12) are the parameters of the unknown rotation RB acting on the input, we can directly optimize them if they are unknown. We select the axis–angle rotation representation because it has only three parameters — the three coordinates of the rotation axis scaled by the rotation angle.
  We propose the following basic recipe to create a steerable model: Train the ancestor MLGP → Fix the learned parameters → Transform the hidden unit parameters into filter banks (8) → Add the interpolation coefficients vk as free parameters to fulfill (13) → Steerable spherical classifier.
sections.heading: 5.3 KNOWN ROTATION EXPERIMENT
sections.text: Using the trained MLGP, we verify the correctness of (13). We first rotate the original data and then use this ground truth rotation to initialize the interpolation coefficients of the constructed steerable model according to (12). Our intuition is that if the steerability constraint (13) is correct, then, given the transformed point set, the activations of the steerable units in the steerable model will be equal to the activations of the geometric neurons in the ancestor MLGP model fed with the point set in the canonical orientation. Hence, the classification accuracies of the ancestor and steerable models on the original and transformed datasets, respectively, should be equal.
  We run this experiment 1000 times. Each time, we generate a random rotation and apply it to the original point set (in case of the 3D skeleton data, we use the test split). We use these rotation parameters to create a steerable model and evaluate it on the transformed point set. To verify the stability of the steerable unit activations, we add uniform noise to the transformed points, n ∼ U(−a, a), where the range of a is motivated by the magnitude of the points in the datasets. For reference, we also present the accuracy of the ancestor model classifying the rotated data. We summarize the results in Table 1. The respective L1 distance results are presented in Appendix B. The analogous Tetris experiment results reported in related work are 100% (over 25 runs and including random translation) (Thomas et al., 2018) and 99±2% (over 17 runs) (Weiler et al., 2018a). However, due to the significant conceptual difference between our approach and these methods, as mentioned at the end of Section 2.1, the results are not directly comparable.
sections.heading: 5.4 ESTIMATED ROTATION EXPERIMENT
sections.text: In this experiment, we feed some rotation information into the system. For example, this information can come from another network branch that performs regression on the rotation, or from a dynamical model as in tracking. In either case, we cannot assume that the rotation is entirely accurate, unlike the experiment in Section 5.3. Therefore, we check the stability with respect to the noise in the rotation. Same as before, we exploit the MLGP trained on the respective point set in the canonical orientation, as described in Section 5.2, to build the steerable model. At each experiment run, we randomly select one shape (from the test subset in case of the 3D skeleton data) and apply a random rotation, RGT, to it. We then perturb this ground truth rotation by varying the rotation angle. We achieve this by sampling the rotation angle from a Gaussian distribution with σ = π/18 = 10° and various means, constructing a rotation of this sampled angle about a random axis, Rnoise, and multiplying with the ground truth: Rinit = Rnoise RGT. We use the resulting rotation Rinit in the axis-angle representation to initialize the steerable model parameters.
  Further, we perform the optimization of the rotation vector parameters, which we call online optimization. Since inference time, we do not have access to the labels. Thus, we need to select a loss based on the consistency of the output. Here, we choose the entropy, i.e., L(p) = −∑Ki=1 pi log(pi), where p ∈ RK is the softmax output of the model. Our intuition is that by minimizing the entropy in the output with respect to the steerable model parameters, we effectively force the model to produce a more confident prediction. For this optimization procedure, we use Adam with learning rate equal to 0.01, and empirically determine the number of epochs of 300 to be sufficient for the Tetris data and 100 for the 3D skeleton data.
  Note that at each run, we provide the model with only one transformed shape, which implies that the accuracy per run is binary, i.e., the shape is either recognized ("1") or misclassified ("0"). To verify the usefulness of the proposed online optimization method, we compare the model classification accuracy before and after optimization. The results for this experiment are summarized in Table 2. The respective L1 distance results are presented in Appendix B.
sections.heading: 6 DISCUSSION AND CONCLUSION
sections.text: Enabled by the complete understanding of the geometry of the spherical classifiers (Banarer et al., 2003b; Perwass et al., 2003; Melnyk et al., 2021), we show in Section 4 that we only need the spherical harmonics of degree up to N = 1 to determine the effect of rotation on the activations in 3D. Using this result, we derive a novel 3D steerability constraint (13). The conducted experiment in Section 5.3 shows that the derived constraint is correct since the constructed steerable model produces accurate predictions for the rotated shapes, provided that the rotation is known. From Table 1, we can see that that rotating the data without steering the neurons leads to substantial degradation of accuracy (from 99.2% to 25.2%), whereas using the steerability equation maintains the high accuracy. Moreover, the steerable model classification error only moderately increases with the level of noise in the input data, which is a clear indication of the robustness of the classifier.
  The experiment presented in Section 5.4 considers a more realistic setting, where the rotation applied to the input is not known exactly but is inaccurately estimated. However, unlike the first experiment, we perform an optimization to improve the initial steerable model prediction. The only three parameters of the model — the axis–angle representation of the unknown external rotation — determining the interpolation coefficients vk, are optimized by minimizing the entropy in the model output. As displayed in Table 2, our steerable model, prior to the optimization, produces rotation-invariant predictions for the transformed shapes for both synthetic and real data, which points to the robustness of our method. The proposed online optimization further improves the model classification accuracy. The lower accuracies for the zero-mean noise (the first row in Table 2) for the skeleton data compared to the π/36 case (the second row) are presumably caused by a small error in the shape orientations after the performed standardization.
  The power of our approach lies in the geometric explainability inherent to the spherical units that constitute our model. Seemingly complicated conformal algebra operations have a surprisingly straightforward interpretation in Euclidean space and allow us to build a steerable feed-forward neural network with a purely geometric motivation. The steerable spherical neurons discussed in our work can be employed in other architectures, e.g., CNNs, and therefore, the equivariance to transformations beyond rotation can be considered.
  ETHICS STATEMENT
  As the nature of the present paper is mainly theoretical, no direct ethical issues arise, nor are there any immediate societal consequences.
  REPRODUCIBILITY STATEMENT
  We provide the implementation of the proposed method, as well as notebooks with the conducted experiments, as part of the supplementary material.
sections.heading: L: [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 0)],
sections.heading: T: [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 1)],
sections.text: zigzag: [(0, 0, 0), (1, 0, 0), (1, 1, 0), (2, 1, 0)].
sections.heading: B KNOWN AND ESTIMATED ROTATION EXPERIMENTS: L1 DISTANCES
sections.text: We compare the hidden unit activations of the ancestor MLGP and constructed steerable models by computing the L1 distance (see Table 3). For convenience, we call the hidden unit activations of the ancestor MLGP fed with the shapes in the canonical orientation ground truth activations. From Table 3, we can see that the L1 distance to the ground truth activations linearly and moderately increases with the level of noise in the input data, which is a clear indication of the robustness of the classifier and corresponds well to the model accuracy results in Table 1. Without steerability (“Ancestor” columns), the respective L1 distances increase by one order of magnitude and basically independently of the noise level, demonstrating the significant improvement by the steerability.
  The results presented in Table 4 indicate that the proposed online optimization, besides improving the classification accuracy (see Table 2), results in the decrease of the L1 distance to the ancestor MLGP softmax output values (referred to as ground truth output) for the Tetris data. Note the low absolute level of the error already for the initial result. In case of the 3D skeleton data, the absolute error level is also very low, but we observe a slight increase of the L1 distances after the optimization despite the improved accuracy (see Table 2). We presume it is caused by the increased number of hidden units in the ancestor model compared to the Tetris case (12 vs. 5), which might not all be equally relevant to the classification output.
title: FULLY STEERABLE 3D SPHERICAL NEURONS