text:
\section{Group Theory and Tensor Field Networks}
\paragraph{Groups}
A \emph{group} is an abstract mathematical concept. Formally a group $(G,\circ)$ consists of a set $G$ and a binary composition operator $\circ: G \times G \to G$ (typically we just use the symbol $G$ to refer to the group). All groups must adhere to the following 4 axioms
\begin{itemize}
    \item \textbf{Closure}: $g\circ h \in G$ for all $g, h \in G$
    \item \textbf{Associativity}: $f \circ (g \circ h) = (f \circ g) \circ h = f \circ g \circ h$ for all $f, g, h \in G$
    \item \textbf{Identity}: There exists an element $e \in G$ such that $e \circ g = g \circ e = g$ for all $g \in G$
    \item \textbf{Inverses}: For each $g \in G$ there exists a $g^{-1} \in G$ such that $g^{-1} \circ g = g \circ g^{-1} = e$
\end{itemize}
In practice, we omit writing the binary composition operator $\circ$, so would write $gh$ instead of $g \circ h$. Groups can be finite or infinite, countable or uncountable, compact or non-compact. Note that they are not necessarily \emph{commutative}; that is, $gh \neq hg$ in general.

\paragraph{Actions/Transformations}
Groups are useful concepts, because they allow us to describe the structure of \emph{transformations}, also sometimes called \emph{actions}. A transformation (operator) $T_g: \mc{X} \to \mc{X}$ is an injective map from a space into itself. It is parameterised by an element $g$ of a group $G$. Transformations obey two laws:
\begin{itemize}
    \item \textbf{Closure}: $T_g \circ T_h$ is a valid transformation for all $g, h \in G$
    \item \textbf{Identity}: There exists at least one element $e \in G$ such that $T_e[\bb{x}] = \bb{x}$ for all $\bb{x} \in \mc{X}$,
\end{itemize}
where $\circ$ denotes composition of transformations. For the expression $T_g[\bb{x}]$, we say that $T_g$ \emph{acts} on $\bb{x}$. It can also be shown that transformations are associative under composition. To codify the structure of a transformation, we note that due to closure we can always write
\begin{align}

    T_g \circ T_h = T_{gh},
\end{align}
If for any $x, y \in \mc{X}$ we can always find a group element $g$, such that $T_g[x] = y$, then we call $\mc{X}$ a \emph{homogeneous space}. Homogeneous spaces are important concepts, because to each pair of points $x, y$ we can always associate at least one group element.

\paragraph{Equivariance and Intertwiners}
As written in the main body of the text, equivariance is a property of functions $f: \mc{X} \to \mc{Y}$. Just to recap, given a set of transformations $T_g: \mc{X} \to \mc{X}$ for $g \in G$, where $G$ is an abstract group, a function $f: \mc{X} \to \mc{Y}$ is called equivariant if for every $g$ there exists a transformation $S_g: \mc{Y} \to \mc{Y}$ such that
\begin{align}
    S_g[f(\bb{x})] = f(T_g[\bb{x}]) \qquad \text{for all } g\in G, \bb{x}\in \mc{X}.
\end{align}
If $f$ is linear and equivariant, then it is called an intertwiner. Two important questions arise: 1) How do we choose $S_g$? 2) once we have $(T_g, S_g)$, how do we solve for $f$? To answer these questions, we need to understand what kinds of $S_g$ are possible. For this, we review representations.

\paragraph{Representations}
A group representation $\rho: G \to GL(N)$ is a map from a group $G$ to the set of $N\times N$ invertible matrices $GL(N)$. Critically $\rho$ is a \emph{group homomorphism}; that is, it satisfies the following property $\rho(g_1 g_2) = \rho(g_1)\rho(g_2)$ for all $g_1, g_2 \in G$. Representations can be used as transformation operators, acting on $N$-dimensional vectors $\bb{x} \in \mbb{R}^N$. For instance, for the group of 3D rotations, known as $SO(3)$, we have that 3D rotation matrices, $\rho(g) = \bb{R}_g$ act on (i.e., rotate) 3D vectors, as
\begin{align}
    T_g[\bb{x}] = \rho(g)\bb{x} = \bb{R}_g \bb{x}, \qquad \text{for all } \bb{x} \in \mc{X}, g \in G.
\end{align}
However, there are many more representations of $SO(3)$ than just the 3D rotation matrices. Among representations, two representations $\rho$ and $\rho'$ of the same dimensionality are said to be \emph{equivalent} if they can be connected by a similarity transformation
\begin{align}
    \rho'(g) = \bb{Q}^{-1} \rho(g) \bb{Q}, \qquad \text{for all } g \in G.
\end{align}
We also say that a representation is \emph{reducible} if is can be written as
\begin{align}
    \rho(g) = \bb{Q}^{-1} (\rho_1(g) \oplus \rho_2(g)) \bb{Q} = \bb{Q}^{-1} \begin{bmatrix} \rho_1(g) & \\ & \rho_2(g) \end{bmatrix} \bb{Q}, \qquad \text{for all } g \in G.
\end{align}
If the representations $\rho_1$ and $\rho_2$ are not reducible, then they are called \emph{irreducible representations} of $G$, or \emph{irreps}. In a sense, they are the atoms among representations, out of which all other representations can be constructed. Note that each irrep acts on a separate subspace, mapping vectors from that space back into it. We say that subspace $\mc{X}_\ell \in \mc{X}$ is \emph{invariant under} irrep $\rho_\ell$, if $\{\rho_\ell(g)\bb{x} \,\, | \,\, \bb{x} \in \mc{X}_\ell, g \in G \} \subseteq \mc{X}_\ell$.

\paragraph{Representation theory of $SO(3)$}
As it turns out, all linear representations of compact groups\footnote{Over a field of characteristic zero.} (such as $SO(3)$) can be decomposed into a direct sum of irreps, as
\begin{align}
    \rho(g) = \bb{Q}^{\top} \left [ \bigoplus_{J} \bb{D}_{J}(g) \right ] \bb{Q}, \label{eq:compact-decomposition}
\end{align}
where $\bb{Q}$ is an orthogonal, $N \times N$, change-of-basis matrix \citep{chirikjian2001engineering}; and each $\bb{D}_J$ for $J = 0,1,2,...$ is a $(2J+1) \times (2J + 1)$ matrix known as a Wigner-D matrix. The Wigner-D matrices are the irreducible representations of $SO(3)$. We also mentioned that vectors transforming according to $\bb{D}_J$ (i.e.\ we set $\bb{Q} = \bb{I}$), are called \emph{type}-$J$ vectors. Type-0 vectors are invariant under rotations and type-1 vectors rotate according to 3D rotation matrices. Note, type-$J$ vectors have length $2J+1$. In the previous paragraph we mentioned that irreps act on orthogonal subspaces $\mc{X}_0, \mc{X}_1, ...$. The orthogonal subspaces corresponding to the Wigner-D matrices are the space of \emph{spherical harmonics}.

\paragraph{The Spherical Harmonics}
The spherical harmonics $\bb{Y}_J: S^2 \to \mbb{C}^{2J+1}$ for $J \geq 0$ are square-integrable complex-valued functions on the sphere $S^2$. They have the satisfying property that they are rotated directly by the Wigner-D matrices as
\begin{align}
    \bb{Y}_J(\bb{R}_g^{-1}\bb{x}) = \bb{D}_{J}^*(g)\bb{Y}_{J}(\bb{x}), \qquad \bb{x} \in S^2, g\in G, \label{eq:spherical-harmonics}
\end{align}
where $\bb{D}_J$ is the $J$\textsuperscript{th} Wigner-D matrix and $\bb{D}_J^*$ is its complex conjugate. They form an orthonormal basis for (the Hilbert space of) square-integrable functions on the sphere $L^2(S^2)$, with inner product given as
\begin{align}
    \la f, h \ra_{S^2} = \int_{S^2} f(\bb{x}) h^*(\bb{x}) \, \mathrm d\bb{x}.
\end{align}
So $\la Y_{Jm}, Y_{J'm'} \ra_{S^2} = \delta_{JJ'} \delta_{mm'}$, where $Y_{Jm}$ is the $m$\textsuperscript{th} element of $\bb{Y}_J$. We can express any function in $L^2(S^2)$ as a linear combination of spherical harmonics, where
\begin{align}
    f(\bb{x}) = \sum_{J \geq 0} \bb{f}_J^\top \bb{Y}_{J}(\bb{x}), \qquad \bb{x} \in S^2,
\end{align}
where each $\bb{f}_J$ is a vector of coefficients of length $2J+1$. And in the opposite direction, we can retrieve the coefficients as
\begin{align}
    \bb{f}_J = \int_{S^2} f(\bb{x}) \bb{Y}_{J}^* (\bb{x}) \, \mathrm{d} \bb{x}
\end{align}
following from the orthonormality of the spherical harmonics.
This is in fact a Fourier transform on the sphere and the the vectors $\bb{f}_J$ can be considered Fourier coefficients. Critically, we can represent rotated functions as
\begin{align}
    f(\bb{R}_g^{-1}\bb{x}) = \sum_{J \geq 0} \bb{f}_J^\top \bb{D}_{J}^*(g)\bb{Y}_{J}(\bb{x}), \qquad \bb{x} \in S^2, g\in G.
\end{align}

\paragraph{The Clebsch-Gordan Decomposition}
In the main text we introduced the \emph{Clebsch-Gordan coefficients}. These are used in the construction of the equivariant kernels. They arise in the situation where we have a tensor product of Wigner-D matrices, which as we will see is part of the equivariance constraint on the form of the equivariant kernels. In representation theory a tensor product of representations is also a representation, but since it is not an easy object to work with, we seek to decompose it into a direct sum of irreps, which are easier. This decomposition is of the form of \Cref{eq:compact-decomposition}, written
\begin{align}
    \bb{D}_k(g) \otimes \bb{D}_\ell(g) = \bb{Q}^{\ell k\top} \left [ \bigoplus_{J = |k - \ell|}^{k + \ell} \bb{D}_J(g) \right ] \bb{Q}^{\ell k}.
\end{align}
In this specific instance, the change of basis matrices $\bb{Q}^{\ell k}$ are given the special name of the Clebsch-Gordan coefficients. These can be found in many mathematical physics libraries.

\paragraph{Tensor Field Layers}
In Tensor Field Networks \cite{ThomasSKYKR18} and 3D Steerable CNNs \cite{WeilerGWBC18}, the authors solve for the intertwiners between SO(3) equivariant point clouds. Here we run through the derivation again in our own notation.

We begin with a point cloud $f(\bb{x}) = \sum_{j=1}^N\bb{f}_j \delta(\bb{x} - \bb{x}_j)$, where $\bb{f}_j$ is an equivariant point feature. Let's say that $\bb{f}_j$ is a type-$k$ feature, which we write as $\bb{f}_j^k$ to remind ourselves of the fact. Now say we perform a convolution $*$ with kernel $\bb{W}^{\ell k}: \bb{R}^3 \to \mbb{R}^{(2\ell+1)\times(2k +1)}$, which maps from type-$k$ features to type-$\ell$ features. Then
\begin{align}
    \bb{f}_\text{out,i}^\ell &= [\bb{W}^{\ell k} * \bb{f}_{\text{in}}^k](\bb{x}) \\
    &= \int_{\mbb{R}^3} \bb{W}^{\ell k}(\bb{x}' - \bb{x}_i)\bb{f}_{\text{in}}^k(\bb{x}') \, \mathrm{d} \bb{x}' \\
    &= \int_{\mbb{R}^3} \bb{W}^{\ell k}(\bb{x}' - \bb{x}_i) \sum_{j=1}^N \bb{f}_{\text{in},j}^k\delta(\bb{x}' - \bb{x}_j) \, \mathrm{d} \bb{x}' \\
    &= \sum_{j=1}^N \int_{\mbb{R}^3} \bb{W}^{\ell k}(\bb{x}' - \bb{x}_i)  \bb{f}_{\text{in},j}^k\delta(\bb{x}' - \bb{x}_j) \, \mathrm{d} \bb{x}' && \text{change of variables } \bb{x}'' = \bb{x}' - \bb{x}_j \\
    &= \sum_{j=1}^N \int_{\mbb{R}^3} \bb{W}^{\ell k}(\bb{x}'' + \bb{x}_j - \bb{x}_i)  \bb{f}_{\text{in},j}^k\delta(\bb{x}'') \, \mathrm{d} \bb{x}'' && \text{sifting theorem }\\
    &= \sum_{j=1}^N \bb{W}^{\ell k}(\bb{x}_j - \bb{x}_i)  \bb{f}_{\text{in},j}^k. \label{eq:tfn-conv}
\end{align}
Now let's apply the equivariance condition to this expression, then
\begin{align}
    \bb{D}_\ell(g) \bb{f}_\text{out,i}^\ell &= \sum_{j=1}^N \bb{W}^{\ell k}(\bb{R}_g^{-1}(\bb{x}_j - \bb{x}_i)) \bb{D}_k(g) \bb{f}_{\text{in},j}^k \\
    \implies \bb{f}_\text{out,i}^\ell &= \sum_{j=1}^N \bb{D}_\ell(g)^{-1} \bb{W}^{\ell k}(\bb{R}_g^{-1}(\bb{x}_j - \bb{x}_i)) \bb{D}_k(g) \bb{f}_{\text{in},j}^k
\end{align}
Now we notice that this expression should also be equal to \Cref{eq:tfn-conv}, which is the convolution with an unrotated point cloud. Thus we end up at
\begin{align}
\boxed{
    \bb{W}^{\ell k}(\bb{R}_g^{-1}\bb{x}) = \bb{D}_\ell(g) \bb{W}^{\ell k}(\bb{x}) \bb{D}_k(g)^{-1},
}
\end{align}
which is sometimes refered to as the \emph{kernel constraint}. To solve the kernel constraint, we notice that it is a linear equation and that we can rearrange it as
\begin{align}
    \text{vec}(\bb{W}^{\ell k}(\bb{R}_g^{-1}\bb{x})) = (\bb{D}_k(g) \otimes \bb{D}_\ell(g)) \text{vec}(\bb{W}^{\ell k}(\bb{x}))
\end{align}
where we used the identity $\text{vec}(\bb{AXB}) = (\bb{B}^\top \otimes \bb{A})\text{vec}(\bb{X})$ and the fact that the Wigner-D matrices are orthogonal. Using the Clebsch-Gordan decomposition we rewrite this as
\begin{align}
    \text{vec}(\bb{W}^{\ell k}(\bb{R}_g^{-1}\bb{x}))  = \bb{Q}^{\ell k\top} \left [ \bigoplus_{J = |k - \ell|}^{k + \ell} \bb{D}_J(g) \right ] \bb{Q}^{\ell k} \text{vec}(\bb{W}^{\ell k}(\bb{R}_g^{-1}\bb{x})) .
\end{align}
Lastly, we can left multiply both sides by $\bb{Q}^{\ell k}$ and denote $\bm{\eta}^{\ell k}(\bb{x}) \triangleq \bb{Q}^{\ell k} \text{vec}(\bb{W}^{\ell k}(\bb{x}))$, noting the the Clebsch-Gordan matrices are orthogonal. At the same time we
\begin{align}
    \bm{\eta}^{\ell k}(\bb{R}_g^{-1}\bb{x}) =  \left [ \bigoplus_{J = |k - \ell|}^{k + \ell} \bb{D}_J(g) \right ] \bm{\eta}^{\ell k}(\bb{x}).
\end{align}
Thus we have that $\bm{\eta}^{\ell k}_J(\bb{R}_g^{-1}\bb{x})$ the $J$\textsuperscript{th} subvector of $\bm{\eta}^{\ell k}(\bb{R}_g^{-1}\bb{x})$ is subject to the constraint
\begin{align}
    \bm{\eta}_J^{\ell k}(\bb{R}_g^{-1}\bb{x}) =  \bb{D}_J(g) \bm{\eta}_J^{\ell k}(\bb{x}),
\end{align}
which is exactly the transformation law for the spherical harmonics from \Cref{eq:spherical-harmonics}! Thus one way how $\bb{W}^{\ell k}(\bb{x})$ can be constructed is
\begin{align}
    \text{vec} \left ( \bb{W}^{\ell k}(\bb{x}) \right ) = \bb{Q}^{\ell k \top} \bigoplus_{J = |k - \ell|}^{k + \ell} \bb{Y}_J(\bb{x}).
\end{align}

\section{Recipe for Building an Equivariant Weight Matrix}
One of the core operations in the SE(3)-Transformer is multiplying a feature vector $\bb{f}$, which transforms according to $SO(3)$, with a matrix $\bb{W}$ while preserving equivariance:
\begin{align}
S_g[\bb{W} * \bb{f}](\bb{x}) = [\bb{W} * T_g[\bb{f}]](\bb{x}),
\end{align}
where $T_g[\bb{f}](\bb{x}) = \rho_\text{in}(g)\bb{f}(\bb{R}_g^{-1}\bb{x})$ and $S_g[\bb{f}](\bb{x}) = \rho_\text{out}(g)\bb{f}(\bb{R}_g^{-1}\bb{x})$. Here, as in the previous section we showed how such a matrix $\bb{W}$ could be constructed when mapping between features of type-$k$ and type-$\ell$, where $\rho_\text{in}(g)$ is a block diagonal matrix of type-$k$ Wigner-D matrices and similarly $\rho_\text{in}(g)$ is made of type-$\ell$ Wigner-D matrices. $\bb{W}$ is dependent on the relative position $\bb{x}$ and underlies the linear equivariance constraints, but is also has learnable components, which we did not show in the previous section. In this section, we show how such a matrix is constructed in practice.

Previously we showed that
\begin{align}
    \text{vec} \left ( \bb{W}^{\ell k}(\bb{x}) \right ) = \bb{Q}^{\ell k \top} \bigoplus_{J = |k - \ell|}^{k + \ell} \bb{Y}_J(\bb{x}),
\end{align}
which is an equivariant mapping between vectors of types $k$ and $\ell$. In practice, we have multiple input vectors $\{\bb{f}_c^k\}_c$ of type-$k$ and multiple output vectors of type-$\ell$. For simplicity, however, we ignore this and pretend we only have a single input and single output. Note that $\bb{W}^{\ell k}$ has no learnable components. Note that the kernel constraint only acts in the angular direction, but not in the radial direction, so we can introduce scalar radial functions $\varphi_{J}^{\ell k}: \bb{R}_{\geq 0} \to \bb{R}$ (one for each $J$), such that
\begin{align}
    \text{vec} \left ( \bb{W}^{\ell k}(\bb{x}) \right ) = \bb{Q}^{\ell k \top} \bigoplus_{J = |k - \ell|}^{k + \ell} \varphi_J^{\ell k}(\|\bb{x}\|) \bb{Y}_J(\bb{x}),
\end{align}
The radial functions $\varphi_{J}^{\ell k}(\|\bb{x}\|)$ act as an independent, learnable scalar factor for each degree $J$. The vectorised matrix has dimensionality $(2\ell+1)(2k+1)$. We can unvectorise the above yielding
\begin{align}
    \bb{W}^{\ell k}(\bb{x}) &= \text{unvec}\left(\bb{Q}^{\ell k \top} \bigoplus_{J = |k - \ell|}^{k + \ell} \varphi_J^{\ell k}(\|\bb{x}\|) \bb{Y}_J(\bb{x})\right) \\
    &= \sum_{J = |k - \ell|}^{k + \ell} \varphi_J^{\ell k}(\|\bb{x}\|) \text{unvec}\left( \bb{Q}_J^{\ell k \top}  \bb{Y}_J(\bb{x}) \right )
\end{align}
where $\bb{Q}_J^{\ell k}$ is a $(2\ell+1)(2k+1) \times (2J+1)$ slice from $\bb{Q}^{\ell k}$, corresponding to spherical harmonic $\bb{Y}_J.$ As we showed in the main text, we can also rewrite the unvectorised Clebsch-Gordan--spherical harmonic matrix-vector product as
\begin{align}
    \text{unvec}\left( \bb{Q}_J^{\ell k \top}  \bb{Y}_J(\bb{x}) \right ) = \sum_{m=-J}^J \bb{Q}_{Jm}^{\ell k \top}  Y_{Jm}(\bb{x}).
\end{align}
In contrast to \citet{WeilerGWBC18}, we do not voxelise space and therefore $\bb{x}$ will be different for each pair of points in each point cloud. However, the same $\bb{Y}_J\left(\bb{x}\right)$ will be used multiple times in the network and even multiple times in the same layer. Hence, precomputing them at the beginning of each forward pass for the entire network can significantly speed up the computation. The Clebsch-Gordan coefficients do not depend on the relative positions and can therefore be precomputed once and stored on disk. Multiple libraries exist which approximate those coefficients numerically.

\section{Accelerated Computation of Spherical Harmonics}
\label{sec:sh_accelerated}
\begin{figure}[t]
    \centering
    \begin{subfigure}[c]{0.45\linewidth}
        \centering
        \includegraphics[width=\linewidth]{images/speed_cpu.pdf}
        \subcaption{Speed comparison on the CPU.}
        \label{fig:speed_cpu}
    \end{subfigure}
    \hspace*{\fill}
    \begin{subfigure}[c]{0.45\linewidth}
        \centering
        \includegraphics[width=\linewidth]{images/speed_gpu.pdf}
        \subcaption{Speed comparison on the GPU.}
        \label{fig:speed_gpu}
    \end{subfigure}
    \caption{Spherical harmonics computation of our own implementation compared to the \texttt{lie-learn} library. We found that speeding up the computation of spherical harmonics is critical to scale up both Tensor Field Networks \citep{ThomasSKYKR18} and SE(3)-Transformers to solve real-world machine learning tasks.}
    \label{fig:speed_sh}
\end{figure}

The spherical harmonics (SH) typically have to be computed on the fly for point cloud methods based on irreducible computations, a bottleneck of TFNs \citep{ThomasSKYKR18}. \citet{ThomasSKYKR18} ameliorate this by restricting the maximum type of feature to type-2, trading expressivity for speed. \citet{WeilerGWBC18} circumvent this challenge by voxelising the input, allowing them to pre-compute spherical harmonics for fixed relative positions. This is at the cost of detail as well as exact rotation and translation equivariance.

The number of spherical harmonic lookups in a network based on irreducible representations can quickly become large (number of layers $\times$ number of points $\times$ number of neighbours $\times$ number of channels $\times$ number of degrees needed). This motivates parallelised computation on the GPU - a feature not supported by common libraries. To that end, we wrote our own spherical harmonics library in \texttt{Pytorch}, which can generate spherical harmonics on the GPU. We found this critical to being able to run the SE(3)-Transformer and Tensor Field network baselines in a reasonable time. This library is accurate to within machine precision against the \texttt{scipy} counterpart \texttt{scipy.special.sph\_harm} and is significantly faster.
E.g., for a ScanObjectNN model, we achieve $\sim 22\times$ speed up of the forward pass compared to a network built with SH from the \texttt{lielearn} library. A speed comparison isolating the computation of the spherical harmonics is shown in \Cref{fig:speed_sh}. Code is available at \url{https://github.com/FabianFuchsML/se3-transformer-public}. In the following, we outline our method to generate them.

The tesseral/real spherical harmonics are given as
\begin{align}
    Y_{Jm}(\theta, \phi) = \sqrt{\frac{2 J + 1}{4 \pi}\frac{(J-m)!}{(J+m)!}} P_J^{|m|}(\cos \theta) \cdot \left \{ \begin{matrix*}[l]
     \sin(|m| \phi)  & m < 0,\\
    1 & m = 0, \\
    \cos(m \phi) & m > 0,
    \end{matrix*} \right .
\end{align}
where $P_J^{|m|}$ is the associated Legendre polynomial (ALP), $\theta \in [0, 2\pi)$ is azimuth, and $\phi \in [0,\pi]$ is a polar angle. The term $P_J^{|m|}$ is by far the most expensive component to compute and can be computed recursively. To speed up the computation, we use dynamic programming storing intermediate results in a memoization.

We make use of the following recursion relations in the computation of the ALPs:
\begin{align}
    P_J^{|J|}(x) &= (-1)^{|J|} \cdot (1-x^2)^{|J|/2} \cdot (2|J|-1)!! && \text{boundary: $J=m$} \label{eq:leg-boundary} \\
    P_J^{-m}(x) &= (-1)^J \frac{(\ell - m)!}{(\ell + m)!} P_J^{m}(x) && \text{negate $m$} \label{eq:leg-negate}\\
    P_J^{|m|}(x) &= \frac{2J-1}{J-|m|} x P_{J-1}^{m}(x) + \mbb{I}[J - |m| > 1] \frac{J + |m| - 1}{J - |m|} P_{J-2}^{m}(x) && \text{recurse} \label{eq:leg-recurse}
\end{align}
where the \emph{semifactorial} is defined as $x!! = x (x-2) (x-4) \cdots$, and $\mbb{I}$ is the indicator function. These relations are helpful because they define a recursion.

To understand how we recurse, we consider an example. \Cref{fig:subproblem} shows the space of $J$ and $m$. The black vertices represent a particular ALP, for instance, we have highlighted $P_3^{-1}(x)$. When $m <0$, we can use \Cref{eq:leg-negate} to compute $P_3^{-1}(x)$ from $P_3^1(x)$. We can then use
\Cref{eq:leg-recurse} to compute $P_3^1(x)$ from $P_2^1(x)$ and $P_1^1(x)$. $P_2^1(x)$ can also be computed from \Cref{eq:leg-recurse} and the boundary value $P_1^1(x)$ can be computed directly using \Cref{eq:leg-boundary}. Crucially, all intermediate ALPs are stored for reuse. Say we wanted to compute $P_4^{-1}(x)$, then we could use \Cref{eq:leg-negate} to find it from $P_4^{-1}(x)$, which can be recursed from the stored values $P_3^1(x)$ and $P_2^1(x)$, without needing to recurse down to the boundary.

\begin{figure}
    \centering
    \includegraphics{images/AssociatedLegendre.pdf}
    \caption{Subproblem graph for the computatin of the associated Legendre polynomials. To compute $P_3^{-1}(x)$, we compute $P_3^{1}(x)$, for which we need $P_2^{1}(x)$ and $P_1^{1}(x)$. We store each intermediate computation, speeding up average computation time by a factor of $\sim10$ on CPU.}
    \label{fig:subproblem}
\end{figure}

\section{Experimental Details}
\subsection{ScanObjectNN}
\label{as}
\subsubsection{SE(3)-Transformer and Tensor Field Network}
A particularity of object classification from point clouds is the large number of points the algorithm needs to handle. We use up to 200 points out of the available 2024 points per sample and create neighbourhoods with up to 40 nearest neighbours. It is worth pointing out that especially in this setting, adding self-attention (i.e. when comparing the SE(3) Transformer to Tensor Field Networks) significantly increased the stability. As a result, whenever we swapped out the attention mechanism for a convolution to retrieve the Tensor Field network baseline, we had to decrease the model size to obtain stable training. However, we would like to stress that all the Tensor Field networks we trained were significantly bigger than in the original paper \cite{ThomasSKYKR18}, mostly enabled by the faster computation of the spherical harmonics.

For the ablation study in \Cref{fig:scanobject_rotate}, we trained networks with 4 hidden equivariant layers with 5 channels each, and up to representation degree 2. This results in a hidden feature size per point of
\begin{align}
    5\cdot \sum_{\ell=0}^2 (2\ell + 1) = 45
\end{align}
We used 200 points of the point cloud and neighbourhood size 40. For the Tensor Field network baseline, in order to achieve stable training, we used a smaller model with 3 instead of 5 channels, 100 input points and neighbourhood size 10, but with representation degrees up to 3.

We used 1 head per attention mechanism yielding one attention weight for each pair of points but across all channels and degrees (for an implementation of multi-head attention, see \Cref{sec:QM9_details}). For the query embedding, we used the identity matrix. For the key embedding, we used a square equivariant matrix preserving the number of degrees and channels per degree.

For the quantitative comparison to the start-of-the-art in \Cref{tab:Scan}, we used 128 input points and neighbourhood size 10 for both the Tensor Field network baseline and the SE(3)-Transformer. We used farthest point sampling with a random starting point to retrieve the 128 points from the overall point cloud. We used degrees up to 3 and 5 channels per degree, which we again had to reduce to 3 channels  for the Tensor Field network to obtain stable training. We used a norm based non-linearity for the Tensor Field network (as in \cite{ThomasSKYKR18}) and no extra non-linearity (beyond the \textit{softmax} in the self-attention algorithm) for the SE(3) Transformer.

For all experiments, the final layer of the equivariant encoder maps to 64 channels of degree 0 representations. This yields a 64-dimensional SE(3) \textit{invariant} representation per point. Next, we pool over the point dimension followed by an MLP with one hidden layer of dimension 64, a ReLU and a 15 dimensional output with a cross entropy loss. We trained for 60000 steps with batch size 10. We used the Adam optimizer \cite{KingmaB14} with a start learning of \texttt{1e-2} and a reduction of the learning rate by 70\% every 15000 steps. Training took up to 2 days on a system with 4 CPU cores, 30 GB of RAM, and an NVIDIA GeForce GTX 1080 Ti GPU.

The input to the Tensorfield network and the Se(3) Transformer are relative x-y-z positions of each point w.r.t. their neighbours. To guarantee equivariance, these inputs are provided as fields of degree 1. For the `+z` versions, however, we deliberately break the SE(3) equivariance by providing additional and relative z-position as two additional scalar fields (i.e. degree 0), as well as relative x-y positions as a degree 1 field (where the z-component is set to 0).

\subsubsection{Number of Input Points}
\label{sec:app_n_points}
Limiting the input to 128/200 points in our experiments on ScanObjectNN was not primarily due to computational limitations: we conducted experiments with up to 2048 points, but without performance improvements. We suspect this is due to the global pooling. Examining cascaded pooling via attention is a future research direction. Interestingly, when limiting other methods to using 128 points, the SE(3)-Transformer outperforms the baselines (PointCNN: $80.3\pm 0.8\%$, PointGLR: $81.5 \pm 1.0\%$, DGCNN: $82.2\pm0.8\%$, \textbf{ours}: $85.0\pm 0.7\%$). It is worth noting that these methods were explicitly designed for the well-studied task of point classification whereas the SE(3)-Transformer was applied as is. Combining different elements from the current state-of-the-art methods with the geometrical inductive bias of the SE(3)-Transformer could potentially yield additional performance gains, especially with respect to leveraging inputs with more points. It is also worth noting that the SE(3)-Transformer was remarkably stable with respect to lowering the number of input points in an ablation study (16 points: $79.2\%$, 32 points: $81.4\%$, 64 points: $82.5\%$, 128 points: $85.0\%$, 256 points: $82.6\%$).

\subsubsection{Sample Complexity}
Equivariance is known to often lead to smaller sample complexity, meaning that less training data is needed (Fig. 10 in \citet{WorrallGRB2016}, Fig. 4 in \citet{WinkelsCohen}, Fig. 4 in \citet{WeilerGWBC18}). We conducted experiments with different amounts of training samples $N_{\text{samples}}$ from the ScanObjectNN dataset. The results showed that for all $N_{\text{samples}}$, the SE(3)-Transformer outperformed the Set Transformer, a non-equivariant network based on attention. The performance delta was also slightly higher for the smallest $N_{\text{samples}}$ we tested ($3.1\%$ of the samples available in the training split of ScanObjectNN) than when using all the data indicating that the SE(3)-Transformer performs particularly well on small amounts of training data. However, performance differences can be due to multiple reasons. Especially for small datasets, such as ScanObjectNN, where the performance does not saturate with respect to amount of data available, it is difficult to draw conclusions about sample complexity of one model versus another. In summary, we found that our experimental results are in line with the claim that equivariance decreases sample complexity in this specific case but do not give definitive support.

\subsubsection{Baselines}
\paragraph{DeepSet Baseline} We originally replicated the implementation proposed in \cite{Zaheer2017} for their object classification experiment on ModelNet40 \citep{modelnet40}. However, most likely due to the relatively small number of objects in the ScanObjectNN dataset, we found that reducing the model size helped the performance significantly. The reported model had 128 units per hidden layer (instead of 256) and no dropout but the same number of layers and type of non-linearity as in \cite{Zaheer2017}.

\paragraph{Set Transformer Baseline} We used the same architecture as \cite{SetTransformer} in their object classification experiment on ModelNet40 \citep{modelnet40} with an ISAB (induced set attention block)-based encoder followed by PMA (pooling by multihead attention) and an MLP.

\subsection{Relational Inference}
Following \citet{KipfFWWZ18}, we simulated trajectories for \(5\) charged, interacting particles. Instead of a 2d simulation setup, we considered a 3d setup.
Positive and negative charges were drawn as Bernoulli trials (\(p=0.5\)).
We used the provided code base \texttt{https://github.com/ethanfetaya/nri} with the following modifications:
While we randomly sampled initial positions inside a \([-5,5]^3\) box, we removed the bounding-boxes during the simulation. We generated \(5\text{k}\) simulation samples for training and \(1\text{k}\) for testing. Instead of phrasing it as a time-series task, we posed it as a regression task: The input data is positions and velocities at a random time step as well as the signs of the charges. The labels (which the algorithm is learning to predict) are the positions and velocities 500 simulation time steps into the future.

\paragraph{Training Details}
We trained each model for 100,000 steps with batch size 128 using an Adam optimizer \cite{KingmaB14}. We used a fixed learning rate throughout training and conducted a separate hyper parameter search for each model to find a suitable learning rate.

\paragraph{SE(3)-Transformer Architecture}
We trained an SE(3)-Transformer with 4 equivariant layers, where the hidden layers had representation degrees $\{0,1,2,3\}$ and 3 channels per degree. The input is handled as two type-1 fields (for positions and velocities) and one type-0 field (for charges). The learning rate was set to \texttt{3e-3}. Each layer included attentive self-interaction.

We used 1 head per attention mechanism yielding one attention weight for each pair of points but across all channels and degrees (for an implementation of multi-head attention, see \Cref{sec:QM9_details}). For the query embedding, we used the identity matrix. For the key embedding, we used a square equivariant matrix preserving the number of degrees and channels per degree.

\paragraph{Baseline Architectures}
All our baselines fulfill permutation invariance (ordering of input points), but only the Tensor Field network and the linear baseline are SE(3) equivariant. For the \textbf{Tensor Field Network}\cite{ThomasSKYKR18} baseline, we used the same hyper parameters as for the SE(3) Transformer but with a linear self-interaction and an additional norm-based nonlinearity in each layer as in \citet{ThomasSKYKR18}. For the \textbf{DeepSet}\cite{Zaheer2017} baseline, we used 3 fully connected layers, a pooling layer, and two more fully connected layers with 64 units each. All fully connected layers act pointwise. The pooling layer uses max pooling to aggregate information from all points, but concatenates this with a skip connection for each point. Each hidden layer was followed by a LeakyReLU. The learning rate was set to \texttt{1e-3}. For the \textbf{Set Transformer}\cite{SetTransformer}, we used 4 self-attention blocks with 64 hidden units and 4 heads each. For each point this was then followed by a fully connected layer (64 units), a LeakyReLU and another fully connected layer. The learning rate was set to \texttt{3e-4}.

For the \textbf{linear baseline}, we simply propagated the particles linearly according to the simulation hyperparamaters. The linear baseline can be seen as removing the interactions between particles from the prediction. Any performance improvement beyond the linear baseline can therefore be interpreted as an indication for relational reasoning being performed.

\subsection{QM9} \label{sec:QM9_details} The QM9 regression dataset \citep{RamakrishnanDRvL14} is a publicly available chemical property prediction task consisting of 134k small drug-like organic molecules with up to 29 atoms per molecule. There are 5 atomic species (Hydrogen, Carbon, Oxygen, Nitrogen, and Flourine) in a molecular graph connected by chemical bonds of 4 types (single, double, triple, and aromatic bonds). `Positions' of each atom, measured in {\aa}ngtr\"oms, are provided. We used the exact same train/validation/test splits as \citet{anderson2019cormorant} of sizes 100k/18k/13k.

The architecture we used is shown in Table \ref{tab:qm9_architecture}. It consists of 7 multihead attention layers interspersed with norm nonlinearities, followed by a TFN layer, max pooling, and two linear layers separated by a ReLU. For each attention layer, shown in \Cref{fig:mab_qm9}, we embed the input to half the number of feature channels before applying multiheaded attention \cite{VaswaniSPUJGKP17}. Multiheaded attention is a variation of attention, where we partition the queries, keys, and values into $H$ \emph{attention heads}. So if our embeddings have dimensionality $(4,16)$ (denoting 4 feature types with 16 channels each) and we use $H=8$ attention heads, then we partition the embeddings to shape $(4,2)$. We then combine each of the 8 sets of shape $(4,2)$ queries, keys, and values individually and then concatenate the results into a single vector of the original shape $(4,16)$. The keys and queries are edge embeddings, and thus the embedding matrices are of TFN type (c.f.\ \Cref{eq:equivariant-kernel}). For TFN type layers, the radial functions are learnable maps. For these we used neural networks with architecture shown in \Cref{tab:qm9_radial}.

For the norm nonlinearities \cite{WorrallGRB2016}, we use
\begin{align}
\label{eq:norm-nonlinearity}
    \texttt{Norm ReLU}(\bb{f}^{\ell}) = \texttt{ReLU}(\texttt{LN} \left ( \|\bb{f}^{\ell}\| \right ) ) \cdot \frac{\bb{f}^{\ell}}{\|\bb{f}^{\ell}\|}, \qquad \text{where }
    \|\bb{f}^\ell\| = \sqrt {\sum_{m=-\ell}^\ell (f_m^{\ell})^2 },
\end{align}
where \texttt{LN} is layer norm \cite{BaKH16} applied across all features within a feature type. For the TFN baseline, we used the exact same architecture but we replaced each of the multiheaded attention blocks with a TFN layer with the same output shape.

The input to the network is a sparse molecular graph, with edges represented by the molecular bonds. The node embeedings are a 6 dimensional vector composed of a 5 dimensional one-hot embedding of the 5 atomic species and a 1 dimension integer node embedding for number of protons per atom. The edges embeddings are a 5 dimensional vector consisting of a 4 dimensional one-hot embedding of bond type and a positive scalar for the Euclidean distance between the two atoms at the ends of the bond. For each regression target, we normalised the values by mean and dividing by the standard deviation of the training set.

We trained for 50 epochs using Adam \cite{KingmaB14} at initial learning rate \texttt{1e-3} and a single-cycle cosine rate-decay to learning rate \texttt{1e-4}. The batch size was 32, but for the TFN baseline we used batch size 16, to fit the model in memory. We show results on the 6 regression tasks not requiring thermochemical energy subtraction in \Cref{tab:QM9}. As is common practice, we optimised architectures and hyperparameters on $\varepsilon_\text{HOMO}$ and retrained each network on the other tasks. Training took about 2.5 days on an NVIDIA GeForce GTX 1080 Ti GPU with 4 CPU cores and 15 GB of RAM.

\begin{table}[h]
    \centering
    \caption{QM9 Network architecture: $d_\text{out}$ is the number of feature types of degrees $0,1,...,d_\text{out}-1$ at the output of the corresponding layer and $C$ is the number of multiplicities/channels per feature type. For the norm nonlinearity we use ReLUs, preceded by equivariant layer norm \cite{WeilerGWBC18} with learnable affine transform.}
    \label{tab:qm9_architecture}
    \begin{tabular}{cllll}
        \toprule
        \textsc{No. repeats} & \textsc{Layer Type} & $d_\text{out}$ & $C$\\
        1x & Input & 1 & 6 \\
        \midrule
        \multirow{2}{*}{1x}& Attention: 8 heads & 4 & 16 \\
        &Norm Nonlinearity & 4 & 16 \\
        \midrule
        \multirow{2}{*}{6x}& Attention: 8 heads & 4 & 16 \\
        &Norm Nonlinearity & 4 & 16 \\
        \midrule
        1x & TFN layer & 1 & 128 \\
        \midrule
        1x   & Max pool & 1 & 128 \\
        1x   & Linear &1 & 128\\
        1x   & ReLU & 1 & 128\\
        1x   & Linear &1 & 1 \\
        \bottomrule
    \end{tabular}
\end{table}

\begin{table}[h]
    \centering
    \caption{QM9 Radial Function Architecture. $C$ is the number of output channels at each layer. Layer norm \cite{BaKH16} is computed per pair of input and output feature types, which have $C_\text{in}$ and $C_\text{out}$ channels each.}
    \label{tab:qm9_radial}
    \begin{tabular}{llll}
        \toprule
        \textsc{Layer Type} & $C$\\
        \midrule
        Input & 6 \\
        \midrule
        Linear & 32 \\
        Layer Norm & 32 \\
        ReLU    & 32 \\
        \midrule
        Linear & 32 \\
        Layer Norm & 32 \\
        ReLU    & 32 \\
        \midrule
        Linear & $d_\text{out}*C_\text{in}*C_\text{out}$ \\
        \bottomrule
    \end{tabular}
\end{table}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{images/MAttentionBlock.pdf}
    \caption{Attention block for the QM9 dataset. Each component is listed with a tuple of numbers representing the output feature types and multiplicities, so $(4,32)$ means feature types $0,1,2,3$ (with dimensionalities $1,3,5,7$), with 32 channels per type. }
    \label{fig:mab_qm9}
\end{figure}

\subsection{General Remark}
Across experiments on different datasets with the SE(3)-Transformer, we made the observation that the number of representation degrees have a significant but saturating impact on performance. A big improvement was observed when switching from degrees $\{0,1\}$ to $\{0,1,2\}$. Adding type-3 latent representations gave small improvements, further representation degrees did not change the performance of the model. However, higher representation degrees have a significant impact on memory usage and computation time. We therefore recommend representation degrees up to 2, when computation time and memory usage is a concern, and 3 otherwise.
\section{Conclusion}
\vspace{-1mm}
We have presented an attention-based neural architecture designed specifically for point cloud data. This architecture is guaranteed to be robust to rotations and translations of the input, obviating the need for training time data augmentation and ensuring stability to arbitrary choices of coordinate frame. The use of self-attention allows for anisotropic, data-adaptive filters, while the use of neighbourhoods enables scalability to large point clouds. We have also introduced the interpretation of the attention mechanism as a data-dependent nonlinearity, adding to the list of equivariant nonlinearties which we can use in equivariant networks. Furthermore, we provide code for a speed up of spherical harmonics computation of up to 3 orders of magnitudes. This speed-up allowed us to train significantly larger versions of both the SE(3)-Transformer and the Tensor Field network \citep{ThomasSKYKR18} and to apply these models to real-world datasets.

Our experiments showed that adding attention to a roto-translation-equivariant model consistently led to higher accuracy and increased training stability. Specifically for large neighbourhoods, attention proved essential for model convergence. On the other hand, compared to convential attention, adding the equivariance constraints also increases performance in all of our experiments while at the same time providing a mathematical guarantee for robustness with respect to rotations of the input data.

\section{Introduction}

Self-attention mechanisms \citep{VaswaniSPUJGKP17} have enjoyed a sharp rise in popularity in recent years. Their relative implementational simplicity coupled with high efficacy on a wide range of tasks such as language modeling \citep{VaswaniSPUJGKP17}, image recognition \citep{ParmarRVBLS19}, or graph-based problems \citep{VelikovicCCRLB2017}, make them an attractive component to use. However, their generality of application means that for specific tasks, knowledge of existing underlying structure is unused. In this paper, we propose the \emph{SE(3)-Transformer} shown in \Cref{fig:teaser}, a self-attention mechanism specifically for 3D point cloud and graph data, which adheres to \emph{equivariance constraints}, improving robustness to nuisance transformations and general performance.

Point cloud data is ubiquitous across many fields, presenting itself in diverse forms such as 3D object scans \citep{uy-scanobjectnn-iccv19}, 3D molecular structures \citep{RamakrishnanDRvL14}, or $N$-body particle simulations \citep{KipfFWWZ18}. Finding neural structures which can adapt to the varying number of points in an input, while respecting the irregular sampling of point positions, is challenging. Furthermore, an important property is that these structures should be invariant to global changes in overall input pose; that is, 3D translations and rotations of the input point cloud should not affect the output. In this paper, we find that the explicit imposition of equivariance constraints on the self-attention mechanism addresses these challenges. The SE(3)-Transformer uses the self-attention mechanism as a data-dependent filter particularly suited for sparse, non-voxelised point cloud data, while respecting and leveraging the symmetries of the task at hand.

Self-attention itself is a pseudo-linear map between sets of points. It can be seen to consist of two components: input-dependent \emph{attention weights} and an embedding of the input, called a \emph{value embedding}. In \Cref{fig:teaser}, we show an example of a molecular graph, where attached to every atom we see a value embedding vector and where the attention weights are represented as edges, with width corresponding to the attention weight magnitude. In the SE(3)-Transformer, we explicitly design the attention weights to be invariant to global pose. Furthermore, we design the value embedding to be equivariant to global pose. Equivariance generalises the translational weight-tying of convolutions. It ensures that transformations of a layer's input manifest as equivalent transformations of the output. SE(3)-equivariance in particular is the generalisation of translational weight-tying in 2D known from conventional convolutions to roto-translations in 3D.
This restricts the space of learnable functions to a subspace which adheres to the symmetries of the task and thus reduces the number of learnable parameters.
Meanwhile, it provides us with a richer form of invariance, since relative positional information between features in the input is preserved.

The works closest related to ours are tensor field networks (TFN) \citep{ThomasSKYKR18} and their voxelised equivalent, 3D steerable CNNs \cite{WeilerGWBC18}. These provide frameworks for building SE(3)-equivariant convolutional networks operating on point clouds. Employing self-attention instead of convolutions has several advantages. (1) It allows a natural handling of edge features extending TFNs to the graph setting. (2) This is one of the first examples of a nonlinear equivariant layer. In \Cref{sec:method_attention}, we show our proposed approach relieves the strong angular constraints on the filter compared to TFNs, therefore adding representational capacity. This constraint has been pointed out in the equivariance literature to limit performance severely \citep{e2cnn}. Furthermore, we provide a more efficient implementation, mainly due to a GPU accelerated version of the spherical harmonics. The TFN baselines in our experiments leverage this and use significantly scaled up architectures compared to the ones used in \cite{ThomasSKYKR18}.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/Teaser_v2.pdf}
    \caption{\textbf{A}) Each layer of the SE(3)-Transformer maps from a point cloud to a point cloud (or graph to graph) while guaranteeing equivariance.
        For classification, this is followed by an invariant pooling layer and an MLP. \textbf{B}) In each layer, for each node, attention is performed. Here, the red node attends to its neighbours. Attention weights (indicated by line thickness) are invariant w.r.t. input rotation.
    }
    \label{fig:teaser}
\end{figure}

Our contributions are the following:
\begin{itemize}
	\item We introduce a novel self-attention mechanism, guaranteeably invariant to global rotations and translations of its input. It is also equivariant to permutations of the input point labels.
	\item We show that the SE(3)-Transformer resolves an issue with concurrent SE(3)-equivariant neural networks, which suffer from angularly constrained filters.
	\item We introduce a \texttt{Pytorch} implementation of spherical harmonics, which is 10x faster than \texttt{Scipy} on CPU and $100 - 1000\times$ faster on GPU. This directly addresses a bottleneck of TFNs \citep{ThomasSKYKR18}. E.g., for a ScanObjectNN model, we achieve $\approx 22 \times$ speed up of the forward pass compared to a network built with SH from the \texttt{lielearn} library (see \Cref{sec:sh_accelerated}).
	\item Code available at \url{https://github.com/FabianFuchsML/se3-transformer-public}
\end{itemize}

\section{Background And Related Work}
In this section we introduce the relevant background materials on self-attention, graph neural networks, and equivariance. We are concerned with point cloud based machine learning tasks, such as object classification or segmentation. In such a task, we are given a point cloud as input, represented as a collection of $n$ coordinate vectors $\bb{x}_i \in \mbb{R}^{3}$ with optional per-point features $\bb{f}_i \in \mbb{R}^{d}$.

\subsection{The Attention Mechanism}
The standard \emph{attention mechanism} \citep{VaswaniSPUJGKP17} can be thought of as consisting of three terms: a set of query vectors $\bb{q}_i \in \mbb{R}^p$ for $i=1,...,m$, a set of key vectors $\bb{k}_j \in \mbb{R}^{p}$ for $j=1,...,n$, and a set of value vectors $\bb{v}_j \in \mbb{R}^{r}$ for $j=1,...,n$, where $r$ and $p$ are the dimensions of the low dimensional embeddings. We commonly interpret the key $\bb{k}_j$ and the value $\bb{v}_j$ as being `attached' to the same point $j$. For a given query $\bb{q}_i$, the attention mechanism can be written as
\begin{align}
	\text{Attn}\left(\bb{q}_i, \{\bb{k}_j\}, \{\bb{v}_j\}\right) = \sum_{j=1}^n \alpha_{ij} \bb{v}_j, \qquad \alpha_{ij} = \frac{\exp(\bb{q}_i^{\top} \bb{k}_j)}{\sum_{j'=1}^n \exp(\bb{q}_i^{\top} \bb{k}_{j'})} \label{eq:self-attention}
\end{align}

where we used a softmax as a nonlinearity acting on the weights.
In general, the number of query vectors does not have to equal the number of input points \citep{SetTransformer}. In the case of \emph{self-attention} the query, key, and value vectors are embeddings of the input features, so
\begin{align}
	\bb{q} = h_Q(\bb{f}), && \bb{k} = h_K(\bb{f}), && \bb{v} = h_V(\bb{f}), \label{eq:embeddings}
\end{align}
where $\{h_Q, h_K, h_V\}$ are, in the most general case, neural networks \citep{steenkiste2018relational}. For us, query $\bb{q}_i$ is associated with a point $i$ in the input, which has a geometric location $\bb{x}_i$. Thus if we have $n$ points, we have $n$ possible queries. For query $\bb{q}_i$, we say that node $i$ \emph{attends} to all other nodes $j \neq i$.

Motivated by a successes across a wide range of tasks in deep learning such as language modeling \citep{VaswaniSPUJGKP17}, image recognition \citep{ParmarRVBLS19}, graph-based problems \citep{VelikovicCCRLB2017}, and relational reasoning \citep{steenkiste2018relational,Mohart},
a recent stream of work has applied forms of self-attention algorithms to point cloud data \citep{Yang2019, ShapeContextNet, SetTransformer}.
One such example is the Set Transformer \citep{SetTransformer}. When applied to object classification on ModelNet40 \citep{modelnet40}, the input to the Set Transformer are the cartesian coordinates of the points. Each layer embeds this positional information further while dynamically querying information from other points. The final per-point embeddings are downsampled and used for object classification.

\textbf{Permutation equivariance}
A key property of self-attention is \emph{permutation equivariance}. Permutations of point labels $1,...,n$ lead to permutations of the self-attention output. This guarantees the attention output does not depend arbitrarily on input point ordering. \citet{Wagstaff2019} recently showed that this mechanism can theoretically approximate \textit{all} permutation equivariant functions. The SE(3)-transformer is a special case of this attention mechanism, inheriting permutation equivariance.
However, it limits the space of learnable functions to rotation and translation equivariant ones.

\subsection{Graph Neural Networks}

Attention scales quadratically with point cloud size, so it is useful to introduce neighbourhoods: instead of each point attending to \textit{all} other points, it only attends to its nearest neighbours. Sets with neighbourhoods are naturally represented as graphs. Attention has previously been introduced on graphs under the names of intra-, self-, vertex-, or graph-attention \citep{lin2017structured, VaswaniSPUJGKP17, VelikovicCCRLB2017, hoshen2017vain, shaw2018selfattention}. These methods were unified by \citet{wang2017nonlocal} with the non-local neural network. This has the simple form
\begin{align}
    \bb{y}_i = \frac{1}{\mc{C}(\{\bb{f}_j \in \mc{N}_i\})}\sum_{j\in\mc{N}_i} w(\bb{f}_i, \bb{f}_j) h(\bb{f}_j) \label{eq:nonlocal}
\end{align}
where $w$ and $h$ are neural networks and $\mc{C}$ normalises the sum as a function of all features in the neighbourhood $\mc{N}_i$. This has a similar structure to attention, and indeed we can see it as performing attention per neighbourhood. While non-local modules do not explicitly incorporate edge-features, it is possible to add them, as done in \citet{VelikovicCCRLB2017} and \citet{hoshen2017vain}.

\subsection{Equivariance}
Given a set of transformations $T_g: \mc{V} \to \mc{V}$ for $g \in G$, where $G$ is an abstract group, a function $\phi : \mc{V} \to \mc{Y}$ is called equivariant if for every $g$ there exists a transformation $S_g: \mc{Y} \to \mc{Y}$ such that
\begin{align}
    S_g[\phi(v)] = \phi(T_g[v]) \qquad \text{for all } g\in G, v\in \mc{V} .\label{eq:equivariance-constraint}
\end{align}
The indices $g$ can be considered as parameters describing the transformation. Given a pair $(T_g, S_g)$, we can solve for the family of equivariant functions $\phi$ satisfying Equation \ref{eq:equivariance-constraint}. Furthermore, if $(T_g, S_g)$ are linear and the map $\phi$ is also linear, then a very rich and developed theory already exists for finding $\phi$~\citep{CohenW2016}. In the equivariance literature, deep networks are built from interleaved linear maps $\phi$ and equivariant nonlinearities. In the case of 3D roto-translations it has already been shown that a suitable structure for $\phi$ is a \emph{tensor field network} \citep{ThomasSKYKR18}, explained below. Note that \citet{Romero2020} recently introduced a 2D roto-translationally equivariant attention module for pixel-based image data.

\textbf{Group Representations}
In general, the transformations $(T_g, S_g)$ are called \emph{group representations}. Formally, a group representation $\rho: G \to GL(N)$ is a map from a group $G$ to the set of $N\times N$ invertible matrices $GL(N)$. Critically $\rho$ is a \emph{group homomorphism}; that is, it satisfies the following property $\rho(g_1 g_2) = \rho(g_1)\rho(g_2)$ for all $g_1, g_2 \in G$. Specifically for 3D rotations $G=SO(3)$, we have a few interesting properties: 1) its representations are orthogonal matrices, 2) all representations can be decomposed as
\begin{align}
    \rho(g) = \bb{Q}^{\top} \left [ \bigoplus_{\ell} \bb{D}_{\ell}(g) \right ] \bb{Q}, \label{eq:so3-representations}
\end{align}
where $\bb{Q}$ is an orthogonal, $N \times N$, change-of-basis matrix \citep{chirikjian2001engineering}; each $\bb{D}_\ell$ for $\ell = 0,1,2,...$ is a $(2\ell+1) \times (2\ell + 1)$ matrix known as a Wigner-D matrix\footnote{The `D' stands for \emph{Darstellung}, German for representation}; and the $\bigoplus$ is the \emph{direct sum} or concatenation of matrices along the diagonal. The Wigner-D matrices are \emph{irreducible representations} of SO(3)---think of them as the `smallest' representations possible.
Vectors transforming according to $\bb{D}_\ell$ (i.e.\ we set $\bb{Q} = \bb{I}$), are called \emph{type}-$\ell$ vectors. Type-0 vectors are invariant under rotations and type-1 vectors rotate according to 3D rotation matrices. Note, type-$\ell$ vectors have length $2\ell+1$. They can be stacked, forming a feature vector $\bb{f}$ transforming according to \Cref{eq:so3-representations}.

\textbf{Tensor Field Networks}
\label{sec:background-tfn}
Tensor field networks (TFN) \citep{ThomasSKYKR18} are neural networks, which map point clouds to point clouds under the constraint of SE(3)-equivariance, the group of 3D rotations and translations. For point clouds, the input is a vector field $\bb{f}: \mbb{R}^3 \to \mbb{R}^d$ of the form
\begin{align}
    \bb{f}(\bb{x}) = \sum_{j=1}^N \bb{f}_j \delta(\bb{x} - \bb{x}_j), \label{eq:point-cloud}
\end{align}
where $\delta$ is the Dirac delta function, $\{\bb{x}_j\}$ are the 3D point coordinates and $\{\bb{f}_j\}$ are point features, representing such quantities as atomic number or point identity. For equivariance to be satisfied, the features of a TFN transform under \Cref{eq:so3-representations}, where $\bb{Q}=\bb{I}$. Each $\bb{f}_j$ is a concatenation of vectors of different \emph{types}, where a subvector of type-$\ell$ is written $\bb{f}_j^\ell$. A TFN layer computes the convolution of a continuous-in-space, learnable weight kernel $\bb{W}^{\ell k}: \mathbb{R}^3 \to \mathbb{R}^{(2\ell+1)\times(2k+1)}$ from type-$k$ features to type-$\ell$ features. The type-$\ell$ output of the TFN layer at position $\bb{x}_i$ is
\begin{align}
    \bb{f}_{\text{out},i}^{\ell} = \sum_{k\geq 0} \underbrace{\int \bb{W}^{\ell k}(\bb{x}'-\bb{x}_i) \bb{f}_{\text{in}}^k(\bb{x}') \, \mathrm{d}\bb{x}'}_{k \to \ell \text{ convolution}} = \sum_{k\geq 0}\sum_{j=1}^n \underbrace{\bb{W}^{\ell k}(\bb{x}_j - \bb{x}_i) \bb{f}_{\text{in},j}^k,}_{\text{node $j \to$ node $i$ message}} \label{eq:tf-layer}
\end{align}
We can also include a sum over input channels, but we omit it here. \citet{WeilerGWBC18, ThomasSKYKR18} and \citet{kondor2018nbody} showed that the kernel $\bb{W}^{\ell k}$ lies in the span of an equivariant basis $\{\bb{W}_J^{\ell k}\}_{J=|k-\ell|}^{k+\ell}$. The kernel is a linear combination of these basis kernels, where the $J$\textsuperscript{th} coefficient is a learnable function $\varphi_J^{\ell k}: \mathbb{R}_{\geq 0} \to \mathbb{R}$ of the radius $\|\bb{x}\|$. Mathematically this is
\begin{align}
    \bb{W}^{\ell k}(\bb{x}) = \sum_{J = |k-\ell|}^{k+\ell} \varphi_J^{\ell k}(\|\bb{x}\|) \bb{W}_{J}^{\ell k}(\bb{x}), \qquad \text{where } \bb{W}_{J}^{\ell k}(\bb{x}) = \sum_{m=-J}^J Y_{Jm}(\bb{x} / \|\bb{x}\|) \bb{Q}_{Jm}^{\ell k}. \label{eq:equivariant-kernel}
\end{align}
Each basis kernel $\bb{W}_{J}^{\ell k}: \mathbb{R}^3 \to \mathbb{R}^{(2\ell+1)\times(2k+1)}$ is formed by taking a linear combination of Clebsch-Gordan matrices $\bb{Q}_{Jm}^{\ell k}$ of shape $(2\ell+1)\times(2k+1)$, where the $J,m$\textsuperscript{th} linear combination coefficient is the $m$\textsuperscript{th} dimension of the $J$\textsuperscript{th} spherical harmonic $Y_J: \mathbb{R}^3 \to \mathbb{R}^{2J+1}$. Each basis kernel $\bb{W}_{J}^{\ell k}$ completely constrains the form of the learned kernel in the angular direction, leaving the only learnable degree of freedom in the radial direction. Note that $\bb{W}_{J}^{\ell k}(\bb{0}) \neq \bb{0}$ only when $k=\ell$ and $J=0$, which reduces the kernel to a scalar $w$ multiplied by the identity, $\bb{W}^{\ell\ell} = w^{\ell\ell}\bb{I}$, referred to as \emph{self-interaction}~\citep{ThomasSKYKR18}. As such we can rewrite the TFN layer as
\begin{align}
    \bb{f}_{\text{out},i}^{\ell} = \underbrace{w^{\ell\ell} \bb{f}_{\text{in},i}^{\ell}}_{\text{self-interaction}} + \sum_{k\geq 0}\sum_{j\neq i}^n \bb{W}^{\ell k}(\bb{x}_j - \bb{x}_i) \bb{f}_{\text{in},j}^k, \label{eq:tf-layer2}
\end{align}
\Cref{eq:tf-layer} and \Cref{eq:tf-layer2} present the convolution in message-passing form, where messages are aggregated from all nodes and feature types. They are also a form of nonlocal graph operation as in \Cref{eq:nonlocal}, where the weights are functions on edges and the features $\{\bb{f}_i\}$ are node features. We will later see how our proposed attention layer unifies aspects of convolutions and graph neural networks.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/4steps.pdf}
    \caption{Updating the node features using our equivariant attention mechanism in four steps. A more detailed description, especially of step 2, is provided in the Appendix. Steps 3 and 4 visualise a graph network perspective: features are passed from nodes to edges to compute keys, queries and values, which depend both on features and relative positions in a rotation-equivariant manner.}
    \label{fig:4steps}
    \vspace{-4mm}
\end{figure}

\section{Method}
Here, we present the \emph{SE(3)-Transformer}. The layer can be broken down into a procedure of steps as shown in \Cref{fig:4steps}, which we describe in the following section. These are the construction of a graph from a point cloud, the construction of equivariant edge functions on the graph, how to propagate SE(3)-equivariant messages on the graph, and how to aggregate them. We also introduce an alternative for the self-interaction layer, which we call \emph{attentive self-interaction}.

\subsection{Neighbourhoods}
Given a point cloud $\{(\bb{x}_i, \bb{f}_i)\}$, we first introduce a collection of neighbourhoods $\mc{N}_i \subseteq \{1,...,N\}$, one centered on each point $i$. These neighbourhoods are computed either via the nearest-neighbours methods or may already be defined. For instance, molecular structures have neighbourhoods defined by their bonding structure. Neighbourhoods reduce the computational complexity of the attention mechanism from quadratic in the number of points to linear. The introduction of neighbourhoods converts our point cloud into a graph. This step is shown as Step 1 of \Cref{fig:4steps}.

\subsection{The SE(3)-Transformer}
\label{sec:method_attention}
The SE(3)-Transformer itself consists of three components. These are 1) edge-wise attention weights $\alpha_{ij}$, constructed to be SE(3)-invariant on each edge $ij$, 2) edge-wise SE(3)-equivariant value messages, propagating information between nodes, as found in the TFN convolution of \Cref{eq:tf-layer}, and 3) a linear/attentive self-interaction layer. Attention is performed on a per-neighbourhood basis as follows:
\begin{align}
\label{eq:att_equi}
    \bb{f}_{\text{out},i}^\ell = \underbrace{\bb{W}_V^{\ell \ell} \bb{f}_{\text{in},i}^{\ell}}_{\text{ \textcircled{\raisebox{-0.9pt}{3}} self-interaction}} + \sum_{k \geq 0} \sum_{j \in \mc{N}_i \setminus i} \underbrace{\alpha_{ij}}_{\text{\textcircled{\raisebox{-0.6pt}{1}} attention}} \underbrace{\bb{W}_V^{\ell k}(\bb{x}_j - \bb{x}_i) \bb{f}_{\text{in},j}^k}_{\text{\textcircled{\raisebox{-0.9pt}{2}} value message}}.
\end{align}
These components are visualised in \Cref{fig:4steps}. If we remove the attention weights then we have a tensor field convolution,
and if we instead remove the dependence of $\bb{W}_V$ on $(\bb{x}_j - \bb{x}_i)$, we have a conventional attention mechanism.
Provided that the attention weights $\alpha_{ij}$ are invariant, \Cref{eq:att_equi} is equivariant to SE(3)-transformations. This is because it is just a linear combination of equivariant value messages. Invariant attention weights can be achieved with a dot-product attention structure shown in \Cref{eq:att_equi_weights}. This mechanism consists of a normalised inner product between a query vector $\bb{q}_i$ at node $i$ and a set of key vectors $\{\bb{k}_{ij}\}_{j \in \mc{N}_i}$ along each edge $ij$ in the neighbourhood $\mc{N}_i$ where
\begin{align}
\label{eq:att_equi_weights}
    \alpha_{ij} = \frac{\exp ( \bb{q}_i^{\top} \bb{k}_{ij} ) }{\sum_{j' \in \mc{N}_i\setminus i} \exp ( \bb{q}_i^{\top} \bb{k}_{ij'})}, \;\;\; \bb{q}_i = \bigoplus_{\ell \geq 0} \sum_{k \geq 0}\bb{W}_Q^{\ell k}\bb{f}_{\text{in},i}^k, \;\;\; \bb{k}_{ij} = \bigoplus_{\ell \geq 0} \sum_{k\geq 0} \bb{W}_K^{\ell k}(\bb{x}_j - \bb{x}_i) \bb{f}_{\text{in},j}^k.
\end{align}
$\bigoplus$ is the direct sum, i.e.\ vector concatenation in this instance. The linear embedding matrices $\bb{W}_Q^{\ell k}$ and $\bb{W}_K^{\ell k}(\bb{x}_j - \bb{x}_i)$ are of TFN type (c.f.\ \Cref{eq:equivariant-kernel}). The attention weights $\alpha_{ij}$ are invariant for the following reason. If the input features $\{ \bb{f}_{\text{in},j} \}$ are SO(3)-equivariant, then the query $\bb{q}_i$ and key vectors $\{\bb{k}_{ij}\}$ are also SE(3)-equivariant, since the linear embedding matrices are of TFN type. The inner product of SO(3)-equivariant vectors, transforming under the same representation $\bb{S}_g$ is invariant, since if $\bb{q} \mapsto \bb{S}_g \bb{q}$ and $\bb{k} \mapsto \bb{S}_g \bb{k}$, then $\bb{q}^\top \bb{S}_g^\top \bb{S}_g \bb{k} = \bb{q}^\top \bb{k}$, because of the orthonormality of representations of SO(3), mentioned in the background section. We follow the common practice from the self-attention literature \citep{VaswaniSPUJGKP17,SetTransformer}, and chosen a softmax nonlinearity to normalise the attention weights to unity, but in general any nonlinear function could be used.

\textbf{Aside: Angular Modulation}
The attention weights add extra degrees of freedom to the TFN kernel in the angular direction. This is seen when \Cref{eq:att_equi} is viewed as a convolution with a data-dependent kernel $\alpha_{ij} \bb{W}_V^{\ell k}(\bb{x})$. In the literature, SO(3) equivariant kernels are decomposed as a sum of products of learnable radial functions $\varphi_J^{\ell k}(\|\bb{x}\|)$ and non-learnable angular kernels $\bb{W}_J^{\ell k}(\bb{x} / \|\bb{x}\|)$ (c.f.\ \Cref{eq:equivariant-kernel}). The fixed angular dependence of $\bb{W}_J^{\ell k}(\bb{x} / \|\bb{x}\|)$ is a strange artifact of the equivariance condition in noncommutative algebras and while necessary to guarantee equivariance, it is seen as overconstraining the expressiveness of the kernels. Interestingly, the attention weights $\alpha_{ij}$ introduce a means to modulate the angular profile of $\bb{W}_J^{\ell k}(\bb{x} / \|\bb{x}\|)$, while maintaining equivariance.

\textbf{Channels, Self-interaction Layers, and Non-Linearities}
Analogous to conventional neural networks, the SE(3)-Transformer can straightforwardly be extended to multiple channels per representation degree $\ell$, so far omitted for brevity. This sets the stage for self-interaction layers. The attention layer (c.f.\ \Cref{fig:4steps} and circles 1 and 2 of \Cref{eq:att_equi}) aggregates information over nodes and input representation degrees $k$. In contrast, the self-interaction layer (c.f.\ circle 3 of \Cref{eq:att_equi}) exchanges information solely between features of the same degree and within one node---much akin to 1x1 convolutions in CNNs. Self-interaction is an elegant form of learnable skip connection, transporting information from query point $i$ in layer $L$ to query point $i$ in layer $L+1$. This is crucial since, in the SE(3)-Transformer, points do not attend to themselves. In our experiments, we use two different types of self-interaction layer: (1) linear and (2) attentive, both of the form
\begin{align}
    \bb{f}_{\text{out},i,c'}^\ell &= \sum_{c} w_{i,c'c}^{\ell\ell} \bb{f}_{\text{in},i,c}^\ell \label{eq:self-interaction}.
\end{align}
\textbf{Linear:} Following \citet{schnet}, output channels are a learned linear combination of input channels using one set of weights $w_{i,c'c}^{\ell\ell} = w_{c'c}^{\ell\ell}$ per representation degree, shared across all points. As proposed in \citet{ThomasSKYKR18}, this is followed by a norm-based non-linearity.

\textbf{Attentive}: We propose an extension of linear self-interaction, \emph{attentive self-interaction}, combining self-interaction and nonlinearity. We replace the learned scalar weights $w_{c'c}^{\ell\ell}$ with attention weights output from an MLP, shown in \Cref{eq:attentive-self-interaction} ($\bigoplus$ means concatenation.). These weights are SE(3)-invariant due to the invariance of inner products of features, transforming under the same representation.
\begin{align}
w_{i,c'c}^{\ell\ell} = \text{MLP}\left( \bigoplus_{c,c'}\bb{f}_{\text{in},i,c'}^{\ell\top}\bb{f}_{\text{in},i,c}^\ell \right)
\label{eq:attentive-self-interaction}
\end{align}

\subsection{Node and Edge Features}
Point cloud data often has information attached to points (node-features) and connections between points (edge-features), which we would both like to pass as inputs into the first layer of the network.
Node information can directly be incorporated via the tensors $\bb{f}_j$ in \Cref{eq:point-cloud,,eq:att_equi}.
For incorporating edge information, note that $\bb{f}_j$ is part of multiple neighbourhoods. One can replace $\bb{f}_j$ with $\bb{f}_{ij}$ in \Cref{eq:att_equi}. Now, $\bb{f}_{ij}$ can carry different information depending on which neighbourhood $\mc{N}_i$ we are currently performing attention over. In other words, $\bb{f}_{ij}$ can carry information both about node $j$ but also about edge $ij$. Alternatively, if the edge information is scalar, it can be incorporated into the weight matrices $\bb{W}_V$ and $\bb{W}_K$ as an input to the radial network (see step 2 in \Cref{fig:4steps}).

\input{experiments}

\input{conclusion}

\input{impact}

\section*{Acknowledgements and Funding Disclosure}
We would like to express our gratitude to the Bosch Center for Artificial Intelligence and Konincklijke Philips N.V. for funding our work and contributing to open research by publishing our paper. Fabian Fuchs worked on this project while on a research sabbatical at the Bosch Center for Artificial Intelligence. His PhD is funded by Kellogg College Oxford and the EPSRC AIMS Centre
for Doctoral Training at Oxford University.

\newpage
\bibliographystyle{plainnat}

\section{Experiments}
We test the efficacy of the SE(3)-Transformer on three datasets, each testing different aspects of the model. The N-body problem is an equivariant task: rotation of the input should result in rotated predictions of locations and velocities of the particles. Next, we evaluate on a real-world object classification task. Here, the network is confronted with large point clouds of noisy data with symmetry only around the gravitational axis. Finally, we test the SE(3)-Transformer on a molecular property regression task, which shines light on its ability to incorporate rich graph structures. We compare to publicly available, state-of-the-art results as well as a set of our own baselines. Specifically, we compare to the Set-Transformer \citep{SetTransformer}, a non-equivariant attention model, and Tensor Field Networks \citep{ThomasSKYKR18}, which is similar to SE(3)-Transformer but does not leverage attention.

Similar to \cite{Sosnovik20, Worrall19}, we measure the exactness of equivariance by applying uniformly sampled SO(3)-transformations to input and output. The distance between the two, averaged over samples, yields the equivariance error. Note that, unlike in \citet{Sosnovik20}, the error is not squared:
\begin{align}
    \Delta_{EQ}=\left\| L_{s} \Phi(f)-\Phi L_{s}(f)\right\|_{2} /\left\| L_{s} \Phi(f) \|_{2}\right.
\end{align}

\subsection{N-Body Simulations}

In this experiment, we use an adaptation of the dataset from \citet{KipfFWWZ18}. Five particles each carry either a positive or a negative charge and exert repulsive or attractive forces on each other. The input to the network is the position of a particle in a specific time step, its velocity, and its charge. The task of the algorithm is then to predict the relative location and velocity 500 time steps into the future. We deliberately formulated this as a regression problem to avoid the need to predict multiple time steps iteratively. Even though it certainly is an interesting direction for future research to combine equivariant attention with, e.g., an LSTM, our goal here was to test our core contribution and compare it to related models. This task sets itself apart from the other two experiments by not being invariant but equivariant: When the input is rotated or translated, the output changes respectively (see \Cref{fig:nri_rotation_example}).

We trained an SE(3)-Transformer with 4 equivariant layers, each followed by an attentive self-interaction layer (details are provided in the Appendix). \Cref{tab:RI} shows quantitative results.
Our model outperforms both an attention-based, but not rotation-equivariant approach (Set Transformer) and a equivariant approach which does not levarage attention (Tensor Field). The equivariance error shows that our approach is indeed fully rotation equivariant up to the precision of the computations.

\begin{table}[]
{\caption{Predicting future locations and velocities in an electron-proton simulation.}
\label{tab:RI}}
\resizebox{\columnwidth}{!}{
\begin{tabular}{llrrrrr}
    \toprule
       &                     & Linear               & DeepSet \cite{Zaheer2017}              & Tensor Field \cite{ThomasSKYKR18}         & Set Transformer \cite{SetTransformer}      & \textbf{SE(3)-Transformer} \\

       \midrule
      & MSE $x$             & $0.0691$             & $0.0639$             & $0.0151$             & $0.0139$             & $\bm{0.0076}$              \\
       \textbf{Position} & std                 & -                    & $0.0086$             & $0.0011$             & $0.0004$             & $0.0002$              \\
       & $\Delta_{EQ}$       & -                    & $0.038$              & $1.9 \cdot 10^{-7}$  & $0.167$              & $3.2 \cdot 10^{-7}$   \\
       \midrule
      & MSE $v$             & $0.261$             & $0.246$              & $0.125$              & $0.101$              & $\bm{0.075}$              \\
       \textbf{Velocity} & std                 & -                    & $0.017$              & $0.002$              & $0.004$              & $0.001$               \\
       & $\Delta_{EQ}$       & -                    & $1.11$               & $5.0 \cdot 10^{-7}$  & $0.37$               & $6.3 \cdot 10^{-7}$   \\
       \bottomrule
\end{tabular}}
\vspace{-3mm}
\end{table}

\subsection{Real-World Object Classification on ScanObjectNN}

\begin{figure}[t]
    \centering
    \begin{subfigure}[c]{0.45\linewidth}
        \centering
        \includegraphics[width=\linewidth]{images/accuracy_over_angle.pdf}
        \subcaption{Training \textit{without} data augmentation.}
        \label{fig:scanobject_rotate_eval}
    \end{subfigure}
    \hspace*{\fill}
    \begin{subfigure}[c]{0.45\linewidth}
        \centering
        \includegraphics[width=\linewidth]{images/accuracy_over_angle_augmented.pdf}
        \subcaption{Training \textit{with} data augmentation.}
        \label{fig:scanobject_rotate_both}
    \end{subfigure}
    \caption{
        ScanObjectNN: $x$-axis shows data augmentation on the test set. The $x$-value corresponds to the maximum rotation around a random axis in the $x$-$y$-plane.
        If both training and test set are not rotated ($x=0$ in \textbf{a}), breaking the symmetry of the SE(3)-Transformer by providing the $z$-component of the coordinates as an additional, scalar input improves the performance significantly. Interestingly, the model learns to ignore the additional, symmetry-breaking input when the training set presents a rotation-invariant problem (strongly overlapping dark red circles and dark purple triangles in \textbf{b}).
    }
    \label{fig:scanobject_rotate}
\end{figure}

Object classification from point clouds is a well-studied task. Interestingly, the vast majority of current neural network methods work on scalar coordinates without incorporating vector specific inductive biases. Some recent works explore rotation invariant point cloud classification \cite{You2019,zhang2019rotation,ClusterNet,RaoFractal}. Our method sets itself apart by using roto-translation equivariant layers acting directly on the point cloud without prior projection onto a sphere \cite{RaoFractal,You2019,cohen2018spherical}. This allows for weight-tying and increased sample efficiency while transporting information about local and global orientation through the network - analogous to the translation equivariance of CNNs on 2D images. To test our method, we choose ScanObjectNN, a recently introduced dataset for real-world object classification.
The benchmark provides point clouds of 2902 objects across 15 different categories. We only use the coordinates of the points as input and object categories as training labels. We train an SE(3)-Transformer with 4 equivariant layers with linear self-interaction followed by max-pooling and an MLP. Interestingly, the task is not fully rotation invariant, in a statistical sense, as the objects are aligned with respect to the gravity axis. This results in a performance loss when deploying a fully SO(3) invariant model (see \Cref{fig:scanobject_rotate_eval}). In other words: when looking at a new object, it helps to know where `up' is.
We create an SO(2) invariant version of our algorithm by additionally feeding the $z$-component as an type-0 field and the $x$, $y$ position as an additional type-1 field (see Appendix). We dub this model \textit{SE(3)-Transformer +z}. This way, the model can `learn' which symmetries to adhere to by suppressing and promoting different inputs (compare \Cref{fig:scanobject_rotate_eval} and \Cref{fig:scanobject_rotate_both}). In \Cref{tab:Scan}, we compare our model to the current state-of-the-art in object classification\footnote{At time of submission, PointGLR was a recently published preprint \citep{PointGLR}. The performance of the following models was taken from the official benchmark of the dataset as of June 4th, 2020 (\url{https://hkust-vgd.github.io/benchmark/}): 3DmFV \citep{3dmfv}, PointNet \citep{Pointnet}, SpiderCNN \citep{SpiderCNN}, PointNet++ \citep{Pointnetpp}, DGCN \citep{DGCNN}.}. Despite the dataset not playing to the strengths of our model (full SE(3)-invariance) and a much lower number of input points, the performance is competitive with models specifically designed for object classification - PointGLR \citep{PointGLR}, for instance, is pre-trained on the larger ModelNet40 dataset \citep{modelnet40}. For a discussion of performance vs. number of input points used, see \Cref{sec:app_n_points}.

\begin{table}[t]
\vspace{-1mm}
{\caption{Classification accuracy on the 'object only' category of the ScanObjectNN dataset\textsuperscript{4}. The performance of the SE(3)-Transformer is averaged over 5 runs (standard deviation 0.7\%).}
\label{tab:Scan}}
\centering
\resizebox{0.98\columnwidth}{!}{
\vspace{2mm}
\begin{tabular}{lrrrrrrrrrrrrr}
\toprule
 & \mcrot{1}{l}{30}{Tensor Field} & \mcrot{1}{l}{30}{DeepSet} & \mcrot{1}{l}{30}{\textbf{SE(3)-Transf.}} & \mcrot{1}{l}{30}{3DmFV} & \mcrot{1}{l}{30}{Set Transformer} & \mcrot{1}{l}{30}{PointNet} & \mcrot{1}{l}{30}{SpiderCNN} & \mcrot{1}{l}{30}{Tensor Field +z} & \mcrot{1}{l}{30}{PointNet++} & \mcrot{1}{l}{30}{\textbf{SE(3)-Transf.+z}} & \mcrot{1}{l}{30}{PointCNN} & \mcrot{1}{l}{30}{DGCNN} & \mcrot{1}{l}{30}{PointGLR}\\
\midrule
\vspace{1mm}
No. Points & 128 & 1024 & \textbf{128} & 1024 & 1024 & 1024 & 1024 & 128 & 1024 & \textbf{128}  & 1024  & 1024  & 1024 \\
\vspace{1mm}
Accuracy & 63.1\% & 71.4\% & \textbf{72.8} \% & 73.8\% & 74.1\% & 79.2\% & 79.5\% & 81.0\% & 84.3\% & \textbf{85.0\%} & 85.5\% & 86.2\% & 87.2\% \\
\bottomrule
\end{tabular}
}
\vspace{-2mm}
\end{table}

\newpage
\subsection{QM9}

\begin{wraptable}[10]{R}{0.65\linewidth}
\vspace{-8mm}
{\caption{QM9 Mean Absolute Error. Top: Non-equivariant models. Bottom: Equivariant models. SE(3)-Tr. is averaged over 5 runs.}
\label{tab:QM9}}
\resizebox{0.65\columnwidth}{!}{
\begin{tabular}{lrrrrrr}
    \toprule
    \textsc{Task}   & $\alpha$ & $\Delta \varepsilon$ & $\varepsilon_\text{HOMO}$ & $\varepsilon_\text{LUMO}$ & $\mu$ & $C_\nu$ \\
    \textsc{Units}  & $\text{bohr}^3$ & meV & meV & meV & D & cal/mol K  \\
    \midrule
    WaveScatt~\citep{HirnMP17}   & .160  & 118   & 85    & 76    & .340  & .049  \\
    NMP~\citep{Gilmer2017}        & .092  & 69    & 43    & 38    & .030  & .040   \\
    SchNet~\citep{schnet}     & .235  & 63    & 41    & 34    & .033  & .033   \\
    \midrule
    Cormorant~\cite{anderson2019cormorant}   & .085  & 61    & 34    & 38    & .038  & .026 \\
    LieConv(T3)~\citep{Finzi2020} & .084 & 49   & 30   & 25    & .032  & .038 \\
    TFN~\citep{ThomasSKYKR18} & .223  & 58    & 40    & 38    & .064  & .101 \\
    \textbf{SE(3)-Transformer}          & .142$\pm$.002  & 53.0$\pm$0.3    & 35.0$\pm$.9    & 33.0$\pm$.7    & .051$\pm$.001  & .054$\pm$.002 \\
    \bottomrule
\end{tabular}
}
\end{wraptable}

The QM9 regression dataset \citep{RamakrishnanDRvL14} is a publicly available chemical property prediction task. There are 134k molecules with up to 29 atoms per molecule. Atoms are represented as a 5 dimensional one-hot node embeddings in a molecular graph connected by 4 different chemical bond types  (more details in Appendix). `Positions' of each atom are provided. We show results on the test set of \citet{anderson2019cormorant} for 6 regression tasks in \Cref{tab:QM9}. Lower is better. The table is split into non-equivariant (top) and equivariant models (bottom). Our nearest models are Cormorant and TFN (own implementation). We see that while not state-of-the-art, we offer competitive performance, especially against Cormorant and TFN, which transform under irreducible representations of SE(3) (like us), unlike LieConv(T3), using a left-regular representation of SE(3), which may explain its success.

\section*{Broader Impact}

The main contribution of the paper is a mathematically motivated attention mechanism which can be used for deep learning on point cloud based problems. We do not see a direct potential of \textit{negative} impact to the society. However, we would like to stress that this type of algorithm is inherently suited for classification and regression problems on molecules. The SE(3)-Transformer therefore lends itself to application in drug research. One concrete application we are currently investigating is to use the algorithm for early-stage suitability classification of molecules for inhibiting the reproductive cycle of the coronavirus. While research of this sort always requires intensive testing in wet labs, computer algorithms can be and are being used to filter out particularly promising compounds from large databases of millions of molecules.

redpajama_set_name: RedPajamaArXiv


text:
Canadian 'so grateful' to be back in B.C. after eight-month detention in Syria
Kristian Baxter of Nanaimo, B.C., returned to Vancouver on Saturday, days after he was released from Syrian custody while visiting as a tourist last December
Kristian Lee Baxter, who was released this week after being detained in Syrian, poses for a photograph in Vancouver after returning to Canada on Saturday August 10, 2019. Baxter of Nanaimo, B.C., was held in Syrian prisons for eight months after visiting the war-ravaged country as a tourist last December.	DARRYL DYCK / THE CANADIAN PRESS
VANCOUVER — A Canadian man who spent eight months in Syrian prisons said he wept three times on his flight back because he was so excited to finally be coming home.
Kristian Baxter of Nanaimo, B.C., returned to Vancouver on Saturday, just days after he was released from Syrian custody thanks in part to Lebanese mediation. The 45-year-old was detained while visiting the war-ravaged country as a tourist last December.
"I'm just thrilled to be a Canadian citizen and I'm just so happy to see green grass and trees and flowers and the big Canadian flag over there," he said during a brief and emotional interview on his way home to Nanaimo.
Kristian Lee Baxter (C), a Canadian formerly held captive by the Syrian government, shakes hands with the Lebanese Head of General Security Major-General Abbas Ibrahim (L), during a press conference with Canada's ambassador to Lebanon Emanuelle Lamoureux (R), following Baxter's release after an eight-month captivity, in the Lebanese capital Beirut on August 9, 2019. Mohammad Al-Sahili / AFP/Getty Images
Baxter appeared pale but in good spirits, holding his arm around his mother Andrea Leclair and his hand on stepfather Jean-Guy Leclair's shoulder as he spoke. He hadn't slept in three days, he said, and was overwhelmed with gratitude for every government official, negotiator, lawyer and family member who worked "tirelessly" for his release.
A warning against travel to Syria has been in place since the war broke out in the Middle Eastern nation in 2011. Canada does not have an embassy in the country.
Lebanon's General Security Chief Abbas Ibrahim said at a news conference in Beirut yesterday that Baxter was detained for what Syrian authorities considered a "major violation" of local laws, adding that authorities there may have considered the incident security related.
Baxter's release marked the second time Lebanon has helped free a foreigner held in Syria.
I can't wait to resume my life again
Lawyer John Weston of Pan Pacific Law Corp., who has been working with the family, said they aren't aware of any charges that were brought against Baxter in Syria but believe he may have broken some travel regulations.
Baxter declined to go into detail about his arrest or conditions of his detention, but he said he rarely had anyone to speak with and wrote a diary on the walls of his cell to keep his mind busy. He didn't realize anyone knew where he was and believed he would have to make his own way out of Syria, if he was ever released.
On his way to the bathroom one day while in detention when a Syrian general called out, "Kristian."
"They never used my name, so that was bizarre," he said. "And he goes, 'you need vitamin D, you need vitamins."'
The general said they were also going clean his room and give him a mattress, to which Baxter replied that all he cared about was going home, he said.
The general told Baxter that he would be going home in a week or two, but when Baxter asked for details, the general said he didn't have any, he said.
Canadian citizen, Kristian Lee Baxter, who was being held in Syria reacts as he walks next to Canadian Ambassador to Lebanon, Emmanuelle Lamoureux and Major General Abbas Ibrahim, Lebanon's internal security chief, after being released, in Beirut, Lebanon August 9, 2019. Mohamed Azakir / Reuters
Three or four days later, Baxter said, his hair and beard, which had both grown long over the months, were cut. Later, he heard the general on the other side of his cell door.
"He came to my room and they had a little slot where they pass you food," Baxter said.
"He pushed his hands through and he said, 'You're going home, I'm so happy for you."'
Baxter still thought it might be a trick when he was told he was being taken across the border into Lebanon to be released. His greatest fear, he said, was that he would be released in Syria and kidnapped. He had no money, no friends in the country and does not speak Arabic, he said.
'I thought I would be there forever': Canadian Kristian Lee Baxter, held in Syria since last year, freed in Lebanon
The hard truth for Canadians imprisoned abroad: we can only do so much for you
Trudeau leans on Trump to help Canadians detained in China at G20 summit
He believed it was real when he said Emmanuelle Lamoureux, the Canadian ambassador to Lebanon, introduced herself.
"I said, 'my name is Kristian,' and she goes, 'I know who you are.' And then I just broke down and hugged her and I cried, I just couldn't stop."
Baxter said that when he arrived at the Vancouver airport, he was escorted to a security room to find his mother and stepfather waiting.
Andrea Leclair, who has advocated for her son since he stopped replying to messages while travelling last December, said the experience has taken an emotional toll.
"It's been hellish. We just worked tirelessly and it's just all paid off," she said.
Baxter said he's looking forward to eating steak, surfing in Tofino and taking advantage of all of British Columbia's outdoor opportunities.
"I can't wait to resume my life again."
Five business tax credits scrapped; UCP abandons targeted programs in favour... Redwood Meadows joins Tsuut'ina fight against Springbank dry dam
'A step in the right direction': Councillor pitches task force to curb gun violence | ...
Police investigating random sexual assault in southwest Calgary home | Calgary Herald
Jumbo Glacier deal enshrines Indigenous protected area, consigns mega-resort to history | Calgary He
Recent string of Quebec domestic homicides spur call for action to protect women | Calgary ...

redpajama_set_name: RedPajamaCommonCrawl


Welcome to your 2 bedroom/2.5 bath townhouse. Located at Colony at San Jose. Enjoy resort style living with indoor and outdoor pools, fitness center, clubhouse and more. Tile flooring downstairs and solid wood staircase. Washer and dryer included. Both bedrooms have two separate closets. Come home and relax in your screened in patio. Must see to appreciate.


There are many things to consider when booking places to stay, because the type of accommodation can affect your entire holiday. Whether you have a pet, children, a large group, travelling on a budget or getting away for a romantic weekend with a loved one, Mission Beach has a good selection on offer for whatever your situation and requirements.
Mission Beach is located on the great geen way between Cairns and Townsville. An 1.5 hr drive from Cairns and 2.5hrs from Townsville. Castaways is positition on the beach 400mt from Mission Beach village.


Gamutvision can analyze the performance of printer/paper/ICC profile combinations from ICC profiles themselves: you don't need to have the printer or paper in your possession. All you need is the ICC profile, which can often be dowloaded from the internet. With the profile and Gamutvision you to do a number of remarkable things.
Find out about what performance to expect from a printer/paper combination before you buy.
Determine the quality of ICC profiles and learn which rendering intents work best.
Find out if your printer is performing as specified— by comparing test prints with Gamutvision simulations. Gamutvision won't tell you why your prints don't look right; it can't tell whether it's your printer hardware or software settings, but it will show you what is happening. This can be a huge help in tracing problems.
Once in a while you may stumble on a defective ICC profile. Epson's standard profiles for their Ultrachrome and Ultrachrome K3 printers are generally excellent, but SPR2400 EnhMatte BstPhoto.icc, the profile for the Epson R2400 printer with Enhanced Matte paper at the best quality print setting, has a defect when used with perceptual rendering intent (the intent most frequently recommended for full-tone photogrpahic images). The problem pops right out at you when you run Gamutvision, using the ICC profile as input.
The white band in the bright green-cyan region doesn't belong! There is a corrresponding pothole in the 3D La*b* gamut plot. It's not a common color; you wouldn't notice a problem in most prints, but it might show up unexpectedly as white blotches in images of turquoise jewelry or a tropical lagoon. You could spend hours and burn lots of paper trying to diagnose the problem.
Of course, this is just a simulation. You need to make a print to verify the problem.
The results of printing the Print Test pattern (shown below) on the Epson R2400 (6.5x10 inches) with the SPR2400 EnhMatte BstPhoto.icc profile, Enhanced Matte paper, and perceptual rendering intent are shown on the right. The print was scanned with the Epson 3200 flatbed scanner and saved as an sRGB file. The similarity of the scanned print with the image above— derived from the profile— is quite stunning; nearly perfect to the eye.
Here are the results of analyzing the scanned print using Gamutvision's Print Test feature (very similar to the Imatest Print Test module).
it is derived from a scanned print rather than the ICC profile.
The results are nearly identical to the first image. The print was scanned on an Epson 3200 flatbed scanner set for ICM color management with output to Adobe RGB (1998). (This differs slightly from the image shown above, which was scanned to the sRGB color space for proper web viewing.) The comparison is even more impressive when we view the 2D La*b* saturation map.
Of course the curves aren't identical, but they're remarkably close, especially when you consider that the the Epson 3200 is a relatively inexpensive scanner. It's not a cheap low-end scanner— it has very decent film scanning capability (now replaced by the 4990), but it's not an expensive laboratory instrument either. It costs about as much as a good compact digital camera.
This profile is usable as long as you don't use the prceptual rendering intent. The Relative Colorimetric intent works fine. Results are shown below.


Explosive Secrets About Using Apple Cider Vinegar For Psoriasis To Stop The Itch And Flakes Revealed!
If you suffer from psoriasis, you MUST have apple cider vinegar in your home. If you don't, why don't you? Using apple cider vinegar for psoriasis is one of the best natural remedies to eliminate the itch as well as the flakes that can be produced by psoriasis. The great thing is that you can use it both internally and externally. Here's why you shouldn't waste any time. Drop whatever you're doing and go out or go online and grab yourself some apple cider vinegar.
Everyone who is into holistic health usually touts how awesome apple cider vinegar is. I have personally been drinking a tablespoon of vinegar mixed with water everyday for years now and plan to for the rest of my life in addition to using it externally every now and then.
One thing to remember about using natural remedies is that there is no scientific proof to support all these claims. But you also need to remember that there are no research studies performed because there is not way to profit from many of these natural remedies because they cannot be patented because anyone can make acv or many other natural remedies because nature cannot be patented. There is no financial incentive to conduct studies on these various natural remedies including acv.
What's so great about apple cider vinegar for psoriasis?
The acidic nature of apple cider vinegar may explain its effectiveness against psoriasis as well as many other diseases and conditions. This acidic nature is also antibacterial, antifungal as well as antiviral.
ACV also contains a myriad of vitamins and minerals that our bodies need as well as enzymes and very important anti-inflammatory properties all of which can help to fight various ailments including psoriasis.
Many Eastern practitioners are of the belief that all sickness and disease including psoriasis are caused by an acidic nature in the body and doing everything to help the body become more alkaline can help to fight off various diseases and conditions and create a healthier body.
When people think of acv, one thing they are certain of is that it is acidic and they are partially right because the acidic nature of acv is what helps with healing when used topically. But when taken internally, ACV actually has an alkaline effect in the body by helping to raise the pH levels from an acidic one to a more alkaline one which can help the body become healthier.
There is a certain segment of society that believes that psoriasis results from candida overgrowth (yeast infection) which enables toxins to thrive in the body. Toxins can increase the risk of developing various diseases and conditions and possibly psoriasis. Candida and acv do not mix. The former cannot survive in the presence of the latter.
One of the important benefits of using apple cider vinegar in the body is that it helps to detoxify the body.
When using apple cider vinegar for psoriasis treatment internally, most people usually take a tablespoon or two every day of acv mixed with water. YOU MUST DILUTE IT!
Undiluted acv can wear at the tooth enamel unless you wear dentures. Undiluted acv can also cause burns to the sensitive areas of the throat and mouth, therefore it is recommended to mix it with a liquid before drinking it.
Most people commonly mix one or two tablespoons of acv with 8 to 12 ounces of water, tea or milk (soy, almond milk, etc, are better than dairy milk especially for those with psoriasis). You may add honey to improve the taste if you wish.
When using apple cider vinegar for psoriasis by applying it externally, there are a number of ways that this can help since external application of acv helps to promote blood circulation, is antifungal and antibacterial and helps to restore normal pH levels on the skin.
Appling it directly to flakes or by soaking certain areas such as hands and feet in a bowl of acv.
Mixing ACV with baking soda and applying to itchy lesions to halt the itch in its tracks.
If you are using phototherapy for psoriasis and get burned, acv can help to soothe and heal the burn quite effectively.
One thing to keep in mind when applying acv externally is that it can sting, so you may want to avoid applying it to open lesions unless you have nerves of steel.
An apple cider vinegar rinse has been beneficial to help eliminate flakes and an itchy scalp. You can use undiluted acv on the scalp and hair but most people find this too strong and instead mix equal parts acv and water and apply to the hair and scalp as a final rinse after shampoo and conditioner.
Some people leave this on for 30 minutes to an hour under a shower cap or plastic bag and then rinse while others do not rinse this out. Choose what works for you.
Don't worry, the smell does go away if you do decide to leave it in. It does not linger.
Use this vinegar rinse two to three times a week depending on the thickness of the scales that accumulate on the scalp as well as the itchiness.
Most of what is available in grocery stores is useless. Use that to clean or cook with. For use as a natural remedy, you will need acv that is made from organic apples. It should state that it is raw, unfiltered and that it is fermented using traditional methods i.e. unpasteurized.
Most of all, it should state that it contains the "mother" which is where most of the healthful properties of acv reside. The most popular brand seems to be the Bragg Organic Raw Apple Cider Vinegar With The "Mother" which I have used for many years now and highly recommend and it also comes in a glass bottle which is much better than unhealthy plastic.
If you like making various remedies at home from scratch, you can also make your own acv if you don't have easy access to good quality apple cider vinegar or are into DIY. You should be able to find instructions on how to make acv at home online or offline.
Amazing finds which you need in your life!
Description: Skin Nourishing, Non-Soap Cleanser with 21 Herbal Extracts and Essentials. Apple Cider Vinegar has been used for centuries as a medicinal folk remedy and when combined with potent natural plant extracts make this a rewarding body cleanser.
Description: Scalp Nourishing Cleanser with Tea Tree, Licorice Root and Calendula Flower Immediately improve the condition of your scalp with the first use. Residue from hair care products, minerals in your water and the environment are left behind and begin to slowly build up on your scalp.
The Psoriasis Vinegar Body Rinse Cleanser and the Psoriasis Scalp Vinegar Rinse Cleanser which are shown above are amazing and you should give them a try not only because they use the best and freshest ingredients and contain none of the harmful chemicals like parabens, sulfates or GMOs and are vegan guaranteed in addition to being made in the USA.
They are handmade fresh and guaranteed to be natural and organic. I cannot say enough good things about this brand and these products. Definitely get acv to drink and also get these for external application.
Not only do they contain all the benefits of acv but they also contain many other herbs that are beneficial for treating psoriatic skin such as sea buckthorn, tea tree, foraha, karanja oil, cucumber, oregano, licorice root, willow bark, manuka essential oil, aloe vera, and so on and so forth.
I have psoriasis on my legs. I've tried many itch relieve cream but non of them works. I even went to dermatologist but still, the steriod medicine they prescribed didn't work either. Then I saw this product and was thinking to just give it a try. Surprisely, the marks on my legs started to fade 5 or 6 days after I used the vinegar rinse, and now are almost faded after 10 days of using. I am so happy I found this product, or else I would still suffer from itching. Thanks!!
After trying steroid creams and other medications for 13 MONTHS…this product was the only thing that worked. She has only been using it for 2 weeks and her skin is almost completely smooth. THIS PRODUCT IS A GREAT!!!!
As you can see, you need apple cider vinegar for psoriasis in your life and while not backed up by science, it has been very crucial is the fight against psoriasis and many other diseases and conditions for many people. Don't wait to try this natural remedy.
Note: If you don't have access to apple cider vinegar, you can use white vinegar for psoriasis. Some people have had positive experiences with the latter so it is an option but only if you cannot get ACV which contains more healing properties.
Besides the two products mentioned above, discover more of the best products that you need in your life for treating psoriasis naturally by clicking here.


Lovely Awkward: A Year of Wine, Romance and Life Among the French: Father-daughter writing team!
Here's an interview with the two of us on CBC Radio, discussing Reality Check in Detroit.
Posted by Kerry MacGregor at 8:43 a.m.