summaries: Recently several works have proposed semi-supervised learning methods to leverage unlabeled biological sequences for learning their general-purpose representations. In this work, the authors proposed the Self-GenomeNet, a novel contrastive learning method for nucleotides based on the reverse-complement (RC) context prediction. First, given a sequence, they divide it into two subsequences and transform one into its RC. Then, the model is trained to distinguish their representations from those of other random nucleotide sequences. The authors claimed that the proposed method considerably outperforms previous self-supervised baseline models on three benchmark datasets in both self-supervised and semi-supervised evaluation

summaries: This paper presents a self-supervised learning approach using contrastive loss for representation learning of genomic sequences. The contrastive loss has been used for self-supervised learning in the NLP and computer vision domains and the paper presents its application for genomics. Self-supervised contrastive learning tries to maximize the agreement between augmented views of a sample. Therefore, Self-GenomeNet splits a sequence into two subsequences, and the learned representation of subsequence 1 is compared to subsequence 2 and its revers compliment (positive samples) as well as other sequences (negative samples). The described method has two considerations specific to the genomics tasks - (1) it handles variable sequences (2) it incorporates reverse complement information of the sequences. The method is applied on two prediction tasks and one transfer learning task using sequences from viral, bacterial, and human genomes. Its performance is compared to the supervised model, generative language model, and self-supervised learning models - CPC and Contrastive-sc. The results show improved classification performance over the baseline for both supervised retraining and semi-supervised training settings.

summaries: This submission introduces self-genomenet, a self-supervised training method for learning from DNA sequences. The self-supervision is done by predicting the end of a sequence from its start (both broken into smaller subsequences), through a contrastive loss against other random sequences. The predicted part is also reverse-complemented (RC), making the network learn the expected reverse-complement invariance of the prediction function. The method is extensively tested on several learning tasks, where it shows good performances.

summaries: The authors proposed a self-supervised learning method for nucleotide-level genomic data utilizing reverse-complement of genomic sequences. The proposed method achieved a considerable performance improvement. In addition, the authors proposed an architecture called Self-GenomeNet that handles varying-length genome sequences.

abstractText: In this paper, we introduce a novel contrastive self-supervised method, namely Self-GenomeNet, for nucleotide genome representation learning. To the best of our knowledge, Self-GenomeNet is the first self-supervised framework that learns a representation of nucleotide genomic data, using domain-specific characteristics. Our proposed method learns and parameterizes the latent space by leveraging the reverse-complement of genomic sequences. During the training procedure, we force our framework to capture semantic representations with a novel context network on top of features extracted by an encoder network. The network is trained with an unsupervised contrastive loss. Extensive experiments with different datasets show that our method with self-supervised and semi-supervised settings outperforms state-of-the-art deep learning methods. Furthermore, we show that the learned representations generalize well and can be transferred to new datasets and tasks. The source code of Self-GenomeNet and all experiments are provided as supplementary material.

authors:
Babak Alipanahi
Andrew Delong
Matthew T Weirauch
Brendan J Frey

title: Predicting the sequence specificities of dna-and rna-binding proteins by deep learning

venue: Nature biotechnology,

authors:
Tom B Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell

title: Language models are few-shot learners

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

venue: In International conference on machine learning,

authors:
Madalina Ciortan
Matthieu Defrance

title: Contrastive self-supervised clustering of scrna-seq data

venue: BMC bioinformatics,

authors:
Andrew M Dai
Quoc V Le

title: Semi-supervised sequence learning

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: IEEE conference on computer vision and pattern recognition,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Gökcen Eraslan
Žiga Avsec
Julien Gagneur
Fabian J Theis

title: Deep learning: new computational modelling techniques for genomics

venue: Nature Reviews Genetics,

authors:
Vladimir Gligorijević
P Douglas Renfrew
Tomasz Kosciolek
Julia Koehler Leman
Daniel Berenberg
Tommi Vatanen
Chris Chandler
Bryn C Taylor
Ian M Fisk
Hera Vlamakis

title: Structurebased protein function prediction using graph convolutional networks

venue: Nature communications,

authors:
Xavier Glorot
Yoshua Bengio

title: Understanding the difficulty of training deep feedforward neural networks

venue: In Proceedings of the thirteenth international conference on artificial intelligence and statistics,

authors:
Michael Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

authors:
John A Hartigan
Manchek A Wong

title: Algorithm as 136: A k-means clustering algorithm

venue: Journal of the royal statistical society. series c (applied statistics),

authors:
Hamid Reza Hassanzadeh
May D Wang

title: Deeperbind: Enhancing prediction of sequence specificities of dna binding proteins

venue: In 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM),

title: Data-efficient image recognition with contrastive predictive coding

venue: In International Conference on Machine Learning,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
David R Kelley
Yakir A Reshef
Maxwell Bileschi
David Belanger
Cory Y McLean
Jasper Snoek

title: Sequential regulatory activity prediction across chromosomes with convolutional neural networks

venue: Genome research,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Jun Li
Yufeng Yao
H Howard Xu
Limin Hao
Zixin Deng
Kumar Rajakumar
Hong-Yu Ou

title: Secret6: a web-based resource for type vi secretion systems found in bacteria

venue: Environmental microbiology,

authors:
Amy X Lu
Haoran Zhang
Marzyeh Ghassemi
Alan M Moses

title: Self-supervised contrastive learning of protein representations by mutual information maximization

venue: BioRxiv,

authors:
Vincent Mallet
Jean-Philippe Vert

title: Reverse-complement equivariant networks for dna sequences

venue: bioRxiv,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Xiaoyong Pan
Peter Rijnbeek
Junchi Yan
Hong-Bin Shen

title: Prediction of rna-protein sequence and structure binding preferences using deep convolutional and recurrent neural networks

venue: BMC genomics,

authors:
Marie-Madlen Pust
Burkhard Tümmler

title: Identification of core and rare species in metagenome samples based on shotgun metagenomic sequencing, fourier transforms and spectral comparisons

venue: ISME Communications,

authors:
Michael A Quail
Miriam Smith
Paul Coupland
Thomas D Otto
Simon R Harris
Thomas R Connor
Anna Bertoni
Harold P Swerdlow
Yong Gu

title: A tale of three next generation sequencing platforms: comparison of ion torrent, pacific biosciences and illumina miseq sequencers

authors:
Daniel Quang
Xiaohui Xie

title: Danq: a hybrid convolutional and recurrent deep neural network for quantifying the function of dna sequences

venue: Nucleic acids research,

authors:
Alexander Rives
Joshua Meier
Tom Sercu
Siddharth Goyal
Zeming Lin
Jason Liu
Demi Guo
Myle Ott
C Lawrence Zitnick
Jerry Ma

title: Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

venue: Proceedings of the National Academy of Sciences,

authors:
Avanti Shrikumar
Peyton Greenside
Anshul Kundaje

title: Reverse-complement parameter sharing improves deep learning models for genomics

venue: bioRxiv, page

authors:
Marina Sokolova
Guy Lapalme

title: A systematic analysis of performance measures for classification tasks

venue: Information processing & management,

authors:
Ameni Trabelsi
Mohamed Chaabane
Asa Ben-Hur

title: Comprehensive evaluation of deep learning architectures for prediction of dna/rna sequence binding specificities

authors:
Rongjie Wang
Tianyi Zang
Yadong Wang

title: Human mitochondrial genome compression using machine learning techniques

venue: Human genomics,

authors:
Hannah Zhou
Avanti Shrikumar
Anshul Kundaje

title: Towards a better understanding of reversecomplement equivariance for deep learning models in regulatory genomics

authors:
Jian Zhou
Olga G Troyanskaya

title: Predicting effects of noncoding variants with deep learning– based sequence model

venue: Nature methods,

heading: 1 INTRODUCTION

text:
In bioinformatics, learning from unlabeled data can reduce development costs in many deep learning applications, since it reduces the number of required annotated samples that often require manual experimental validation, such as the functional annotation of genes (Gligorijević et al., 2021) or chromatin effects of single nucleotide polymorphisms (SNPs) (Zhou and Troyanskaya, 2015). Selfsupervised learning is amongst the most promising approaches for learning from limited amounts of labeled data and is an attractive approach in bioinformatics due to the availability of large quantities of unlabeled sequence data. In contrast to supervised methods, these techniques learn a representation of the data without relying on human annotation. Early self-supervised methods focused on solving a pretext task (Devlin et al., 2018) that allows utilizing the data itself to generate labels, and use supervised methods to solve unsupervised problems. The representations learned by performing this task can be used as a starting point for downstream supervised tasks such as taxonomic prediction or gene annotation. Several different self-supervised representation learning methods were proposed e.g. in the fields of natural language processing (NLP) and computer vision (CV). However, self-supervised learning has not yet seen such widespread adoption in bioinformatics and remains an important and challenging endeavor for the representation of genome data.
Existing methods for representation learning for omics-data tends to use methods from NLP or CV with slight architecture adjustments and additional preprocessing (e.g. Rives et al., 2021; Lu et al., 2020; Ciortan and Defrance, 2021). However, these methods so far do not take many biological characteristics into account. In particular, the genome sequence embodied by one strand on the DNA molecule is always accompanied by another strand going in the opposite direction with complementary nucleotides. This means that both the forward and the so-called reverse-complement (RC) sequences are valid DNA sequences and equally correspond to several properties of the genome, such as regulatory and taxonomical characteristics. Therefore, machine learning models developed to distinguish different aspects of genomic sequences ideally take the RC into account, for example by being equivariant with respect to it (Mallet and Vert, 2021; Zhou et al., 2021). Our proposed Self-GenomeNet uses a context network that learns representations that also take advantage of the RC.
We propose a simple contrastive self-supervised algorithm for learning representations of nucleotide-level genomic sequences (Figure 1). Our method divides a sequence into two nonintersecting subsequences, where one of them is transformed to its RC. Then, the contextual representations of these subsequences are required to distinguish each other from the representations of other, random sequences. We exploit the inherent symmetry in the genome sequence implied by the RC to achieve a natural simplification over possible methods that do not take it into account, all while encoding RC-awareness in our architecture. Moreover, we train our network with varyinglength subsequences both for the input and the target subsequence. Because, genome sequences can have semantic structure at varying ranges of length, representations that respect both local and global structures are preferable. For this, the context network makes use of a recurrent neural network layer, which makes the computationally efficient training based on all used length scales within a single step possible. Our self-supervised method utilizes contrastive loss to maximize the similarity between representations of subsequence and its RC while minimizing the similarity between representations derived from unrelated sequences.
Contributions: We present the first self-supervised framework tailored specifically to DNA sequence data. Our main findings and contributions can be summarized as follows: (1) We introduce a contrastive self-supervised method that is specialized for nucleotide-level genomics data. (2) We propose a context-aware network that makes use of inherent symmetries in genomics data in the form of the reverse-complement and improves the quality of learning representation. (3) SelfGenomeNet is evaluated in self-supervised and semi-supervised settings on bioinformatic benchmarks and datasets and achieved superior performance in several tasks. (4) We show the learned representations generalize well when transferred to a new dataset and task.

heading: 2 RELATED WORK AND BACKGROUND

text:
Self-supervised learning Deep learning models based on self-supervised learning have become a standard method of choice in natural language processing (NLP). Powerful models such as GPT3 (Brown et al., 2020) and BERT (Devlin et al., 2018) are pre-training in a self-supervised manner on a large text corpus. They are then often fine-tuned for various tasks using smaller task-specific datasets, where they show performance that is superior to purely supervised methods. Recently, selfsupervised visual representation learning (Chen et al., 2020) outperformed supervised methods and achieved state-of-the-art performance on ImageNet (Deng et al., 2009). Self-supervised contrastive learning is one of the most successful methods where it maximizes the agreement of learned representations between differently augmented views of the same example (Chen et al., 2020; Oord et al., 2018). Motivated by this, we propose a novel contrastive self-supervised model similar to (Oord et al., 2018) but differing in network architecture, representation learning strategy, and with applications specifically in bioinformatics.
Self-supervised learning in bioinformatics Existing self-supervised learning techniques applied on sequential RNA and protein data are adopted from other application fields of deep learning. (Rives
et al., 2021) adopted BERT (Devlin et al., 2018) as a pre-trained model for predicting masked amino acids and achieved superior performance on sequence prediction task. Contrastive-sc (Ciortan and Defrance, 2021) is a method adapted from computer vision for cell clustering based on single-cell RNA sequencing data. Contrastive-sc creates two copies of sequences with randomly masked nucleotides and trained the network using a contrastive loss. As a downstream task, self-supervised embeddings are used for clustering with K-Means algorithm (Hartigan and Wong, 1979). All these methods for self-supervision described so far have in common that they rely on randomly masking part of their input to create a pretext task. This differs from Contrastive Predictive Coding (CPC) (Oord et al., 2018), which splits the input into parts such as image patches or subsequences, and predicts the representation of some of the parts, contrastively, from the representation of other parts. CPCProt (Lu et al., 2020) adapted the CPC model for protein data.Similarly, Self-GenomeNet is trained by the contrastive loss in which the positive pairs include the input sequence and a reversecomplement of the sequence. Unlike previous works, our network is trained with varying-length targets to explore both local and global structures of sequences.
Deep learning methods for genome sequences In genomics, many deep learning architectures use convolutional layers to capture local patterns in data, whereas recurrent layers learn long-range dependencies in the sequences (Eraslan et al., 2019). Zhou and Troyanskaya (2015) created the DeepSEA dataset for predicting chromatin effects of sequence alterations, and used it to train a model consisting of multiple convolutional layers. Later, DanQ (Quang and Xie, 2016) improved the deep learning framework of DeepSEA using a bidirectional LSTM (Hochreiter and Schmidhuber, 1997) unit on top of a convolutional layer. iDeepS (Pan et al., 2018), DeepBind (Alipanahi et al., 2015), and DeeperBind (Hassanzadeh and Wang, 2016) use a sequence CNNs and RNNs to predict RNA binding proteins sites. DeepDNA (Wang et al., 2019) uses an LSTM layer on top of a CNN layer to compress human mitochondrial genome data. Instead of recurrent layers, it is also possible to use dilated convolutional layers to model long-range dependencies in genome data, such as done in Basenji (Kelley et al., 2018), which is used to predict cell-type–specific epigenetic and transcriptional profiles in large mammalian genomes. Trabelsi et al. (2019) performed a comprehensive study on network architecture where the ECBLSTM network, composed of an embedding layer, CNNs, and LSTM layer outperformed other networks on several genomic tasks. Here, we also used the DeepSEA dataset to compared the performance of our method with several other self-supervision techniques. The architecture of Self-GenomeNet requires a recurrent layer for it to create representations of varying-length subsequences, and is therefore based off the DanQ-architecture (Quang and Xie, 2016), which has shown good performance on the DeepSEA task.
Reverse Complement DNA molecules in living organisms always occur as a double strand, where both strands have opposite directions and consist of nucleotides that are complementary to each other. The sequence on one strand is therefore accompanied by its reverse complement (RC) on the other: The sequence in reverse, and with the A, C, G, T nucleotides replaced by T, G, C, A respectively. Since both forward and reverse-complement sequences are found in the structure, they are equally responsible for several properties of the genome, such as regulatory and taxonomical characteristics. However, next-generation sequencing techniques (NGS) output sub-sequences randomly drawn from both strands, since it usually involves breaking the genome into a collection of small DNA fragments that are sequenced individually. Therefore, both the original sequence data and the RC of this sequence are valid inputs to the machine learning models developed to work on NGS datasets (Shrikumar et al., 2017; Zhou et al., 2021). In this paper, we exploit this genome-specific feature in contrastive self-supervised learning framework helps to improve the learning representation in several genomic tasks.

heading: 3 METHOD

text: We first describe the motivation of our proposed method before explaining its details in Section 3.2.

heading: 3.1 MOTIVATION

text:
Most successful self-supervised learning approaches are built for computer vision or NLP and are therefore not optimal for bioinformatics data. For example, the self-supervision task of CPC (Oord et al., 2018) or generative language models (Dai and Le, 2015) are based on predicting the representation of a short, constant length subsequence or a single token, which could be an N-gram or a single nucleotide. We hypothesize single-length short sequence targets hinder the model from learn-
ing global semantics effectively. We therefore propose training representations that are based on target subsequences whose lengths are varying. While predicting future observations of constant sequence length from input sequences which are changing in length is common for methods like CPC and single character models, predicting representations of varying length sequences is a novelty in this area. Similar to Oord et al. (2018), we use a recurrent context network on top of an encoding network that encodes to constant-size patches. However, we use the output of the context network to predict the output of the same context network applied to other patches, as opposed to predicting only the output of the encoding network. The fact that the RC of an input sequence is a valid genome sequence as well simplifies this approach: instead of having to fit both one forward and one reverse context network that predict each other, only a single context-network is trained that is both applied to the forward as well as to the RC sequence.

heading: 3.2 SELF-GENOMENET

text:
The Self-GenomeNet architecture learns representations of varying-length genome sequences by encoding them in both convolutional and recurrent neural network layers, which are trained using a contrastive loss. The architecture enforces the symmetry implied by the RC while also making training more efficient. As depicted in Fig. 1, our proposed method initially creates two subsequences from an input sequence. The subsequences are taken by a representation network composed of the encoder network (fθ) and the context network (Cφ). Later, on top of the embedding representation, the linear prediction layer (qm) estimates the embedding of the other subsequence using a contrastive loss against random other subsequences. More specifically, given are an unlabeled genome sequence of nucleotides S1:N = [s1, s2, . . . , sN ] and its RC SN :1 = [sN , sN−1, . . . , s1], where si ∈ {A,C,G,T} and si gives the complementary nucleotide, e.g. A = T. We aim to produce an embedding of a subsequence S1:t that contrastively predicts the embedding of the RC of the remaining subsequence, SN :t′ for some t′ > t. Predicting SN :t′ , i.e. the RC of St′:N , instead of just the reverse of St′:N , has the benefit that it is possible to use the same embedding and context network for both the first and the second subsequence, as SN :t′ is a valid genome sequence, whereas SN :t′ is not. We use a recurrent layer to build the representations of both S and S, so representations of different length subsequences are calculated at once by using the hidden state of the recurrent unit at different points in the sequence. This makes it possible to contrastively predict subsequences of varying lengths against each other efficiently in a single training step. This has an advantage over using two forward sequences S1:t and St′:N , where using different starting point values t′ in one step would be much more computationally demanding, since the values of recurrent units would have to be calculated fresh for every t′. The entire procedure is symmetric since our model learns sequence and the RC of sequence. This symmetry provides the possibility to use a context network as well
as the prediction networks with shared weights both when reading the subsequence in forward direction as in the left subsequence in Fig. 2 as well as when reading the reverse-complement of the sequence on the right. This is visualized in Fig. 3 in Appendix. (i) The same recurrent context network is used for predicting and target subsequences, (ii) the target subsequence can also predict the predicting subsequence, (iii) the same prediction network(s) used for predictions of the target subsequences from the predicting subsequences can also be used for the predictions of the predicting subsequences from the target subsequences. In other words, with the help of the symmetry induced by RC, substantial weight sharing is provided.
To produce the representation, the sequence is first divided into P overlapping patches Sp(j) with p(j) = (j · a + 1):(j · a + l), where l is the patch-length and a the patch stride length, and where a < l for overlapping patches. Patches are first encoded with a convolutional neural network fθ(·). The resulting sequence of vectors [fθ(Sp(0)), fθ(Sp(1)), . . . , fθ(Sp(P−1))] is fed into a recurrent context network Cφ(·), giving rise to embeddings zi = Cφ ( {fθ(Sp(u))}u<i ) for 1 ≤ i ≤ P .
Patches of the RC, Sp(j) = [sj·a+l, . . . , sj·a+1] are equally fed into first fθ then Cφ, giving rise to zi = Cφ ( {fθ(Sp(u))}u≥i ) . zi is then a representation of S1:((i−1)·a+l); likewise, zi represents
SN :(i·a+1).
Training the encoder and context networks consists of predicting zi+m from zi, with m > 0, contrastively against corresponding embeddings from other, negative example sequences S(k)−, i.e. against z(k)−i+m (note: m is the stride length between the patches). For this, a linear prediction layer qm is used, and the Noise Contrastive Estimation or InfoNCE loss (Gutmann and Hyvärinen, 2010), which maximizes the mutual information shared between forward sequence and its matching RC sequence, is used:
Li,m = − log exp
( zTi+mqm(zi) ) exp ( zTi+mqm(zi) ) + ∑ k exp (( z (k)− i+m )T qm(zi)
) . (1) Negative samples z(k)−i+m are efficiently produced by comparing against representations of other sequences loaded in the same mini-batch. Note that every sequence in the minibatch produces two negative examples for other sequences, giving 2(B − 1) negative samples when using minibatch size B: The embedding of the RC of the last (P − i − m) patches starting from path (i + m) – i.e., the analog to zi+m coming from other sequences – and the embedding of the first (P − i−m) patches of the normal forward-sequence, so the analog to zP−i−m. This is because of the invariance of the training method with respect to the RC. Embeddings are, however, always only contrasted with embeddings of sequences that have the same length as the positive example. This is to prevent the network from learning to encode the represented length directly to gain an advantage, which would not be an intrinsically interesting feature for downstream tasks.
One loss term is introduced for each index i, denoting the number of patches represented by zi. Furthermore, because it is possible to skip a different number of patches m between the representations being matched, one can sum over multiple values of m, each with its individual weights for predictor qm. Because of the symmetry of the setup, the model is both predicting zi+m from zi as described above, as well as zi from zi+m. The corresponding loss Li,m then uses negative examples with the same length as zi. The ultimate loss for each individual sequence S is therefore:
L = ∑ i,m ( Li,m + Li,m ) . (2)
It was proven by (Oord et al., 2018), optimizing a contrastive loss by predicting future patches Sp(j) maximizes the lower bound of mutual information between the patches and the embeddings. The theoretical proof regarding how we maximize the lower bound of mutual information between the embeddings zi and varying-length reverse-complement targets {Sp(u)}u≥j for several values of j is straight-forward by exchanging {Sp(u)}u≥j and Sp(j) (they use xk+t as notation) in the equations of the proof derived by Oord et al. (2018). By adopting the equations accordingly, we derive,
I({Sp(u)}u≥j , zi) ≥ log(N)− L (3)
where I({Sp(u)}u≥j , zi) is the mutual information between the learned representation zi and the reverse-complement of the consecutive sequence {Sp(u)}u≥j . N is the number of random samples in the contrastive pretraining and L is the loss function given in the Eq.2. Note that in our method, differently from CPC (Oord et al., 2018), as the value of j changes, the number of patches in the set and therefore the total length of the targeted patches vary. By doing this, we maximize the mutual information between varying-length targets and the learned representations.

heading: 4 EXPERIMENT SETUP

text:
Datasets and Tasks We evaluate our method with the following datasets:
(1) DeepSEA dataset: introduced by Zhou and Troyanskaya (2015), contains around 5 million human genome sequences, where each data sample contains 1000 nucleotides as input and a label vector for 919 binary chromatin features. We evaluated our model with the DeepSEA dataset because it is a dataset often used by genomic deep learning methods for evaluation (e.g. Zhou and Troyanskaya, 2015; Quang and Xie, 2016; Kelley et al., 2018). It represents the task of identifying the class of certain regions within a genome sequence.
(2) Virus dataset: We downloaded all publicly available viral genomes from GenBank (Sayers et al., 2021). We allocated the FASTA files into training, validation, and test sets by approximate ratios of 70%, 20%, and 10%. We divided the dataset into two taxonomical classes as a classification task: bacteriophages vs. viruses that are not bacteriophages, based on provided annotations. The bacteriophage class contained approximately 1.0 billion nucleotides, the non-phage virus dataset approximately 0.5 billion nucleotides. The Virus dataset is a machine learning task that is complementary to it: instead of regions of a genome, the class of the whole sequence should be determined, based on a given subsequence of fixed length. Species classification tasks often arise in metagenomics, where NGS is used to analyse DNA found in environmental samples (Pust and Tümmler, 2021). NGS typically produces 150nt samples (Quail et al., 2012), therefore a sequence length of 150nt is used with this dataset, which is drawn from the sequences at random. The length of 150nt would therefore allow a direct application of our model in metagenomics.
(3) Bacteria dataset: We downloaded all publicly available bacteria genomes from GenBank, comprising approximately 83 billion nucleotides, similarly to how we collected the Virus dataset. This dataset was only used for unsupervised pre-training for transfer learning, so no supervised task was defined. The sequence length was set to 1000 nt, to compare the capabilities of the methods for different sequence lengths.
(4) T6SS effector protein dataset: We performed experiments on T6SS dataset that publicly available bacteria dataset (SecReT6, Li et al. 2015) to demonstrate that our method works well for a real-world dataset with actual label scarcity. Here the task is the identification of effector proteins. We used T6SS effector proteins as the positive samples to identify, whereas T6SS immunity proteins, T6SS regulators, and T6SS accessory proteins are negative samples. We divided the training, validation, and test set by approximate ratios of 60%, 20%, and 20%. The sequence length is 1000nt in all experiments. We provide more details for pre-processing and motivation of the datasets in Appendix B.
Network architecture As mentioned in Section 3, Self-GenomeNet composed of a representation block followed by an embedding layer and a prediction network. The representation block includes encoder network fθ(·) and context network Cφ(·). For the self-supervised training on the Virus dataset, the encoder network is a convolution layer with 1024 filters, a kernel size of 24, and a stride of 6. Only a single CNN-layer is used, so the kernel parameters directly correspond to patch parameters described above, i.e. l = 24 and a = 6. The context network is an LSTM layer with a hidden layer size of 512. For the DeepSEA benchmark, we followed the DanQ network architecture by Quang and Xie (2016). The encoder network has an additional max-pooling layer followed by a dropout layer.
For each value of m, a different fully connected layer is used as the prediction layer qm. In any selfsupervised pretraining, the default choice is to have only one prediction network q1 due to longer pretraining time when multiple prediction layers are used. Therefore the default is estimating the neighbor subsequence of the predicting subsequence without any skipped nucleotide between them. However, we additionally test pretraining with multiple prediction networks and therefore estimate
even further subsequences for the DeepSEA dataset. The reported results are the best performance of Self-GenomeNet. We provide more information in Appendix A.
Optimization We used Adam optimizer (Kingma and Ba, 2014) with β1 = 0.9, β2 = 0.999 and a learning rate of 0.0001 for all experiments except the DeepSEA dataset that we used the RMSprop optimizer. The dropout rate of 0.2 and 0.5 are used respectively for the dropout layers as in the DanQ model (Quang and Xie, 2016). For all datasets, we used the default Keras weight initialization, which is the Glorot uniform initialization (Glorot and Bengio, 2010). During the self-supervised pre-training, we used the largest fitting minibatch size (power of 2) that worked in a GeForce RTX 2080 Ti, which was 512 for the virus dataset and 128 for the Bacteria and DeepSEA datasets.
For the baselines experiments, we used the same hyperparameter settings for all our experiments such as Adam hyperparameters or learning rate. When not applicable, we followed the recommendations in the papers of the baseline methods regarding the hyperparameter selection. We provided more detail on optimization in Appendix B.
Evaluation For all experiments, we report the performance in terms of F1 score. However, We use Receiver Operating Characteristics (ROC) Area Under Curve (AUC) and Precision-Recall (PR) AUC for DeepSEA dataset for all 919 binary outputs, as these metrics are used by Quang and Xie (2016). In this paper, it was also shown that the sparsity of positive binary targets(∼ 2%) in the dataset inflated the ROC AUC, and that PR AUC is a better indicator of performance since it does not take the number of true negatives into account. We macro-averaged recall RecallM (Sokolova and Lapalme (2009)), which is equivalent to balanced accuracy, and F1 score for dataset evaluations on Virus.

heading: 5 RESULTS

text:
We compare our proposed Self-GenomeNet with CPC (Oord et al., 2018) as the CPC is the closest self-supervision method to our method which can also be applied in different application fields of deep learning. Moreover, we provide comparison with Contrastive-sc (Ciortan and Defrance, 2021) and generative language model since these methods originally proposed for computer vision or natural language processing respectively but applied in bioinformatics several times. Both single nucleotides, denoted as (single nt), and 3-gram targets are predicted for pretraining using a language model (See Fig. 4).We follow the standard evaluation protocol for self-supervised learning and evaluate the unsupervised learned representation with a linear classification and semi-supervised tasks, as well as using transfer learning to different datasets and different tasks (Henaff, 2020).
Linear Evaluation The common evaluation protocol for self-supervised learning is based on freezing the representation networks (both base-encoder and context network) after unsupervised pretraining and then train a supervised linear classifier on top of them. The linear classifier is a fully connected layer followed by softmax-classification using cross-entropy loss, which is plugged on top of the context network after removing the prediction layer. Table 1 shows the comparison of our model against the baselines under the linear evaluation on Virus and DeepSEA dataset.
Semi-supervised Learning We evaluate the performance of our models on a semi-supervised learning task. In this task, we pretrain our model on unlabeled datasets and fine-tune a classification or prediction model using a subset of datasets with labels. Differently from the linear evaluation protocol, the pretrained base network is also fine-tuned. We follow the semi-supervised protocol of (Henaff, 2020) and use the same fixed splits of respectively 1% and 10% of labeled training data in DeepSEA and Virus datasets.
Dataset DeepSEA Virus Labeled 1% 10% 1% 10%
Transfer to Other Tasks We further assess the generalization capacity of the learned representation on viral detection by pre-training Self-GenomeNet on the Bacteria datasets. Our motivation for transfer learning is to take advantage of abundant bacterial data, which are much better described and studied than their viral counterparts, and optimize the model for tasks on viruses(such as viral identification or taxonomic classifications). We also used the T6SS dataset to demonstrate that our method works well for a real-world dataset with actual label scarcity. To this end, the Self-GenomeNet is trained in self-supervised fashion on the bacteria dataset and then we train in a supervised manner using the phage and non-phage labels on the virus dataset and evaluate the performance on this dataset. Table 3 provides a comparison of transfer learning performance of our self-supervised approach for the task of classification of viruses.
Dataset Virus T6SS Base Network Fine-tuned Fixed Fine-tuned Fixed

heading: 6 ABLATION STUDIES AND DISCUSSIONS

text:
We present multiple ablation studies on Self-GenomeNet to give an intuition of its behavior and performance: (1) the impact of reverse-complement targets against other potential targets; (2) predicting varying-length targets against single-length targets; (3) robustness of our algorithm in the low-data regime. All the ablation study is performed on the virus dataset under linear evaluation setting.
Learning from the reverse complement of sequences Self-GenomeNet uses the embedding representation of a context network for downstream tasks which learns by predicting the reverse complement of neighbor subsequence S̄N :t′ as a target for its predictions. We investigated the effect of using target embeddings derived by RC of neighbor subsequences against two other potential ways of creating embeddings derived from varying-length targets. To this end, besides our method, we performed two other experiments where the recurrent context network reads the ”target” subsequences in different ways: in forward and reverse directions. In forward setting, the context network reads the target subsequence St′:N in the same direction as it reads the predicting subsequence S1:t; whereas reverse direction refers to reading the target sequence other way around, as in SN :t′ . Table 4 compares the performance of our trained network in these ways.
Predicting varying-length targets We investigate the effect of having varying-length target subsequences by comparing our method against the constant-length target subsequences. As mentioned, having non-varying length targets is indeed a common feature for self-supervision techniques such as CPC or language models. Nevertheless, we would like to investigate the effect of varying-length targets, when other features of Self-GenomeNet is unchanged. To create nonvarying-length target subsequences we divide the sequences into two nearly equal-in-size subsequences. Specifically, we split the sequences of 150 into 72 and 78, so that subsequences with length 72 and 78 predicts the embeddings derived from the reverse-complement of each other. In the Self-GenomeNet however, for the given architecture, subsequences with length x ∈ {24, 30, 36, ...126} predicts a subsequence with length 150−x. 72 and 78 was chosen because they are the two closest values that can be investigated using our previous architecture without any padding (splitting into 2 subsequence with 75 is not possible due to stride value of 6). We observed for linear evaluation setting on the virus dataset the performance dropped from 69.1% to 64.9% (macro-averaged recall), when only same-size targets are predicted. A straight-forward explanation to this is that maximizing the mutual information between the learned embeddings and varying-length target subsequences (both short and long) helps the machine learning model to learn more comprehensive range of semantics and therefore boosts the performance in the downstream supervised tasks.
Data efficient We study and compare the performance of Self-GenomeNet with other methods in the low-data regime. We created label-scarcity artificially on the Virus and DeepSEA datasets by subsetting the number of labeled training samples down to 0.1%. We train the models shown in Table 5 using 0.1% of the dataset after pretraining on the whole dataset in self-supervised way. The reported results are when the base networks are fine-tuned. Based on the results reported in Table 5, our method outperforms other methods by substantial margin in the low-data regime.

heading: 7 CONCLUSION

text:
In this paper, we presented Self-GenomeNet, a framework for contrastive learning of nucleotidelevel genomic data. Self-GenomeNet is composed of an encoding network followed by a context network and uses a prediction layer for training. The network extracts compact latent representations by predicting representations of the reverse complement of future observations. Self-GenomeNet is optimized end-to-end using unsupervised contrastive loss. We tested the learned representations by linear evaluation, semi-supervised learning, and transfer to other tasks and achieved better performance on the four public bioinformatics benchmarks compared to other self-supervision methods. Moreover, we showed the ability of our method for training in the low-data regime, tested the effect of predicting varying-length target subsequences, and the effect of estimating the output of the context network which reads the predicted subsequence in forward or reverse direction compared to reading reverse-complement of the subsequences.
Reproducibility Statement: The source code is available in the supplementary material.

heading: APPENDIX A PREDICTION NETWORK

text:
Single prediction network or multiple prediction networks: For the pretraining in DeepSEA dataset, we trained the network with only one prediction network q1 (qm where m = 1) as well as with multiple prediction networks m ∈ {1, 4, 7, 10, 13, 16} for DeepSEA. As mentioned in the manuscript, multiple prediction networks are used when further subsequences are predicted, in addition to the closest subsequence to the predicting subsequence. We report the results of SelfGenomeNet when only one single network is used in the tables 6, 7, 8. Only single prediction network is used during the pretraining stage of the other datasets.
Dataset DeepSEA Labeled 1% 10%
Here we also report the pretraining time in hours in Table 9 for our method and the baselines on DeepSEA dataset using a GeForce RTX 2080 Ti. One can immediately see, the pretraining time for the simplest version of Self-GenomeNet with one prediction network (m = 1) is shorter than or comparable to the pretraining time of the other self-supervision methods, whereas the performance is superior.

heading: APPENDIX B PROCESSING OF DATASETS AND FILES

text:
For virus-bacteria dataset, data samples are created by reading consecutive samples taken from random regions of FASTA files. Compared to division of FASTA files into same-length non-intersecting sequences, this technique increases number of samples that can be created out of FASTA files. In order to provide randomness, either maximum 90% and 10% of the file is processed for virus files and bacteria files respectively, or maximum 64 data samples are created from a file, before next FASTA file is processed.
For phage/non-phage classification on virus dataset, we set the sequence length to 150. The motivation to limit the input length to 150 nt in some of our evaluations is to be close to a potential application, since this is a typical read length that gets produced during Next Generation Sequencing which is frequently used. The length of 150 nt, therefore, allows a direct application of the sequencing output to perform model predictions that can be built using pre-trained models as presented in our work. We also evaluate some experiments with a sequence length of 1,000 nt, because this is a meaningful length of the output of genome assemblers, where these 150 nt long reads are assembled together using overlap information or mapping to reference sequences to produce longer fragments.
The short sequence length can be justified to build neural networks directly on the read output of a sequencer (150 nt), rather than on the output of the assembling process since the latter comes with several limitations. it requires (i) computational resources and often (ii) the presence of reference genomes, which are often absent when an environmental sample is sequenced; furthermore (iii) assembling Softwares perform poorly on repetitive regions of bacterial genomes that are of particular interest to study since these often belong to the bacterial defense system. Therefore, traditional bioinformatics software is taking raw read input rather than preprocessed contigs.

heading: APPENDIX C OPTIMIZATION OF SELF-SUPERVISED METHODS

text:
For all architectures and all our experiments, we used the same encoder and context models for a fair comparison. Moreover, we took most hyperparameters as given in the original method.
As depicted in Fig 5, we adopted the Henaff (2020) implementation of CPC for 1-dimensional data and use an RNN (LSTM) and a single fully connected layer on top of LSTM layer for prediction as in Oord et al. (2018). It is shown by (Oord et al., 2018) that the performance for the baseline models decreases significantly when the span of predicted future observations is too short, however it decreases slightly when they are too long. To this extend, the predicted span for the patches are selected to be not too short as well as not too long compared to the full length of the sequence. For pretraining on the virus dataset, every second feature from 4th to 10th derived by encoder network is predicted with different linear layers. Thus, the representations derived from the following 60 nucleotides are predicted, when the full sequence length is 150. For bacteria pretraining, the following 2nd to 10th features are predicted, making the prediction span of 200 nucleotides for the sequence length of 1000 nucleotides. For the DeepSEA dataset, every second feature from 3rd to 20th features are predicted, making the span up to 260 nucleotides for the sequence length of 1000.
For Contrastive-sc (Ciortan and Defrance, 2021) (Fig. 5), we used the masking rate of 0.9 as recommended by the paper and did not use additional linear layer on top of the final outputs of LSTM.
For the language modelling (Fig. 4), we followed different strategy for different datasets. For the virus and bacteria dataset, the model predicts the following 60 and 200 nucleotides respectively, which are taken from outside of the sequence. For DeepSEA, the nucleotides outside the sequence is not provided within the dataset. Thus, the last 260 nucleotides are estimated from the first 740 nucleotides during pretraining. Please note that the span of predicted observations is the same for the CPC and the language modelling.
In the supervised training of virus datasets, we firstly train only the final linear layer while the pretrained base model is frozen. Once this layer is optimized, starting from this point, we tune the whole network. For DeepSEA dataset, we directly optimize the whole model for both freezed and unfreezed base-networks.
C.1 ADDITIONAL EXPERIMENTAL RESULTS
In this section we further compare our proposed Self-GenomeNet with the baseline methods for linear evaluation (Fig. 6), semi-supervised learning (Fig. 8, Fig. 9), transfer learning (Fig. 10,Fig. 11), and data efficiency (Fig. 7).

title: SELF-GENOMENET: SELF-SUPERVISED LEARNING


summaries: This paper proposes to make the geometric neurons of Melnyk et al.'21 to be steerable so that objects undergoing arbitrary rotations can be classified with higher accuracy. This is done in multiple stages. First, the neurons of Melnyk et al. are trained to convergence with a hyperspherical output layer. Then, the frozen weights are transformed such that the input of a *steerable neuron* can be written as a linear combinations of the rotated versions of itself. The experiments act as a sanity check while demonstrating the validity of the algorithm on a simple human pose dataset.

summaries: The paper proposes a method for constructing steerable spherical neurons, building on the recent Geometric Neurons developed in Melnyk et al 2021. The main technical result in the paper is a steerability constraint for a geometric neuron, as given by eq. (13) in the paper. This constraint is used in an implementation whereby a steerable model is constructed and is then use for two tasks. The first task involves the use of steerable spherical neutrons for the classification of 3D Tetris objects seen under rotations, and the second is a similar experiment applied to 3D skeleton data. The results demonstrate a very large performance boost when using the steerable versions. The second experiment builds on this idea to construct a version that adapts a possibly imperfect initial rotation estimate, using the representation. This second experiment is in the spirit of demonstrating equivariance under 3D rotations with perturbations.

summaries: ## Summary and contributions. Authors propose 3D "spherical neurons" leading to rotationally equivariant layers. They do so by building on the spherical and geometric neurons introduced in Melnyk et al. (2021), which leverages the conformal space (for $\mathbb{R}^n$) to perform operations. The authors then solve for the steerability constraint for this neuron and empirically show that the proposed approach overperforms Melnyk et al. (2021) on rotated 3D data.

summaries: The paper aims to derive a steerability constraint for spherical neurons (3D point classifiers with spherical decision boundary). The steerability constraint enables test-time optimization of a pre-trained classifier to make predictions equivariant to 3D rotation perturbations applied to the input. When input rotation perturbations are unknown, the authors propose a method to recover the unknown rotations and therefore make rotation-invariant predictions. The experiments on a few small scale datasets verifies some of the claims.

abstractText: Emerging from low-level vision theory, steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations. In our work, we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds. Focusing on 3D geometry, we derive a 3D steerability constraint for hypersphere neurons, which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets. Exploiting the rotational equivariance, we show how our model parameters are fully steerable at inference time. We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and, after online optimization, invariant class predictions for known point sets in unknown orientations.

authors:
Brandon Anderson
Truong Son Hy
Risi Kondor

title: Cormorant: Covariant molecular neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Vladimir Banarer
Christian Perwass
Gerald Sommer

title: Design of a multilayered feed-forward neural network using hypersphere neurons

venue: In International Conference on Computer Analysis of Images and Patterns,

title: The hypersphere neuron

venue: In ESANN, pp

authors:
William T Freeman
Edward H Adelson

title: The design and use of steerable filters

venue: IEEE Transactions on Pattern analysis and machine intelligence,

authors:
Fabian Fuchs
Daniel Worrall
Volker Fischer
Max Welling

title: Se(3)-transformers: 3d rototranslation equivariant attention networks

venue: Advances in Neural Information Processing Systems,

title: Geometric operations implemented by conformal geometric algebra neural nodes

venue: Proc. SICE Symposium on Systems and Information 2008,

authors:
Roger Alan Horn
Charles Royal Johnson

title: Topics in Matrix Analysis

authors:
Bowen Jing
Stephan Eismann
Patricia Suriana
Raphael JL Townshend
Ron Dror

title: Learning from protein structure with geometric vector perceptrons

venue: arXiv preprint arXiv:2009.01411,

title: Adam: A method for stochastic optimization

venue: In Proceedings of the International Conference on Learning Representations (ICLR),

authors:
Hans Knutsson
Leif Haglund
Håkan Bårman
Gösta H. Granlund

title: A framework for anisotropic adaptive filtering and analysis of image sequences and volumes

venue: In [Proceedings]

authors:
Risi Kondor
Zhen Lin
Shubhendu Trivedi

title: Clebsch–Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network

authors:
Hongbo Li
David Hestenes
Alyn Rockwood

title: Generalized homogeneous coordinates for computational geometry

venue: In Geometric Computing with Clifford Algebras,

title: A universal model for conformal geometries of Euclidean, spherical and double-hyperbolic spaces. In Geometric computing with Clifford algebras, pp. 77–104

authors:
Diego Marcos
Michele Volpi
Nikos Komodakis
Devis Tuia

title: Rotation Equivariant Vector Field Networks

venue: In Proceedings of the IEEE International Conference on Computer Vision (ICCV),

authors:
Pavlo Melnyk
Michael Felsberg
Mårten Wadenbäck

title: Embed Me if You Can: A Geometric Perceptron

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An imperative style, high-performance deep learning library

title: Deformable kernels for early vision

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Christian Perwass
Vladimir Banarer
Gerald Sommer

title: Spherical decision surfaces using conformal modelling

venue: In Joint Pattern Recognition Symposium,

title: Group integration techniques in pattern analysis - a kernel view

venue: PhD thesis, University of Freiburg, Freiburg im Breisgau,

authors:
Eero P. Simoncelli
William T. Freeman

title: The steerable pyramid: a flexible architecture for multiscale derivative computation

venue: In Proceedings., International Conference on Image Processing,

authors:
Eero P. Simoncelli
William T. Freeman
Edward H. Adelson
David J. Heeger

title: Shiftable multiscale transforms

venue: IEEE Transactions on Information Theory,

authors:
Patrick C. Teo
Yacov Hel-Or

title: Lie generators for computing steerable functions

venue: Pattern Recognition Letters,

authors:
Nathaniel Thomas
Tess Smidt
Steven Kearnes
Lusann Yang
Li Li
Kai Kohlhoff
Patrick Riley

title: Tensor field networks: Rotation-and translation-equivariant neural networks for 3D point clouds

venue: arXiv preprint arXiv:1802.08219,

authors:
Luc Van Gool
Theo Moons
Eric Pauwels
André Oosterlinck

title: Vision and Lie’s approach to invariance

venue: Image and Vision Computing,

authors:
Maurice Weiler
Gabriele Cesa

title: General E(2)-Equivariant Steerable CNNs

venue: arXiv preprint arXiv:1911.08251,

authors:
Maurice Weiler
Mario Geiger
Max Welling
Wouter Boomsma
Taco S Cohen

title: 3D steerable CNNs: Learning rotationally equivariant features in volumetric data

authors:
Maurice Weiler
Fred A Hamprecht
Martin Storath

title: Learning steerable filters for rotation equivariant CNNs

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Lu Xia
Chia-Chih Chen
Jagdishkumar K. Aggarwal

title: View invariant human action recognition using histograms of 3d joints

venue: In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Yongheng Zhao
Tolga Birdal
Jan Eric Lenssen
Emanuele Menegatti
Leonidas Guibas
Federico Tombari

title: Quaternion equivariant capsule networks for 3d point clouds

venue: In European Conference on Computer Vision,

heading: 1 INTRODUCTION

text:
We present a novel feed-forward model consisting of steerable 3D neurons for point cloud classification, an important and challenging problem with many applications such as autonomous driving, human-robot interaction, and mixed-reality installations. We achieve the steerability by using geometric neurons (Melnyk et al., 2021), leading to a rotation equivariant network. Besides becoming a geometrically explainable approach, further benefits of the proposed method include rotation invariant classification and, therefore, lowered data augmentation requirements during learning.
We make use of a conformal embedding to obtain higher-order decision surfaces. Following the motivation in the recent work of Melnyk et al. (2021), we focus on 3D geometry and spherical decision surfaces, arguing for their natural suitability for problems in Euclidean space. We show how a spherical neuron (classifier), i.e., the hypersphere neuron (Banarer et al., 2003b) or its generalization for 3D input point sets — the geometric neuron (Melnyk et al., 2021) — can be turned into a steerable neuron. We prove that the aforementioned spherical neurons in any dimension require only up to first-degree spherical harmonics to accommodate the effect of rotation. This allows us to derive a 3D steerability constraint for the spherical neurons and to describe a recipe for creating a steerable model from a pretrained spherical classifier.
Using the synthetic Tetris dataset (Thomas et al., 2018) and the skeleton data from the UTKinectAction3D dataset (Xia et al., 2012), we first verify the derived constraint and check its stability with respect to perturbations in the input. We further conduct an experiment in a realistic setting, where we initialize the steerable model parameters using a noisy or imperfect prediction of the transformation applied to the input and optimize the parameters in an unsupervised way.
The contributions of our work are as follows: (a) We prove that the activation of spherical neurons on rotated input only varies by up to first-degree spherical harmonics (Section 4.1.1). (b) Based on a minimal set of four spherical neurons that are rotated to the corresponding vertices of a regular tetrahedron (Section 4.1.2), we derive the main result of our paper (Section 4.1.3) – a 3D steerability constraint for spherical neurons (13). (c) We propose a method to turn a trained spherical classifier into a fully steerable model (Section 5.2) that produces predictions invariant to 3D rotations, even in the presence of noise (Section 5.3), which we verify experimentally using both synthetic and real 3D data.

heading: 2 RELATED WORK

heading: 2.1 STEERABILITY AND EQUIVARIANCE

text:
Steerability is a powerful concept from early vision and image processing (Freeman et al., 1991; Knutsson et al., 1992; Simoncelli et al., 1992; Perona, 1995; Simoncelli & Freeman, 1995; Teo & Hel-Or, 1998) that resonates in the era of deep learning. The utility of steerable filters and the main theorems for their construction are presented in the seminal work of Freeman et al. (1991). Geometric equivariance in the context of computer vision has been an object of extensive research over the last decades (Van Gool et al., 1995). Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator. For this reason, Lie theory can be used to to study steerable filters for equivariance and invariance (Reisert, 2008).
Nowadays, equivariance-related research extends into deep learning, e.g., the SE(3)-equivariant models of Fuchs et al. (2020), Thomas et al. (2018), and Zhao et al. (2020), and the SO(3)-equivariant network of Anderson et al. (2019), as well as Marcos et al. (2017) and Kondor et al. (2018). Also steerable filter concepts are increasingly used in current works, e.g., Cohen & Welling (2016) considered image data and proposed a CNN architecture to produce equivariant representations with steerable features, which involves fewer parameters than traditional CNNs. They considered the dihedral group (discrete rotation, translation, and reflection), and the steerable representations in their work are proposed as formation of elementary feature types. One limitation of their approach is that rotations are restricted to four orientations, i.e., by π/2. More recently, Weiler et al. (2018b) utilized group convolutions and introduced steerable filter convolutional neural networks (SFCNNs) operating on images to jointly attain equivariance under translations and discrete rotations. In their work, the filter banks are learned rather than fixed. Further, the work of Weiler & Cesa (2019) proposed a unified framework for E(2)-equivariant steerable CNNs and presented their general theory.
The steerable CNNs for 3D data proposed by Weiler et al. (2018a) are closely related to our work. The authors employed a combination of scalar, vector and tensor fields as features transformed by SO(3) representations and presented a model that is equivariant to SE(3) transformations. They also considered different types of nonlinearities suitable for nonscalar components of the feature space. The novel SE(3)-equivariant approach by Fuchs et al. (2020) introduced a self-attention mechanism that is invariant to global rotations and translations of its input and solves the limitation of angularly constrained filters in other equivariant methods, e.g., Thomas et al. (2018). Noteworthy, the work of Jing et al. (2020) proposed the geometric vector perceptron (GVP) consisting of two linear transformations for the input scalar and vector features, followed by nonlinearities. The GVP scalar and vector outputs are invariant and equivariant, respectively, with respect to an arbitrary composition of rotations and reflections in the 3D Euclidean space.
The key point distinguishing our approach from other equivariant networks is that we use conformal modeling (Li et al., 2001a; Hitzer, 2008), with which the scalar product of two points yields their Euclidean distance (allowing us to construct spherical decision surfaces) and nonlinear conformal transformations on Rn can be linearized (allowing us to steer the decision spheres). Besides, we do not constrain the space of learnable parameters (as opposed to, e.g., Thomas et al. (2018), Weiler et al. (2018a), and Fuchs et al. (2020)), but construct our steerable model from a freely trained base network, as we discuss in detail in Section 4.

heading: 2.2 CONFORMAL MODELING AND THE HYPERSPHERE NEURON

text:
The utility of conformal embedding for Euclidean geometry and the close connection to Minkowski spaces are thoroughly discussed by Li et al. (2001a). An important result is that one can construct hyperspherical decision surfaces using representations in the conformal space (Li et al., 2001b), as done in the work of Perwass et al. (2003). The hypersphere neuron proposed by Banarer et al. (2003b) is such a spherical classifier. Remarkably, since a hypersphere can be seen as a generalization of a hyperplane, the standard neuron can be considered as a special case of the hypersphere neuron. Stacking multiple hypersphere neurons in a feed-forward network results in a multilayer hypersphere perceptron (MLHP), which was shown by Banarer et al. (2003a) to outperform the standard MLP for some classification tasks. However, its application to point sets was not discussed. This motivated the work of Melnyk et al. (2021) on the geometric neuron, where the learned parameters were shown to represent a combination of spherical decision surfaces. Moreover, the geometric neuron activations
were proved to be isometric in the 3D Euclidean space. In our work, we use the latter observation as the necessary condition for deriving the steerability constraint. Making spherical classifiers steerable adds to the practical value of the prior work by Melnyk et al. (2021).

heading: 3 BACKGROUND

heading: 3.1 STEERABILITY

text:
As per Freeman et al. (1991), a 2D function f(x, y) is said to steer if it can be written as a linear combination of rotated versions of itself, i.e., when it satisfies the constraint
fθ(x, y) =
M∑
j=1
vj(θ)f θj (x, y) , (1)
where vj(θ) are the interpolation functions, θj are the basis function orientations, and M is the number of basis function terms required to steer the function. An alternative formulation can be found in the work of Knutsson et al. (1992). In 3D, the steering equation becomes
fR(x, y, z) =
M∑
j=1
vj(R)fRj (x, y, z) , (2)
where fR(x, y, z) is f(x, y, z) rotated by R ∈ SO(3), and each Rj ∈ SO(3) orients the corresponding jth basis function.
Theorems 1, 2 and 4 in Freeman et al. (1991) describe the conditions under which the steerability constraints (1) and (2) hold, and how to determine the minimum number of basis functions for the 2D and 3D case, respectively.

heading: 3.2 CONFORMAL EMBEDDING

text:
We refer the reader to Section 3 in the work of Melnyk et al. (2021) for more details, and only briefly introduce important notation in this section. The conformal space for the Euclidean Rn counterpart can be formed as MEn ≡ Rn+1,1 = Rn ⊕ R1,1, where R1,1 is the Minkowski plane (Li et al., 2001a) with orthonormal basis defined as {e+, e−} and null basis {e0, e∞} representing the origin e0 = 1 2 (e− − e+) and point at infinity e∞ = e− + e+. Thus, a Euclidean vector x ∈ Rn can be embedded in the conformal space MEn as
X = C(x) = x + 1 2 ∥x∥2 e∞ + e0 , (3)
where X ∈ MEn is called normalized. The conformal embedding (3) represents the stereographic projection of x onto a projection sphere in MEn and is homogeneous, i.e., all embedding vectors in the equivalence class [X] = { Z ∈ Rn+1,1 : Z = γX, γ ∈ R \ {0} } are taken to represent the same vector x. Importantly, given the conformal embedding X and some Y = y + 12∥y∥2 e∞ + e0, their scalar product in the conformal space corresponds to their Euclidean distance, X · Y = − 12∥x − y∥2. This interpretation of the scalar product in the conformal space is the main motivation in constructing spherical classifiers.

heading: 3.3 SPHERICAL CLASSIFIERS

text:
By spherical classifiers, we collectively refer to hypersphere (Banarer et al., 2003a) and geometric (Melnyk et al., 2021) neurons, which have spherical decision surfaces.
As discussed by Banarer et al. (2003a), by embedding both a data vector x ∈ Rn and a hypersphere S ∈ MEn in Rn+2 as
X = ( x1, . . . , xn,−1,− 1
2 ∥x∥2
) ∈ Rn+2, S = ( c1, . . . , cn, 1
2 (∥c∥2 − r2), 1
) ∈ Rn+2 , (4)
where c = (c1, . . . , cn) ∈ Rn is the hypersphere center and r ∈ R is the radius, their scalar product X · S in the conformal space MEn can be computed equivalently in Rn+2 as X⊤S:
X · S = X⊤S = −1 2 ∥x − c∥2 + 1 2 r2 . (5)
This result enables the implementation of a hypersphere neuron in MEn using the standard dot product in Rn+2. The hypersphere vector components are treated as independent learnable parameters during training. Thus, a spherical classifier effectively learns non-normalized hyperspheres of the form S̃ = (s1, . . . , sn+2) ∈ Rn+2. Due to the homogeneity of the representation, both normalized and non-normalized hyperspheres represent the same decision surface. More details can be found in Section 3.2 in the work of Melnyk et al. (2021).
∑
z
S̃1
S̃2 S̃k
x1 −1 `1 x2 −1 `2 xk −1 `k
Furthermore, Melnyk et al. (2021) demonstrated that the geometric neuron activations are isometric in 3D. That is, rotating the input is equivalent to rotating the decision spheres. This result is a necessary condition to consider rotation and translation equivariance of models constructed with geometric neurons and forms the basis for our methodology.
In the following sections, we use the same notation for a 3D rotation R represented in the Euclidean space R3, the homogeneous (projective) space P(R3), and ME3 ∼= R5, depending on the context. This is possible since we can add the required number of ones to the diagonal of the original rotation matrix without changing the transformation representation.

heading: 4 METHOD

text: To build a 3D steerable model, we perform the following steps: We first train an MLGP model (Melnyk et al., 2021), which consists of spherical neurons. After optimizing the model parameters, we freeze them and transform according to the 3D steerability constraint we derive. Finally, by combining the resulting parameters in filter banks and adding interpolation coefficients as free parameters, we create a steerable model.

heading: 4.1 FULLY STEERABLE 3D SPHERICAL NEURONS

text: In this section, we identify the conditions under which a geometric neuron as a function of its 3D input can be steered. In other words, we derive an expression that gives us the response of a hypothetical geometric neuron for some input, using rotated versions of the learned geometric neuron parameters. We start by considering the steerability conditions for a single sphere classifying the corresponding input point X, i.e., f(X) = X⊤S, where X and S are embedded in ME3 ∼= R5 according to (4). Since the geometric neuron output is a linear combination of these functions, as per (6), we will use the identified conditions to build a steerable feed-forward network by reassembling modified spheres.

heading: 4.1.1 BASIS CONSTRUCTION

text:
To formulate a steerability constraint for a spherical neuron (sphere), first, we need to determine the minimum number of basis functions, i.e., the number of terms M in (2). This number only depends on the degree of the spherical harmonics that are required to compute the steered result (Freeman et al., 1991). Thus, we need to determine the required degrees. Theorem. Let S ∈ Rn+2 be an nD classifier with center c ∈ Rn (R := ∥c∥) and radius r, and x ∈ Rn be a point represented by X ∈ Rn+2, see (4). Let further S′ be the classifier that is obtained by rotating S in nD space, then X⊤S′ and X⊤S are related by spherical harmonics up to first degree.
Proof. Without loss of generality, the rotation is defined by the plane of rotation π by the angle θ. Denote the projection of a vector v ∈ Rn onto π by vπ and define v⊥π = v− vπ . From (5) we obtain
2X⊤S = r2 −∥x − c∥2 = r2 − ∥∥(x − c)⊥π ∥∥2 − ∥∥(x − c)π ∥∥2 . A rotation in π only affects the rightmost term above and there exists a ϕ ∈ [0, 2π) such that
∥∥(x − c)π ∥∥2 =∥xπ − cπ∥2 =∥xπ∥2 +∥cπ∥2 − 2∥xπ∥ ∥cπ∥ cosϕ .
With a similar argument, we obtain
2X⊤S′ = r2 − ∥∥(x − c)⊥π ∥∥2 −∥xπ∥2 −∥cπ∥2 + 2∥xπ∥ ∥cπ∥ cos(ϕ+ θ) .
This result is valid in any dimension, but we are primarily interested in n = 2 and n = 3, as illustrated in Figure 2. Following the result of Theorem 4 in Freeman et al. (1991) and using N = 1, we have that M = (N + 1)2 = 4 basis functions suffice in the 3D case (2).

heading: 4.1.2 SPHERICAL FILTER BANKS IN 3D

text:
In 3D, we thus select four rotated versions of the function f(X), as the basis functions. The rotations {Rj}4j=1 must be chosen to satisfy the condition (b) in Theorem 4 (Freeman et al., 1991). Therefore, we transform f(X) such that the resulting four spheres are spaced in three dimensions equally, i.e., form a regular tetrahedron with the vertices (1, 1, 1), (1,−1,−1), (−1, 1,−1), and (−1,−1, 1), as shown in Figure 3 b). We stack the homogeneous coordinates of the tetrahedron vertices mj in a matrix column-wise (scaled by 1/2) to get the orthogonal matrix
M = [ m1 m2 m3 m4 ] = 1
2

1 1 −1 −1 1 −1 1 −1 1 −1 −1 1 1 1 1 1
  . (7)
We will use this matrix operator M to compute the linear coefficients in the vector space generated by the vertices of the regular tetrahedron (Granlund & Knutsson, 1995). This will be necessary to find the appropriate interpolation functions and formulate the steerability constraint in Section 4.1.3.
The four rotated versions of the function f(X) will constitute the basis functions that we call a spherical filter bank. To construct this filter bank, we choose the following convention. The originally learned spherical classifier f(X) = X⊤S is first rotated to (1, 1, 1) (see Figure 3 b)) with the corresponding transformation denoted as RO. Next, we rotate the transformed sphere into the other three vertices of the regular tetrahedron and transform back to the original coordinate system (see Figure 3 a) for the case of (1,−1,−1)). The resulting filter bank for one spherical classifier is thus composed as the following 20× 1 matrix:
B(S) = [ R⊤O RTi RO S ] , (8)
where each of {RTi}3i=0 is the rotation isomorphism in R5 corresponding to a 3D rotation from (1, 1, 1) to the vertex i+ 1 of the regular tetrahedron. Therefore, RT0 = I5.

heading: 4.1.3 3D STEERABILITY CONSTRAINT

text:
The steerability constraint can be formulated as follows. For an arbitrary orientation R applied to the input of the function f(X), we want the output of the spherical filter bank B(S) in (8) to be interpolated with vj(R) such that the response is equal to the original function output, i.e.,
f(X) = fR(RX) = M∑
j=1
vj(R) fRj (RX) = (v(R)⊗ RX)⊤B(S), (9)
where X ∈ R5 is a single, appropriately embedded, 3D point, v(R) ∈ R4 is a vector of the interpolation coefficients, and ⊗ denotes the Kronecker product (Horn & Johnson, 1991). The coefficients v(R) should conform to the basis function construction (condition (b) in Theorem 4 in Freeman et al. (1991)), which is why they are computed with M defined in (7). Given X ∈ R5 as input and an unknown rotation R acting on it, the steering equation (9) implies
(v(R)⊗ RX)⊤B(S) = (v(I)⊗ X)⊤B(S) . (10) Given a tetrahedron rotation, e.g., RT1, as shown in the diagram in Figure 3 a), we can define the unknown rotation accordingly as R = R⊤ORT1RO. In this case, it is easy to see that to satisfy the constraint (10), v(R) must be (0, 1, 0, 0), i.e., the second filter in the filter bank B(S) must be chosen. This can be achieved by transforming a constant vector m1 = ( 12 , 1 2 , 1 2 , 1 2 ) by rotation RT1 and multiplying it by the basis matrix M as follows:
v(R) = M⊤(RT1 m1) = M⊤ (1 2 ,−1 2 ,−1 2 , 1 2 ) = (0, 1, 0, 0) . (11)
Note that, in general, a geometric neuron (6) takes a set of embedded points as input. Therefore, with the setup above, R will be different for each input shape point k if the same v is used for all k, which contradicts that the shape is transformed by a rigid body motion, i.e., the same RB for all k. Thus, we need to consider a suitable vector vk for each input shape point k, such that the resulting RB is the same for all k. This can be achieved recalling how we construct the basis functions in the spherical filter bank (8): we need to consider the respective initial rotation RkO. The desired interpolation coefficients vk are thus computed as
vk(RB) = M⊤ (RkORBR k⊤ O m1) . (12)
The resulting vk(RB) interpolate the responses of the tetrahedron-copies of the originally learned sphere Sk to replace the rotated sphere.
By plugging (9) into (6), we can now establish the main result of our paper – the steerability constraint for a geometric neuron that takes a set of N embedded points {Xk}k as input:
fR(RX) = N∑
k=1
γk f k R(RXk) =
N∑
k=1
γk (v k(R)⊗ RXk)⊤B(Sk) . (13)

heading: 5 EXPERIMENTS

text:
We use a single GTX 1050 Ti GPU and conduct two types of experiments to confirm our findings presented in Section 4.
original
derotated

heading: 5.1 DATASETS

text:
3D Tetris Following the experiments reported by Thomas et al. (2018), Weiler et al. (2018a), and Melnyk et al. (2021), we use the synthetic point set of eight 3D Tetris shapes consisting of four points each (see, e.g., Figure 3 in Melnyk et al. (2021) and the numerical description in Appendix A).
3D skeleton data We also perform experiments on real-world data to substantiate the validity of our theoretical results. We use the UTKinect-Action3D dataset introduced by Xia et al. (2012), in particular, the 3D skeletal joint locations extracted from Kinect depth maps. For each action sequence and from each frame, we extract the skeleton consisting of twenty points and assign it to the class of actions (ten categories) this frame is part of. Therefore, we formulate the task as shape recognition (i.e., a kind of action recognition from a static point cloud), where each shape is of size 20 × 3. Since the orientations of the shapes vary significantly across the sequences, we perform the following standardization: We first center each shape at the origin, and then compute the orientation from its three hip joints and derotate the shape in the xy-plane (viewer coordinate system) accordingly. We illustrate the effect of derotation in Figure 4. From each action sequence, we randomly select 50% of the skeletons for the test set and 20% of the remainder as validation data. The resulting data split is as follows: 2295 training shapes, 670 shapes for validation, and 3062 test shapes.

heading: 5.2 STEERABLE MODEL CONSTRUCTION

text:
To construct and test steerable models, we perform the same steps for both datasets. The minor differences are the choice of training hyperparameters and the presence of validation and test subsets in the 3D skeleton dataset.
We first train a two-layer MLGP model (Melnyk et al., 2021), where the hidden layer consists of geometric neurons and the output layer of hypersphere neurons, to classify the shapes. Since the architecture choice is not the objective of the experiments, when building the MLGP, we use only one configuration with five hidden units for the Tetris data and twelve hidden units (determined by using the validation data) for the 3D skeleton dataset throughout the experiments. Similar to Melnyk et al. (2021), we do not use any activation function in the hidden layer due to the nonlinearity of the conformal embedding. We implement both MLGP models in PyTorch (Paszke et al., 2019) and keep the default parameter initialization for the linear layers. We train both models by minimizing the cross-entropy loss function and use the Adam optimizer (Kingma & Ba, 2015) with the default hyperparameters (the learning rate is 0.001). The Tetris MLGP learns to classify the eight shapes in the canonical orientation perfectly after 2000 epochs, whereas the Skeleton model trained for 10000 epochs achieves a test set accuracy of 92.9%. We refer to both trained models as ancestor MLGP.
For both, we then freeze the trained parameters and construct a steerable model. Note that we form steerable units only in the hidden layer and keep the output, i.e., classification, layer hypersphere neurons as they are. The steerability is not required for the subsequent layers as the output of the first layer becomes rotation-independent. The steerable units are formed from the corresponding frozen parameters as the (fixed) filter banks according to (8).
The only free parameters of this constructed steerable model are interpolation coefficients vk(RB) ∈ R4 defined in (12), where k indexes the learned hidden layer parameters (spheres) in the ancestor MLGP model. But since the only variables in (12) are the parameters of the unknown rotation RB acting on the input, we can directly optimize them if they are unknown. We select the axis–angle rotation representation because it has only three parameters — the three coordinates of the rotation axis scaled by the rotation angle.
We propose the following basic recipe to create a steerable model: Train the ancestor MLGP → Fix the learned parameters → Transform the hidden unit parameters into filter banks (8) → Add the interpolation coefficients vk as free parameters to fulfill (13) → Steerable spherical classifier.

heading: 5.3 KNOWN ROTATION EXPERIMENT

text:
Using the trained MLGP, we verify the correctness of (13). We first rotate the original data and then use this ground truth rotation to initialize the interpolation coefficients of the constructed steerable model according to (12). Our intuition is that if the steerability constraint (13) is correct, then, given the transformed point set, the activations of the steerable units in the steerable model will be equal to the activations of the geometric neurons in the ancestor MLGP model fed with the point set in the canonical orientation. Hence, the classification accuracies of the ancestor and steerable models on the original and transformed datasets, respectively, should be equal.
We run this experiment 1000 times. Each time, we generate a random rotation and apply it to the original point set (in case of the 3D skeleton data, we use the test split). We use these rotation parameters to create a steerable model and evaluate it on the transformed point set. To verify the stability of the steerable unit activations, we add uniform noise to the transformed points, n ∼ U(−a, a), where the range of a is motivated by the magnitude of the points in the datasets. For reference, we also present the accuracy of the ancestor model classifying the rotated data. We summarize the results in Table 1. The respective L1 distance results are presented in Appendix B. The analogous Tetris experiment results reported in related work are 100% (over 25 runs and including random translation) (Thomas et al., 2018) and 99±2% (over 17 runs) (Weiler et al., 2018a). However, due to the significant conceptual difference between our approach and these methods, as mentioned at the end of Section 2.1, the results are not directly comparable.

heading: 5.4 ESTIMATED ROTATION EXPERIMENT

text:
In this experiment, we feed some rotation information into the system. For example, this information can come from another network branch that performs regression on the rotation, or from a dynamical model as in tracking. In either case, we cannot assume that the rotation is entirely accurate, unlike the experiment in Section 5.3. Therefore, we check the stability with respect to the noise in the rotation. Same as before, we exploit the MLGP trained on the respective point set in the canonical orientation, as described in Section 5.2, to build the steerable model. At each experiment run, we randomly select one shape (from the test subset in case of the 3D skeleton data) and apply a random rotation, RGT, to it. We then perturb this ground truth rotation by varying the rotation angle. We achieve this by sampling the rotation angle from a Gaussian distribution with σ = π/18 = 10° and various means, constructing a rotation of this sampled angle about a random axis, Rnoise, and multiplying with the ground truth: Rinit = Rnoise RGT. We use the resulting rotation Rinit in the axis-angle representation to initialize the steerable model parameters.
Further, we perform the optimization of the rotation vector parameters, which we call online optimization. Since inference time, we do not have access to the labels. Thus, we need to select a loss based on the consistency of the output. Here, we choose the entropy, i.e., L(p) = −∑Ki=1 pi log(pi), where p ∈ RK is the softmax output of the model. Our intuition is that by minimizing the entropy in the output with respect to the steerable model parameters, we effectively force the model to produce a more confident prediction. For this optimization procedure, we use Adam with learning rate equal to 0.01, and empirically determine the number of epochs of 300 to be sufficient for the Tetris data and 100 for the 3D skeleton data.
Note that at each run, we provide the model with only one transformed shape, which implies that the accuracy per run is binary, i.e., the shape is either recognized ("1") or misclassified ("0"). To verify the usefulness of the proposed online optimization method, we compare the model classification accuracy before and after optimization. The results for this experiment are summarized in Table 2. The respective L1 distance results are presented in Appendix B.

heading: 6 DISCUSSION AND CONCLUSION

text:
Enabled by the complete understanding of the geometry of the spherical classifiers (Banarer et al., 2003b; Perwass et al., 2003; Melnyk et al., 2021), we show in Section 4 that we only need the spherical harmonics of degree up to N = 1 to determine the effect of rotation on the activations in 3D. Using this result, we derive a novel 3D steerability constraint (13). The conducted experiment in Section 5.3 shows that the derived constraint is correct since the constructed steerable model produces accurate predictions for the rotated shapes, provided that the rotation is known. From Table 1, we can see that that rotating the data without steering the neurons leads to substantial degradation of accuracy (from 99.2% to 25.2%), whereas using the steerability equation maintains the high accuracy. Moreover, the steerable model classification error only moderately increases with the level of noise in the input data, which is a clear indication of the robustness of the classifier.
The experiment presented in Section 5.4 considers a more realistic setting, where the rotation applied to the input is not known exactly but is inaccurately estimated. However, unlike the first experiment, we perform an optimization to improve the initial steerable model prediction. The only three parameters of the model — the axis–angle representation of the unknown external rotation — determining the interpolation coefficients vk, are optimized by minimizing the entropy in the model output. As displayed in Table 2, our steerable model, prior to the optimization, produces rotation-invariant predictions for the transformed shapes for both synthetic and real data, which points to the robustness of our method. The proposed online optimization further improves the model classification accuracy. The lower accuracies for the zero-mean noise (the first row in Table 2) for the skeleton data compared to the π/36 case (the second row) are presumably caused by a small error in the shape orientations after the performed standardization.
The power of our approach lies in the geometric explainability inherent to the spherical units that constitute our model. Seemingly complicated conformal algebra operations have a surprisingly straightforward interpretation in Euclidean space and allow us to build a steerable feed-forward neural network with a purely geometric motivation. The steerable spherical neurons discussed in our work can be employed in other architectures, e.g., CNNs, and therefore, the equivariance to transformations beyond rotation can be considered.
ETHICS STATEMENT
As the nature of the present paper is mainly theoretical, no direct ethical issues arise, nor are there any immediate societal consequences.
REPRODUCIBILITY STATEMENT
We provide the implementation of the proposed method, as well as notebooks with the conducted experiments, as part of the supplementary material.

heading: L: [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 0)],

heading: T: [(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 1, 1)],

text: zigzag: [(0, 0, 0), (1, 0, 0), (1, 1, 0), (2, 1, 0)].

heading: B KNOWN AND ESTIMATED ROTATION EXPERIMENTS: L1 DISTANCES

text:
We compare the hidden unit activations of the ancestor MLGP and constructed steerable models by computing the L1 distance (see Table 3). For convenience, we call the hidden unit activations of the ancestor MLGP fed with the shapes in the canonical orientation ground truth activations. From Table 3, we can see that the L1 distance to the ground truth activations linearly and moderately increases with the level of noise in the input data, which is a clear indication of the robustness of the classifier and corresponds well to the model accuracy results in Table 1. Without steerability (“Ancestor” columns), the respective L1 distances increase by one order of magnitude and basically independently of the noise level, demonstrating the significant improvement by the steerability.
The results presented in Table 4 indicate that the proposed online optimization, besides improving the classification accuracy (see Table 2), results in the decrease of the L1 distance to the ancestor MLGP softmax output values (referred to as ground truth output) for the Tetris data. Note the low absolute level of the error already for the initial result. In case of the 3D skeleton data, the absolute error level is also very low, but we observe a slight increase of the L1 distances after the optimization despite the improved accuracy (see Table 2). We presume it is caused by the increased number of hidden units in the ancestor model compared to the Tetris case (12 vs. 5), which might not all be equally relevant to the classification output.

title: FULLY STEERABLE 3D SPHERICAL NEURONS