Steerable Transformers
<|reference_start|>Steerable Transformers: In this work we introduce Steerable Transformers, an extension of the Vision Transformer mechanism that maintains equivariance to the special Euclidean group $\mathrm{SE}(d)$. We propose an equivariant attention mechanism that operates on features extracted by steerable convolutions. Operating in Fourier space, our network utilizes Fourier space non-linearities. Our experiments in both two and three dimensions show that adding a steerable transformer encoder layer to a steerable convolution network enhances performance.<|reference_end|>
@article{kundu2024steerable,
    title={Steerable Transformers},
    author={Soumyabrata Kundu and Risi Kondor},
    journal={arXiv preprint arXiv:2405.15932},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15932},
    primaryClass={cs.CV}
}