RotRNN: Modelling Long Sequences with Rotations
<|reference_start|>RotRNN: Modelling Long Sequences with Rotations: Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets.<|reference_end|>
@article{biegun2024rotrnn:,
    title={RotRNN: Modelling Long Sequences with Rotations},
    author={Kai Biegun, Rares Dolga, Jake Cunningham, David Barber},
    journal={arXiv preprint arXiv:2407.07239},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07239},
    primaryClass={cs.LG stat.ML}
}