Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
outperforms in few-shot learning, but RWKV is better in zero-shot accuracy on SuperGLUE tasks. Moreover, Hyena performs on par with a Transformer on language modeling on WikiText103. Regarding runtime, the cross-over point between Hyena and attention occurs at 2048, and Hyena and flash attention