Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
range from 4086 to 8196. My 2 cents: Hyena is an interesting approach for extending input length through scalable computing. Nonetheless, further investigations on a larger scale are necessary to confirm its efficacy as a viable alternative to the Transformer model. For now, the RWKV model offers