The RWKV language model: An RNN with the advantages of a transformer
> though in practice, the model might have a hard time generalizing to much longer context lengths than it saw during trainingWhat's the benefit of RWKV then? The results are practically identical to transformers.