Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
few-shot learning. RetNet replaces the KV cache of Transformers with recurrence and saves memory. Also, chunk-wise retention makes inference significantly scalable with increasing batch size and input length. They also show that RetNet is computationally way more efficient than Transformer and