Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
computationally heavy, and it needs to store a KV cache that is also heavy in memory at inference. This is a limiting factor, especially in problems with extended context sizes. Transformers' space complexity increases quadratically with the increasing context size. The Transformer is a key