Artificial Intelligence, Thesequence, Llm, Machine Learning.
figure, Infini Attention stores the entire context, while Transformer-XL only captures the latest states. The Results Infini-Transformer models were evaluated on benchmarks involving extremely long input sequences: long-context language modeling, 1M length passkey context block retrieval, and 500K