TPTT: Transforming Pretrained Transformer into Titans
Recent advances in large language models (LLMs) have led to remarkable
progress in natural language processing, but their computational and memory
demands remain a significant challenge, particularly for long-context
inference. We introduce TPTT (Transforming Pretrained Transformer into Titans),
a novel framework for enhancing pretrained Transformer models with efficient
linearized attention mechanisms and advanced memory management. TPTT employs
techniques such as Memory as Gate (MaG) and mixed linearized attention (LiZA).
It is fully compatible with the Hugging Face Transformers library, enabling
seamless adaptation of any causal LLM through parameter-efficient fine-tuning
(LoRA) without full retraining. We show the effectiveness of TPTT on the MMLU
benchmark with models of approximately 1 billion parameters, observing
substantial improvements in both efficiency and accuracy. For instance,
Titans-Llama-3.2-1B achieves a 20% increase in Exact Match (EM) over its
baseline. Statistical analyses and comparisons with recent state-of-the-art
methods confirm the practical scalability and robustness of TPTT. Code is
available at https://github.com/fabienfrfr/tptt . Python package at
https://pypi.org/project/tptt/ .
685ab1900e4ad7e21975878c
Fabien Furfaro
claimed_verified
2025-06-25T12:56:29.914Z
662ac66c0ee2fbf668ef9fe7
/avatars/80d097210627c0322e1f33122c7a1174.svg
Fabien Furfaro
user
ffurfaro
TPTT: Transforming Pretrained Transformer into Titans
Recent advances in large language models (LLMs) have led to remarkable
progress in natural language processing, but their computational and memory
demands remain a significant challenge, particularly for long-context
inference. We introduce TPTT (Transforming Pretrained Transformer into Titans),
a novel framework for enhancing pretrained Transformer models with efficient
linearized attention mechanisms and advanced memory management. TPTT employs
techniques such as Memory as Gate (MaG) and mixed linearized attention (LiZA).
It is fully compatible with the Hugging Face Transformers library, enabling
seamless adaptation of any causal LLM through parameter-efficient fine-tuning
(LoRA) without full retraining. We show the effectiveness of TPTT on the MMLU
benchmark with models of approximately 1 billion parameters, observing
substantial improvements in both efficiency and accuracy. For instance,
Titans-Llama-3.2-1B achieves a 20% increase in Exact Match (EM) over its
baseline. Statistical analyses and comparisons with recent state-of-the-art
methods confirm the practical scalability and robustness of TPTT. Code is
available at https://github.com/fabienfrfr/tptt . Python package at
https://pypi.org/project/tptt/ .
TPTT enhances large language models with efficient linearized attention and advanced memory management, improving both efficiency and accuracy for long-context inference.
Transformers large language models (LLMs) linearized attention mechanisms Memory as Gate (MaG) mixed linearized attention (LiZA) parameter-efficient fine-tuning (LoRA) causal LLM MMLU benchmark Exact Match (EM)