MemLong: Memory-Augmented Retrieval for Long Text Modeling
<|reference_start|>MemLong: Memory-Augmented Retrieval for Long Text Modeling: Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation. This work introduces MemLong: Memory-Augmented Retrieval for Long Text Generation, a method designed to enhance the capabilities of long-context language modeling by utilizing an external retriever for historical information retrieval. MemLong combines a non-differentiable ``ret-mem'' module with a partially trainable decoder-only language model and introduces a fine-grained, controllable retrieval attention mechanism that leverages semantic-level relevant chunks. Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs. More importantly, MemLong can extend the context length on a single 3090 GPU from 4k up to 80k. Our code is available at https://github.com/Bui1dMySea/MemLong<|reference_end|>
@article{liu2024memlong:,
    title={MemLong: Memory-Augmented Retrieval for Long Text Modeling},
    author={Weijie Liu, Zecheng Tang, Juntao Li, Kehai Chen, Min Zhang},
    journal={arXiv preprint arXiv:2408.16967},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16967},
    primaryClass={cs.CL cs.AI}
}