MELODI: Exploring Memory Compression for Long Contexts
<|reference_start|>MELODI: Exploring Memory Compression for Long Contexts: We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8.<|reference_end|>
@article{chen2024melodi:,
    title={MELODI: Exploring Memory Compression for Long Contexts},
    author={Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David
  Racz, Jesper Andersen},
    journal={arXiv preprint arXiv:2410.03156},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03156},
    primaryClass={cs.LG cs.AI}
}