MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
<|reference_start|>MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models: Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. We introduce MemoryPrompt, a leaner approach in which the LM is complemented by a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with a sequence of vectors, akin to soft prompts, without requiring LM finetuning. Tested on a task designed to probe a LM's ability to keep track of multiple fact updates, a MemoryPrompt-augmented LM outperforms much larger LMs that have access to the full input history. We also test MemoryPrompt on a long-distance dialogue dataset, where its performance is comparable to that of a model conditioned on the entire conversation history. In both experiments we also observe that, unlike full-finetuning approaches, MemoryPrompt does not suffer from catastrophic forgetting when adapted to new tasks, thus not disrupting the generalist capabilities of the underlying LM.<|reference_end|>
@article{rakotonirina2024memoryprompt:,
    title={MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained
  Language Models},
    author={Nathana"el Carraz Rakotonirina, Marco Baroni},
    journal={arXiv preprint arXiv:2402.15268},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.15268},
    primaryClass={cs.CL cs.AI cs.LG}
}