MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources
<|reference_start|>MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources: Leveraging external knowledge is crucial for achieving high performance in knowledge-intensive tasks, such as question answering. The retrieve-and-read approach is widely adopted for integrating external knowledge into a language model. However, this approach suffers from increased computational cost and latency due to the long context length, which grows proportionally with the number of retrieved knowledge. Furthermore, existing retrieval-augmented models typically retrieve information from a single type of knowledge source, limiting their scalability to diverse knowledge sources with varying structures. In this work, we introduce an efficient memory-augmented transformer called MATTER, designed to retrieve relevant knowledge from multiple heterogeneous knowledge sources. Specifically, our model retrieves and reads from both unstructured sources (paragraphs) and semi-structured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed. Furthermore, MATTER achieves competitive results compared to conventional read-and-retrieve models while having 100x throughput during inference.<|reference_end|>
@article{lee2024matter:,
    title={MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge
  Sources},
    author={Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, Jens Lehmann},
    journal={arXiv preprint arXiv:2406.04670},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.04670},
    primaryClass={cs.CL cs.AI}
}