Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection
Yun Zhu Jia-Chen Gu Caitlin Sikora Ho Ko Yinxiao Liu Chu-Cheng Lin Lei Shu Liangchen Luo Lei Meng Bang Liu Jindong Chen
Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing a dramatic increase in latency. In this paper, we propose a novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Speciﬁcally, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. It is notable that Sparse RAG combines the assessment of each individual document and the generation of the response into a single process. The designed sparse mechanism in a RAG system can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, ﬁltering out undesirable contexts enhances the model’s focus on relevant context, inherently improving its generation quality. Evaluation results on four datasets show that Sparse RAG can be used to strike an optimal balance between generation quality and computational efﬁciency, demonstrating its generalizability across tasks.
Retrieval-Augmented Generation, Sparse Context, Efficiency
We propose a novel paradigm named Sparse RAG, which seeks to cut computation costs of RAG through sparsity.