Inference Scaling for Long-Context Retrieval Augmented Generation
The scaling of inference computation has unlocked the potential of
long-context large language models (LLMs) across diverse settings. For
knowledge-intensive tasks, the increased compute is often allocated to
incorporate more external knowledge. However, without effectively utilizing
such knowledge, solely expanding context does not always enhance performance.
In this work, we investigate inference scaling for retrieval augmented
generation (RAG), exploring strategies beyond simply increasing the quantity of
knowledge. We focus on two inference scaling strategies: in-context learning
and iterative prompting. These strategies provide additional flexibility to
scale test-time computation (e.g., by increasing retrieved documents or
generation steps), thereby enhancing LLMs' ability to effectively acquire and
utilize contextual information. We address two key questions: (1) How does RAG
performance benefit from the scaling of inference computation when optimally
configured? (2) Can we predict the optimal test-time compute allocation for a
given budget by modeling the relationship between RAG performance and inference
parameters? Our observations reveal that increasing inference computation leads
to nearly linear gains in RAG performance when optimally allocated, a
relationship we describe as the inference scaling laws for RAG. Building on
this, we further develop the computation allocation model to estimate RAG
performance across different inference configurations. The model predicts
optimal inference parameters under various computation constraints, which align
closely with the experimental results. By applying these optimal
configurations, we demonstrate that scaling inference compute on long-context
LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.
670710e7b9c9f7ddac2f0081
Zhenrui Yue
extracted_confirmed
2024-10-09T23:26:40.279Z
644c4eff9a7c19cfaf0c8f4a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2gHtfhPFjSsnwxVtBnxe2.jpeg
Zhenrui Yue
user
yuezrhb
670710e7b9c9f7ddac2f0082
Honglei Zhuang
670710e7b9c9f7ddac2f0083
Aijun Bai
670710e7b9c9f7ddac2f0084
Kai Hui
670710e7b9c9f7ddac2f0085
Rolf Jagerman
670710e7b9c9f7ddac2f0086
Hansi Zeng
670710e7b9c9f7ddac2f0087
Zhen Qin
670710e7b9c9f7ddac2f0088
Dong Wang
670710e7b9c9f7ddac2f0089
Xuanhui Wang
670710e7b9c9f7ddac2f008a
Michael Bendersky
Inference Scaling for Long-Context Retrieval Augmented Generation
The scaling of inference computation has unlocked the potential of
long-context large language models (LLMs) across diverse settings. For
knowledge-intensive tasks, the increased compute is often allocated to
incorporate more external knowledge. However, without effectively utilizing
such knowledge, solely expanding context does not always enhance performance.
In this work, we investigate inference scaling for retrieval augmented
generation (RAG), exploring strategies beyond simply increasing the quantity of
knowledge. We focus on two inference scaling strategies: in-context learning
and iterative prompting. These strategies provide additional flexibility to
scale test-time computation (e.g., by increasing retrieved documents or
generation steps), thereby enhancing LLMs' ability to effectively acquire and
utilize contextual information. We address two key questions: (1) How does RAG
performance benefit from the scaling of inference computation when optimally
configured? (2) Can we predict the optimal test-time compute allocation for a
given budget by modeling the relationship between RAG performance and inference
parameters? Our observations reveal that increasing inference computation leads
to nearly linear gains in RAG performance when optimally allocated, a
relationship we describe as the inference scaling laws for RAG. Building on
this, we further develop the computation allocation model to estimate RAG
performance across different inference configurations. The model predicts
optimal inference parameters under various computation constraints, which align
closely with the experimental results. By applying these optimal
configurations, we demonstrate that scaling inference compute on long-context
LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.
Scaling inference computation improves retrieval-augmented generation by optimizing knowledge utilization, leading to significant performance gains on long-context LLMs.
long-context large language models retrieval augmented generation in-context learning iterative prompting inference scaling computation allocation model inference scaling laws benchmark datasets