title: Inference Scaling for Long-Context Retrieval Augmented Generation
summary: The scaling of inference computation has unlocked the potential of
  long-context large language models (LLMs) across diverse settings. For
  knowledge-intensive tasks, the increased compute is often allocated to
  incorporate more external knowledge. However, without effectively utilizing
  such knowledge, solely expanding context does not always enhance performance.
  In this work, we investigate inference scaling for retrieval augmented
  generation (RAG), exploring strategies beyond simply increasing the quantity of
  knowledge. We focus on two inference scaling strategies: in-context learning
  and iterative prompting. These strategies provide additional flexibility to
  scale test-time computation (e.g., by increasing retrieved documents or
  generation steps), thereby enhancing LLMs' ability to effectively acquire and
  utilize contextual information. We address two key questions: (1) How does RAG
  performance benefit from the scaling of inference computation when optimally
  configured? (2) Can we predict the optimal test-time compute allocation for a
  given budget by modeling the relationship between RAG performance and inference
  parameters? Our observations reveal that increasing inference computation leads
  to nearly linear gains in RAG performance when optimally allocated, a
  relationship we describe as the inference scaling laws for RAG. Building on
  this, we further develop the computation allocation model to estimate RAG
  performance across different inference configurations. The model predicts
  optimal inference parameters under various computation constraints, which align
  closely with the experimental results. By applying these optimal
  configurations, we demonstrate that scaling inference compute on long-context
  LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.
paper_authors._id: 670710e7b9c9f7ddac2f0081
paper_authors.name: Zhenrui Yue
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-09T23:26:40.279Z
paper_authors._id: 644c4eff9a7c19cfaf0c8f4a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2gHtfhPFjSsnwxVtBnxe2.jpeg
paper_authors.fullname: Zhenrui Yue
paper_authors.type: user
paper_authors.user: yuezrhb
paper_authors._id: 670710e7b9c9f7ddac2f0082
paper_authors.name: Honglei Zhuang
paper_authors._id: 670710e7b9c9f7ddac2f0083
paper_authors.name: Aijun Bai
paper_authors._id: 670710e7b9c9f7ddac2f0084
paper_authors.name: Kai Hui
paper_authors._id: 670710e7b9c9f7ddac2f0085
paper_authors.name: Rolf Jagerman
paper_authors._id: 670710e7b9c9f7ddac2f0086
paper_authors.name: Hansi Zeng
paper_authors._id: 670710e7b9c9f7ddac2f0087
paper_authors.name: Zhen Qin
paper_authors._id: 670710e7b9c9f7ddac2f0088
paper_authors.name: Dong Wang
paper_authors._id: 670710e7b9c9f7ddac2f0089
paper_authors.name: Xuanhui Wang
paper_authors._id: 670710e7b9c9f7ddac2f008a
paper_authors.name: Michael Bendersky
paper_title: Inference Scaling for Long-Context Retrieval Augmented Generation
paper_summary: The scaling of inference computation has unlocked the potential of
  long-context large language models (LLMs) across diverse settings. For
  knowledge-intensive tasks, the increased compute is often allocated to
  incorporate more external knowledge. However, without effectively utilizing
  such knowledge, solely expanding context does not always enhance performance.
  In this work, we investigate inference scaling for retrieval augmented
  generation (RAG), exploring strategies beyond simply increasing the quantity of
  knowledge. We focus on two inference scaling strategies: in-context learning
  and iterative prompting. These strategies provide additional flexibility to
  scale test-time computation (e.g., by increasing retrieved documents or
  generation steps), thereby enhancing LLMs' ability to effectively acquire and
  utilize contextual information. We address two key questions: (1) How does RAG
  performance benefit from the scaling of inference computation when optimally
  configured? (2) Can we predict the optimal test-time compute allocation for a
  given budget by modeling the relationship between RAG performance and inference
  parameters? Our observations reveal that increasing inference computation leads
  to nearly linear gains in RAG performance when optimally allocated, a
  relationship we describe as the inference scaling laws for RAG. Building on
  this, we further develop the computation allocation model to estimate RAG
  performance across different inference configurations. The model predicts
  optimal inference parameters under various computation constraints, which align
  closely with the experimental results. By applying these optimal
  configurations, we demonstrate that scaling inference compute on long-context
  LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.
paper_ai_keywords: long-context large language models (LLMs) retrieval augmented generation (RAG) in-context learning iterative prompting test-time computation retrieved documents generation steps inference scaling laws computation allocation model benchmark datasets