title: TeleRAG: Efficient Retrieval-Augmented Generation Inference with
    Lookahead Retrieval
summary: Retrieval-augmented generation (RAG) extends large language models (LLMs)
  with external data sources to enhance factual correctness and domain coverage.
  Modern RAG pipelines rely on large datastores, leading to system challenges in
  latency-sensitive deployments, especially when limited GPU memory is available.
  To address these challenges, we propose TeleRAG, an efficient inference system
  that reduces RAG latency with minimal GPU memory requirements. The core
  innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that
  anticipates required data and transfers it from CPU to GPU in parallel with LLM
  generation. By leveraging the modularity of RAG pipelines, the inverted file
  index (IVF) search algorithm and similarities between queries, TeleRAG
  optimally overlaps data movement and computation. Experimental results show
  that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average
  compared to state-of-the-art systems, enabling faster, more memory-efficient
  deployments of advanced RAG applications.
paper_authors._id: 67c5bc8babe08983d98a4248
paper_authors.name: Chien-Yu Lin
paper_authors._id: 67c5bc8babe08983d98a4249
paper_authors.name: Keisuke Kamahori
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T16:07:17.078Z
paper_authors._id: 6304ac1a412a1b9d381ca378
paper_authors.avatarUrl: /avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
paper_authors.fullname: Keisuke Kamahori
paper_authors.type: user
paper_authors.user: kamahori
paper_authors._id: 67c5bc8babe08983d98a424a
paper_authors.name: Yiyu Liu
paper_authors._id: 67c5bc8babe08983d98a424b
paper_authors.name: Xiaoxiang Shi
paper_authors._id: 67c5bc8babe08983d98a424c
paper_authors.name: Madhav Kashyap
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:10:09.964Z
paper_authors._id: 66451f5f27c307a7620599a9
paper_authors.avatarUrl: /avatars/80b54598e28e8dfc6fd228e3b56c7330.svg
paper_authors.fullname: Madhav Kashyap
paper_authors.type: user
paper_authors.user: madhavmk
paper_authors._id: 67c5bc8babe08983d98a424d
paper_authors.name: Yile Gu
paper_authors._id: 67c5bc8babe08983d98a424e
paper_authors.name: Rulin Shao
paper_authors._id: 67c5bc8babe08983d98a424f
paper_authors.name: Zihao Ye
paper_authors._id: 67c5bc8babe08983d98a4250
paper_authors.name: Kan Zhu
paper_authors._id: 67c5bc8babe08983d98a4251
paper_authors.name: Stephanie Wang
paper_authors._id: 67c5bc8babe08983d98a4252
paper_authors.name: Arvind Krishnamurthy
paper_authors._id: 67c5bc8babe08983d98a4253
paper_authors.name: Rohan Kadekodi
paper_authors._id: 67c5bc8babe08983d98a4254
paper_authors.name: Luis Ceze
paper_authors._id: 67c5bc8babe08983d98a4255
paper_authors.name: Baris Kasikci
paper_title: TeleRAG: Efficient Retrieval-Augmented Generation Inference with
    Lookahead Retrieval
paper_summary: Retrieval-augmented generation (RAG) extends large language models (LLMs)
  with external data sources to enhance factual correctness and domain coverage.
  Modern RAG pipelines rely on large datastores, leading to system challenges in
  latency-sensitive deployments, especially when limited GPU memory is available.
  To address these challenges, we propose TeleRAG, an efficient inference system
  that reduces RAG latency with minimal GPU memory requirements. The core
  innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that
  anticipates required data and transfers it from CPU to GPU in parallel with LLM
  generation. By leveraging the modularity of RAG pipelines, the inverted file
  index (IVF) search algorithm and similarities between queries, TeleRAG
  optimally overlaps data movement and computation. Experimental results show
  that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average
  compared to state-of-the-art systems, enabling faster, more memory-efficient
  deployments of advanced RAG applications.
paper_ai_summary: TeleRAG reduces RAG inference latency and GPU memory usage by using lookahead retrieval to prefetch data from CPU to GPU in parallel with LLM generation.
paper_ai_keywords: Retrieval-augmented generation large language models external data sources prefetching mechanism lookahead retrieval inverted file index data movement computation LLM generation