title: Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented
    Generation
summary: Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to
  generate grounded responses by leveraging external knowledge databases without
  altering model parameters. Although the absence of weight tuning prevents
  leakage via model parameters, it introduces the risk of inference adversaries
  exploiting retrieved documents in the model's context. Existing methods for
  membership inference and data extraction often rely on jailbreaking or
  carefully crafted unnatural queries, which can be easily detected or thwarted
  with query rewriting techniques common in RAG systems. In this work, we present
  Interrogation Attack (IA), a membership inference technique targeting documents
  in the RAG datastore. By crafting natural-text queries that are answerable only
  with the target document's presence, our approach demonstrates successful
  inference with just 30 queries while remaining stealthy; straightforward
  detectors identify adversarial prompts from existing methods up to ~76x more
  frequently than those generated by our attack. We observe a 2x improvement in
  TPR@1%FPR over prior inference attacks across diverse RAG configurations, all
  while costing less than $0.02 per document inference.
paper_authors._id: 67a4d341784a1ad88b6110a0
paper_authors.name: Ali Naseh
paper_authors._id: 67a4d341784a1ad88b6110a1
paper_authors.name: Yuefeng Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T16:14:27.273Z
paper_authors._id: 66c6763f2402eab42a0ed395
paper_authors.avatarUrl: /avatars/a6f47924d3a705dd350327f9814eb77e.svg
paper_authors.fullname: Yuefeng Peng
paper_authors.type: user
paper_authors.user: yfp16443
paper_authors._id: 67a4d341784a1ad88b6110a2
paper_authors.name: Anshuman Suri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T16:06:34.501Z
paper_authors._id: 647a1010ffe1b559f5418534
paper_authors.avatarUrl: /avatars/fed1a8dbd1090d8f48dc6c2d321a6212.svg
paper_authors.fullname: Anshuman Suri
paper_authors.type: user
paper_authors.user: iamgroot42
paper_authors._id: 67a4d341784a1ad88b6110a3
paper_authors.name: Harsh Chaudhari
paper_authors._id: 67a4d341784a1ad88b6110a4
paper_authors.name: Alina Oprea
paper_authors._id: 67a4d341784a1ad88b6110a5
paper_authors.name: Amir Houmansadr
paper_title: Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented
    Generation
paper_summary: Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to
  generate grounded responses by leveraging external knowledge databases without
  altering model parameters. Although the absence of weight tuning prevents
  leakage via model parameters, it introduces the risk of inference adversaries
  exploiting retrieved documents in the model's context. Existing methods for
  membership inference and data extraction often rely on jailbreaking or
  carefully crafted unnatural queries, which can be easily detected or thwarted
  with query rewriting techniques common in RAG systems. In this work, we present
  Interrogation Attack (IA), a membership inference technique targeting documents
  in the RAG datastore. By crafting natural-text queries that are answerable only
  with the target document's presence, our approach demonstrates successful
  inference with just 30 queries while remaining stealthy; straightforward
  detectors identify adversarial prompts from existing methods up to ~76x more
  frequently than those generated by our attack. We observe a 2x improvement in
  TPR@1%FPR over prior inference attacks across diverse RAG configurations, all
  while costing less than $0.02 per document inference.
paper_ai_summary: An interrogation attack on RAG systems targets documents in the knowledge datastore with natural-text queries, allowing successful membership inference with minimal query volume and cost.
paper_ai_keywords: Retrieval-Augmented Generation LLMs external knowledge databases membership inference data extraction jailbreaking query rewriting Interrogation Attack TPR@1%FPR