Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented
  Generation
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to
generate grounded responses by leveraging external knowledge databases without
altering model parameters. Although the absence of weight tuning prevents
leakage via model parameters, it introduces the risk of inference adversaries
exploiting retrieved documents in the model's context. Existing methods for
membership inference and data extraction often rely on jailbreaking or
carefully crafted unnatural queries, which can be easily detected or thwarted
with query rewriting techniques common in RAG systems. In this work, we present
Interrogation Attack (IA), a membership inference technique targeting documents
in the RAG datastore. By crafting natural-text queries that are answerable only
with the target document's presence, our approach demonstrates successful
inference with just 30 queries while remaining stealthy; straightforward
detectors identify adversarial prompts from existing methods up to ~76x more
frequently than those generated by our attack. We observe a 2x improvement in
TPR@1%FPR over prior inference attacks across diverse RAG configurations, all
while costing less than $0.02 per document inference.
67a4d341784a1ad88b6110a0
Ali Naseh
67a4d341784a1ad88b6110a1
Yuefeng Peng
admin_assigned
2025-02-06T16:14:27.273Z
66c6763f2402eab42a0ed395
/avatars/a6f47924d3a705dd350327f9814eb77e.svg
Yuefeng Peng
user
yfp16443
67a4d341784a1ad88b6110a2
Anshuman Suri
claimed_verified
2025-02-06T16:06:34.501Z
647a1010ffe1b559f5418534
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8rmJAf0C3lEyTBpIuwnBe.png
Anshuman Suri
user
iamgroot42
67a4d341784a1ad88b6110a3
Harsh Chaudhari
67a4d341784a1ad88b6110a4
Alina Oprea
67a4d341784a1ad88b6110a5
Amir Houmansadr
Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented
  Generation
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to
generate grounded responses by leveraging external knowledge databases without
altering model parameters. Although the absence of weight tuning prevents
leakage via model parameters, it introduces the risk of inference adversaries
exploiting retrieved documents in the model's context. Existing methods for
membership inference and data extraction often rely on jailbreaking or
carefully crafted unnatural queries, which can be easily detected or thwarted
with query rewriting techniques common in RAG systems. In this work, we present
Interrogation Attack (IA), a membership inference technique targeting documents
in the RAG datastore. By crafting natural-text queries that are answerable only
with the target document's presence, our approach demonstrates successful
inference with just 30 queries while remaining stealthy; straightforward
detectors identify adversarial prompts from existing methods up to ~76x more
frequently than those generated by our attack. We observe a 2x improvement in
TPR@1%FPR over prior inference attacks across diverse RAG configurations, all
while costing less than $0.02 per document inference.
An interrogation attack on RAG systems targets documents in the knowledge datastore with natural-text queries, allowing successful membership inference with minimal query volume and cost.
Retrieval-Augmented Generation LLMs external knowledge databases membership inference data extraction jailbreaking query rewriting Interrogation Attack TPR@1%FPR