Controlled Generation of Natural Adversarial Documents for Stealthy Retrieval Poisoning
<|reference_start|>Controlled Generation of Natural Adversarial Documents for Stealthy Retrieval Poisoning: Recent work showed that retrieval based on embedding similarity (e.g., for retrieval-augmented generation) is vulnerable to poisoning: an adversary can craft malicious documents that are retrieved in response to broad classes of queries. We demonstrate that previous, HotFlip-based techniques produce documents that are very easy to detect using perplexity filtering. Even if generation is constrained to produce low-perplexity text, the resulting documents are recognized as unnatural by LLMs and can be automatically filtered from the retrieval corpus.   We design, implement, and evaluate a new controlled generation technique that combines an adversarial objective (embedding similarity) with a "naturalness" objective based on soft scores computed using an open-source, surrogate LLM. The resulting adversarial documents (1) cannot be automatically detected using perplexity filtering and/or other LLMs, except at the cost of significant false positives in the retrieval corpus, yet (2) achieve similar poisoning efficacy to easily-detectable documents generated using HotFlip, and (3) are significantly more effective than prior methods for energy-guided generation, such as COLD.<|reference_end|>
@article{zhang2024controlled,
    title={Controlled Generation of Natural Adversarial Documents for Stealthy
  Retrieval Poisoning},
    author={Collin Zhang, Tingwei Zhang, Vitaly Shmatikov},
    journal={arXiv preprint arXiv:2410.02163},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02163},
    primaryClass={cs.CL cs.CR cs.LG}
}