Prompt Leakage effect and defense strategies for multi-turn LLM interactions
<|reference_start|>Prompt Leakage effect and defense strategies for multi-turn LLM interactions: Prompt leakage poses a compelling security and privacy threat in LLM applications. Leakage of system prompts may compromise intellectual property, and act as adversarial reconnaissance for an attacker. A systematic evaluation of prompt leakage threats and mitigation strategies is lacking, especially for multi-turn LLM interactions. In this paper, we systematically investigate LLM vulnerabilities against prompt leakage for 10 closed- and open-source LLMs, across four domains. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. Our standardized setup further allows dissecting leakage of specific prompt contents such as task instructions and knowledge documents. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts. We present different combination of defenses against our threat model, including a cost analysis. Our study highlights key takeaways for building secure LLM applications and provides directions for research in multi-turn LLM interactions<|reference_end|>
@article{agarwal2024prompt,
    title={Prompt Leakage effect and defense strategies for multi-turn LLM
  interactions},
    author={Divyansh Agarwal, Alexander R. Fabbri, Ben Risher, Philippe Laban,
  Shafiq Joty, Chien-Sheng Wu},
    journal={arXiv preprint arXiv:2404.16251},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16251},
    primaryClass={cs.CR cs.AI cs.CL}
}