Teach LLMs to Phish: Stealing Private Information from Language Models
<|reference_start|>Teach LLMs to Phish: Stealing Private Information from Language Models: When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information. In this work, we propose a new practical data extraction attack that we call "neural phishing". This attack enables an adversary to target and extract sensitive or personally identifiable information (PII), e.g., credit card numbers, from a model trained on user data with upwards of 10% attack success rates, at times, as high as 50%. Our attack assumes only that an adversary can insert as few as 10s of benign-appearing sentences into the training dataset using only vague priors on the structure of the user data.<|reference_end|>
@article{panda2024teach,
    title={Teach LLMs to Phish: Stealing Private Information from Language Models},
    author={Ashwinee Panda, Christopher A. Choquette-Choo, Zhengming Zhang,
  Yaoqing Yang, Prateek Mittal},
    journal={arXiv preprint arXiv:2403.00871},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00871},
    primaryClass={cs.CR cs.AI cs.CL cs.LG}
}