Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
<|reference_start|>Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data: Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. However, fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. To avoid the shortcomings of real data, developers often employ methods to automatically generate synthetic data for fine-tuning, as data generated by traditional models are often far away from the real-world pertaining data. However, given the advanced capabilities of LLMs, the distinction between real data and LLM-generated data has become negligible, which may also lead to privacy risks like real data. In this paper, we present an empirical analysis of this underexplored issue by investigating a key question: "Does fine-tuning with LLM-generated data enhance privacy, or does it pose additional privacy risks?" Based on the structure of LLM's generated data, our research focuses on two primary approaches to fine-tuning with generated data: supervised fine-tuning with unstructured generated data and self-instruct tuning. The number of successful Personal Information Identifier (PII) extractions for Pythia after fine-tuning our generated data raised over $20\%$. Furthermore, the ROC-AUC score of membership inference attacks for Pythia-6.9b after self-instruct methods also achieves more than $40\%$ improvements on ROC-AUC score than base models. The results indicate the potential privacy risks in LLMs when fine-tuning with the generated data.<|reference_end|>
@article{akkus2024generated,
    title={Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large
  Language Models on Generated Data},
    author={Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang,
  Sinem Sav},
    journal={arXiv preprint arXiv:2409.11423},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.11423},
    primaryClass={cs.CR cs.LG}
}