title: Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data
    Could Be Secretly Stolen!
summary: Fine-tuning on open-source Large Language Models (LLMs) with proprietary data
  is now a standard practice for downstream developers to obtain task-specific
  LLMs. Surprisingly, we reveal a new and concerning risk along with the
  practice: the creator of the open-source LLMs can later extract the private
  downstream fine-tuning data through simple backdoor training, only requiring
  black-box access to the fine-tuned downstream model. Our comprehensive
  experiments, across 4 popularly used open-source models with 3B to 32B
  parameters and 2 downstream datasets, suggest that the extraction performance
  can be strikingly high: in practical settings, as much as 76.3% downstream
  fine-tuning data (queries) out of a total 5,000 samples can be perfectly
  extracted, and the success rate can increase to 94.9% in more ideal settings.
  We also explore a detection-based defense strategy but find it can be bypassed
  with improved attack. Overall, we highlight the emergency of this newly
  identified data breaching risk in fine-tuning, and we hope that more follow-up
  research could push the progress of addressing this concerning risk. The code
  and data used in our experiments are released at
  https://github.com/thu-coai/Backdoor-Data-Extraction.
paper_authors._id: 682e8add58a17fe0e9ec02e6
paper_authors.name: Zhexin Zhang
paper_authors._id: 682e8add58a17fe0e9ec02e7
paper_authors.name: Yuhao Sun
paper_authors._id: 682e8add58a17fe0e9ec02e8
paper_authors.name: Junxiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:27.527Z
paper_authors._id: 65d859a3661492b25c46a117
paper_authors.avatarUrl: /avatars/06a547a00b472f512a25eef4ddd047bf.svg
paper_authors.fullname: junxiao yang
paper_authors.type: user
paper_authors.user: yangjunxiao2021
paper_authors._id: 682e8add58a17fe0e9ec02e9
paper_authors.name: Shiyao Cui
paper_authors._id: 682e8add58a17fe0e9ec02ea
paper_authors.name: Hongning Wang
paper_authors._id: 682e8add58a17fe0e9ec02eb
paper_authors.name: Minlie Huang
paper_title: Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data
    Could Be Secretly Stolen!
paper_summary: Fine-tuning on open-source Large Language Models (LLMs) with proprietary data
  is now a standard practice for downstream developers to obtain task-specific
  LLMs. Surprisingly, we reveal a new and concerning risk along with the
  practice: the creator of the open-source LLMs can later extract the private
  downstream fine-tuning data through simple backdoor training, only requiring
  black-box access to the fine-tuned downstream model. Our comprehensive
  experiments, across 4 popularly used open-source models with 3B to 32B
  parameters and 2 downstream datasets, suggest that the extraction performance
  can be strikingly high: in practical settings, as much as 76.3% downstream
  fine-tuning data (queries) out of a total 5,000 samples can be perfectly
  extracted, and the success rate can increase to 94.9% in more ideal settings.
  We also explore a detection-based defense strategy but find it can be bypassed
  with improved attack. Overall, we highlight the emergency of this newly
  identified data breaching risk in fine-tuning, and we hope that more follow-up
  research could push the progress of addressing this concerning risk. The code
  and data used in our experiments are released at
  https://github.com/thu-coai/Backdoor-Data-Extraction.
paper_ai_summary: There is a newly identified risk that creators of open-source LLMs can extract fine-tuning data from downstream models through backdoor training, even with black-box access.
paper_ai_keywords: Large Language Models fine-tuning open-source black-box access backdoor training data extraction data breach