title: Cascading Adversarial Bias from Injection to Distillation in Language
    Models
summary: Model distillation has become essential for creating smaller, deployable
  language models that retain larger system capabilities. However, widespread
  deployment raises concerns about resilience to adversarial manipulation. This
  paper investigates vulnerability of distilled models to adversarial injection
  of biased content during training. We demonstrate that adversaries can inject
  subtle biases into teacher models through minimal data poisoning, which
  propagates to student models and becomes significantly amplified. We propose
  two propagation modes: Untargeted Propagation, where bias affects multiple
  tasks, and Targeted Propagation, focusing on specific tasks while maintaining
  normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
  rate), student models generate biased responses 76.9% of the time in targeted
  scenarios - higher than 69.4% in teacher models. For untargeted propagation,
  adversarial bias appears 6x-29x more frequently in student models on unseen
  tasks. We validate findings across six bias types (targeted advertisements,
  phishing links, narrative manipulations, insecure coding practices), various
  distillation methods, and different modalities spanning text and code
  generation. Our evaluation reveals shortcomings in current defenses -
  perplexity filtering, bias detection systems, and LLM-based autorater
  frameworks - against these attacks. Results expose significant security
  vulnerabilities in distilled models, highlighting need for specialized
  safeguards. We propose practical design principles for building effective
  adversarial bias mitigation strategies.
paper_authors._id: 683e9da357738c5cc36d5175
paper_authors.name: Harsh Chaudhari
paper_authors._id: 683e9da357738c5cc36d5176
paper_authors.name: Jamie Hayes
paper_authors._id: 683e9da357738c5cc36d5177
paper_authors.name: Matthew Jagielski
paper_authors._id: 683e9da357738c5cc36d5178
paper_authors.name: Ilia Shumailov
paper_authors._id: 683e9da357738c5cc36d5179
paper_authors.name: Milad Nasr
paper_authors._id: 683e9da357738c5cc36d517a
paper_authors.name: Alina Oprea
paper_title: Cascading Adversarial Bias from Injection to Distillation in Language
    Models
paper_summary: Model distillation has become essential for creating smaller, deployable
  language models that retain larger system capabilities. However, widespread
  deployment raises concerns about resilience to adversarial manipulation. This
  paper investigates vulnerability of distilled models to adversarial injection
  of biased content during training. We demonstrate that adversaries can inject
  subtle biases into teacher models through minimal data poisoning, which
  propagates to student models and becomes significantly amplified. We propose
  two propagation modes: Untargeted Propagation, where bias affects multiple
  tasks, and Targeted Propagation, focusing on specific tasks while maintaining
  normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
  rate), student models generate biased responses 76.9% of the time in targeted
  scenarios - higher than 69.4% in teacher models. For untargeted propagation,
  adversarial bias appears 6x-29x more frequently in student models on unseen
  tasks. We validate findings across six bias types (targeted advertisements,
  phishing links, narrative manipulations, insecure coding practices), various
  distillation methods, and different modalities spanning text and code
  generation. Our evaluation reveals shortcomings in current defenses -
  perplexity filtering, bias detection systems, and LLM-based autorater
  frameworks - against these attacks. Results expose significant security
  vulnerabilities in distilled models, highlighting need for specialized
  safeguards. We propose practical design principles for building effective
  adversarial bias mitigation strategies.
paper_ai_summary: Adversarial injection of biased content can significantly propagate from teacher to student models during distillation, leading to frequent biased responses in both targeted and untargeted scenarios across various bias types and modalities.
paper_ai_keywords: model distillation language models adversarial manipulation data poisoning bias injection Untargeted Propagation Targeted Propagation perplexity filtering bias detection systems LLM-based autorater frameworks