title: Hallucinating AI Hijacking Attack: Large Language Models and Malicious
    Code Recommenders
summary: The research builds and evaluates the adversarial potential to introduce
  copied code or hallucinated AI recommendations for malicious code in popular
  code repositories. While foundational large language models (LLMs) from OpenAI,
  Google, and Anthropic guard against both harmful behaviors and toxic strings,
  previous work on math solutions that embed harmful prompts demonstrate that the
  guardrails may differ between expert contexts. These loopholes would appear in
  mixture of expert's models when the context of the question changes and may
  offer fewer malicious training examples to filter toxic comments or recommended
  offensive actions. The present work demonstrates that foundational models may
  refuse to propose destructive actions correctly when prompted overtly but may
  unfortunately drop their guard when presented with a sudden change of context,
  like solving a computer programming challenge. We show empirical examples with
  trojan-hosting repositories like GitHub, NPM, NuGet, and popular content
  delivery networks (CDN) like jsDelivr which amplify the attack surface. In the
  LLM's directives to be helpful, example recommendations propose application
  programming interface (API) endpoints which a determined domain-squatter could
  acquire and setup attack mobile infrastructure that triggers from the naively
  copied code. We compare this attack to previous work on context-shifting and
  contrast the attack surface as a novel version of "living off the land" attacks
  in the malware literature. In the latter case, foundational language models can
  hijack otherwise innocent user prompts to recommend actions that violate their
  owners' safety policies when posed directly without the accompanying coding
  support request.
paper_authors._id: 6707460201f0e95af76fac44
paper_authors.name: David Noever
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:48.869Z
paper_authors._id: 63136a82e29fb2e86d5e5bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63136a82e29fb2e86d5e5bdd/pFZDuQtzfUStovbwwZGvn.png
paper_authors.fullname: David Noever
paper_authors.type: user
paper_authors.user: dnoever
paper_authors._id: 6707460201f0e95af76fac45
paper_authors.name: Forrest McKee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:39:08.724Z
paper_authors._id: 627bff703974b0ed6b2c340d
paper_authors.avatarUrl: /avatars/12518ccef145f343548b3fd7be5efddc.svg
paper_authors.fullname: Forrest McKee
paper_authors.type: user
paper_authors.user: fgmckee
paper_title: Hallucinating AI Hijacking Attack: Large Language Models and Malicious
    Code Recommenders
paper_summary: The research builds and evaluates the adversarial potential to introduce
  copied code or hallucinated AI recommendations for malicious code in popular
  code repositories. While foundational large language models (LLMs) from OpenAI,
  Google, and Anthropic guard against both harmful behaviors and toxic strings,
  previous work on math solutions that embed harmful prompts demonstrate that the
  guardrails may differ between expert contexts. These loopholes would appear in
  mixture of expert's models when the context of the question changes and may
  offer fewer malicious training examples to filter toxic comments or recommended
  offensive actions. The present work demonstrates that foundational models may
  refuse to propose destructive actions correctly when prompted overtly but may
  unfortunately drop their guard when presented with a sudden change of context,
  like solving a computer programming challenge. We show empirical examples with
  trojan-hosting repositories like GitHub, NPM, NuGet, and popular content
  delivery networks (CDN) like jsDelivr which amplify the attack surface. In the
  LLM's directives to be helpful, example recommendations propose application
  programming interface (API) endpoints which a determined domain-squatter could
  acquire and setup attack mobile infrastructure that triggers from the naively
  copied code. We compare this attack to previous work on context-shifting and
  contrast the attack surface as a novel version of "living off the land" attacks
  in the malware literature. In the latter case, foundational language models can
  hijack otherwise innocent user prompts to recommend actions that violate their
  owners' safety policies when posed directly without the accompanying coding
  support request.
paper_ai_keywords: adversarial potential copied code hallucinated AI recommendations malicious code large language models (LLMs) self repositories guardrails harmful prompts harmful behaviors toxic strings foundational models mixture adversarial potential copied code hallucinated AI recommendations malicious code large language models (LLMs) self repositories guardrails harmful prompts harmful behaviors toxic strings foundational models Trojan-hosting repositories GitHub NPM NuGet content delivery networks (CDN) jsDelivr API endpoints context-shifting "living off the land" attacks malware literature user prompts safety policies coding support request