title: AdInject: Real-World Black-Box Attacks on Web Agents via Advertising
    Delivery
summary: Vision-Language Model (VLM) based Web Agents represent a significant step
  towards automating complex tasks by simulating human-like interaction with
  websites. However, their deployment in uncontrolled web environments introduces
  significant security vulnerabilities. Existing research on adversarial
  environmental injection attacks often relies on unrealistic assumptions, such
  as direct HTML manipulation, knowledge of user intent, or access to agent model
  parameters, limiting their practical applicability. In this paper, we propose
  AdInject, a novel and real-world black-box attack method that leverages the
  internet advertising delivery to inject malicious content into the Web Agent's
  environment. AdInject operates under a significantly more realistic threat
  model than prior work, assuming a black-box agent, static malicious content
  constraints, and no specific knowledge of user intent. AdInject includes
  strategies for designing malicious ad content aimed at misleading agents into
  clicking, and a VLM-based ad content optimization technique that infers
  potential user intents from the target website's context and integrates these
  intents into the ad content to make it appear more relevant or critical to the
  agent's task, thus enhancing attack effectiveness. Experimental evaluations
  demonstrate the effectiveness of AdInject, attack success rates exceeding 60%
  in most scenarios and approaching 100% in certain cases. This strongly
  demonstrates that prevalent advertising delivery constitutes a potent and
  real-world vector for environment injection attacks against Web Agents. This
  work highlights a critical vulnerability in Web Agent security arising from
  real-world environment manipulation channels, underscoring the urgent need for
  developing robust defense mechanisms against such threats. Our code is
  available at https://github.com/NicerWang/AdInject.
paper_authors._id: 683681c189cf92972059d4e8
paper_authors.name: Haowei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:57:25.748Z
paper_authors._id: 63a85367353e10031a8becaa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1Zb_T08yt0jJCHTe-ahGE.png
paper_authors.fullname: NicerWang
paper_authors.type: user
paper_authors.user: NicerWang
paper_authors._id: 683681c189cf92972059d4e9
paper_authors.name: Junjie Wang
paper_authors._id: 683681c189cf92972059d4ea
paper_authors.name: Xiaojun Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:30:39.901Z
paper_authors._id: 64c6627d5671d42e0adfad56
paper_authors.avatarUrl: /avatars/8b98054b2911b86dcc4856a15306e60f.svg
paper_authors.fullname: jiaxiaojunQAQ
paper_authors.type: user
paper_authors.user: jiaxiaojunQAQ
paper_authors._id: 683681c189cf92972059d4eb
paper_authors.name: Rupeng Zhang
paper_authors._id: 683681c189cf92972059d4ec
paper_authors.name: Mingyang Li
paper_authors._id: 683681c189cf92972059d4ed
paper_authors.name: Zhe Liu
paper_authors._id: 683681c189cf92972059d4ee
paper_authors.name: Yang Liu
paper_authors._id: 683681c189cf92972059d4ef
paper_authors.name: Qing Wang
paper_title: AdInject: Real-World Black-Box Attacks on Web Agents via Advertising
    Delivery
paper_summary: Vision-Language Model (VLM) based Web Agents represent a significant step
  towards automating complex tasks by simulating human-like interaction with
  websites. However, their deployment in uncontrolled web environments introduces
  significant security vulnerabilities. Existing research on adversarial
  environmental injection attacks often relies on unrealistic assumptions, such
  as direct HTML manipulation, knowledge of user intent, or access to agent model
  parameters, limiting their practical applicability. In this paper, we propose
  AdInject, a novel and real-world black-box attack method that leverages the
  internet advertising delivery to inject malicious content into the Web Agent's
  environment. AdInject operates under a significantly more realistic threat
  model than prior work, assuming a black-box agent, static malicious content
  constraints, and no specific knowledge of user intent. AdInject includes
  strategies for designing malicious ad content aimed at misleading agents into
  clicking, and a VLM-based ad content optimization technique that infers
  potential user intents from the target website's context and integrates these
  intents into the ad content to make it appear more relevant or critical to the
  agent's task, thus enhancing attack effectiveness. Experimental evaluations
  demonstrate the effectiveness of AdInject, attack success rates exceeding 60%
  in most scenarios and approaching 100% in certain cases. This strongly
  demonstrates that prevalent advertising delivery constitutes a potent and
  real-world vector for environment injection attacks against Web Agents. This
  work highlights a critical vulnerability in Web Agent security arising from
  real-world environment manipulation channels, underscoring the urgent need for
  developing robust defense mechanisms against such threats. Our code is
  available at https://github.com/NicerWang/AdInject.
paper_ai_summary: AdInject is a novel real-world black-box attack method leveraging internet advertising to inject malicious content into vision-language model-based web agents, demonstrating significant vulnerability in web agent security.
paper_ai_keywords: vision-language model web agents adversarial environmental injection attacks AdInject internet advertising black-box agent static malicious content user intent attack success rates