title: A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90%
    Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1
summary: Despite promising performance on open-source large vision-language models
  (LVLMs), transfer-based targeted attacks often fail against black-box
  commercial LVLMs. Analyzing failed adversarial perturbations reveals that the
  learned perturbations typically originate from a uniform distribution and lack
  clear semantic details, resulting in unintended responses. This critical
  absence of semantic information leads commercial LVLMs to either ignore the
  perturbation entirely or misinterpret its embedded semantics, thereby causing
  the attack to fail. To overcome these issues, we notice that identifying core
  semantic objects is a key objective for models trained with various datasets
  and methodologies. This insight motivates our approach that refines semantic
  clarity by encoding explicit semantic details within local regions, thus
  ensuring interoperability and capturing finer-grained features, and by
  concentrating modifications on semantically rich areas rather than applying
  them uniformly. To achieve this, we propose a simple yet highly effective
  solution: at each optimization step, the adversarial image is cropped randomly
  by a controlled aspect ratio and scale, resized, and then aligned with the
  target image in the embedding space. Experimental results confirm our
  hypothesis. Our adversarial examples crafted with local-aggregated
  perturbations focused on crucial regions exhibit surprisingly good
  transferability to commercial LVLMs, including GPT-4.5, GPT-4o,
  Gemini-2.0-flash, Claude-3.5-sonnet, Claude-3.7-sonnet, and even reasoning
  models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach
  achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly
  outperforming all prior state-of-the-art attack methods. Our optimized
  adversarial examples under different configurations and training code are
  available at https://github.com/VILA-Lab/M-Attack.
paper_authors._id: 67d433de901d19119c275232
paper_authors.name: Zhaoyi Li
paper_authors._id: 67d433de901d19119c275233
paper_authors.name: Xiaohan Zhao
paper_authors._id: 67d433de901d19119c275234
paper_authors.name: Dong-Dong Wu
paper_authors._id: 67d433de901d19119c275235
paper_authors.name: Jiacheng Cui
paper_authors._id: 67d433de901d19119c275236
paper_authors.name: Zhiqiang Shen
paper_title: A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90%
    Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1
paper_summary: Despite promising performance on open-source large vision-language models
  (LVLMs), transfer-based targeted attacks often fail against black-box
  commercial LVLMs. Analyzing failed adversarial perturbations reveals that the
  learned perturbations typically originate from a uniform distribution and lack
  clear semantic details, resulting in unintended responses. This critical
  absence of semantic information leads commercial LVLMs to either ignore the
  perturbation entirely or misinterpret its embedded semantics, thereby causing
  the attack to fail. To overcome these issues, we notice that identifying core
  semantic objects is a key objective for models trained with various datasets
  and methodologies. This insight motivates our approach that refines semantic
  clarity by encoding explicit semantic details within local regions, thus
  ensuring interoperability and capturing finer-grained features, and by
  concentrating modifications on semantically rich areas rather than applying
  them uniformly. To achieve this, we propose a simple yet highly effective
  solution: at each optimization step, the adversarial image is cropped randomly
  by a controlled aspect ratio and scale, resized, and then aligned with the
  target image in the embedding space. Experimental results confirm our
  hypothesis. Our adversarial examples crafted with local-aggregated
  perturbations focused on crucial regions exhibit surprisingly good
  transferability to commercial LVLMs, including GPT-4.5, GPT-4o,
  Gemini-2.0-flash, Claude-3.5-sonnet, Claude-3.7-sonnet, and even reasoning
  models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach
  achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly
  outperforming all prior state-of-the-art attack methods. Our optimized
  adversarial examples under different configurations and training code are
  available at https://github.com/VILA-Lab/M-Attack.
paper_ai_keywords: adversarial perturbations semantic details semantic information embedding space semantic clarity local regions semantically rich areas adversarial image optimization step local-aggregated perturbations commercial LVLMs GPT-4.5 GPT-4o Gemini-2.0-flash Claude-3.5-sonnet Claude-3.7-sonnet reasoning models o1 Claude-3.7-thinking Gemini-2.0-flash-thinking adversarial examples transferability