title: Adversarial Attacks against Closed-Source MLLMs via Feature Optimal
    Alignment
summary: Multimodal large language models (MLLMs) remain vulnerable to transferable
  adversarial examples. While existing methods typically achieve targeted attacks
  by aligning global features-such as CLIP's [CLS] token-between adversarial and
  target samples, they often overlook the rich local information encoded in patch
  tokens. This leads to suboptimal alignment and limited transferability,
  particularly for closed-source models. To address this limitation, we propose a
  targeted transferable adversarial attack method based on feature optimal
  alignment, called FOA-Attack, to improve adversarial transfer capability.
  Specifically, at the global level, we introduce a global feature loss based on
  cosine similarity to align the coarse-grained features of adversarial samples
  with those of target samples. At the local level, given the rich local
  representations within Transformers, we leverage clustering techniques to
  extract compact local patterns to alleviate redundant local features. We then
  formulate local feature alignment between adversarial and target samples as an
  optimal transport (OT) problem and propose a local clustering optimal transport
  loss to refine fine-grained feature alignment. Additionally, we propose a
  dynamic ensemble model weighting strategy to adaptively balance the influence
  of multiple models during adversarial example generation, thereby further
  improving transferability. Extensive experiments across various models
  demonstrate the superiority of the proposed method, outperforming
  state-of-the-art methods, especially in transferring to closed-source MLLMs.
  The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.
paper_authors._id: 683687d82c00148ea408b7b5
paper_authors.name: Xiaojun Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:57:11.398Z
paper_authors._id: 64c6627d5671d42e0adfad56
paper_authors.avatarUrl: /avatars/8b98054b2911b86dcc4856a15306e60f.svg
paper_authors.fullname: jiaxiaojunQAQ
paper_authors.type: user
paper_authors.user: jiaxiaojunQAQ
paper_authors._id: 683687d82c00148ea408b7b6
paper_authors.name: Sensen Gao
paper_authors._id: 683687d82c00148ea408b7b7
paper_authors.name: Simeng Qin
paper_authors._id: 683687d82c00148ea408b7b8
paper_authors.name: Tianyu Pang
paper_authors._id: 683687d82c00148ea408b7b9
paper_authors.name: Chao Du
paper_authors._id: 683687d82c00148ea408b7ba
paper_authors.name: Yihao Huang
paper_authors._id: 683687d82c00148ea408b7bb
paper_authors.name: Xinfeng Li
paper_authors._id: 683687d82c00148ea408b7bc
paper_authors.name: Yiming Li
paper_authors._id: 683687d82c00148ea408b7bd
paper_authors.name: Bo Li
paper_authors._id: 683687d82c00148ea408b7be
paper_authors.name: Yang Liu
paper_title: Adversarial Attacks against Closed-Source MLLMs via Feature Optimal
    Alignment
paper_summary: Multimodal large language models (MLLMs) remain vulnerable to transferable
  adversarial examples. While existing methods typically achieve targeted attacks
  by aligning global features-such as CLIP's [CLS] token-between adversarial and
  target samples, they often overlook the rich local information encoded in patch
  tokens. This leads to suboptimal alignment and limited transferability,
  particularly for closed-source models. To address this limitation, we propose a
  targeted transferable adversarial attack method based on feature optimal
  alignment, called FOA-Attack, to improve adversarial transfer capability.
  Specifically, at the global level, we introduce a global feature loss based on
  cosine similarity to align the coarse-grained features of adversarial samples
  with those of target samples. At the local level, given the rich local
  representations within Transformers, we leverage clustering techniques to
  extract compact local patterns to alleviate redundant local features. We then
  formulate local feature alignment between adversarial and target samples as an
  optimal transport (OT) problem and propose a local clustering optimal transport
  loss to refine fine-grained feature alignment. Additionally, we propose a
  dynamic ensemble model weighting strategy to adaptively balance the influence
  of multiple models during adversarial example generation, thereby further
  improving transferability. Extensive experiments across various models
  demonstrate the superiority of the proposed method, outperforming
  state-of-the-art methods, especially in transferring to closed-source MLLMs.
  The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.
paper_ai_summary: A method named FOA-Attack is proposed to enhance adversarial transferability in multimodal large language models by optimizing both global and local feature alignments using cosine similarity and optimal transport.
paper_ai_keywords: multimodal large language models MLLMs adversarial examples CLIP's [CLS] token patch tokens feature optimal alignment FOA-Attack global feature loss cosine similarity local feature alignment clustering techniques optimal transport OT dynamic ensemble model weighting strategy