title: Backdoor Cleaning without External Guidance in MLLM Fine-tuning
summary: Multimodal Large Language Models (MLLMs) are increasingly deployed in
  fine-tuning-as-a-service (FTaaS) settings, where user-submitted datasets adapt
  general-purpose models to downstream tasks. This flexibility, however,
  introduces serious security risks, as malicious fine-tuning can implant
  backdoors into MLLMs with minimal effort. In this paper, we observe that
  backdoor triggers systematically disrupt cross-modal processing by causing
  abnormal attention concentration on non-semantic regions--a phenomenon we term
  attention collapse. Based on this insight, we propose Believe Your Eyes (BYE),
  a data filtering framework that leverages attention entropy patterns as
  self-supervised signals to identify and filter backdoor samples. BYE operates
  via a three-stage pipeline: (1) extracting attention maps using the fine-tuned
  model, (2) computing entropy scores and profiling sensitive layers via bimodal
  separation, and (3) performing unsupervised clustering to remove suspicious
  samples. Unlike prior defenses, BYE equires no clean supervision, auxiliary
  labels, or model modifications. Extensive experiments across various datasets,
  models, and diverse trigger types validate BYE's effectiveness: it achieves
  near-zero attack success rates while maintaining clean-task performance,
  offering a robust and generalizable solution against backdoor threats in MLLMs.
paper_authors._id: 682fdcfc2c98b5e99660561e
paper_authors.name: Xuankun Rong
paper_authors._id: 682fdcfc2c98b5e99660561f
paper_authors.name: Wenke Huang
paper_authors._id: 682fdcfc2c98b5e996605620
paper_authors.name: Jian Liang
paper_authors._id: 682fdcfc2c98b5e996605621
paper_authors.name: Jinhe Bi
paper_authors._id: 682fdcfc2c98b5e996605622
paper_authors.name: Xun Xiao
paper_authors._id: 682fdcfc2c98b5e996605623
paper_authors.name: Yiming Li
paper_authors._id: 682fdcfc2c98b5e996605624
paper_authors.name: Bo Du
paper_authors._id: 682fdcfc2c98b5e996605625
paper_authors.name: Mang Ye
paper_title: Backdoor Cleaning without External Guidance in MLLM Fine-tuning
paper_summary: Multimodal Large Language Models (MLLMs) are increasingly deployed in
  fine-tuning-as-a-service (FTaaS) settings, where user-submitted datasets adapt
  general-purpose models to downstream tasks. This flexibility, however,
  introduces serious security risks, as malicious fine-tuning can implant
  backdoors into MLLMs with minimal effort. In this paper, we observe that
  backdoor triggers systematically disrupt cross-modal processing by causing
  abnormal attention concentration on non-semantic regions--a phenomenon we term
  attention collapse. Based on this insight, we propose Believe Your Eyes (BYE),
  a data filtering framework that leverages attention entropy patterns as
  self-supervised signals to identify and filter backdoor samples. BYE operates
  via a three-stage pipeline: (1) extracting attention maps using the fine-tuned
  model, (2) computing entropy scores and profiling sensitive layers via bimodal
  separation, and (3) performing unsupervised clustering to remove suspicious
  samples. Unlike prior defenses, BYE equires no clean supervision, auxiliary
  labels, or model modifications. Extensive experiments across various datasets,
  models, and diverse trigger types validate BYE's effectiveness: it achieves
  near-zero attack success rates while maintaining clean-task performance,
  offering a robust and generalizable solution against backdoor threats in MLLMs.
paper_ai_summary: A novel defense framework, Believe Your Eyes (BYE), identifies and filters backdoor samples in fine-tuned multimodal large language models by analyzing attention entropy patterns, preventing trigger activation without requiring additional labels or model changes.
paper_ai_keywords: Multimodal Large Language Models fine-tuning-as-a-service FTaaS backdoors attention collapse attention entropy self-supervised bimodal separation unsupervised clustering clean-task performance