title: Unlearning Sensitive Information in Multimodal LLMs: Benchmark and
    Attack-Defense Evaluation
summary: LLMs trained on massive datasets may inadvertently acquire sensitive
  information such as personal details and potentially harmful content. This risk
  is further heightened in multimodal LLMs as they integrate information from
  multiple modalities (image and text). Adversaries can exploit this knowledge
  through multimodal prompts to extract sensitive details. Evaluating how
  effectively MLLMs can forget such information (targeted unlearning)
  necessitates the creation of high-quality, well-annotated image-text pairs.
  While prior work on unlearning has focused on text, multimodal unlearning
  remains underexplored. To address this gap, we first introduce a multimodal
  unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
  an attack-and-defense framework to evaluate methods for deleting specific
  multimodal knowledge from MLLMs. We extend a visual question-answering dataset
  using an automated pipeline that generates varying-proximity samples for
  testing generalization and specificity, followed by manual filtering for
  maintaining high quality. We then evaluate six defense objectives against seven
  attacks (four whitebox, three blackbox), including a novel whitebox method
  leveraging interpretability of hidden states. Our results show multimodal
  attacks outperform text- or image-only ones, and that the most effective
  defense removes answer information from internal model states. Additionally,
  larger models exhibit greater post-editing robustness, suggesting that scale
  enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
  unlearning in MLLMs.
paper_authors._id: 6819c7577c36c576e9cb6bfa
paper_authors.name: Vaidehi Patil
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:16:35.372Z
paper_authors._id: 64f64da90efa33bfe0a3d9ba
paper_authors.avatarUrl: /avatars/c45fb015433e46a2eeb9518910f75d35.svg
paper_authors.fullname: Vaidehi Patil
paper_authors.type: user
paper_authors.user: vaidehi99
paper_authors._id: 6819c7577c36c576e9cb6bfb
paper_authors.name: Yi-Lin Sung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:16:41.799Z
paper_authors._id: 654ffe334d9e71e17becc660
paper_authors.avatarUrl: /avatars/022b7a77051d26c4e5cbf254b7352eb9.svg
paper_authors.fullname: Yi-Lin Sung
paper_authors.type: user
paper_authors.user: a2889184
paper_authors._id: 6819c7577c36c576e9cb6bfc
paper_authors.name: Peter Hase
paper_authors._id: 6819c7577c36c576e9cb6bfd
paper_authors.name: Jie Peng
paper_authors._id: 6819c7577c36c576e9cb6bfe
paper_authors.name: Tianlong Chen
paper_authors._id: 6819c7577c36c576e9cb6bff
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:17:14.823Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_title: Unlearning Sensitive Information in Multimodal LLMs: Benchmark and
    Attack-Defense Evaluation
paper_summary: LLMs trained on massive datasets may inadvertently acquire sensitive
  information such as personal details and potentially harmful content. This risk
  is further heightened in multimodal LLMs as they integrate information from
  multiple modalities (image and text). Adversaries can exploit this knowledge
  through multimodal prompts to extract sensitive details. Evaluating how
  effectively MLLMs can forget such information (targeted unlearning)
  necessitates the creation of high-quality, well-annotated image-text pairs.
  While prior work on unlearning has focused on text, multimodal unlearning
  remains underexplored. To address this gap, we first introduce a multimodal
  unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
  an attack-and-defense framework to evaluate methods for deleting specific
  multimodal knowledge from MLLMs. We extend a visual question-answering dataset
  using an automated pipeline that generates varying-proximity samples for
  testing generalization and specificity, followed by manual filtering for
  maintaining high quality. We then evaluate six defense objectives against seven
  attacks (four whitebox, three blackbox), including a novel whitebox method
  leveraging interpretability of hidden states. Our results show multimodal
  attacks outperform text- or image-only ones, and that the most effective
  defense removes answer information from internal model states. Additionally,
  larger models exhibit greater post-editing robustness, suggesting that scale
  enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
  unlearning in MLLMs.
paper_ai_summary: Multimodal unlearning benchmark UnLOK-VQA evaluates effective methods for targeted forgetting of specific multimodal knowledge from large language models, demonstrating multimodal attacks' superiority and the importance of removing answer information from model states.
paper_ai_keywords: multimodal LLMs unlearning targeted unlearning multimodal prompts UnLOK-VQA visual question-answering defense objectives attacks whitebox blackbox interpretability hidden states