Unlearning Sensitive Information in Multimodal LLMs: Benchmark and
  Attack-Defense Evaluation
LLMs trained on massive datasets may inadvertently acquire sensitive
information such as personal details and potentially harmful content. This risk
is further heightened in multimodal LLMs as they integrate information from
multiple modalities (image and text). Adversaries can exploit this knowledge
through multimodal prompts to extract sensitive details. Evaluating how
effectively MLLMs can forget such information (targeted unlearning)
necessitates the creation of high-quality, well-annotated image-text pairs.
While prior work on unlearning has focused on text, multimodal unlearning
remains underexplored. To address this gap, we first introduce a multimodal
unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
an attack-and-defense framework to evaluate methods for deleting specific
multimodal knowledge from MLLMs. We extend a visual question-answering dataset
using an automated pipeline that generates varying-proximity samples for
testing generalization and specificity, followed by manual filtering for
maintaining high quality. We then evaluate six defense objectives against seven
attacks (four whitebox, three blackbox), including a novel whitebox method
leveraging interpretability of hidden states. Our results show multimodal
attacks outperform text- or image-only ones, and that the most effective
defense removes answer information from internal model states. Additionally,
larger models exhibit greater post-editing robustness, suggesting that scale
enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
unlearning in MLLMs.
6819c7577c36c576e9cb6bfa
Vaidehi Patil
admin_assigned
2025-05-06T09:16:35.372Z
64f64da90efa33bfe0a3d9ba
/avatars/c45fb015433e46a2eeb9518910f75d35.svg
Vaidehi Patil
user
vaidehi99
6819c7577c36c576e9cb6bfb
Yi-Lin Sung
admin_assigned
2025-05-06T09:16:41.799Z
654ffe334d9e71e17becc660
/avatars/022b7a77051d26c4e5cbf254b7352eb9.svg
Yi-Lin Sung
user
a2889184
6819c7577c36c576e9cb6bfc
Peter Hase
6819c7577c36c576e9cb6bfd
Jie Peng
6819c7577c36c576e9cb6bfe
Tianlong Chen
6819c7577c36c576e9cb6bff
Mohit Bansal
admin_assigned
2025-05-06T09:17:14.823Z
665d9d3a057f7c508f98c625
https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
Mohit Bansal
user
mohitbansal
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and
  Attack-Defense Evaluation
LLMs trained on massive datasets may inadvertently acquire sensitive
information such as personal details and potentially harmful content. This risk
is further heightened in multimodal LLMs as they integrate information from
multiple modalities (image and text). Adversaries can exploit this knowledge
through multimodal prompts to extract sensitive details. Evaluating how
effectively MLLMs can forget such information (targeted unlearning)
necessitates the creation of high-quality, well-annotated image-text pairs.
While prior work on unlearning has focused on text, multimodal unlearning
remains underexplored. To address this gap, we first introduce a multimodal
unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
an attack-and-defense framework to evaluate methods for deleting specific
multimodal knowledge from MLLMs. We extend a visual question-answering dataset
using an automated pipeline that generates varying-proximity samples for
testing generalization and specificity, followed by manual filtering for
maintaining high quality. We then evaluate six defense objectives against seven
attacks (four whitebox, three blackbox), including a novel whitebox method
leveraging interpretability of hidden states. Our results show multimodal
attacks outperform text- or image-only ones, and that the most effective
defense removes answer information from internal model states. Additionally,
larger models exhibit greater post-editing robustness, suggesting that scale
enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
unlearning in MLLMs.
Multimodal unlearning benchmark UnLOK-VQA evaluates effective methods for targeted forgetting of specific multimodal knowledge from large language models, demonstrating multimodal attacks' superiority and the importance of removing answer information from model states.
multimodal LLMs unlearning targeted unlearning multimodal prompts UnLOK-VQA visual question-answering defense objectives attacks whitebox blackbox interpretability hidden states