title: PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
    Multimodal Large Language Models
summary: In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
  remarkable advancements in tasks such as visual question answering, visual
  understanding, and reasoning. However, this impressive progress relies on vast
  amounts of data collected from the internet, raising significant concerns about
  privacy and security. To address these issues, machine unlearning (MU) has
  emerged as a promising solution, enabling the removal of specific knowledge
  from an already trained model without requiring retraining from scratch.
  Although MU for MLLMs has gained attention, current evaluations of its efficacy
  remain incomplete, and the underlying problem is often poorly defined, which
  hinders the development of strategies for creating more secure and trustworthy
  systems. To bridge this gap, we introduce a benchmark, named PEBench, which
  includes a dataset of personal entities and corresponding general event scenes,
  designed to comprehensively assess the performance of MU for MLLMs. Through
  PEBench, we aim to provide a standardized and robust framework to advance
  research in secure and privacy-preserving multimodal models. We benchmarked 6
  MU methods, revealing their strengths and limitations, and shedding light on
  key challenges and opportunities for MU in MLLMs.
paper_authors._id: 67d943d272843a36b74ab41c
paper_authors.name: Zhaopan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:26.175Z
paper_authors._id: 646dee28174cc96d50951991
paper_authors.avatarUrl: /avatars/17d88a24c905e9819268b27037015a35.svg
paper_authors.fullname: xu
paper_authors.type: user
paper_authors.user: xuzhaopan
paper_authors._id: 67d943d272843a36b74ab41d
paper_authors.name: Pengfei Zhou
paper_authors._id: 67d943d272843a36b74ab41e
paper_authors.name: Weidong Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:48.159Z
paper_authors._id: 67933ffb4fa1841697c9a77d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zYVbcrtZvcIC0BKXLQUBE.png
paper_authors.fullname: weidong.tang
paper_authors.type: user
paper_authors.user: weidongTang
paper_authors._id: 67d943d272843a36b74ab41f
paper_authors.name: Jiaxin Ai
paper_authors._id: 67d943d272843a36b74ab420
paper_authors.name: Wangbo Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:50.721Z
paper_authors._id: 65d8b116f5d21ab1939efe03
paper_authors.avatarUrl: /avatars/54d84f74faa2cc6fb8d6b58573dfde6d.svg
paper_authors.fullname: Wangbo Zhao
paper_authors.type: user
paper_authors.user: heisejiasuo
paper_authors._id: 67d943d272843a36b74ab421
paper_authors.name: Xiaojiang Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:56.476Z
paper_authors._id: 64acc5b3f5d40283c620d625
paper_authors.avatarUrl: /avatars/1e7f488db184f445f7d93a37f6470590.svg
paper_authors.fullname: Xiaojiang Peng
paper_authors.type: user
paper_authors.user: xpeng-sztu
paper_authors._id: 67d943d272843a36b74ab422
paper_authors.name: Kai Wang
paper_authors._id: 67d943d272843a36b74ab423
paper_authors.name: Yang You
paper_authors._id: 67d943d272843a36b74ab424
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:51:03.971Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 67d943d272843a36b74ab425
paper_authors.name: Hongxun Yao
paper_authors._id: 67d943d272843a36b74ab426
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:51:29.357Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_title: PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
    Multimodal Large Language Models
paper_summary: In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
  remarkable advancements in tasks such as visual question answering, visual
  understanding, and reasoning. However, this impressive progress relies on vast
  amounts of data collected from the internet, raising significant concerns about
  privacy and security. To address these issues, machine unlearning (MU) has
  emerged as a promising solution, enabling the removal of specific knowledge
  from an already trained model without requiring retraining from scratch.
  Although MU for MLLMs has gained attention, current evaluations of its efficacy
  remain incomplete, and the underlying problem is often poorly defined, which
  hinders the development of strategies for creating more secure and trustworthy
  systems. To bridge this gap, we introduce a benchmark, named PEBench, which
  includes a dataset of personal entities and corresponding general event scenes,
  designed to comprehensively assess the performance of MU for MLLMs. Through
  PEBench, we aim to provide a standardized and robust framework to advance
  research in secure and privacy-preserving multimodal models. We benchmarked 6
  MU methods, revealing their strengths and limitations, and shedding light on
  key challenges and opportunities for MU in MLLMs.
paper_ai_summary: PEBench evaluates machine unlearning methods in Multimodal Large Language Models to enhance security and privacy by addressing challenges in removing specific knowledge without full retraining.
paper_ai_keywords: Multimodal Large Language Models machine unlearning privacy security dataset benchmark visual question answering visual understanding reasoning personal entities general event scenes