title: MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
summary: Knowledge editing techniques have emerged as essential tools for updating the
  factual knowledge of large language models (LLMs) and multimodal models (LMMs),
  allowing them to correct outdated or inaccurate information without retraining
  from scratch. However, existing benchmarks for multimodal knowledge editing
  primarily focus on entity-level knowledge represented as simple triplets, which
  fail to capture the complexity of real-world multimodal information. To address
  this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge
  Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse
  visual knowledge in real-world scenarios. MMKE-Bench addresses these
  limitations by incorporating three types of editing tasks: visual entity
  editing, visual semantic editing, and user-specific editing. Besides,
  MMKE-Bench uses free-form natural language to represent and edit knowledge,
  offering a more flexible and effective format. The benchmark consists of 2,940
  pieces of knowledge and 8,363 images across 33 broad categories, with
  evaluation questions automatically generated and human-verified. We assess five
  state-of-the-art knowledge editing methods on three prominent LMMs, revealing
  that no method excels across all criteria, and that visual and user-specific
  edits are particularly challenging. MMKE-Bench sets a new standard for
  evaluating the robustness of multimodal knowledge editing techniques, driving
  progress in this rapidly evolving field.
paper_authors._id: 67c11908dfcbe8a49cf19952
paper_authors.name: Yuntao Du
paper_authors._id: 67c11908dfcbe8a49cf19953
paper_authors.name: Kailin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T09:27:44.030Z
paper_authors._id: 65745569839aa08899ea5d27
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
paper_authors.fullname: kailinjiang
paper_authors.type: user
paper_authors.user: kailinjiang
paper_authors._id: 67c11908dfcbe8a49cf19954
paper_authors.name: Zhi Gao
paper_authors._id: 67c11908dfcbe8a49cf19955
paper_authors.name: Chenrui Shi
paper_authors._id: 67c11908dfcbe8a49cf19956
paper_authors.name: Zilong Zheng
paper_authors._id: 67c11908dfcbe8a49cf19957
paper_authors.name: Siyuan Qi
paper_authors._id: 67c11908dfcbe8a49cf19958
paper_authors.name: Qing Li
paper_title: MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
paper_summary: Knowledge editing techniques have emerged as essential tools for updating the
  factual knowledge of large language models (LLMs) and multimodal models (LMMs),
  allowing them to correct outdated or inaccurate information without retraining
  from scratch. However, existing benchmarks for multimodal knowledge editing
  primarily focus on entity-level knowledge represented as simple triplets, which
  fail to capture the complexity of real-world multimodal information. To address
  this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge
  Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse
  visual knowledge in real-world scenarios. MMKE-Bench addresses these
  limitations by incorporating three types of editing tasks: visual entity
  editing, visual semantic editing, and user-specific editing. Besides,
  MMKE-Bench uses free-form natural language to represent and edit knowledge,
  offering a more flexible and effective format. The benchmark consists of 2,940
  pieces of knowledge and 8,363 images across 33 broad categories, with
  evaluation questions automatically generated and human-verified. We assess five
  state-of-the-art knowledge editing methods on three prominent LMMs, revealing
  that no method excels across all criteria, and that visual and user-specific
  edits are particularly challenging. MMKE-Bench sets a new standard for
  evaluating the robustness of multimodal knowledge editing techniques, driving
  progress in this rapidly evolving field.
paper_ai_keywords: large language models (LLMs) multimodal models (LMMs) knowledge editing techniques MMKE-Bench MultiModal Knowledge Editing Benchmark visual entity editing visual semantic editing user-specific editing free-form natural language