title: Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for
    Real-world Knowledge Injection
summary: Despite significant advances in large language models (LLMs), their knowledge
  memorization capabilities remain underexplored, due to the lack of standardized
  and high-quality test ground. In this paper, we introduce a novel, real-world
  and large-scale knowledge injection benchmark that evolves continuously over
  time without requiring human intervention. Specifically, we propose WikiDYK,
  which leverages recently-added and human-written facts from Wikipedia's "Did
  You Know..." entries. These entries are carefully selected by expert Wikipedia
  editors based on criteria such as verifiability and clarity. Each entry is
  converted into multiple question-answer pairs spanning diverse task formats
  from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290
  facts and 77,180 questions, which is also seamlessly extensible with future
  updates from Wikipedia editors. Extensive experiments using continued
  pre-training reveal a surprising insight: despite their prevalence in modern
  LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge
  memorization capabilities compared to Bidirectional Language Models (BiLMs),
  exhibiting a 23% lower accuracy in terms of reliability. To compensate for the
  smaller scales of current BiLMs, we introduce a modular collaborative framework
  utilizing ensembles of BiLMs as external knowledge repositories to integrate
  with LLMs. Experiment shows that our framework further improves the reliability
  accuracy by up to 29.1%.
paper_authors._id: 682d2f9f3b5f51f42185dd7d
paper_authors.name: Yuwei Zhang
paper_authors._id: 682d2f9f3b5f51f42185dd7e
paper_authors.name: Wenhao Yu
paper_authors._id: 682d2f9f3b5f51f42185dd7f
paper_authors.name: Shangbin Feng
paper_authors._id: 682d2f9f3b5f51f42185dd80
paper_authors.name: Yifan Zhu
paper_authors._id: 682d2f9f3b5f51f42185dd81
paper_authors.name: Letian Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:21:33.327Z
paper_authors._id: 64323dd503d81fa4d26deaf9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64323dd503d81fa4d26deaf9/x3ES8VXEZJljxDWvFWaAf.png
paper_authors.fullname: Letian Peng
paper_authors.type: user
paper_authors.user: KomeijiForce
paper_authors._id: 682d2f9f3b5f51f42185dd82
paper_authors.name: Jayanth Srinivasa
paper_authors._id: 682d2f9f3b5f51f42185dd83
paper_authors.name: Gaowen Liu
paper_authors._id: 682d2f9f3b5f51f42185dd84
paper_authors.name: Jingbo Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:21:40.883Z
paper_authors._id: 660655119e3555d648f6c6b5
paper_authors.avatarUrl: /avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
paper_authors.fullname: Jingbo Shang
paper_authors.type: user
paper_authors.user: shangjingbo
paper_title: Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for
    Real-world Knowledge Injection
paper_summary: Despite significant advances in large language models (LLMs), their knowledge
  memorization capabilities remain underexplored, due to the lack of standardized
  and high-quality test ground. In this paper, we introduce a novel, real-world
  and large-scale knowledge injection benchmark that evolves continuously over
  time without requiring human intervention. Specifically, we propose WikiDYK,
  which leverages recently-added and human-written facts from Wikipedia's "Did
  You Know..." entries. These entries are carefully selected by expert Wikipedia
  editors based on criteria such as verifiability and clarity. Each entry is
  converted into multiple question-answer pairs spanning diverse task formats
  from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290
  facts and 77,180 questions, which is also seamlessly extensible with future
  updates from Wikipedia editors. Extensive experiments using continued
  pre-training reveal a surprising insight: despite their prevalence in modern
  LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge
  memorization capabilities compared to Bidirectional Language Models (BiLMs),
  exhibiting a 23% lower accuracy in terms of reliability. To compensate for the
  smaller scales of current BiLMs, we introduce a modular collaborative framework
  utilizing ensembles of BiLMs as external knowledge repositories to integrate
  with LLMs. Experiment shows that our framework further improves the reliability
  accuracy by up to 29.1%.
paper_ai_summary: A new knowledge injection benchmark, WikiDYK, highlights differences in knowledge memorization between CLMs and BiLMs, and a modular collaborative framework enhances LLM reliability.
paper_ai_keywords: large language models knowledge injection benchmark Causal Language Models Bidirectional Language Models knowledge memorization WikiDYK modular collaborative framework