title: Discovering Knowledge Deficiencies of Language Models on Massive
    Knowledge Base
summary: Large language models (LLMs) possess impressive linguistic capabilities but
  often fail to faithfully retain factual knowledge, leading to hallucinations
  and unreliable outputs. Understanding LLMs' knowledge deficiencies by
  exhaustively evaluating against full-scale knowledge bases is computationally
  prohibitive, especially for closed-weight models. We propose stochastic error
  ascent (SEA), a scalable and efficient framework for discovering knowledge
  deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
  than naively probing all knowledge candidates, SEA formulates error discovery
  as a stochastic optimization process: it iteratively retrieves new high-error
  candidates by leveraging the semantic similarity to previously observed
  failures. To further enhance search efficiency and coverage, SEA employs
  hierarchical retrieval across document and paragraph levels, and constructs a
  relation directed acyclic graph to model error propagation and identify
  systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
  than Automated Capability Discovery and 26.7% more than AutoBencher, while
  reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
  confirms the high quality of generated questions, while ablation and
  convergence analyses validate the contribution of each component in SEA.
  Further analysis on the discovered errors reveals correlated failure patterns
  across LLM families and recurring deficits, highlighting the need for better
  data coverage and targeted fine-tuning in future LLM development.
paper_authors._id: 67eb57a56522661171fb4725
paper_authors.name: Linxin Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:45.180Z
paper_authors._id: 64d660308ebc40443813f014
paper_authors.avatarUrl: /avatars/516bb2d2383be99794e366dfb41636b6.svg
paper_authors.fullname: Linxin Song
paper_authors.type: user
paper_authors.user: linxinso
paper_authors._id: 67eb57a56522661171fb4726
paper_authors.name: Xuwei Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T12:26:16.488Z
paper_authors._id: 676f6c0ccb094bb8d8a0948f
paper_authors.avatarUrl: /avatars/78d26c39e1108ee4029639e4202d0ec9.svg
paper_authors.fullname: Xuwei Ding
paper_authors.type: user
paper_authors.user: Xuwei04
paper_authors._id: 67eb57a56522661171fb4727
paper_authors.name: Jieyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T12:26:35.397Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 67eb57a56522661171fb4728
paper_authors.name: Taiwei Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:46:48.349Z
paper_authors._id: 62e1b3cb3eb0730f621a83f6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
paper_authors.fullname: Taiwei Shi
paper_authors.type: user
paper_authors.user: MaksimSTW
paper_authors._id: 67eb57a56522661171fb4729
paper_authors.name: Ryotaro Shimizu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T12:26:41.560Z
paper_authors._id: 65641c8ec683336b76fecb7f
paper_authors.avatarUrl: /avatars/3dcf51fc55f7e5ce2c9e84eaee270277.svg
paper_authors.fullname: Ryotaro Shimizu
paper_authors.type: user
paper_authors.user: latataro
paper_authors._id: 67eb57a56522661171fb472a
paper_authors.name: Rahul Gupta
paper_authors._id: 67eb57a56522661171fb472b
paper_authors.name: Yang Liu
paper_authors._id: 67eb57a56522661171fb472c
paper_authors.name: Jian Kang
paper_authors._id: 67eb57a56522661171fb472d
paper_authors.name: Jieyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T12:27:18.161Z
paper_authors._id: 64ba0230c9a9894feb0cd32b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ba0230c9a9894feb0cd32b/V1XWySRXAhLN4wqXZJ3Ai.jpeg
paper_authors.fullname: Jieyu Zhao
paper_authors.type: user
paper_authors.user: jieyuz
paper_title: Discovering Knowledge Deficiencies of Language Models on Massive
    Knowledge Base
paper_summary: Large language models (LLMs) possess impressive linguistic capabilities but
  often fail to faithfully retain factual knowledge, leading to hallucinations
  and unreliable outputs. Understanding LLMs' knowledge deficiencies by
  exhaustively evaluating against full-scale knowledge bases is computationally
  prohibitive, especially for closed-weight models. We propose stochastic error
  ascent (SEA), a scalable and efficient framework for discovering knowledge
  deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
  than naively probing all knowledge candidates, SEA formulates error discovery
  as a stochastic optimization process: it iteratively retrieves new high-error
  candidates by leveraging the semantic similarity to previously observed
  failures. To further enhance search efficiency and coverage, SEA employs
  hierarchical retrieval across document and paragraph levels, and constructs a
  relation directed acyclic graph to model error propagation and identify
  systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
  than Automated Capability Discovery and 26.7% more than AutoBencher, while
  reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
  confirms the high quality of generated questions, while ablation and
  convergence analyses validate the contribution of each component in SEA.
  Further analysis on the discovered errors reveals correlated failure patterns
  across LLM families and recurring deficits, highlighting the need for better
  data coverage and targeted fine-tuning in future LLM development.
paper_ai_summary: Stochastic Error Ascent (SEA) is a scalable framework that identifies knowledge errors in large language models using semantic similarity and hierarchical retrieval, uncovering more errors with lower costs compared to existing methods.
paper_ai_keywords: stochastic error ascent SEA knowledge deficiencies closed-weight LLMs semantic similarity hierarchical retrieval document level paragraph level relation directed acyclic graph error propagation systematic failure modes