title: Inside-Out: Hidden Factual Knowledge in LLMs
summary: This work presents a framework for assessing whether large language models
  (LLMs) encode more factual knowledge in their parameters than what they express
  in their outputs. While a few studies hint at this possibility, none has
  clearly defined or demonstrated this phenomenon. We first propose a formal
  definition of knowledge, quantifying it for a given question as the fraction of
  correct-incorrect answer pairs where the correct one is ranked higher. This
  gives rise to external and internal knowledge, depending on the information
  used to score individual answer candidates: either the model's observable
  token-level probabilities or its intermediate computations. Hidden knowledge
  arises when internal knowledge exceeds external knowledge. We then present a
  case study, applying this framework to three popular open-weights LLMs in a
  closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
  more factual knowledge internally than what they express externally, with an
  average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
  model can internally know an answer perfectly, yet fail to generate it even
  once, despite large-scale repeated sampling of 1,000 answers. This reveals
  fundamental limitations in the generation capabilities of LLMs, which (3) puts
  a practical constraint on scaling test-time compute via repeated answer
  sampling in closed-book QA: significant performance improvements remain
  inaccessible because some answers are practically never sampled, yet if they
  were, we would be guaranteed to rank them first.
paper_authors._id: 67dd5e658dc9e947cee90f4a
paper_authors.name: Zorik Gekhman
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-21T13:57:32.354Z
paper_authors._id: 61499a8f8517f77f09a3a990
paper_authors.avatarUrl: /avatars/fa15d117e8ce8c50669455d3e483408b.svg
paper_authors.fullname: Zorik
paper_authors.type: user
paper_authors.user: zorik
paper_authors._id: 67dd5e658dc9e947cee90f4b
paper_authors.name: Eyal Ben David
paper_authors._id: 67dd5e658dc9e947cee90f4c
paper_authors.name: Hadas Orgad
paper_authors._id: 67dd5e658dc9e947cee90f4d
paper_authors.name: Eran Ofek
paper_authors._id: 67dd5e658dc9e947cee90f4e
paper_authors.name: Yonatan Belinkov
paper_authors._id: 67dd5e658dc9e947cee90f4f
paper_authors.name: Idan Szpector
paper_authors._id: 67dd5e658dc9e947cee90f50
paper_authors.name: Jonathan Herzig
paper_authors._id: 67dd5e658dc9e947cee90f51
paper_authors.name: Roi Reichart
paper_title: Inside-Out: Hidden Factual Knowledge in LLMs
paper_summary: This work presents a framework for assessing whether large language models
  (LLMs) encode more factual knowledge in their parameters than what they express
  in their outputs. While a few studies hint at this possibility, none has
  clearly defined or demonstrated this phenomenon. We first propose a formal
  definition of knowledge, quantifying it for a given question as the fraction of
  correct-incorrect answer pairs where the correct one is ranked higher. This
  gives rise to external and internal knowledge, depending on the information
  used to score individual answer candidates: either the model's observable
  token-level probabilities or its intermediate computations. Hidden knowledge
  arises when internal knowledge exceeds external knowledge. We then present a
  case study, applying this framework to three popular open-weights LLMs in a
  closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
  more factual knowledge internally than what they express externally, with an
  average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
  model can internally know an answer perfectly, yet fail to generate it even
  once, despite large-scale repeated sampling of 1,000 answers. This reveals
  fundamental limitations in the generation capabilities of LLMs, which (3) puts
  a practical constraint on scaling test-time compute via repeated answer
  sampling in closed-book QA: significant performance improvements remain
  inaccessible because some answers are practically never sampled, yet if they
  were, we would be guaranteed to rank them first.
paper_ai_summary: LLMs encode more internal factual knowledge than they express externally, with some knowledge so deeply hidden that it is never generated, despite repeated sampling.
paper_ai_keywords: large language models LLMs knowledge token-level probabilities intermediate computations external knowledge internal knowledge hidden knowledge closed-book QA generation capabilities