title: Estimating Knowledge in Large Language Models Without Generating a
    Single Token
summary: To evaluate knowledge in large language models (LLMs), current methods query
  the model and then evaluate its generated responses. In this work, we ask
  whether evaluation can be done before the model has generated any
  text. Concretely, is it possible to estimate how knowledgeable a model is about
  a certain entity, only from its internal computation? We study this question
  with two tasks: given a subject entity, the goal is to predict (a) the ability
  of the model to answer common questions about the entity, and (b) the
  factuality of responses generated by the model about the entity. Experiments
  with a variety of LLMs show that KEEN, a simple probe trained over internal
  subject representations, succeeds at both tasks - strongly correlating with
  both the QA accuracy of the model per-subject and FActScore, a recent
  factuality metric in open-ended generation. Moreover, KEEN naturally aligns
  with the model's hedging behavior and faithfully reflects changes in the
  model's knowledge after fine-tuning. Lastly, we show a more interpretable yet
  equally performant variant of KEEN, which highlights a small set of tokens that
  correlates with the model's lack of knowledge. Being simple and lightweight,
  KEEN can be leveraged to identify gaps and clusters of entity knowledge in
  LLMs, and guide decisions such as augmenting queries with retrieval.
paper_authors._id: 6672f26f5f184b6936630454
paper_authors.name: Daniela Gottesman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:18:25.138Z
paper_authors._id: 637f37b5b3cb8158e1725697
paper_authors.avatarUrl: /avatars/18780d63b12aa8a5171130d09e214b25.svg
paper_authors.fullname: Daniela
paper_authors.type: user
paper_authors.user: dhgottesman
paper_authors._id: 6672f26f5f184b6936630455
paper_authors.name: Mor Geva
paper_title: Estimating Knowledge in Large Language Models Without Generating a
    Single Token
paper_summary: To evaluate knowledge in large language models (LLMs), current methods query
  the model and then evaluate its generated responses. In this work, we ask
  whether evaluation can be done before the model has generated any
  text. Concretely, is it possible to estimate how knowledgeable a model is about
  a certain entity, only from its internal computation? We study this question
  with two tasks: given a subject entity, the goal is to predict (a) the ability
  of the model to answer common questions about the entity, and (b) the
  factuality of responses generated by the model about the entity. Experiments
  with a variety of LLMs show that KEEN, a simple probe trained over internal
  subject representations, succeeds at both tasks - strongly correlating with
  both the QA accuracy of the model per-subject and FActScore, a recent
  factuality metric in open-ended generation. Moreover, KEEN naturally aligns
  with the model's hedging behavior and faithfully reflects changes in the
  model's knowledge after fine-tuning. Lastly, we show a more interpretable yet
  equally performant variant of KEEN, which highlights a small set of tokens that
  correlates with the model's lack of knowledge. Being simple and lightweight,
  KEEN can be leveraged to identify gaps and clusters of entity knowledge in
  LLMs, and guide decisions such as augmenting queries with retrieval.
paper_ai_keywords: subject representations KEEN QA accuracy FActScore hedging behavior fine-tuning tokens interpretability