Estimating Knowledge in Large Language Models Without Generating a
  Single Token
To evaluate knowledge in large language models (LLMs), current methods query
the model and then evaluate its generated responses. In this work, we ask
whether evaluation can be done before the model has generated any
text. Concretely, is it possible to estimate how knowledgeable a model is about
a certain entity, only from its internal computation? We study this question
with two tasks: given a subject entity, the goal is to predict (a) the ability
of the model to answer common questions about the entity, and (b) the
factuality of responses generated by the model about the entity. Experiments
with a variety of LLMs show that KEEN, a simple probe trained over internal
subject representations, succeeds at both tasks - strongly correlating with
both the QA accuracy of the model per-subject and FActScore, a recent
factuality metric in open-ended generation. Moreover, KEEN naturally aligns
with the model's hedging behavior and faithfully reflects changes in the
model's knowledge after fine-tuning. Lastly, we show a more interpretable yet
equally performant variant of KEEN, which highlights a small set of tokens that
correlates with the model's lack of knowledge. Being simple and lightweight,
KEEN can be leveraged to identify gaps and clusters of entity knowledge in
LLMs, and guide decisions such as augmenting queries with retrieval.
6672f26f5f184b6936630454
Daniela Gottesman
claimed_verified
2024-06-19T20:18:25.138Z
637f37b5b3cb8158e1725697
/avatars/18780d63b12aa8a5171130d09e214b25.svg
Daniela
user
dhgottesman
6672f26f5f184b6936630455
Mor Geva
Estimating Knowledge in Large Language Models Without Generating a
  Single Token
To evaluate knowledge in large language models (LLMs), current methods query
the model and then evaluate its generated responses. In this work, we ask
whether evaluation can be done before the model has generated any
text. Concretely, is it possible to estimate how knowledgeable a model is about
a certain entity, only from its internal computation? We study this question
with two tasks: given a subject entity, the goal is to predict (a) the ability
of the model to answer common questions about the entity, and (b) the
factuality of responses generated by the model about the entity. Experiments
with a variety of LLMs show that KEEN, a simple probe trained over internal
subject representations, succeeds at both tasks - strongly correlating with
both the QA accuracy of the model per-subject and FActScore, a recent
factuality metric in open-ended generation. Moreover, KEEN naturally aligns
with the model's hedging behavior and faithfully reflects changes in the
model's knowledge after fine-tuning. Lastly, we show a more interpretable yet
equally performant variant of KEEN, which highlights a small set of tokens that
correlates with the model's lack of knowledge. Being simple and lightweight,
KEEN can be leveraged to identify gaps and clusters of entity knowledge in
LLMs, and guide decisions such as augmenting queries with retrieval.
KEEN, a simple probe of internal subject representations, predicts an LLM's ability to answer questions and generate factual responses about an entity before text generation.
LLMs KEEN internal subject representations QA accuracy FActScore hedging behavior fine-tuning entity knowledge augmenting queries with retrieval