ChroKnowledge: Unveiling Chronological Knowledge of Language Models in
  Multiple Domains
Large language models (LLMs) have significantly impacted many aspects of our
lives. However, assessing and ensuring their chronological knowledge remains
challenging. Existing approaches fall short in addressing the accumulative
nature of knowledge, often relying on a single time stamp. To overcome this, we
introduce ChroKnowBench, a benchmark dataset designed to evaluate
chronologically accumulated knowledge across three key aspects: multiple
domains, time dependency, temporal state. Our benchmark distinguishes between
knowledge that evolves (e.g., scientific discoveries, amended laws) and
knowledge that remain constant (e.g., mathematical truths, commonsense facts).
Building on this benchmark, we present ChroKnowledge (Chronological
Categorization of Knowledge), a novel sampling-based framework for evaluating
and updating LLMs' non-parametric chronological knowledge. Our evaluation
shows: (1) The ability of eliciting temporal knowledge varies depending on the
data format that model was trained on. (2) LLMs partially recall knowledge or
show a cut-off at temporal boundaries rather than recalling all aspects of
knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting
to elicit chronological knowledge by traversing step-by-step through the
surrounding time spans. We observe that our framework successfully updates the
overall knowledge across the entire timeline in both the biomedical domain
(+11.9%) and the general domain (+2.8%), demonstrating its effectiveness in
refining temporal knowledge. This non-parametric approach also enables
knowledge updates not only in open-source models but also in proprietary LLMs,
ensuring comprehensive applicability across model types. We perform a
comprehensive analysis based on temporal characteristics of ChroKnowPrompt and
validate the potential of various models to elicit intrinsic temporal knowledge
through our method.
670df14b4968ab91d81a72e4
Yein Park
670df14b4968ab91d81a72e5
Chanwoong Yoon
670df14b4968ab91d81a72e6
Jungwoo Park
claimed_verified
2024-10-18T08:25:26.432Z
60f8435644e75317cc02ed51
/avatars/68b7fc077fe2bda6607b1c470add8140.svg
Jungwoo Park
user
affjljoo3581
670df14b4968ab91d81a72e7
Donghyeon Lee
670df14b4968ab91d81a72e8
Minbyul Jeong
claimed_verified
2024-10-17T11:26:59.904Z
64587be872b60ae7a3817858
https://cdn-avatars.huggingface.co/v1/production/uploads/64587be872b60ae7a3817858/BbdOOxOCEzWTvEpkWp8MM.png
Minbyul Jeong
user
Minbyul
670df14b4968ab91d81a72e9
Jaewoo Kang
ChroKnowledge: Unveiling Chronological Knowledge of Language Models in
  Multiple Domains
Large language models (LLMs) have significantly impacted many aspects of our
lives. However, assessing and ensuring their chronological knowledge remains
challenging. Existing approaches fall short in addressing the accumulative
nature of knowledge, often relying on a single time stamp. To overcome this, we
introduce ChroKnowBench, a benchmark dataset designed to evaluate
chronologically accumulated knowledge across three key aspects: multiple
domains, time dependency, temporal state. Our benchmark distinguishes between
knowledge that evolves (e.g., scientific discoveries, amended laws) and
knowledge that remain constant (e.g., mathematical truths, commonsense facts).
Building on this benchmark, we present ChroKnowledge (Chronological
Categorization of Knowledge), a novel sampling-based framework for evaluating
and updating LLMs' non-parametric chronological knowledge. Our evaluation
shows: (1) The ability of eliciting temporal knowledge varies depending on the
data format that model was trained on. (2) LLMs partially recall knowledge or
show a cut-off at temporal boundaries rather than recalling all aspects of
knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting
to elicit chronological knowledge by traversing step-by-step through the
surrounding time spans. We observe that our framework successfully updates the
overall knowledge across the entire timeline in both the biomedical domain
(+11.9%) and the general domain (+2.8%), demonstrating its effectiveness in
refining temporal knowledge. This non-parametric approach also enables
knowledge updates not only in open-source models but also in proprietary LLMs,
ensuring comprehensive applicability across model types. We perform a
comprehensive analysis based on temporal characteristics of ChroKnowPrompt and
validate the potential of various models to elicit intrinsic temporal knowledge
through our method.
ChroKnowBench and ChroKnowledge address the challenges of chronological knowledge in large language models by introducing a benchmark dataset and sampling-based framework, respectively, to evaluate and update temporal knowledge effectively.
ChroKnowBench ChroKnowledge chronological knowledge benchmark dataset sampling-based framework temporal knowledge biomedical domain general domain ChroKnowPrompt temporal characteristics