title: ChroKnowledge: Unveiling Chronological Knowledge of Language Models in
    Multiple Domains
summary: Large language models (LLMs) have significantly impacted many aspects of our
  lives. However, assessing and ensuring their chronological knowledge remains
  challenging. Existing approaches fall short in addressing the accumulative
  nature of knowledge, often relying on a single time stamp. To overcome this, we
  introduce ChroKnowBench, a benchmark dataset designed to evaluate
  chronologically accumulated knowledge across three key aspects: multiple
  domains, time dependency, temporal state. Our benchmark distinguishes between
  knowledge that evolves (e.g., scientific discoveries, amended laws) and
  knowledge that remain constant (e.g., mathematical truths, commonsense facts).
  Building on this benchmark, we present ChroKnowledge (Chronological
  Categorization of Knowledge), a novel sampling-based framework for evaluating
  and updating LLMs' non-parametric chronological knowledge. Our evaluation
  shows: (1) The ability of eliciting temporal knowledge varies depending on the
  data format that model was trained on. (2) LLMs partially recall knowledge or
  show a cut-off at temporal boundaries rather than recalling all aspects of
  knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting
  to elicit chronological knowledge by traversing step-by-step through the
  surrounding time spans. We observe that our framework successfully updates the
  overall knowledge across the entire timeline in both the biomedical domain
  (+11.9%) and the general domain (+2.8%), demonstrating its effectiveness in
  refining temporal knowledge. This non-parametric approach also enables
  knowledge updates not only in open-source models but also in proprietary LLMs,
  ensuring comprehensive applicability across model types. We perform a
  comprehensive analysis based on temporal characteristics of ChroKnowPrompt and
  validate the potential of various models to elicit intrinsic temporal knowledge
  through our method.
paper_authors._id: 670df14b4968ab91d81a72e4
paper_authors.name: Yein Park
paper_authors._id: 670df14b4968ab91d81a72e5
paper_authors.name: Chanwoong Yoon
paper_authors._id: 670df14b4968ab91d81a72e6
paper_authors.name: Jungwoo Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:25:26.432Z
paper_authors._id: 60f8435644e75317cc02ed51
paper_authors.avatarUrl: /avatars/68b7fc077fe2bda6607b1c470add8140.svg
paper_authors.fullname: Jungwoo Park
paper_authors.type: user
paper_authors.user: affjljoo3581
paper_authors._id: 670df14b4968ab91d81a72e7
paper_authors.name: Donghyeon Lee
paper_authors._id: 670df14b4968ab91d81a72e8
paper_authors.name: Minbyul Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:26:59.904Z
paper_authors._id: 64587be872b60ae7a3817858
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64587be872b60ae7a3817858/BbdOOxOCEzWTvEpkWp8MM.png
paper_authors.fullname: Minbyul Jeong
paper_authors.type: user
paper_authors.user: Minbyul
paper_authors._id: 670df14b4968ab91d81a72e9
paper_authors.name: Jaewoo Kang
paper_title: ChroKnowledge: Unveiling Chronological Knowledge of Language Models in
    Multiple Domains
paper_summary: Large language models (LLMs) have significantly impacted many aspects of our
  lives. However, assessing and ensuring their chronological knowledge remains
  challenging. Existing approaches fall short in addressing the accumulative
  nature of knowledge, often relying on a single time stamp. To overcome this, we
  introduce ChroKnowBench, a benchmark dataset designed to evaluate
  chronologically accumulated knowledge across three key aspects: multiple
  domains, time dependency, temporal state. Our benchmark distinguishes between
  knowledge that evolves (e.g., scientific discoveries, amended laws) and
  knowledge that remain constant (e.g., mathematical truths, commonsense facts).
  Building on this benchmark, we present ChroKnowledge (Chronological
  Categorization of Knowledge), a novel sampling-based framework for evaluating
  and updating LLMs' non-parametric chronological knowledge. Our evaluation
  shows: (1) The ability of eliciting temporal knowledge varies depending on the
  data format that model was trained on. (2) LLMs partially recall knowledge or
  show a cut-off at temporal boundaries rather than recalling all aspects of
  knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting
  to elicit chronological knowledge by traversing step-by-step through the
  surrounding time spans. We observe that our framework successfully updates the
  overall knowledge across the entire timeline in both the biomedical domain
  (+11.9%) and the general domain (+2.8%), demonstrating its effectiveness in
  refining temporal knowledge. This non-parametric approach also enables
  knowledge updates not only in open-source models but also in proprietary LLMs,
  ensuring comprehensive applicability across model types. We perform a
  comprehensive analysis based on temporal characteristics of ChroKnowPrompt and
  validate the potential of various models to elicit intrinsic temporal knowledge
  through our method.
paper_ai_summary: ChroKnowBench and ChroKnowledge address the challenges of chronological knowledge in large language models by introducing a benchmark dataset and sampling-based framework, respectively, to evaluate and update temporal knowledge effectively.
paper_ai_keywords: ChroKnowBench ChroKnowledge chronological knowledge benchmark dataset sampling-based framework temporal knowledge biomedical domain general domain ChroKnowPrompt temporal characteristics