DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs
<|reference_start|>DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs: LLMs acquire knowledge from massive data snapshots collected at different timestamps. Their knowledge is then commonly evaluated using static benchmarks. However, factual knowledge is generally subject to time-sensitive changes, and static benchmarks cannot address those cases. We present an approach to dynamically evaluate the knowledge in LLMs and their time-sensitiveness against Wikidata, a publicly available up-to-date knowledge graph. We evaluate the time-sensitive knowledge in twenty-four private and open-source LLMs, as well as the effectiveness of four editing methods in updating the outdated facts. Our results show that 1) outdatedness is a critical problem across state-of-the-art LLMs; 2) LLMs output inconsistent answers when prompted with slight variations of the question prompt; and 3) the performance of the state-of-the-art knowledge editing algorithms is very limited, as they can not reduce the cases of outdatedness and output inconsistency.<|reference_end|>
@article{mousavi2024dyknow:,
    title={DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs},
    author={Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi},
    journal={arXiv preprint arXiv:2404.08700},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.08700},
    primaryClass={cs.CL cs.AI}
}