AKEW: Assessing Knowledge Editing in the Wild
<|reference_start|>AKEW: Assessing Knowledge Editing in the Wild: Knowledge editing injects knowledge updates into language models to keep them correct and up-to-date. However, its current evaluations deviate significantly from practice: their knowledge updates solely consist of structured facts derived from meticulously crafted datasets, instead of practical sources -- unstructured texts like news articles, and they often overlook practical real-world knowledge updates. To address these issues, in this paper we propose AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing. AKEW fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets. It further introduces new datasets featuring both counterfactual and real-world knowledge updates. Through extensive experiments, we demonstrate the considerable gap between state-of-the-art knowledge-editing methods and practical scenarios. Our analyses further highlight key insights to motivate future research for practical knowledge editing.<|reference_end|>
@article{wu2024akew:,
    title={AKEW: Assessing Knowledge Editing in the Wild},
    author={Xiaobao Wu, Liangming Pan, William Yang Wang, Anh Tuan Luu},
    journal={arXiv preprint arXiv:2402.18909},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18909},
    primaryClass={cs.CL cs.AI}
}