MuLan: A Study of Fact Mutability in Language Models
<|reference_start|>MuLan: A Study of Fact Mutability in Language Models: Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of time-contingent knowledge to/from LLMs.<|reference_end|>
@article{fierro2024mulan:,
    title={MuLan: A Study of Fact Mutability in Language Models},
    author={Constanza Fierro, Nicolas Garneau, Emanuele Bugliarello, Yova
  Kementchedjhieva, Anders S{o}gaard},
    journal={arXiv preprint arXiv:2404.03036},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03036},
    primaryClass={cs.CL}
}