Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization
<|reference_start|>Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization: To ensure large language models contain up-to-date knowledge, they need to be updated regularly. However, model editing is challenging as it might also affect knowledge that is unrelated to the new data. State-of-the-art methods identify parameters associated with specific knowledge and then modify them via direct weight updates. However, these locate-and-edit methods suffer from heavy computational overhead and lack theoretical validation. In contrast, directly fine-tuning the model on requested edits affects the model's behavior on unrelated knowledge, and significantly damages the model's generation fluency and consistency. To address these challenges, we propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization. Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.<|reference_end|>
@article{wang2024better,
    title={Better Call SAUL: Fluent and Consistent Language Model Editing with
  Generation Regularization},
    author={Mingyang Wang, Lukas Lange, Heike Adel, Jannik Str"otgen, Hinrich
  Sch"utze},
    journal={arXiv preprint arXiv:2410.02433},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02433},
    primaryClass={cs.CL cs.LG}
}