AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
<|reference_start|>AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models: Large language models (LLMs) often exhibit hallucinations due to incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, a prevailing paradigm is the locating-then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupt the originally preserved knowledge within LLMs, especially in sequential editing scenarios. To address this, we introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, show that AlphaEdit boosts the performance of most locating-then-editing methods by an average of 36.4% with a single line of additional code for projection solely. Our code is available at: https://github.com/jianghoucheng/AlphaEdit.<|reference_end|>
@article{fang2024alphaedit:,
    title={AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models},
    author={Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang,
  Xiangnan He, Tat-seng Chua},
    journal={arXiv preprint arXiv:2410.02355},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02355},
    primaryClass={cs.CL cs.AI}
}