What Do the Circuits Mean? A Knowledge Edit View
<|reference_start|>What Do the Circuits Mean? A Knowledge Edit View: In the field of language model interpretability, circuit discovery is gaining popularity. Despite this, the true meaning of these circuits remains largely unanswered. We introduce a novel method to learn their meanings as a holistic object through the lens of knowledge editing. We extract circuits in the GPT-2 base model for classification tasks related to syntax and model safety, and study their knowledge property via a model edit dataset containing hierarchical entities. We find that these circuits contain entity knowledge but resist new knowledge, demonstrating a "confirmation bias" behavior. Additionally, we examine the impact of circuit size, discovering that an ideal "theoretical circuit" where essential knowledge is concentrated likely incorporates more than 5% but less than 50% of the model's parameters. We also assess the overlap between circuits from different datasets, finding moderate similarities. We proceed with analyzing the modular components of the circuits, finding that up to 60% of the circuits consist of layer normalization modules rather than attention or MLP modules, adding evidence to the ongoing debates regarding knowledge localization. In summary, our findings offer novel insights into the meanings of the circuits, and introduce directions for further interpretability and safety research of language models.<|reference_end|>
@article{ge2024what,
    title={What Do the Circuits Mean? A Knowledge Edit View},
    author={Huaizhi Ge, Frank Rudzicz, Zining Zhu},
    journal={arXiv preprint arXiv:2406.17241},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17241},
    primaryClass={cs.CL}
}