Interpreting Low-level Vision Models with Causal Effect Maps
<|reference_start|>Interpreting Low-level Vision Models with Causal Effect Maps: Deep neural networks have significantly improved the performance of low-level vision tasks but also increased the difficulty of interpretability. A deep understanding of deep models is beneficial for both network design and practical reliability. To take up this challenge, we introduce causality theory to interpret low-level vision models and propose a model-/task-agnostic method called Causal Effect Map (CEM). With CEM, we can visualize and quantify the input-output relationships on either positive or negative effects. After analyzing various low-level vision tasks with CEM, we have reached several interesting insights, such as: (1) Using more information of input images (e.g., larger receptive field) does NOT always yield positive outcomes. (2) Attempting to incorporate mechanisms with a global receptive field (e.g., channel attention) into image denoising may prove futile. (3) Integrating multiple tasks to train a general model could encourage the network to prioritize local information over global context. Based on the causal effect theory, the proposed diagnostic tool can refresh our common knowledge and bring a deeper understanding of low-level vision models. Codes are available at https://github.com/J-FHu/CEM.<|reference_end|>
@article{hu2024interpreting,
    title={Interpreting Low-level Vision Models with Causal Effect Maps},
    author={Jinfan Hu, Jinjin Gu, Shiyao Yu, Fanghua Yu, Zheyuan Li, Zhiyuan You,
  Chaochao Lu, Chao Dong},
    journal={arXiv preprint arXiv:2407.19789},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19789},
    primaryClass={cs.CV}
}