Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation
<|reference_start|>Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation: Relieving the reliance of neural network training on a global back-propagation (BP) has emerged as a notable research topic due to the biological implausibility and huge memory consumption caused by BP. Among the existing solutions, local learning optimizes gradient-isolated modules of a neural network with local errors and has been proved to be effective even on large-scale datasets. However, the reconciliation among local errors has never been investigated. In this paper, we first theoretically study non-greedy layer-wise training and show that the convergence cannot be assured when the local gradient in a module w.r.t. its input is not reconciled with the local gradient in the previous module w.r.t. its output. Inspired by the theoretical result, we further propose a local training strategy that successively regularizes the gradient reconciliation between neighboring modules without breaking gradient isolation or introducing any learnable parameters. Our method can be integrated into both local-BP and BP-free settings. In experiments, we achieve significant performance improvements compared to previous methods. Particularly, our method for CNN and Transformer architectures on ImageNet is able to attain a competitive performance with global BP, saving more than 40% memory consumption.<|reference_end|>
@article{yang2024towards,
    title={Towards Interpretable Deep Local Learning with Successive Gradient
  Reconciliation},
    author={Yibo Yang, Xiaojie Li, Motasem Alfarra, Hasan Hammoud, Adel Bibi,
  Philip Torr, Bernard Ghanem},
    journal={arXiv preprint arXiv:2406.05222},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05222},
    primaryClass={cs.LG cs.NE}
}