DeeperForward: Enhanced Forward-Forward Training for Deeper and Better Performance
Liang Sun Yang Zhang Weizhao He Jiajun Wen Linlin Shen Weicheng Xie
While backpropagation effectively trains models, it presents challenges related to bio-plausibility, resulting in high memory demands and limited parallelism. Recently, Hinton (2022) proposed the Forward-Forward (FF) algorithm for high-parallel local updates. FF leverages squared sums as the local update target, termed goodness, and employs L2 normalization to decouple goodness and extract new features. However, this design encounters issues with feature scaling and deactivated neurons, limiting its application mainly to shallow networks. This paper proposes a novel goodness design utilizing **layer normalization** and **mean goodness** to overcome these challenges, demonstrating performance improvements even in 17-layer CNNs. Experiments on CIFAR-10, MNIST, and Fashion-MNIST show significant advantages over existing FF-based algorithms, highlighting the potential of FF in deep models. Furthermore, the model parallel strategy is proposed to achieve highly efficient training based on the property of local updates.
backpropagation-free, forward-forward, forward learning, brain-inspired computing
We propose DeeperForward which extends FF to CNNs up to 17 layers using a new design of goodness, improving performance with layer depth and achieving the best performance among recent FF methods.