Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition
Inspired by "predictive coding" -a theory in neuroscience, we develop a bidirectional and dynamic neural network with local recurrent processing, namely predictive coding network (PCN). Unlike feedforward-only convolutional neural networks, PCN includes both feedback connections, which carry top-down predictions, and feedforward connections, which carry bottom-up errors of prediction. Feedback and feedforward connections enable adjacent layers to interact locally and recurrently to refine representations towards minimization of layer-wise prediction errors. When unfolded over time, the recurrent processing gives rise to an increasingly deeper hierarchy of non-linear transformation, allowing a shallow network to dynamically extend itself into an arbitrarily deep network. We train and test PCN for image classification with SVHN, CIFAR and ImageNet datasets. Despite notably fewer layers and parameters, PCN achieves competitive performance compared to classical and state-of-the-art models. Further analysis shows that the internal representations in PCN converge over time and yield increasingly better accuracy in object recognition. Errors of top-down prediction also reveal visual saliency or bottom-up attention.
Summary: This paper proposes a method for object recognition based on the predictive coding principle from neuroscience. They use a recurrent neural network (RNN) in each layer computation of a convolutional neural network (CNN). The RNNs performs bottom-up and top-down inference of the features at each layer to minimize layer-wise prediction errors. In experiments, they authors show competitive performance against state-of-the-art recognition methods, and also outperform a baseline based on global predictive coding rather than the local predictive coding approach used here. Additionally, experiments showing analysis of the recurrent cycles within each layer in relation to classification performance.  Pros: Novel local predictive coding based architecture/formulation that improves previous predictive coding based methods Competitive performance in relation to the state-of-the-art recognition models with less parameters. Interesting analysis of the recurrent cycles within the network  Comments: Fewer number of layers argument: Throughout the paper the number of layers is used as argument to highlight the advantage of the proposed method. However, doesn’t the fact that there are RNNs within each layer implicitly add up to the number of layer within the total computation graph? When RNNs are being unrolled through time, we are technically unrolling multi-layer network with inputs at each layer and special gates to move into the next layer of computation. Can the authors give their insight on this?  Similar number of parameters comparison with baselines: In the comparison tables with the state-of-the-art, it would be nice to have comparisons with the same number of parameters. Either reduce the baselines number of parameters or (if possible) increase the number of parameters of the proposed method.  Clarity: In figure 1b, it would be nice if the authors can make it clear that the arrows between the light-gray boxes means the feature is being sent to the next computation block (if I am not mistaken) (e.g., r_l^{T-1} -> r_l^{T-1}).  Overall, I like this paper, however, it would be nice if the authors can address the issues brought up by the reviewer. The use of predictive coding for layers to converge to optimal features to improve the recognition task is attractive.