Feedforward convolutional neural network has achieved a great success in many computer vision tasks. While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system. In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure. We found that such feedback connections could enable lower layers to "rethink" about their representations given the top-down contextual information. We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classification, partially occluded object recognition and fine-grained image classification. We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.
This paper presents a novel deep learning module for recurrent processes. The general idea and motivation are generally appealing but the experimental validation is a mess. Architectures and hyper-parameters are casually changed from experiment to experiment (please refer to Do CIFAR-10 Classifiers Generalize to CIFAR-10? By Recht et al 2018 to understand why this is a serious problem.) Some key evaluations are missing (see below). Key controls are also lacking. This study is not the first to look into recurrent / feedback processes. Indeed some (but not all) prior work is cited in the introduction. Some of these should be used as baselines as opposed to just feedforward networks. TBut with all that said, even addressing these concerns would not be sufficient for this paper to pass threshold since overall the improvements are relatively modest (e.g., see Fig. 4 right panel where the improvements are a fraction of a % or left panel with a couple % fooling rates improvements) for a module that adds significant computational cost to an architecture runtime. As a side note, I would advise to tone down some of the claims such as "our network could outperform baseline feedforward networks by a large margin”...
****
Additional comments:
The experiments are all over the place. What is the SOA on CIFAR-10 and CIFAR-100? If different from VGG please provide a strong rationale for testing the circuit on VGG and not SOA. In general, the experimental validation would be much stronger if consistent improvements were shown across architectures.
Accuracy is reported for CIFAR-10 and CIFAR-100 for 1 and 2 feedback iterations and presumably with the architecture shown in Fig. 1. Then robustness to noise and adversarial attacks tested on ImageNet and with a modification of the architecture. According to the caption of Fig. 4, this is done with 5 timesteps this time! Accuracy on ImageNet needs to be reported ** especially ** if classification accuracy is not improved (as I expect). 
Then experiments on fine-grained with ResNet-34! What architecture is this? Is this yet another number of loops and feedback iterations? When reporting that "Our model can get a top-1 error of 25.1, while that of the ResNet-34 model is 26.5.” Please provide published accuracy for the baseline algorithm.
For the experiment on occlusions, the authors report using “a multi-recurrent model which is similar to the model mentioned in the Imagenet task”. Sorry but this is not good enough.
Table 4 has literally no explanation. What is FF? What are unroll times?
As a side note, VGG-GAP does not seem to be defined anywhere.
When stating "We investigated VGG16 (Simonyan & Zisserman, 2014), a standard CNN that closely approximate the ventral visual hierarchical stream, and its recurrent variants for comparison.”, the authors probably meant “coarsely” not “closely".