Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence. This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons. However, the relative contributions of these connections to perceptual grouping are poorly understood. We address this question by systematically evaluating neural network architectures featuring combinations of these connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping. We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up processing. Horizontal connections resolve this limitation on tasks with Gestalt cues by supporting incremental spatial propagation of activities, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object. Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.
This paper proposed a dataset and designed a relevant network structure to analyze the function of horizontal and top-down connections for perceptual grouping. The used two datasets smartly isolate the requirements for exploiting Gestalt cues and object-based strategies. Appendix A detailed describes the cABC dataset, and the control experiments in Appendix B further validate the designing of the cABC. The proposed network flexibly integrates three types of connections and successfully solves both two challenges. The visualization results in Figure 4, S8 and S9 are insightful and also validate the intuitions. Overall the paper is clearly written and easy to follow.
Although the use of different types of connections in the proposed network is clear, I think the author should conduct more analysis for the standard networks and datasets.
In section 3, the author claims that the two challenges will cause a high computational burden for feedforward models like ResNets. However, in Figure 5, the results show that ResNet-152 and ResNet-50 could easily solve the Pathfinder. For cABC, ResNet-18 could solve and ResNet-50 and ResNet-152 will fail while the author illustrates this is due to ResNet-50 and ResNet-152 overfit to cABC. Those phenomena are quite misaligning with the arguments. 
- Can we find a suitable ResNet structure that could solve both two challenges? 
- Can we increase the depth of U-Net and make it solve both two challenges? 
- Can we add more data or increase the difficulty for cABC and prevent the overfitting for ResNet-50 and ResNet-152? 
- Can we keep increasing the difficulties of cABC and straining ResNet-18 and UNet? Does that difficulty level would also strain TD+H-CNN?  
- We could also perform similar experiments on Pathfinder that increasing its difficulty for ResNet-50 and ResNet-152. 
Also, if we train the TD+H-CNN on standard image datasets such as cifar-10 or ImageNet, what about the recurrent activities and the final acc?
=========================================================
After Rebuttal:
I thank the author for the response. 
I hope the comments are useful for preparing a future version of this work when you have enough time.