The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes. Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc. A key architectural feature combating such environmental irregularities is long-range horizontal connections that aid the perception of the global form of objects. In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity. We introduce the Texturized Challenge -a new benchmark to evaluate object recognition performance under perceptual noise -which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing. Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images. We also present the emergence of several biologically plausible horizontal connectivity patterns, namely centeron surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500). Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.
The paper proposes, inspired by the Primates brain, to add horizontal inhibitory and excitatory connections. In practice, the work proposes a variant of convolutional LSTM cells, that incorporates additional convolutions. 
Overall, the paper is hard to read, the experimental setting does not fully convince.  It seems hard to reproduce the results using the information given in the paper. A lot of the method section focuses on intuition, using vague vocabulary which makes it hard to understand concretely what is done in practice. In particular, the contributions are not clear enough. The difference between the proposed approach and existing works needs to be made clearer. 
The idea is interesting, however, and the paper would benefit greatly from addressing those issues. 
Main points
The claims seem somewhat bigger than the actual contributions. In practice, the contribution is a modification to an LSTM cell.
The paper is not easy to read, and mixes various terms without introducing them (e.g presynaptic activity is used to introduce the method but never introduced, not even in related work). It would be good to use standard notation and math font (e.g. small bold letter for vectors, etc) and to define notations. In Figure 1, the kronecker symbold represents convolutions but in (1), the hadamard product (*) represents convolutions. These inconsistensies make the paper harder to follow.
The problem of clarity extends to the experimental section, where the experiments are not clearly explained, 
The method section could be more detailed, in particular, a mathematical comparison of LSTM vs the proposed approachh would be useful. Figure 1 is unclear: what do the red, dark blue and light blue lines correspond to mathematically?
The related attempts in ML and Deep learning should be reviewed ( e.g. Lateral Inhibition-Inspired Convolutional Neural Networkfor Visual Attention and Saliency Detection, AAAI 2018).
The experimental setting is not convincing. It seems that a simple state-of-the-art CNN architecture would do better than the proposed approaches. Comparing a ConvLSTM with a single convolution does not seem fair.
Very little implementation details are given. The authors mention a ConvLSTM2D layer is used. However, the inputs are static images: how exactly was the experiment done? Where does the time come from since the dataset considered is static?
About emergent neurally plausible horizontal connections: that section is interesting but would benefit from being more detailed and rigorous. There is no actual measure or study of the emergence of said connections.
3.2 is misleading as it corresponds to a setting that is never used in the experimental setting. 
Most of the results are qualitative, except for Table 1. It would be useful to have quantitative comparisons on established benchmarks.