Cortical microcircuits as gated-recurrent neural networks
Cortical circuits exhibit intricate recurrent architectures that are remarkably similar across different brain areas. Such stereotyped structure suggests the existence of common computational principles. However, such principles have remained largely elusive. Inspired by gated-memory networks, namely long short-term memory networks (LSTMs), we introduce a recurrent neural network in which information is gated through inhibitory cells that are subtractive (subLSTM). We propose a natural mapping of subLSTMs onto known canonical excitatory-inhibitory cortical microcircuits. Our empirical evaluation across sequential image classification and language modelling tasks shows that subLSTM units can achieve similar performance to LSTM units. These results suggest that cortical circuits can be optimised to solve complex contextual problems and proposes a novel view on their computational function. Overall our work provides a step towards unifying recurrent networks as used in machine learning with their biological counterparts.
A new recurrent neural network model is presented. It has similar functional features as LSTMs
			but additive gating instead of multiplicative gating for the input and output gates.
			With this additive gating mechanism there are some striking similarities to cortical circuits.
			In my opinion, this papers could be really interesting for both computational neuroscientists and machine learners.
			For small network sizes the proposed model performs as good or better than simple LSTMs on some non-trivial tasks.
			It is, however, somewhat disappointing that For the language modelling task it seems, however, that the multiplicative forget gates are still needed for good performance in larger networks.
			Further comments and questions:
			1. I guess the sigma-subRNN have also controlled forget gates (option i on line 104). 
			   If yes, it would be fair to highlight this fact more clearly in the results/discussion section.
			   It is somewhat disappointing that this not so biologically plausible feature 
			   seems to be needed to achieve good performances on the language modelling tasks.
			2. Would it be possible to implement the controlled forget gate also with a subtractive gate.
			   Maybe something like c_t = relu(c_{t-1} - f_t) + (z_t - i_t), 
			   which could be interpreted as a strong inhibition (f_t) reseting the reverberating loop before new input (z_t - i_t) arrives.
			   But maybe the sign of c_t is important....
			3. line 111: Why is it important to have rectified linear transfer functions?
			   Doesn't sigma-subRNN perform better in some cases? 
			   From the perspective of biological plausibility the transfer function should have the range [0, f_max]
			   where f_max is the maximal firing rate of a neuron (see e.g. the transfer function of a LIF neuron with absolute refractoriness).
			4. line 60 "inhibitory neurons act subtractively": I think one should acknowledge that
			   the picture is not so simple in biological systems (e.g. Chance et al. 2002 https://doi.org/10.1016/S0896-6273(02)00820-6 or 
			   Müllner et al. 2015 https://doi.org/10.1016/j.neuron.2015.07.003).
			5. In my opinion it is not needed to explicitely state the well known Gershgorin circle theorem in the main text.
			6. Fig 3f: BGRNN? Should this be subRNN?
I read through the author’s rebuttal and my colleagues’ review, and decided to not change my assessment of the paper.