We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons. Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information. A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones. Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns. In this manner, we enable the slope of the activation function to be context dependent. This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.
Summary: this submission proposes a modification of neural network architectures that allows the modulation of activation functions of a given layer as a function of the activations in the previous layer. The author provide different version of their approach adapted to CNN, DenseNets and LSTM, and show it outperforms a vanilla version of these algorithms.
Evaluation: In the classical context of supervised learning tasks investigated in this submission, it is unclear to me what could be the benefit of introducing such “modulators”, as vanilla ANNs already have the capability of modulating the excitability of their neurons. Although the results show significant, but quite limited, improvements with respect to the chosen baseline, more extensive baseline comparisons are needed.
Details comments:
1.	Underlying principles of the approach
It is unclear to me why the proposed approach should bring a significant improvement to the existing architectures. First, from a neuroscientific perspective, neuromodulators allow the brain to go through different states, including arousal, sleep, and different levels of stress. While it is relatively clear that state modulation has some benefits to a living system, it is less so for an ANN focused on a supervised learning task. Why should the state change instead of focusing on the optimal way to perform the task? If the authors want to use a neuroscientific argument, I would suggest to elaborate based on the precise context of the tasks they propose to solve. 
In addition, as mentioned several times in the paper, neuromodulation is frequently associated to changes in cell excitability. While excitability is a concept that can be associated to multiple mechanisms, a simple way to model changes in excitability is to modify the threshold that must be reached by the membrane potential of a given neuron in order for the cell to fire. Such simple change in excitability can be easily implemented in ANNs architectures by affecting one afferent neuron in the previous layer to the modification of this firing threshold (simply adding a bias term). As a consequence, if there is any benefit to the proposed architecture, it is very likely to originate specifically from the multiplicative interactions used to implement modulation in this paper. However, approximation of such multiplicative interactions can also be implemented using multiple layers network equipped with non-linear activations. Overall, it would be good to discuss these aspects in great detail in the introduction and/or discussion of the paper, and possibly find a more convincing justification for the approach.
2.	Weak baseline comparison results
In the CNN experiments, modulated networks are only compared with a single vanilla counterpart equipped with ReLu. There are at least two obvious additional baseline comparison that would be useful: what if the Re-Lu activations are replaced with fixed sigmoids? And what if batch-normalization is switched on/off (I could not find whether it was used at all). Indeed it, we should exclude benefits that are simply due to the non-linearity of the sigmoid, and batch normalization also implements a form of modulation at training that may provide benefits equivalent to modulation (or on the contrary, batch norm could implement a modulation in the wrong way). It would be better to look at all possible combinations of these architecture choices.
Due to lack of details in the paper and my personal lack of expertise in LSTMs, I will not comment on baselines for that part but I assume similar modifications can be done.
Overall, given the weak improvements in performance, it is questionable whether this extra degree of complexity should be added to the architecture. Additionally, I could not find the precise description of the statistical tests performed. Ideally, the test, the number of samples, the exact p-value, and whether the method of correction for multiple comparison should be included each time a p-value is mentioned.