BACKPROPAMINE: TRAINING SELF-MODIFYING NEU- RAL NETWORKS WITH DIFFERENTIABLE NEUROMODU- LATED PLASTICITY
The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.
This work presents Backpropamine, a neuromodulated plastic LSTM training regime. It extends previous research on differentiable Hebbian plasticity by introducing a neuromodulatory term to help gate information into the Hebbian synapse. The neuromodulatory term is placed under network control, allowing it to be time varying (and hence to be sensitive to the input, for example). Another variant proposes updating the Hebbian synapse with modulated exponential average of the Hebbian product. This average is linked to the notion of an eligibility trace, and ties into some recent biological work that shows the role of dopamine in retroactively modulating synaptic plasticity.
Overall the work is nicely motivated and clearly presented. There are some interesting ties to biological work -- in particular, to retroactive plasticity phenomena. There should be sufficient details for a reader to implement this model, thought there are some minor details missing regarding the experimental setup, which will be addressed below.
The authors test their model on three tasks: cue-award association, maze learning, and Penn Treebank (PTB). In the cue-award association task the retroactive and simple modulation networks perform well, while the non-modulated and non-plastics fail. For the maze navigation task the modulated networks perform better than the non-modulated networks, though the effect is less pronounced. Finally, on PTB the authors report improvements over baseline LSTMs. 
One of the main claims of this paper is that neuromodulated plastic LSTMs...outperform standard LSTMs on a benchmark language modeling task”, and that therefore “differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks”. This claim is unfortunately unfounded for a very important reason: the LSTM performance is not at all close to that which can be achieved by LSTMs in general. The authors cite such models in the appendix (Melor et al), but claim that “much larger models” are needed, potentially with other mechanisms, such as dropout. Though this may be true, these models still undermine the claim that “neuromodulated plastic LSTMs...outperform standard LSTMs on a benchmark language modeling task”. This claim is simply not true, and more care is needed in reporting the results here in the wider context of the literature. Also, I am left wondering what are considered the parameters of the models -- are only the neuromodulatory terms considered as the additional trainable parameters compared to baseline LSTMs? How are the Hebbian synapses themselves considered in this calculation? If the Hebbian synapses are not considered, then the authors need a control with matched memory-capacities to account for the extra capacity afforded by the Hebbian synapses. Given the ties between Hebbian synapses and attention (see Ba et al), an important control here could be an LSTM with Bahdanau (2014) style attention. 
Finally, the style (font) of the paper does not adhere to the ICLR style template, and must be changed.
Overall, the ideas presented in the paper are intriguing, and further research down this line is encouraged. However, in its current state the work lacks sufficiently strong baselines to support the paper’s claims; thus, the merits of this approach cannot yet be properly assessed.