MODULAR CONTINUAL LEARNING IN A UNIFIED VISUAL ENVIRONMENT
A core aspect of human intelligence is the ability to learn new tasks quickly and switch between them flexibly. Here, we describe a modular continual reinforcement learning paradigm inspired by these abilities. We first introduce a visual interaction environment that allows many types of tasks to be unified in a single framework. We then describe a reward map prediction scheme that learns new tasks robustly in the very large state and action spaces required by such an environment. We investigate how properties of module architecture influence efficiency of task learning, showing that a module motif incorporating specific design principles (e.g. early bottlenecks, low-order polynomial nonlinearities, and symmetry) significantly outperforms more standard neural network motifs, needing fewer training examples and fewer neurons to achieve high levels of performance. Finally, we present a meta-controller architecture for task switching based on a dynamic neural voting scheme, which allows new modules to use information learned from previouslyseen tasks to substantially improve their own learning efficiency.
Reading this paper feels like reading at least two closely-related papers compressed into one, with overflow into the appendix (e.g. one about the EMS module, one about the the recurrent voting, etc).
There were so many aspects/components, that I am not entirely confident I fully understood how they all work together, and in fact I am pretty confident there was at least some part of this that I definitely did not understand. Reading it 5-20 more times would most likely help.
For example, consider the opening example of Section 3. In principle, this kind of example is great, and more of these would be very useful in this paper. This particular one raises a few questions:
-Eq 5 makes it so that $(W \Psi)$ and $(a_x)$ need to be positive or negative together.  Why use ReLu's here at all? Why not just $sign( (W \Psi) a_x) $? Multiplying them will do the same thing, and is much simpler. I am probably missing something here, would like to know what it is... (Or, if the point of the artificial complexity is to give an example of the 3 basic principles, then perhaps point this out, or point out why the simpler version I just suggested would not scale up, etc)
-what exactly, in this example, does $\Psi$ correspond to? In prev discussion, $\Psi$ is always written with subscripts to denote state history (I believe), so this is an opportunity to explain what is different here. 
-Nitpick: why is a vector written as $W$? (or rather, what is the point of bold vs non-bold here?)
-a non-bold version of $Psi$, a few lines below, seems to correspond to the 4096 features of VGG's FC6, so I am still not sure what the bold version represents
-The defs/eqns at the beginning of section 3.1 (Sc, CReLu, etc) were slightly hard to follow and I wonder whether there were any typos, e.g. was CReS meant to refer directly to Sc, but used the notation ${ReLu}^2$ instead? 
Each of these on its own would be easier to overlook, but there is a compounding effect here for me, as a reader, such that by further on in the paper, I am rather confused.
I also wonder whether any of the elements described, have more "standard" interpretations/notations. For example, my slight confusion propagated further: after above point, I then did not have a clear intuition about $l_i$ in the EMS module. I get that symmetry has been built in, e.g. by the definitions of CReS and CReLu, etc, but I still don't see how it all works together, e.g. are late bottleneck architectures *exactly* the same as MLPs, but where inputs have simply been symmetrized, squared, etc? Nor do I have intuition about multiplicative symmetric interactions between visual features and actions, although I do get the sense that if I were to spend several hours implementing/writing out toy examples, it would clarify it significantly (in fact, I wouldn't be too surprised if it turns out to be fairly straightforward, as in my above comment indicating a seeming equivalence to simply multiplying two terms and taking the resulting sign). If the paper didn't need to be quite as dense, then I would suggest providing more elucidation for the reader, either with intuitions or examples or clearer relationships to more familiar formulations.
Later, I did find that some of the info I *needed* in order to understand the results (e.g. exactly what is meant by a "symmetry ablation", how was that implemented?) was in fact in the appendices (of which there are over 8 pages).
I do wonder how sensitive the performance of the overall system is to some of the details, like, e.g. the low-temp Boltzmann sampling rather than identity function, as described at the end of S2.
My confidence in this review is somewhere between 2 and 3.
The problem is an interesting one, the overall approach makes sense, it is clear the authors have done a very substantial  amount of work, and very diligently so (well-done!), some of the ideas are interesting and seem creative, but I am not sure I understand the glue of the details, and that might be very important here in order to assess it effectively.