Prefrontal cortex (PFC) is a part of the brain which is responsible for behavior repertoire. Inspired by PFC functionality and connectivity, as well as human behavior formation process, we propose a novel modular architecture of neural networks with a Behavioral Module (BM) and corresponding end-to-end training strategy. This approach allows efficient learning of behaviors and preferences representation. This property is particularly useful for user modeling (as for dialog agents) and recommendation tasks, as allows learning personalized representations of different user states. In the experiment with video games playing, the results show that the proposed method allows separation of main task's objectives and behaviors between different BMs. The experiments also show network extendability through independent learning of new behavior patterns. Moreover, we demonstrate a strategy for an efficient transfer of newly learned BMs to unseen tasks.
This paper introduces a "behavior module" (BM) which is a small network that encodes preferences over actions and runs parallel to the fully connected layers of a policy. The paper shows this architecture working in Atari games, where the same policy can be used to achieve different action preferences over a game while still playing well. It also includes a thorough recap of past modular approaches. 
The motivation for the BM is that we may want deep networks to be able to decompose "strategy" and "behavior", where behavior may influence decisions without affecting the performance. In this framework, the BM is trained on a reward of correctness + personalized “satisfaction”.
The experiments model behavior as preferences over how many actions to play simultaneously. The trained BMs can be transferred to new tasks without finetuning. The ideas here also have some similarity to the few shot learning literature.
Comments on the experiments:
1. The Table 2  do not show a smooth interpolation between reward scaling and AMSR vs BD. This is surprising because the performance on the game should be highest went it is weighted to the most. This indicates to me that the results are actually high variance, the 0.8 vs 0.88 in stage 2 of 0.25r vs 0.5r means that is probably at least +/- 0.08 standard deviation. Adding standard deviations to these numbers is important for scientific interpretability.
2. I expect some BMs should perform much better than others (as they have been defined by number of actions to play at once). I would like to see (maybe in the appendix) a table similar to table 2 for for individual BMs. I currently assume the numbers are averaged over all BMs.
3. Similarly, I would like to see the BD for BM0 (e.g., if a policy is not optimized for any behavior, how close does it get to the other behaviors on average). This is an important lower bound that we can compare the other BD to. 
4. An obvious baseline missing is to directly weight the Q values of the action outputs  (instead of having an additional network)  by the designed behavior rewards. There is an optimal way to do this because of experimental choices. 
Questions:
1.For BM2, you write " Up and Down (or Right and Left)" did you mean "Up and Right"? How can Up and Down be played at the same time?
Overall, this paper uses neuroscience to motivate a behavior module. However, the particular application and problem settings falls short of these abstract "behaviors". Currently, the results are just showing that RL optimizes whatever reward function is provided, and that architectural decomposition allows for transfer, which was already showed in (Devin 2017). An experiment which would better highlight the behavior part of the BM architecture is the following:
1. Collect datasets of demonstrations (e.g. on atari) from different humans.
2. Train a policy to accomplish the task (with RL)
3. Train BMs on each human to accomplish the task in the style of each human.
This would show that the BMs can capture actual behavior. 
The dialog examples discussed in the abstract would also be very exciting.
In conclusion, I find the idea interesting, but the experiments do not show that this architecture can do anything new. The abstract and introduction discuss applications that would be much more convincing. I hope to see experiments with a more complex definition of "behavior" that cannot be handcoded into the Q function.