Learning Hierarchical Information Flow with Recurrent Neural Modules
We propose ThalNet, a deep learning model inspired by neocortical communication via the thalamus. Our model consists of recurrent neural modules that send features through a routing center, endowing the modules with the flexibility to share features over multiple time steps. We show that our model learns to route information hierarchically, processing input data by a chain of modules. We observe common architectures, such as feed forward neural networks and skip connections, emerging as special cases of our architecture, while novel connectivity patterns are learned for the text8 compression task. Our model outperforms standard recurrent neural networks on several sequential benchmarks.
The paper introduces a modular architecture that has a central routing module and other in/out modules, akin to the role of thalamus/neocortex regions in the brain. Initial experiments show promises.
I like the idea of the paper, which was nicely written. Some module details are missing (the read function r, merging function m and the module function f), possibly because the space limitation, but it would be very useful to have in a supplement. 
The main cons are (i) the experiments are not super-strong; (ii) the search for the best module/routing functions is quite preliminary. However I believe that the direction is fruitful and worth exploring further.
In related work, there is a contemporary architecture with similar intention (although the specific targets were different, i.e., multi-X learning, where X = instance, view, task, label): "On Size Fit Many: Column Bundle for Multi-X Learning", Trang Pham, Truyen Tran, Svetha Venkatesh. arXiv preprint arXiv: 1702.07021.