Published as a conference paper at ICLR 2018 PARAMETRIZED HIERARCHICAL PROCEDURES FOR NEURAL PROGRAMMING
Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism. Despite the potential to increase the interpretability and the compositionality of the behavior of artificial agents, it remains difficult to learn from demonstrations neural networks that represent computer programs. The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of specific structures of data, and the extremely limited observability. To address these challenges, we propose to model programs as Parametrized Hierarchical Procedures (PHPs). A PHP is a sequence of conditional operations, using a program counter along with the observation to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller. We develop an algorithm for training PHPs from a set of supervisor demonstrations, only some of which are annotated with the internal call structure, and apply it to efficient level-wise training of multi-level PHPs. We show in two benchmarks, NanoCraft and long-hand addition, that PHPs can learn neural programs more accurately from smaller amounts of both annotated and unannotated demonstrations.
I thank the authors for their updates and clarifications.  I stand by my original review and score.  I think their method and their evaluation has some major weaknesses, but I think that it still provides a good baseline to force work in this space towards tasks which can not be solved by simpler models like this.  So while I'm not super excited about the paper I think it is above the accept threshold.
--------------------------------------------------------------------------
This paper extends an existing thread of neural computation research focused on learning resuable subprocedures (or options in RL-speak).  Instead of simply input and output examples, as in most of the work in neural computation, they follow in the vein of the Neural Programmer-Interpreter (Reed and de Freitas, 2016) and Li et. al., 2017, where the supervision contains the full sequence of elementary actions in the domain for all samples, and some samples also contain the hierarchy of subprocedure calls.
The main focus of their work is learning from fewer fully annotated samples than prior work.  They introduce two new ideas in order to enable this:
1.  They limit the memory state of each level in the program heirarchy to simply a counter indicating the number of elementary actions/subprocedure calls taken so far (rather than a full RNN embedded hidden/cell state as in prior work).  They also limit the subprocedures such that they do not accept any arguments.
2.  By considering this very limited set of possible hidden states, they can compute the gradients using a dynamic program that seems to be more accurate than the approximate dynamic program used in Li et. al., 2017.  
The main limitation of the work is this extremely limited memory state, and the lack of arguments.  Without arguments, everything that parameterizes the subprocedures must be in the visible world state.  In both of their domains, this is true, but this places a significant limitation on the algorithms which can be modeled with this technique.  Furthermore, the limited memory state means that the only way a subprocedure can remember anything about the current observation is to call a different subprocedure.  Again, their two evalation tasks fit into this paradigm, but this places very significant limitations on the set of applicable domains.  I would have like to see more discussion on how constraining these limitations would be in practice.  For example, it seems it would be impossible for this architecture to perform the Nanocraft task if the parameters of the task (width, height, etc.) were only provided in the first observation, rather than every observation.  
None-the-less I think this work is an important step in our understanding of the learning dynamics for neural programs.  In particular, while the RNN hidden state memory used by the prior work enables the learning of more complicted programs *in theory*, this has not been shown in practice. So, it's possible that all the prior work is doing is learning to approixmate a much simpler architecture of this form.  Specifically, I think this work can act as a great base-line by forcing future work to focus on domains which cannot be easily solved by a simpler architecture of this form.  This limitation will also force the community to think about which tasks require a more complicated form of memory, and which can be solved with a very simple memory of this form.
I also have the following additional concerns about the paper:
1.  I found the current explanation of the algorithm to be very difficult to understand.  It's extremely difficult to understand the core method without reading the appendix, and even with the appendix I found the explanation of the level-by-level decomposition to be too terse.
2.  It's not clear how their gradient approximation compares to the technique used by Li et. al.  They obviously get better results on the addition and Nanocraft domains, but I would have liked a more clear explanation and/or some experiments providing insights into what enables these improvements (or at least an admission by the authors that they don't really understand what enabled the performance improvements).