We present a generic dynamic architecture that employs a problem specific differentiable forking mechanism to leverage discrete logical information about the problem data structure. We adapt and apply our model to CLEVR Visual Question Answering, giving rise to the DDRprog architecture; compared to previous approaches, our model achieves higher accuracy in half as many epochs with five times fewer learnable parameters. Our model directly models underlying question logic using a recurrent controller that jointly predicts and executes functional neural modules; it explicitly forks subprocesses to handle logical branching. While FiLM and other competitive models are static architectures with less supervision, we argue that inclusion of program labels enables learning of higher level logical operations -our architecture achieves particularly high performance on questions requiring counting and integer comparison. We further demonstrate the generality of our approach though DDRstack -an application of our method to reverse Polish notation expression evaluation in which the inclusion of a stack assumption allows our approach to generalize to long expressions, significantly outperforming an LSTM with ten times as many learnable parameters.
Summary:
The paper presents a generic dynamic architecture for CLEVR VQA and Reverse Polish notation problems. Experiments on CLEVR show that the proposed model DDRprog outperforms existing models, but it requires explicit program supervision. The proposed architecture for RPN, called DDRstack outperforms an LSTM baseline.
Strengths:
— For CLEVR VQA task, the proposed model outperforms the state-of-the-art with significantly less number of parameters.
— For RPN task, the proposed model outperforms baseline LSTM model by a large margin.
Weaknesses:
— The paper doesn’t describe the model clearly. After reading the paper, it’s not clear to me what the components of the model are, what each of them take as input and produce as output, what these modules do and how they are combined. I would recommend to restructure the paper to clearly mention each of the components, describe them individually and then explain how they are combined for both cases - DDRprog and DDRstack. 
— Is the “fork” module the main contribution of the paper? If so, at least this should be described in detail. So, if no fork module is required for a question, the model architecture is effectively same as IEP?
— Machine accuracy is already par with human accuracy on CLEVR and very close to 100%. Why is this problem still important? 
— Given that the performance of state-on-art on CLEVR dataset is already very high ( <5% error) and the performance numbers of the proposed model are not very far from the previous models, it is very important to report the variance in accuracies along with the mean accuracies to determine if the performance of the proposed model is statistically significantly better than the previous models or not.
— In Figure 4, why are the LSTM32/128 curves different for Length 10 and Length 30 till subproblem index 10? They are both trained on the same training data, only test data is of different length and ideally both models should achieve similar accuracy for the first 10 subproblems (same trend as DDRstack).
— Why is DDRstack not compared to StackRNN?
— Can the authors provide training time comparison of their model and other/baseline models? Because that is more important than the number of epochs required in training.
— There are only 3 curves (instead of 4) in Figure 3.
— In a number of places, the authors are referring to left and right program branches. What are they? These names have not being defined formally in the paper.
Overall:
I think the research work in the paper is interesting and significant, but given the current presentation and level of detail in the paper, I don’t think it will be helpful for the research community. By proper restructuring of paper and adding more details, the paper can be converted to a solid submission.