Published as a conference paper at ICLR 2018 COMBINING SYMBOLIC EXPRESSIONS AND BLACK- BOX FUNCTION EVALUATIONS IN NEURAL PROGRAMS
Neural programming involves training neural networks to learn programs, mathematics, or logic from data. Previous works have failed to achieve good generalization performance, especially on problems and programs with high complexity or on large domains. This is because they mostly rely either on black-box function evaluations that do not capture the structure of the program, or on detailed execution traces that are expensive to obtain, and hence the training data has poor coverage of the domain under consideration. We present a novel framework that utilizes black-box function evaluations, in conjunction with symbolic expressions that define relationships between the given functions. We employ tree LSTMs to incorporate the structure of the symbolic expression trees. We use tree encoding for numbers present in function evaluation data, based on their decimal representation. We present an evaluation benchmark for this task to demonstrate our proposed model combines symbolic reasoning and function evaluation in a fruitful manner, obtaining high accuracies in our experiments. Our framework generalizes significantly better to expressions of higher depth and is able to fill partial equations with valid completions.
Summary
This paper presents a dataset of mathematical equations and applies TreeLSTMs to two tasks: verifying and completing mathematical equations. For these tasks, TreeLSTMs outperform TreeNNs and RNNs. In my opinion, the main contribution of this paper is this potentially useful dataset, as well as an interesting way of representing fixed-precision floats. However, the application of TreeNNs and TreeLSTMs is rather straight-forward, so in my (subjective) view there are only a few insights salvageable for the ICLR community and compared to Allamanis et al. (2017) this paper is a rather incremental extension.
Strengths
The authors present a new datasets for mathematical identities. The method for generating additional correct identities could be useful for future research in this area.
I find the representation of fixed-precision floats presented in this paper intriguing. I believe this contribution should be emphasized more as it allows the model to generalize to unseen numbers and I am wondering whether the authors see some wider application of this representation for neural programming models.
I liked the categorization of the related work.
Weaknesses
p2: It is mentioned that the framework is the first to combine symbolic expressions with black-box function evaluations, but I would argue that Neural Programmer-Interpreters (NPI; Reed & De Freitas) are already doing that (see Fig 1 in that paper where the execution trace is a symbolic expression and some expressions "Act(LEFT)" are black-box function applications directly changing the image).
The differences to Allamanis et al. (2017) are not worked out well. For instance, the authors use the TreeNN model from that paper as a baseline but the EqNet model is not mentioned at all. The obvious question is whether EqNets can be applied to the two tasks (verifying and completing mathematical equations) and if so why this has not been done.
The contribution regarding black box function application is unclear to me. On page 6, it is unclear to me what "handles […] function evaluation expressions". As far as I understand, the TreeLSTM learns to the return value of function evaluation expressions in order to predict equality of equations, but this should be clarified.
I find the connection of the proposed model and task to "neural programming" weak. For instance, as far as I understand there is no support for stateful programs. Furthermore, it would be interesting to hear how this work can be applied to existing programming languages such as Haskell. What are the limitations of the architecture? Could it learn to identify equality of two lists in Haskell?
p6: The paragraph on baseline models is rather uninformative. TreeLSTMs have been shown to outperform Tree NN's in various prior work. The statement that "LSTM cell […] helps the model to have a better understanding of the underlying functions in the domain" is vague. LSTM cells compared to fully-connected layers in Tree NNs ameliorate vanishing and exploding gradients along paths in the tree. Furthermore, I would like to see a qualitative analysis of the reasoning capabilities that are mentioned here. Did you observe any systematic differences in the ~4% of equations where the TreeLSTM fails to generalize (Table 3; first column).
Minor Comments
Abstract: "Our framework generalizes significantly better" I think it would be good to already mention in comparison to what this statement is.
p1: "aim to solve tasks such as learn mathematical" -> "aim to solve tasks such as learning mathematical"
p2: You could add a citation for Theano, Tensorflow and Mxnet.
p2: Could you elaborate how equation completion is used in Mathematical Q&A?
p3: Could you expand on "mathematical equation verification and completion […] has broader applicability" by maybe giving some concrete examples.
p3 Eq. 5: What precision do you consider? Two digits?
p3: "division because that they can" -> "division because they can"
p4 Fig. 1: Is there a reason 1 is represented as 10^0 here? Do you need the distinction between 1 (the integer) and 1.0 (the float)?
p5: "we include set of changes" -> "we include the set of changes"
p5: In my view there is enough space to move appendix A to section 2. In addition, it would be great to see more examples of generated identities at this stage (including negative ones).
p5: "We generate all possible equations (with high probability)" – what is probabilistic about this?
p5: I don't understand why function evaluation results in identities of depth 2 and 3. Is it both or one of them?
p6: The modules "symbol" and "number" are not shown in the figure. I assume they refer to projections using Wsymb and Wnum?
p6: "tree structures neural networks" -> "tree structured neural networks"
p6: A reference for the ADAM optimizer should be added.
p6: Which method was used for optimizing these hyperparameters? If a grid search was used, what intervals were used?
p7: "the superiority of Tree LSTM to Tree NN shows that is important to incorporate cells that have memory" is not a novel insight.
p8: When you mention "you give this set of equations to the models look at the top k predictions" I assume you ranked the substituted equations by the probability that the respective model assigns to it?
p8: Do you have an intuition why prediction function evaluations for "cos" seem to plateau certain points? Furthermore, it would be interesting to see what effect the choice of non-linearity on the output of the TreeLSTM has on how accurately it can learn to evaluate functions. For instance, one could replace the tanh with cos and might expect that the model has now an easy time to learn to evaluate cos(x).
p8 Fig 4b; p9: Relating to the question regarding plateaus in the function evaluation: "in Figure 4b […] the top prediction (0.28) is the correct value for tan with precision 2, but even other predictions are quite close" – they are all the same and this bad, right?
p9: "of the state-of-the-art neural reasoning systems" is very broad and in my opinion misleading too. First, there are other reasoning tasks (machine reading/Q&A, Visual Q&A, knowledge base inference etc.) too and it is not obvious how ideas from this paper translate to these domains. Second, for other tasks TreeLSTMs are likely not state-of-the-art (see for example models on the SQuAD leaderboard: https://rajpurkar.github.io/SQuAD-explorer/) .
p9: "exploring recent neural models that explicitly use memory cells" – I think what you mean is models with addressable differentiable memory.
# Update after the rebuttal
Thank you for the in-depth response and clarifications. I am increasing my score by one point. I have looked at the revised paper and I strongly suggest that you add the clarifications and in particular comments regarding comparison to related work (NPI, EqNet etc) to the paper. Regarding Fig. 4b, I am still not sure why all scores are the same (0.9977) -- I assume this is not the desired behavior?