DeepMath -Deep Sequence Models for Premise Selection
We study the effectiveness of neural sequence models for premise selection in automated theorem proving, one of the main bottlenecks in the formalization of mathematics. We propose a two stage approach for this task that yields good results for the premise selection task on the Mizar corpus while avoiding the handengineered features of existing state-of-the-art models. To our knowledge, this is the first time deep learning has been applied to theorem proving on a large scale.
• A demonstration for the first time that neural network models are useful for aiding in large scale automated logical reasoning without the need for hand-engineered features.
• The comparison of various network architectures (including convolutional, recurrent and hybrid models) and their effect on premise selection performance.
• A method of semantic-aware "definition"-embeddings for function symbols that improves the generalization of formulas with symbols occurring infrequently. This model outperforms previous approaches.
• Analysis showing that neural network based premise selection methods are complementary to those with hand-engineered features: ensembling with previous results produce superior results.
This paper considers the problem of premise selection for automated theorem proving: given a set of axioms (premises) and a conjecture, select a subset of axioms to provide the theorem prover that will allow proving the conjecture. The approach is to embed axioms and conjectures into real-valued vectors, and then map the concatenation of embeddings to a classification of whether the axiom is relevant for proving the conjecture. A variety of neural network architectures are explored for representing the axioms and conjectures, including recurrent and convolutional neural network approaches. In addition, a two-stage approach is developed, where embeddings for identifiers are created by embedding the identifier’s definition. Results show that when a small number of premises are returned (up to 64), a k-NN baseline performs best. When up to 1024 premises are returned, one of proposed models proves 62.95% of conjectures versus the k-NN baseline’s 59.34%.
Pros:  * The problem is interesting and does not appear to have previously received attention from the NIPS community  * The paper is clearly written  Cons:  * The approach is not particularly novel; it is mostly a case of applying a range of existing neural network components to the premise selection problem.  * The improvements achieved over a k-NN baseline are small, and I have some reservations about how the experimental results are reported (see below)  Experiments  * Numbers are presented for the performance of previous work in Section 3, but they don’t appear to be comparable to the results reported in the experiments (Section 6). Is there a way to meaningfully compare these numbers? If not, it's confusing to include them.  * Table 1 is not a fair way of reporting results. This presentation is asking us to take a maximum of test performance over all the different neural network variants, and to compare that to the performance of a single baseline method. However, there is quite a lot of variance in the neural network performance relative to the difference between neural network and baseline performance. Thus, there is a bias towards selecting one of the neural networks as the best method, even though it could just be due to the fact that we’re taking a max over many noisy performances. In section 3 it is mentioned that a portfolio of simple learners significantly outperform any single learner. If such an extensive search is being done over tweaks to the proposed method, then an equally extensive search should be done over baseline methods.  Presentation  * Please number Figures 5 and 6  * Figure 5 isn’t readable in black and white  * I find the title off-putting. “DeepMath” is a very general term relative to what the paper actually does (premise selection for automated theorem proving). It sounds like a title meant to grab headlines as opposed to being descriptive of the scientific contributions.