Synthesizing SQL queries from natural language is a long-standing open problem and has been attracting considerable interest recently. Toward solving the problem, the de facto approach is to employ a sequence-to-sequence-style model. Such an approach will necessarily require the SQL queries to be serialized. Since the same SQL query may have multiple equivalent serializations, training a sequenceto-sequence-style model is sensitive to the choice from one of them. This phenomenon is documented as the "order-matters" problem. Existing state-of-the-art approaches rely on reinforcement learning to reward the decoder when it generates any of the equivalent serializations. However, we observe that the improvement from reinforcement learning is limited. In this paper, we propose a novel approach, i.e., SQLNet, to fundamentally solve this problem by avoiding the sequence-to-sequence structure when the order does not matter. In particular, we employ a sketch-based approach where the sketch contains a dependency graph so that one prediction can be done by taking into consideration only the previous predictions that it depends on. In addition, we propose a sequence-to-set model as well as the column attention mechanism to synthesize the query based on the sketch. By combining all these novel techniques, we show that SQLNet can outperform the prior art by 9% to 13% on the WikiSQL task.
This paper proposes a neural network-based approach to converting natural language questions to SQL queries. The idea is to use a small grammar to facilitate the process, together making some independence assumptions. It is evaluated on a recently introduced dataset for natural language to SQL.
Pros:
- good problem, NL2SQL is an important task given how dominant SQL is
- incorporating a grammar ("sketch") is a sensible improvement.
Cons:
- The dataset used makes very strong simplification assumptions. Not  problem per se, but it is not the most challenging SQL dataset. The ATIS corpus is NL2SQL and much more challenging and realistic:
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: the ATIS-3 corpus. In Proceedings of the workshop on Human Language Technology (HLT '94). Association for Computational Linguistics, Stroudsburg, PA, USA, 43-48. DOI: https://doi.org/10.3115/1075812.1075823
- In particular, the assumption that every token in the SQL statement is either an SQL keyword or appears in the natural language statement is rather atypical and unrealistic.
- The use of a grammar in the context of semantic parsing is not novel; see this tutorial for many pointers:
http://yoavartzi.com/tutorial/
- As far as I can tell, the set prediction is essentially predicted each element independently, without taking into account any dependencies. Nothing wrong, but also nothing novel, that is what most semantic parsing/semantic role labeling baseline approaches do. The lack of ordering among the edges, doesn't mean they are independent.
- Given the rather constrained type of questions and SQL statements, it would make sense to compare it against approaches for question answering over knowledge-bases:
https://github.com/scottyih/Slides/blob/master/QA%20Tutorial.pdf
While SQL can express much more complex queries, the ones supported by the grammar here are not very different.
- Pasupat and Liang (2015) also split the data to make sure different tables appear only in training, dev, test and they developed their dataset using crowd sourcing.
- The comparison against Dong and Lapata (2016) is not fair because their model is agnostic and thus applicable to 4 datasets while the one presented here is tailored to the dataset due the grammar/sketch used. Also, suggesting that previous methods might not generalize well sounds odd given that the method proposed seems to use much larger datasets.
- Not sure I agree that mixing the same tables across training/dev/test is more realistic. If anything, it assumes more training data and manual annotation every time a new table is added.