Seq2SQL: Generating Structured Queries From Natural Language Using Reinforcement Learning
Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.
 This work introduces a new semantic parsing dataset, which focuses on generating SQL from natural language. It also proposes a reinforcement-learning based model for this task. First of all, I*d like to emphasize that the creation of a large scale semantic parsing dataset is fantastic, and it is a much appreciated contribution. However, I find its presentation problematic. It claims to supplant existing semantic parsing and language-to-SQL datasets, painting WikiSQL as a more challenging dataset overall. Given the massive simplifications to what is considered SQL in this dataset (no joins, no subqueries, minimal lexical grounding problem), I am reluctant to accept this claim without empirical evidence. For example, how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS? That being said, I am sympathetic to making simplifications to a dataset for the sake of scalability, but it shouldn*t be presented as representative of SQL. On the modeling side, the role of reinforcement learning seems oddly central in the paper, even though though the added complexity is not well motivated. RL is typically needed when there are latent decisions that can affect the outcome in ways that are not known a priori. In this case, we know the reward is invariant to the ordering of the tokens in the WHERE clause. There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples. These should be included as baselines. While the data contribution is great, the claims of the paper need to be revised.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Seq2SQL: Generating Structured Queries From Natural Language Using Reinforcement Learning
Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This work introduces a new semantic parsing dataset, which focuses on generating SQL from natural language. It also proposes a reinforcement-learning based model for this task. First of all, I*d like to emphasize that the creation of a large scale semantic parsing dataset is fantastic, and it is a much appreciated contribution. However, I find its presentation problematic. It claims to supplant existing semantic parsing and language-to-SQL datasets, painting WikiSQL as a more challenging dataset overall. Given the massive simplifications to what is considered SQL in this dataset (no joins, no subqueries, minimal lexical grounding problem), I am reluctant to accept this claim without empirical evidence. For example, how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS? That being said, I am sympathetic to making simplifications to a dataset for the sake of scalability, but it shouldn*t be presented as representative of SQL. On the modeling side, the role of reinforcement learning seems oddly central in the paper, even though though the added complexity is not well motivated. RL is typically needed when there are latent decisions that can affect the outcome in ways that are not known a priori. In this case, we know the reward is invariant to the ordering of the tokens in the WHERE clause. There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples. These should be included as baselines. While the data contribution is great, the claims of the paper need to be revised.
assistant
 This work introduces a new semantic parsing dataset, which focuses on generating SQL from natural language. It also proposes a reinforcement-learning based model for this task. First of all, I*d like to emphasize that the creation of a large scale semantic parsing dataset is fantastic, and it is a much appreciated contribution. However, I find its presentation problematic. It claims to supplant existing semantic parsing and language-to-SQL datasets, painting WikiSQL as a more challenging dataset overall. Given the massive simplifications to what is considered SQL in this dataset (no joins, no subqueries, minimal lexical grounding problem), I am reluctant to accept this claim without empirical evidence. For example, how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS? That being said, I am sympathetic to making simplifications to a dataset for the sake of scalability, but it shouldn*t be presented as representative of SQL. On the modeling side, the role of reinforcement learning seems oddly central in the paper, even though though the added complexity is not well motivated. RL is typically needed when there are latent decisions that can affect the outcome in ways that are not known a priori. In this case, we know the reward is invariant to the ordering of the tokens in the WHERE clause. There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples. These should be included as baselines. While the data contribution is great, the claims of the paper need to be revised.