Grammar as a Foreign Language
Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.
This paper proposes a method to learn a syntactic constituency parser using an attention-enhanced sequence-to-sequence model. The paper claims that it obtains state-of-the-art results on WSJ and performs comparatively to Berkeley parser.The problem of syntactic parsing is an important one in NLP, and is a fundamental pre-processing step for more advanced tasks such as named entity extraction or relation extraction. Therefore, any improvement on syntactic parsing is likely to have a significant impact on the community. To this end the application of LSTM-based sequence-to-sequence coding is both timely and important.Additional strong points of the proposed parser are its speed, and domain adaptability.However, I have several concerns that I would wish the authors could further clarify as mentioned below.The title of the paper and its relevance to the current work is somewhat unclear. I do understand that sequence-to-sequnce mapping has been previously applied for machine translation, but what exactly do you mean by "grammar as a foreign language"? The relevance of the title to the proposed method is never explicitly discussed in the paper. I guess you wanted to hint that you could "learn" the grammar of a language using sequence-to-sequence models. But the evaluation tasks are all monolingual adaptation task and not cross-lingual parsing tasks. There are several statements in the paper that require further clarifications or analysis. For example, it is mentioned that replacing all pos tags by XX gave an F1 improvement of 1 point. Why is this? Removing POS info means you are loosing many important features that have been found to be useful for syntactic parsing.What is the significance that the constructed parser matches Berkeley parser with 90.5 F1? Should not it be the case that the comparison should be done against an external test dataset and not the parser that generated the train data in the first place?The degree of novelty w.r.t. machine learning or deep learning is minimal in the paper considering that it is using a previously proposed sequence-to-sequence encoding method based on LSTM for parsing. On the other hand, the claims of the paper and application task (parsing) is more relevant to the NLP community. The paper would be much appreciated and recognized at an NLP conference such as ACL or EMNLP than at NIPS. Therefore, it would be more suitable to submit this work at an NLP related conference.The problem of domain adaptation of parsers has been considered in the NLP community and arguably the most famous benchmark for evaluating this would be SANCL-2012 "parsing the web" task (see Overview of the 2012 Shared Task on Parsing the Web by Petrov and McDonald 2012). Unfortunately, this line of work is not discussed nor evaluated against as a benchmark dataset in the current paper, which makes it difficult to justify the claims about domain adaptability of the proposed method.The robustness of the parser is also a concern. It is mentioned that all 14 cases where the parser failed to produce a balanced tree are fragments or not ending with proper punctuations. Unfortunately, majority of the text found in social media such as twitter would be such cases. It is not discussed in the paper how you could address this issue (in particular that you claim the domain adaptability of the parser) in a more systematic manner than simply adding brackets to balance it.