Neural conversational models are widely used in applications like personal assistants and chat bots. These models seem to give better performance when operating on word level. However, for fusion languages like French, Russian and Polish, vocabulary size sometimes become infeasible since most of the words have lots of word forms. To reduce vocabulary size we propose a new pipeline for building conversational models: first generate words in standard form and then transform them into a grammatically correct sentence. For this task we propose a neural network architecture that efficiently employs correspondence between standardised and target words and significantly outperforms character-level models while being 2x faster in training and 20% faster at evaluation. The proposed pipeline gives better performance than character-level conversational models according to assessor testing.
In this work, the authors propose a sequence-to-sequence architecture that learns a mapping from a normalized sentence to a grammatically correct sentence. The proposed technique is a simple modification to the standard encoder-decoder paradigm which makes it more efficient and better suited to this task. The authors evaluate their technique using three morphologically rich languages French, Polish and Russian and obtain promising results.
The morphological agreement task would be an interesting contribution of the paper, with wider potential. But one concern that I have is regarding the evaluation metrics used for it. Firstly, word accuracy rate doesn't seem appropriate, as it does not measure morphological agreement. Secondly, sentence accuracy (w.r.t. the sentences from which the normalized sentences are derived) is not indicative of morphological agreement: even "wrong" sentences in the output could be perfectly valid in terms of agreement. A grammatical error rate (fraction of grammatically wrong sentences produced) would probably be a better measure.
Another concern I have is regarding the quality of the baseline: Additional variants of the baseline models should be considered and the best one reported. Specifically, in the conversation task, have the authors considered switching the order of normalized answer and context in the input? Also, the word order of the normalized answer and/or context could be reversed (as is done in sequence-to-sequence translation models).
Also, many experimental details are missing from the draft:
-- What are the sizes of the train/test sets derived from the OpenSubtitles database?
-- Details of the validation sets used to tune the models.
-- In Section 5.4, no details of the question-answer corpus are provided. How many pairs were extracted? How many were used for training and testing?
-- In Section 5.4.1, how many assessors participated in the evaluation and how many questions were evaluated?
-- In some of the tables (e.g. 6, 7, 8) which show example sentences from Polish, Russian and French, please provide some more information in the accompanying text on how to interpret these examples (since most readers may not be familiar with these languages).
Pros:
-- Efficient model
-- Proposed architecture is general enough to be useful for other sequence-to-sequence problems
Cons:
-- Evaluation metrics for the morphological agreement task are unsatisfactory
-- It would appear that the baselines could be improved further using standard techniques