Automated metrics such as BLEU are widely used in the machine translation literature. They have also been used recently in the dialogue community for evaluating dialogue response generation. However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting. Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity. It is thus reasonable to think that these automated metrics would correlate well with human judgment in the task-oriented setting where the generation task consists of translating dialogue acts into a sentence. We conduct an empirical study to confirm whether this is the case. Our findings indicate that these automated metrics have stronger correlation with human judgments in the task-oriented setting compared to what has been observed in the non task-oriented setting. We also observe that these metrics correlate even better for datasets which provide multiple ground truth reference sentences. In addition, we show that some of the currently available corpora for task-oriented language generation can be solved with simple models and advocate for more challenging datasets.
The authors present a solid overview of unsupervised metrics for NLG, and perform a correlation analysis between these metrics and human evaluation scores on two task-oriented dialog generation datasets using three LSTM-based models. They find weak but statistically significant correlations for a subset of the evaluated metrics, an improvement over the situation that has been observed in open-domain dialog generation.
Other than the necessarily condensed model section (describing a model explained at greater length in a different work) the paper is quite clear and well-written throughout, and the authors' explication of metrics like BLEU and greedy matching is straightforward and readable. But the novel work in the paper is limited to the human evaluations collected and the correlation studies run, and the authors' efforts to analyze and extend these results fall short of what I'd like to see in a conference paper.
Some other points:
1. Where does the paper's framework for response generation (i.e., dialog act vectors and delexicalized/lexicalized slot-value pairs) fit into the landscape of task-oriented dialog agent research? Is it the dominant or state-of-the-art approach?
2. The sentence "This model is a variant of the “ld-sc-LSTM” model proposed by Sharma et al. (2017) which is based on an encoder-decoder framework" is ambiguous; what is apparently meant is that Sharma et al. (2017) introduced the hld-scLSTM, not simply the ld-scLSTM.
3. What happens to the correlation coefficients when exact reference matches (a significant component of the highly-rated upper right clusters) are removed?
4. The paper's conclusion naturally suggests the question of whether these results extend to more difficult dialog generation datasets. Can the authors explain why the datasets used here were chosen over e.g. El Asri et al. (2017) and Novikova et al. (2016)?