We review the limitations of BLEU and ROUGE -the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries/translations against hypothesis summaries/translations.
=== Summary ===
The authors motivate the development of new (automatic) metrics to evaluate language generation by using similarity with a given reference: standard metrics like BLEU, ROUGE or METEOR have been shown to have poor correlation with human judgment on a number of tasks and are vulnerable to changes in word re-ordering, semantics-changing word replacement, and syntactic transformations.
  
They then propose a multi-dimensional evaluation criteria to evaluate sentence similarity based on semantic similarity (something that correlates with human judgments of the same), logical equivalence and fluency.
  
The paper then goes on to describe possible directions to tackle several key problems in evaluation: evaluating semantic similarity by using models trained on the GLUE benchmark, evaluating logical equivalence using models trained on the MNLI corpus and fluency based on the CoLA corpus.
=== Decision ===
The problem this paper seeks to tackle is clearly one of great
importance in the field, but I find it hard to argue that this paper
significantly contributes to the existing body of work (more on this
below) and as a result I vote to reject this paper.
There are two possible contributions for this paper: a set of criteria for what makes a good evaluation metric and the concrete proposals to implement these criteria.
  
For the first, I find the proposed criteria to be overly generic and not helpful at providing additional clarity on what makes for a good evaluation: for example, how is semantic similarity different from logical consistency? Does it make sense to compare the semantic similarity of two sentences if one of them isn't even near grammatical? A lot of prior work already argue the shortcomings of the existing metrics this paper is making, e.g. Conroy and Dang (2008), Liu et al. (2016), Novikova et al. (2017). I think it would be valuable to present new axes to decompose the evaluation problem, but more work is needed to clarify and develop the axes presented in this paper.
  
For the second possible contribution, the idea of evaluating language generation along dimensions is not novel and in fact quite standard in the NLP community. The challenge has been showing that there are subset of tasks that can be used a reliable metrics across different domains and systems. Unfortunately, this paper does not actually evaluate its own proposals, making it hard to evaluate how effective its proposals are.