  Although automated metrics are commonly used to evaluate NLG systems, they
often correlate poorly with human judgements. Newer metrics such as BERTScore
have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which
rely on n-gram matching. These newer methods, however, are still limited in
that they do not consider the generation context, so they cannot properly
reward generated text that is correct but deviates from the given reference.
  In this paper, we propose Language Model Augmented Relevance Score (MARS), a
new context-aware metric for NLG evaluation. MARS leverages off-the-shelf
language models, guided by reinforcement learning, to create augmented
references that consider both the generation context and available human
references, which are then used as additional references to score generated
text. Compared with seven existing metrics in three common NLG tasks, MARS not
only achieves higher correlation with human reference judgements, but also
differentiates well-formed candidates from adversarial samples to a larger
degree.

Computation and Language;Machine Learning