  A robust evaluation metric has a profound impact on the development of text
generation systems. A desirable metric compares system output against
references based on their semantics rather than surface forms. In this paper we
investigate strategies to encode system and reference texts to devise a metric
that shows a high correlation with human judgment of text quality. We validate
our new metric, namely MoverScore, on a number of text generation tasks
including summarization, machine translation, image captioning, and
data-to-text generation, where the outputs are produced by a variety of neural
and non-neural systems. Our findings suggest that metrics combining
contextualized representations with a distance measure perform the best. Such
metrics also demonstrate strong generalization capability across tasks. For
ease-of-use we make our metrics available as web service.

Computation and Language