We propose the use of a sequence-to-sequence paraphraser for automatic machine translation evaluation . The paraphrasertakes a human reference as input and then force-decodes and scores an MT system output . We propose training the aforementioned paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot "language pair" (e.g., Russian to Russian). We denote our paraphraser "unbiased" because the mode of our model 's output probability is centered around a copy of the input sequence, which in our case represent the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT19 segment-level shared metrics task in all languages , excluding Gujarati where the model had no training data  . We also explore using our model conditioned on the source instead of the reference, and find that it outperforms every quality estimation as a metric system from the WMT19 shared task on quality estimation by a statistically significant margin in every language pair.
We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference . We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser 's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages ( excluding Gujarati where the model had no training data ) . We also explore using our model for the task of quality estimation as a metric--conditioning on the source instead of the reference--and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.
R
propose the use of a sequence-to-sequence paraphraser for automatic
frame the task of
R
. The paraphrasertakes
as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on
D
as input and then force-decodes and scores an MT system output
D
aforementioned
R
"language pair"
translation task
R
Russian to Russian). We denote our paraphraser "unbiased" because the mode of our model
Czech to Czech). This results in the paraphraser
R
probability is
mode being
R
in our case represent
represents
R
our
does not require human judgements for training. Our
R
WMT19
WMT 2019
R
,
(
A
)
R
conditioned
for the task of quality estimation as a metric--conditioning
R
reference, and
reference--and
R
outperforms every quality estimation as a metric system from the WMT19
significantly outperforms every submission to the WMT 2019
D
by a statistically significant margin