For the automatic evaluation of Generative Question Answering (genQA ) systems, it is essential to assess the correctness of the generated answers . However, n-gram similarity metrics, which are widely used to compare generated texts and references, are prone to misjudge fact-based assessments . Moreover, there is a lack of benchmark datasets to measure the quality of metrics in terms of the correctness. To study a better metric for genQA, we collect high-quality human judgments of correctness on two standard genQA datasets. Using our human-evaluation datasets, we show that existing metrics based on n-gram similarity  do not correlate with human judgments. To alleviate this problem, we propose a new metric for evaluating the correctness of genQA . Specifically, the new metric assigns different weights on each token via keyphrase prediction, thereby judging whether a predicted answer sentence captures the key meaning of the human judge's ground-truth . Our proposed metric shows a significantly higher correlation with human judgment than widely used existing metrics .
In the automatic evaluation of generative question answering (GenQA ) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer . Moreover, there is a lack of benchmark datasets to evaluate the suitability of existing metrics in terms of correctness. To study a better metric for GenQA, we first create high-quality human judgments of correctness on two standard GenQA datasets. Using our human-evaluation datasets, we show that widely used n-gram similarity metrics do not correlate with human judgments. To alleviate this problem, we propose a new metric for evaluating the correctness of GenQA . Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer . Our proposed metric shows a significantly higher correlation with human judgments than existing metrics in various datasets .
R
For
In
R
Generative Question Answering (genQA
generative question answering (GenQA
R
essential
difficult
R
the generated answers . However, n-gram similarity metrics, which are widely used to compare generated texts and references, are prone to misjudge fact-based assessments
generated answers due to the free-form of the answer
R
measure the quality of
evaluate the suitability of existing
D
the
R
genQA, we collect
GenQA, we first create
R
genQA
GenQA
R
existing metrics based on
widely used
A
metrics
R
genQA
GenQA
R
the
our
R
on
to
R
predicted
generated
R
human judge's ground-truth
reference answer
R
judgment than widely used existing metrics
judgments than existing metrics in various datasets