For the automatic evaluation of Generative Question Answering (genQA ) systems, it is essential to assess the correctness of the generated answers . However, n-gram similarity metrics, which are widely used to compare generated texts and references, are prone to misjudge fact-based assessments . Moreover, there is a lack of benchmark datasets to measure the quality of metrics in terms of the correctness. To study a better metric for genQA, we collect high-quality human judgments of correctness on two standard genQA datasets. Using our human-evaluation datasets, we show that existing metrics based on n-gram similarity  do not correlate with human judgments. To alleviate this problem, we propose a new metric for evaluating the correctness of genQA . Specifically, the new metric assigns different weights on each token via keyphrase prediction, thereby judging whether a predicted answer sentence captures the key meaning of the human judge's ground-truth . Our proposed metric shows a significantly higher correlation with human judgment than widely used existing metrics .
In the automatic evaluation of generative question answering (GenQA ) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer . Moreover, there is a lack of benchmark datasets to evaluate the suitability of existing metrics in terms of correctness. To study a better metric for GenQA, we first create high-quality human judgments of correctness on two standard GenQA datasets. Using our human-evaluation datasets, we show that widely used n-gram similarity metrics do not correlate with human judgments. To alleviate this problem, we propose a new metric for evaluating the correctness of GenQA . Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer . Our proposed metric shows a significantly higher correlation with human judgments than existing metrics in various datasets .
R
For
In
coherence
coherence fluency coherence
R
Generative Question Answering (genQA
generative question answering (GenQA
fluency
fluency fluency fluency
R
essential
difficult
clarity
style clarity clarity
R
the generated answers . However, n-gram similarity metrics, which are widely used to compare generated texts and references, are prone to misjudge fact-based assessments
generated answers due to the free-form of the answer
clarity
clarity clarity clarity
R
measure the quality of
evaluate the suitability of existing
clarity
clarity clarity style
D
the
fluency
fluency fluency fluency
R
genQA, we collect
GenQA, we first create
clarity
clarity coherence clarity
R
genQA
GenQA
fluency
fluency fluency clarity
R
existing metrics based on
widely used
clarity
clarity clarity style
A
metrics
clarity
clarity clarity clarity
R
genQA
GenQA
fluency
fluency clarity fluency
R
the
our
clarity
clarity fluency clarity
R
on
to
fluency
fluency fluency coherence
R
predicted
generated
clarity
clarity clarity clarity
R
human judge's ground-truth
reference answer
clarity
clarity clarity clarity
R
judgment than widely used existing metrics
judgments than existing metrics in various datasets
clarity
clarity clarity clarity