  Evaluation of QA systems is very challenging and expensive, with the most
reliable approach being human annotations of correctness of answers for
questions. Recent works (AVA, BEM) have shown that transformer LM encoder based
similarity metrics transfer well for QA evaluation, but they are limited by the
usage of a single correct reference answer. We propose a new evaluation metric:
SQuArE (Sentence-level QUestion AnsweRing Evaluation), using multiple reference
answers (combining multiple correct and incorrect references) for sentence-form
QA. We evaluate SQuArE on both sentence-level extractive (Answer Selection) and
generative (GenQA) QA systems, across multiple academic and industrial
datasets, and show that it outperforms previous baselines and obtains the
highest correlation with human annotations.

Computation and Language;Machine Learning