  We outline the Great Misalignment Problem in natural language processing
research, this means simply that the problem definition is not in line with the
method proposed and the human evaluation is not in line with the definition nor
the method. We study this misalignment problem by surveying 10 randomly sampled
papers published in ACL 2020 that report results with human evaluation. Our
results show that only one paper was fully in line in terms of problem
definition, method and evaluation. Only two papers presented a human evaluation
that was in line with what was modeled in the method. These results highlight
that the Great Misalignment Problem is a major one and it affects the validity
and reproducibility of results obtained by a human evaluation.

Computation and Language