Because the judges do not evaluate the same cases , we could not employ standard inter-annotator agreement measures ( Carletta 1996 ) .
assessing agreement on classification tasks the kappa statistic
J Carletta
In addition , we tried to ensure that the sets of cases shown to the judges were of similar quality , so that the judges ' assessments would be comparable . Because the judges do not evaluate the same cases , we could not employ standard inter-annotator agreement measures ( Carletta 1996 ) . However , it is still necessary to
Because the judges do not evaluate the same cases , we could not employ standard inter-annotator agreement measures ( @@CITATION ) .