feature Cohen 's k corrected k agreement 73.59 98.74 dial act 84.53 98.87 turn 73.52 99.16 Table 2 : Inter-coder agreement on feedback expression annotation Although researchers do not totally agree on how to measure agreement in various types of annotated data and on how to interpret the resulting figures , see Artstein and Poesio ( 2008 ) , it is usually assumed that Cohen 's kappa figures over 60 are good while those over 75 are excellent ( Fleiss , 1971 ) .
measuring nominal scale agreement among many raters
Joseph L Fleiss
The inter-coder agreement figures obtained for the three types of annotation are given in Table 2 . feature Cohen 's k corrected k agreement 73.59 98.74 dial act 84.53 98.87 turn 73.52 99.16 Table 2 : Inter-coder agreement on feedback expression annotation Although researchers do not totally agree on how to measure agreement in various types of annotated data and on how to interpret the resulting figures , see Artstein and Poesio ( 2008 ) , it is usually assumed that Cohen 's kappa figures over 60 are good while those over 75 are excellent ( Fleiss , 1971 ) . Looking at the cases of disagreement we could see that many of these are due to the fact that the annotators had forgotten to remove some of the features automatically proposed by ANVIL from the latest annotated element .
feature Cohen 's k corrected k agreement 73.59 98.74 dial act 84.53 98.87 turn 73.52 99.16 Table 2 : Inter-coder agreement on feedback expression annotation Although researchers do not totally agree on how to measure agreement in various types of annotated data and on how to interpret the resulting figures , see Artstein and Poesio ( 2008 ) , it is usually assumed that Cohen 's kappa figures over 60 are good while those over 75 are excellent ( @@CITATION ) .