= (P(agreement between O1 and O2) P(e)) / (1 P(e)) where: κ(O1,O2) – – P(agreement) = (3+5+0+1+0+2)/30 = 11/30 0.3667 (diagonal elements) ≈ P(e) = (3/30).(15/30) + (17/30).(6/30) + (3/30).(1/30) + (1/30).(3/30) + (3/30).(0/30) + (3/30).(5/30) ≈ 0.1867. (pair-wise margins) So (0.3667-0.1867) / (1-0.1867) 0.22 κ(O1,O2) ≈ ≈ This is a low value, indicating very poor agreement between these two observers. There is reason for concern and action should be taken to improve the agreement. Similarly the values for the other pairs can be calculated: 0.72, 0.22.Observer 1 and 3 are in good agreement. Observer 2 is clearly κ(O1,O3) ≈ κ(O2,O3) ≈ different from 1 and 3 and the reasons for the deviation requires further investigation (e.g. consider need for additional training). 3.8 References Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37-46. Cohen, J. (1968) Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4): 213-220. Bland, J. M. Altman D. G. (1986) Statistical methods for assessing agreement between two methods of clinical measurement, Lancet: 307–310. (ISO 5725-2 based software)
κ(O1,O2) = (P(agreement between O1 and O2) – P(e)) / (1 – P(e)) where: P(agreement) = (3+5+0+1+0+2)/30 = 11/30 ≈ 0.3667 (diagonal elements) P(e) = (3/30).(15/30) + (17/30).(6/30) + (3/30).(1/30) + (1/30).(3/30) + (3/30).(0/30) + (3/30).(5/30) ≈ 0.1867. (pair-wise margins) So κ(O1,O2) ≈ (0.3667-0.1867) / (1-0.1867) ≈ 0.22 This is a low value, indicating very poor agreement between these two observers. There is reason for concern and action should be taken to improve the agreement. Similarly the values for the other pairs can be calculated: κ(O1,O3) ≈ 0.72, κ(O2,O3) ≈ 0.22.Observer 1 and 3 are in good agreement. Observer 2 is clearly different from 1 and 3 and the reasons for the deviation requires further investigation (e.g. consider need for additional training). 3.8 References Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37-46. Cohen, J. (1968) Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4): 213-220. Bland, J. M. Altman D. G. (1986) Statistical methods for assessing agreement between two methods of clinical measurement, Lancet: 307–310. (ISO 5725-2 based software)
tgp_8_3_draft_1.docx