UM  > Faculty of Social Sciences
Affiliated with RCfalse
Interrater Reliability Estimators Tested against True Interrater Reliabilities
Xinshu Zhao; Guangchao Charles Feng; Song Harris Ao; Piper Liping Liu
Source PublicationBMC Medical Research Methodology

Background: Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate.

Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which of the chance-adjusted indices are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, a good index should rely on to estimate chance agreement, or which of the factors the known indices in fact rely on.

The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatch between the assumed and the actual rater behaviors causes the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.

Methods: We conducted a 4×8×3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliability and chance agreement. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.

Results: The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2=.84. It was also the third best approximator, overestimating observed reliability by 13 percentage points. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r2=.312 and underestimated reliability by 31.4~31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating by the raters while the raters instead exhibited involuntary and unwilling random rating.

Conclusion: The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

KeywordIntercoder Reliability Interrater Reliability Reconstructed Experiment Cohen’s Kappa Krippendorff’s Alpha
Indexed BySSCI
Fulltext Access
Document TypeJournal article
CollectionFaculty of Social Sciences
AffiliationUniversity of Macau
First Author AffilicationUniversity of Macau
Recommended Citation
GB/T 7714
Xinshu Zhao,Guangchao Charles Feng,Song Harris Ao,et al. Interrater Reliability Estimators Tested against True Interrater Reliabilities[J]. BMC Medical Research Methodology,2022.
APA Xinshu Zhao,Guangchao Charles Feng,Song Harris Ao,&Piper Liping Liu.(2022).Interrater Reliability Estimators Tested against True Interrater Reliabilities.BMC Medical Research Methodology.
MLA Xinshu Zhao,et al."Interrater Reliability Estimators Tested against True Interrater Reliabilities".BMC Medical Research Methodology (2022).
Files in This Item: Download All
File Name/Size Publications Version Access License
Interrater Reliabili(856KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Xinshu Zhao]'s Articles
[Guangchao Charles Feng]'s Articles
[Song Harris Ao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Xinshu Zhao]'s Articles
[Guangchao Charles Feng]'s Articles
[Song Harris Ao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Xinshu Zhao]'s Articles
[Guangchao Charles Feng]'s Articles
[Song Harris Ao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Interrater Reliability Estimators Tested against True Interrater Reliabilities.pdf
Format: Adobe PDF
File name: data 384.csv
Format: CSV
This file does not support browsing at this time
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.