UM  > Faculty of Social Sciences
Residential Collegefalse
Status已發表Published
Liberal-Conservative Hierarchies for Indices of Inter-Coder Reliability
Xinshu ZHAO1,2; Ke DENG3; Guangchao Charles FENG2; Lixing ZHU2; Visne Kam Che CHAN2
2012-05-01
Conference Name2012 Conference of International Communication Association
Source Publication2012 Conference of International Communication Association
Conference Date2012-05-01
Conference PlacePhoenix, AZ, USA
Abstract

Inter-coder reliability is the most often used quantitative indicator of measurement quality in content studies.  Researchers in psychology, sociology, education, medicine, marketing and other social science disciplines also use reliability to evaluate the quality of diagnosis, tests and other assessments. While more than a dozen indices of inter-coder reliability are available, methodologists debated which indices are more suitable for general use (Zhao, Liu and Deng, in press).    

Zhao, Liu and Deng (in press) analyzed 22 indices of inter-coder reliability, including the first introduced -- percent agreement (ao) and Benini’s β (1901), the most often “reinvented” -- Bennett et al’s S (1954), the most often used -- Scott’s π (1955) and Cohen’s κ (1960), the most strongly advocated-- Krippendorff’s α (1970), and the most recently introduced -- Gwet’s AC1 (2008). Based on their analysis of the mathematics and the previously hidden assumptions, Zhao, Liu and Deng (in press) produced two liberal-conservative hierarchies for the 22 indices. The authors argued that researchers, reviewers and editors need to know, in general, which indices tend to generate higher scores and which tend to generate lower scores, so that they can interpret the numbers produced by these indices on a more informed basis.

This study is to extend Zhao, Liu and Deng’s (in press) work. While we agree that such hierarchies are useful, we argue that hierarchies based on mathematical analysis alone may not tell the whole story. We need alternative methodologies for cross verification. Accordingly, this study uses simulated data to build six more hierarchies. The resulted hierarchies show many similarities with Zhao, Liu and Deng’s (in press) results but also show some differences. In conclusion, these hierarchies together may be useful to workbench researchers who wish to better evaluate the inter-coder reliability indices of their instruments, and we provided software at (http://reliability.hkbu.edu.hk/) that helps to calculate these indices.

KeywordReliability Agreement Cohen’s Kappa Scott’s Pi Krippendorff’s Alpha
Document TypeConference paper
CollectionFaculty of Social Sciences
DEPARTMENT OF COMMUNICATION
Affiliation1.Fudan University
2.Hong Kong Baptist University
3.Harvard University
Recommended Citation
GB/T 7714
Xinshu ZHAO,Ke DENG,Guangchao Charles FENG,et al. Liberal-Conservative Hierarchies for Indices of Inter-Coder Reliability[C],2012.
APA Xinshu ZHAO,Ke DENG,Guangchao Charles FENG,Lixing ZHU,&Visne Kam Che CHAN.(2012).Liberal-Conservative Hierarchies for Indices of Inter-Coder Reliability.2012 Conference of International Communication Association.
Files in This Item: Download All
File Name/Size Publications Version Access License
Liberal-Conservative(479KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Xinshu ZHAO]'s Articles
[Ke DENG]'s Articles
[Guangchao Charles FENG]'s Articles
Baidu academic
Similar articles in Baidu academic
[Xinshu ZHAO]'s Articles
[Ke DENG]'s Articles
[Guangchao Charles FENG]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Xinshu ZHAO]'s Articles
[Ke DENG]'s Articles
[Guangchao Charles FENG]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Liberal-Conservative Hierarchies for Indices of Inter-Coder Reliability.docx
Format: Microsoft Word
This file does not support browsing at this time
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.