Contributions to estimation of measures for assessing rater reliability

Abstract

Reliability measures have been well studied over many years, beginning with an entire chapter devoted to intraclass correlation in the first edition of Fisher (1925). Such measures have been thoroughly studied for two factor models. This dissertation, motivated by a medical research problem, extends point and confidence interval estimation of both intraclass correlation coefficient and interater reliability coefficient to models containing three crossed random factors -- subjects, raters and occasions. The intraclass correlation coefficient is used when decision is made on an absolute basis with rater's scores, while the interater reliability coefficient is defined for decisions made on a relative basis. The estimation is conducted using both ANOVA and MCMC methods. The results from the two methods are compared. The MCMC method is preferred for analyses of small data sets when ICC values are high. Besides, the bias of estimator of intraclass correlation coefficient in one-way random effects model is evaluated.