The estimation of the IRT reliability coefficient and its lower and upper bounds, with comparisons to CTT reliability statistics

Abstract

The primary purpose of this study is to investigate the mathematical characteristics of the test reliability coefficient rho(XX)' as a function of item response theory (IRT) parameters and present the lower and upper bounds of the coefficient. Another purpose is to examine relative performances of the IRT reliability statistics and two classical test theory (CTT) reliability statistics (Cronbach's alpha and Feldt-Gilmer congeneric coefficients) under various testing conditions that result from manipulating large-scale real data. For the first purpose, two alternative ways of exactly quantifying rho(XX)' are compared in terms of computational efficiency and statistical usefulness. In addition, the lower and upper bounds for rho(XX)' are presented in line with the assumptions of essential tau-equivalence and congeneric similarity, respectively. Empirical studies conducted for the second purpose showed across all testing conditions that (1) the IRT reliability coefficient was higher than the CTT reliability statistics; (2) the IRT reliability coefficient was closer to the Feldt-Gilmer coefficient than to the Cronbach's alpha coefficient; and (3) the alpha coefficient was close to the lower bound of IRT reliability. Some advantages of the IRT approach to estimating test-score reliability over the CTT approaches are discussed in the end.