Chapter 14 Reliability

LEARNING OUTCOMES

  • Identify the different approaches to the concept of reliability.
  • Distinguish the key features of reliability for several administrations of a test, inter-rater reliability, and internal consistency.
  • Apply R to compute reliability coefficients of internal consistency and to generate data visualizations.
  • Critically evaluate the different reliability coefficients and appraise the outputs produced with the functions included in the package psych.

Despite being a common-sense concept (i.e., how reliable and consistent is the measurement instrument?), reliability has generated much debate among psychometricians. Reliability is conceptualized as the proportion of observed score variance attributed to true score variance.

\[\begin{aligned} \ Reliability = \frac{\sigma^{2}_{T}}{\sigma^{2}_{X}} \end{aligned}\]

Although the concept of reliability as the ratio between the true and the observed scores' variability is easy to understand, the different ways to operationalize reliability and to estimate the diversity of reliability coefficients complicate things a little bit (Cronbach, 1951; DeVellis, 2017; Huysamen, 2006; Revelle & Zinbarg, 2009; Sijtsma, 2009).

In Classical Test Theory (CTT), test scores are based on the general linear model and test total scores (Crocker & Algina, 1986). This measurement theory is also called True Score Theory because it decomposes the observed score into true scores and measurement error (random error and bias).

\[\begin{aligned} \ X_{i} = \ T_{i} + \ E_{i} \end{aligned}\]

Derived from the former psychometric model, CTT includes four additional assumptions:

  • The expected value of the observed score is the true score.

\[\begin{aligned} \ E(X_{i}) = \ T_{i} \end{aligned}\]

  • True scores and errors are independent.

\[\begin{aligned} \ Cov(T_{i}, \ E_{i}) = \ 0 \end{aligned}\]

  • Errors across test's forms are independent.

\[\begin{aligned} \ Cov(E_{i}, \ E_{j}) = \ 0 \end{aligned}\]

  • The errors in one form are independent of the true score on another form.

\[\begin{aligned} \ Cov(E_{i}, \ T_{j}) = \ 0 \end{aligned}\]

In sum, the assumptions of the CTT model consider that the variance components are independent.

\[\begin{aligned} \sigma^{2}_{X} = \sigma^{2}_{T} + \sigma^{2}_{E} \end{aligned}\]