14.1 Approaches to reliability

Reliability can be operationalized in different ways (Table 14.1). First, we could consider reliability as an estimation of measurement stability. Consequently, we should administer the same test several times. Second, we might consider reliability as the estimation of measurement equivalence in two equal forms. To estimate this type of reliability we must create two equal forms (i.e., alternate forms) of the same test. Third, we could consider reliability as an estimation of consistency across raters. Then, the same task (e.g., classifying objects or people) should be applied to two or more raters/judges. Last, reliability can be conceptualized as a method to estimate the internal consistency of a test. In this case, we need to administer the same test to at least two respondents to generate several scores on the same test.

Table 14.1: Approaches to Reliability

Estimation	Coefficient	Operationalization	Test administration	Source of error
Test-retest	\(Cor(X_{1}, X_{2})\)	Repetition of measurement	Two or more times	Instability over time
Parallel forms	\(Cor(X_{i}, X_{j})\)	Two alternate forms	Twice	Lack of equivalence
Inter-rater			Once	Disagreement among raters
	Cohen-Kappa (\(\kappa\))	Two raters (categorical data)
	Fleiss-Kappa	More than two raters (categorical data)
	Intraclass correlation	Two or more raters (quantitative data)
Internal consistency		Two or more scores in a single test	Once	Heterogeneous item content
Split half tests	Spearman-Brown
	Tau Equivalent
Item covariance	Alpha (\(\alpha\))
	KR--20
	Guttman's Lambdas (\(\lambda\))
	Omega (\(\omega_{h}, \omega_{t}\))
	EFA/CFA
Note. \(Cor(X_{1}, X_{2})\) = Correlation between test and retest. \(Cor(X_{i}, X_{j})\) = Correlation between forms \(i\) and \(j\) of the test. KR--20 = Kuder-Richardson Formula 20 to estimate the reliability coefficient for binary data. EFA/CFA = Exploratory Factor Analysis/Confirmatory Factor Analysis.