14.2 Reliability methods requiring several administrations of a test

14.2.1 Test-retest

The test-retest method operationalizes reliability as stability of the measures over time. For example, the same test could be administered at the beginning and at the end of the academic year. If the instrument is reliable, the time interval between test administrations should have no effect on test scores.

To estimate the reliability of a test-retest, we will compute a Pearson's correlation test between the test and the retest's scores. The estimation of this type of reliability depends on the measurement scale. For instance, for ordinal data we will compute Kendall's Tau-b tests and for binary data we will compute tetrachoric correlations.

The method to estimate reliability as stability over time is frequently used in speed tests. However, this method can be negatively affected by the time interval between the test and retest (e.g., long periods are related to respondents' individual changes) as well as the influence of the first administration on the second (e.g., recall effects, practice).

14.2.2 Parallel forms

The parallel-forms method is designed to overcome some of the limitations of the test-retest approach. For instance, a test aimed at measuring the respondents' ability in verbal comprehension---if administered twice---might be affected by learning and memory effects. For this reason, we could develop two parallel (alternate) forms under the same domain and test specification. These two forms should be administered at the same time and to a large number of respondents. Finally, we should compute Pearson's correlation test to the scores of both parallel forms.

This method assumes equal true scores and error variances of both alternate forms.

\[\begin{aligned} \ X_{i} = \ T_{i} + \ E_{i} \end{aligned}\]

\[\begin{aligned} \ X'_{i} = \ T_{i} + \ E'_{i} \end{aligned}\]

\[\begin{aligned} \sigma^{2}_{e} = \sigma^{2}_{e'} \end{aligned}\]

In sum, the reliability coefficient of a parallel-forms test will be estimated as the Pearson's correlation of two parallel forms (i.e., two parallel tests).

\[\begin{aligned} \rho_{xx'} = \frac{\sigma^{2}_{T}}{\sigma^{2}_{X}} \end{aligned}\]

\[\begin{aligned} \sigma^{2}_{e} = \sigma^{2}_{x} \ \sqrt{1 - \rho_{xx'}} \end{aligned}\]

CAUTION!

This method is problematic because it is highly unlikely to find strict parallelism between two forms. The parallel-forms reliability coefficient includes random errors and the degree of parallelism of the forms. The parallel-forms method is affected by practice effects.