LSAC Resources - Research

Statistical Tests for Conditional Independence in a Hierarchical Model for Speed and Accuracy on Test Items (RR 06-02)

by Wim J. van der Linden and Cees A. W. Glas, University of Twente, Enschede, The Netherlands

Executive Summary

In an earlier research project, a hierarchical framework for modeling speed and accuracy of test-taker responses to test questions (items) in a computerized adaptive testing environment was presented. It is important that the goodness of fit of this framework can be assessed in a statistically sound way. In this project, we focused on the development of statistical tests for the assumptions of conditional or local independence on which the framework is based. The local independence assumption states that a test taker's responses to individual items on a test should be statistically independent once the test taker's ability level has been accounted for.

Three different versions of this assumption were identified: (1) the usual conditional independence between the responses given the test taker's ability; (2) the same type of conditional independence but now between the response times given the test taker's speed; and (3) conditional independence between the response and response time given both the test taker's ability and speed. For each of these three assumptions, a Lagrange multiplier test for a statistical check on its validity was derived. These statistical tests involve statistics that are straightforward to calculate when the items have been calibrated with good precision.

An empirical application of the goodness-of-fit tests for a data set from a large-scale certification test showed that the assumptions were violated for considerable numbers of items. Further analyses of the data set should reveal if the violations are ignorable and due to the high power of the tests or are more substantial and point at unknown behavior by the test takers on some of the items.

• Full Research Report (PDF)