The Probability of Exceedance as a Nonparametric Person-Fit Statistic for Tests of Moderate Length (RR 13-06)
In this report we present a measure to identify unlikely patterns of correct/incorrect answers to test questions (commonly referred to as items). Some examples of why such patterns may occur include the misinterpretation of questions, item preknowledge, answer copying, or guessing behavior. The proposed measure is the probability of exceedance (PE). PE provides information about the probability of a correct/incorrect answer pattern, conditional on the test taker’s total score. Although this concept is not new, it is hardly if ever applied in practice.
In this report we show how the PE of a response vector can be computed and how misfitting response patterns are detected. A simulation study is conducted to investigate the robustness of this procedure.