Detection of Large-Scale Item Preknowledge in Computerized Adaptive Testing via Kullback–Leibler Divergence (RR 12-01)
Detection of test collusion is a new research direction in the area of test security. Test collusion may be described as large-scale sharing of test materials, including answers to test questions (items). Test collusion for high-stakes testing programs can affect the scoring of the exam because of the potentially large number of test takers involved. Therefore, identifying such test takers in order to withdraw their responses from the data is an important task. Current methods of detecting test collusion are based on statistics also used in answer-copying detection. In computerized adaptive testing, these methods lose power because the items presented are not the same for all test takers. This report addresses this problem by introducing a new approach that searches through various partitions of the test and analyzes differences between corresponding distributions of test-taker ability. The suggested approach was found to be effective in detecting groups of test takers with item preknowledge, meaning those with access to a (possibly unknown) subset (or subsets) of items prior to the exam.