Statistical Algorithms for Detection of Test Collusion (RR 11-03)

Executive Summary

The development of statistical methods for detecting test collusion is a new research direction in the area of test security. Test collusion may be described as large-scale sharing of test materials or answers to test questions. The source of the test materials could be a teacher, a test-preparation company, the Internet, or test takers communicating on the day of the exam. The danger of test collusion for high-stakes testing programs is that it can seriously affect the scoring of the exam because of the potentially large number of test takers involved. This is a serious concern for the test users (law schools, universities, companies, government organizations, etc.), who will be given invalid scores for test takers involved in test collusion. Therefore, identifying such test takers in order to remove their responses from the data is an important task.

Test collusion often influences the test performance or speed of involved test takers for the affected portion of the test. However, the portion of the test where the collusion took place is often unknown, and searching through all possible portions will decrease the detection power or increase the false-positive rate of a statistical test. This report introduces an algorithm that resolves this problem by working in two stages. In the first stage, test centers with potential collusion are identified as aberrant. In the second stage, potentially aberrant test takers from the aberrant test centers are identified. A simulation study demonstrates the advantages of using this algorithm for the Law School Admission Test (a case of a partially stolen section of the LSAT was simulated). However, the algorithm is general and can be applied to other testing formats, including computer-based testing (CBT), multiple-stage testing (MST), and computerized adaptive testing (CAT).

Request the Full Report

To request the full report, please email Linda Reustle at lreustle@LSAC.org.