LSAC Resources - Research

Constraining Item Exposure in Computerized Adaptive Testing with Shadow Tests (CT 02-03)

by Wim J. van der Linden and Bernard P. Veldkamp, University of Twente, Enschede, The Netherlands

Executive Summary

A potential danger in computerized adaptive testing (CAT) is that the item selection algorithm may overuse the items with the most attractive statistical characteristics, and the security of these items may be thereby compromised. Therefore, item exposure constraints that assure that none of the items in an item pool are overexposed are an important component of a CAT item selection algorithm. In this paper an alternative to the currently widely applied Sympson-Hetter item exposure control method is presented, which is based on decisions about the eligibility of the items in the pool before the test taker takes the test. If an item is eligible it remains in the pool; if it is ineligible it is removed from the pool for the test taker. These decisions are based on the outcomes of a probabilistic experiment with probabilities of eligibility that constrain the item-exposure rates to be below the target value.

The main advantage of the method is that, unlike the Sympson-Hetter method, no time-consuming simulation studies are necessary to find admissible values for control parameters of items. Instead, the current method is self-adaptive and can be implemented "on the fly" during operational testing. The method counts certain events during the testing of the test takers and uses these counts to automatically adapt the probabilities of item eligibility to their optimal level, which is then maintained during the rest of the testing process.

An extensive simulation study with adaptive tests from a previous pool of LSAT items showed that the probabilities of item eligibility were already stable after 1,000 test takers were tested, and the method produced exposure rates that were below the target for all items.

• Full Research Report (PDF)