Robust Automated Test Assembly for Testlet-Based
Tests: An Illustration With the Analytical Reasoning
Section of the LSAT (RR 13-02)
by Bernard P. Veldkamp and
Muirne C. S. Paap,
University of Twente, Enschede, the Netherlands
Executive Summary
In many high-stakes tests, subsets of questions (i.e., items) grouped around a
common stimulus are often utilized to increase testing efficiency. These subsets of
items are commonly called testlets. Since responses to items belonging to the same
testlet not only depend on the test taker’s ability, but also on the correct reading,
understanding, and interpretation of the stimulus, the assumption that the responses
to these items are independent of one another does not always hold.
A mathematical model called item response theory is often applied in automated
test assembly (ATA) with testlets. Testlet response theory (TRT) models have been
developed to deal with dependency among items within a testlet. This report
addresses some of the questions that arise in the application of TRT models to ATA.
Specifically, a robust ATA method is applied. The results obtained by this method, as
well as the advantages it offers, are discussed. Finally, recommendations about the
use of the new method are given.