Comparability of Test-Taker Proficiency Scores on Computer Adaptive Tests Using Real and Simulated Data (RR 11-04)

Executive Summary

The simulation of correct/incorrect responses to test questions (items) is a commonly used analytical technique with many potential benefits, but it is unclear to what extent these artificially generated datasets compare to real-world test-taker item responses. This study compares real, large-scale test-taker item response data to simulated data to determine the extent to which simulated data are an accurate representation of real-world testing outcomes. Using an original method, this study matched item response data from multiple administrations of the Law School Admission Test (LSAT) to create a single dataset of 534 items and 5,000 synthetic test takers. Numerous IRT-based comparisons between real and simulated data were made by three test-length conditions. Results indicate that simulated data are largely similar to real test data, with minor caveats for shorter-length tests.

Request the Full Report

To request the full report, please email Linda Reustle at lreustle@LSAC.org.