Information-Based Versus Number-Correct Routing in Multistage Classification Tests (RR 07-05)
by Alexander Weissman and Dmitry I. Belov, Law School Admission Council; and Ronald D. Armstrong, Rutgers University
Executive Summary
The last two decades have seen paper-and-pencil (P&P) tests being replaced by computerized adaptive tests (CATs) for many standardized test administrations. CATs have several advantages as compared to conventional P&P tests. CATs determine the items (questions) to administer in real time; thus, each test form is tailored to the test taker's skill level. A test taker's responses to items are recorded during the test, and a regularly updated estimate of the test taker's ability is maintained. A CAT can acquire more information about a test taker's ability while administering fewer items. Other advantages of a CAT are immediate scoring, more frequent (flexible) administrations, and the ability to utilize constructed response items.
A multistage adaptive test (MST) is an ordered collection of testlets that allows for adaptation based on a test taker's ability while exposing a preset number of items and providing a reasonable number of possible forms. (Note that a testlet is a group of items selected prior to test administration and targeted to a specific range of proficiency levels.) This test structure is a hybrid between conventional P&P and CAT whereby the test is divided into stages and a testlet is administered at each stage. The MST contains multiple stages and consists of bins in which testlets are placed. The bins at a given stage are arranged in levels corresponding to ability classifications. Various paths through the testing stages are possible, and each path contains one testlet from each stage.
This paper concentrates on the routing aspect of the MST approach. An MST is not a linear test, and at specified stages of an MST administration a decision has to be made as to the next testlet to be included in the test taker's form. Formal procedures are introduced to match a test taker's ability to future items with the objective of either maximizing information or classifying the test taker to a percentile group of the population's ability. This paper introduces a new routing procedure. Selected existing routing rules are also presented and evaluated with simulation results.