It is crucial for any testing organization to be able to recognize inconsistencies in test-taker performance across multiple standardized test administrations or across sections within a single administration. One method of recognizing inconsistent performance is to construct multiple posterior distributions and to compare the divergence between pairs of such distributions. (Note: In the context of a test taker's performance, a posterior distribution summarizes what we know about the probability associated with certain levels of performance by the test taker.)
Comparing posteriors takes into account all information available from the responses and allows consideration of various partitions of the test taker's responses to test questions (items). Practical partitions include scored versus unscored items, hard versus easy items, unexposed versus exposed items, uncompromised versus compromised items, and items of one type versus items of another type. A large divergence between posteriors indicates a significant change in a test taker's performance. Such changes could be an indication of answer copying, item pre-knowledge, or test-taker pre-identification of the unscored section.
This paper evaluates the use of Kullback–Leibler divergence (KLD) to compare posterior distributions in the context of the Law School Admission Test. The statistical characteristics of KLD are presented and evaluated. KLD has been applied for this purpose in the context of magnetic resonance imaging, human gene analysis, stochastic complexity, and sample size selection. It is concluded that the properties of KLD support its use in the comparison of posterior distributions to identify inconsistent test-taker response cases.
Request the Full Report
To request the full report, please email Linda Reustle at lreustle@LSAC.org.