LSAT Item-Type Validity Study: A Replication (TR 13-02)

This study was conducted to evaluate the predictive validity of each of the current Law School Admission Test (LSAT) item types as well as the interrelationships among them. The current LSAT consists of three item types: Analytical Reasoning (AR), Logical Reasoning (LR), and Reading Comprehension (RC). Even though the correlation of overall LSAT scaled score with first-year average (FYA) in law school is examined on a regular basis at the Law School Admission Council (LSAC), the separate correlations for each of these three item types have only rarely been studied. The interrelationships of the item types have been studied in previous dimensionality analyses of the entire test-taking population for particular LSAT administrations, but few analyses have been conducted to verify that the nature of these relationships is being maintained.

The primary purpose of this report is to replicate the Roussos and Norton (1998) item-type validity investigation using data associated with the 2008, 2009, and 2010 first-year classes of ABA-approved law schools. To this end, correlations of AR, LR, RC, and LSAT total raw score with FYA were calculated for individual law schools. Also, correlations of AR, LR, and RC raw scores with each other were calculated for the applicant pool of each law school. All FYA correlations were corrected for restriction of range to provide an estimate of what the correlation would have been had all the applicants entered a particular law school. Such a correlation is a more accurate measure of the true predictive power of a test score. All correlations involving test sections of different lengths were corrected such that the correlations reflected the value that would have occurred had the individual test sections been the length of the entire LSAT (corrected to a test length of 101 items). This correction allowed for fair comparisons among item-type validity coefficients and between those coefficients and the total test score validity coefficient.

The results of this report are in line with those observed in the Roussos and Norton (1998) study, with the major result indicating that each of the operational LSAT item types has a substantial correlation with FYA, and that each is needed to obtain the reported overall correlation because no two item types are perfectly correlated with each other. The item type with the greatest predictive validity was LR with a validity coefficient of 0.511, followed by RC with a validity coefficient of 0.404, and finally AR with a validity coefficient of 0.277. The results also verified that the interrelationships among the item types in the law school applicant pools were the same as those previously found for all test takers for a fixed LSAT form. The results verified that LR and RC remain very highly correlated (0.736), while AR is less correlated with LR and RC, but still strongly so, with correlations of 0.566 and 0.444, respectively.