Research Library

A Foundation of Validity

Beginning with the very first notion of a standardized test for admission to law school, validity was a primary focus. In his May 17, 1945 letter to the College Entrance Examination Board (CEEB) suggesting the development of such a test, Frank H. Bowles, Director of Admissions at Columbia University, stated the seven criteria listed below.

Since the inception of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) has sought to evaluate and ensure its validity for use in the law school admission process. As predictive validity is an important component in the overall evaluation of test validity, LSAC has carried out predictive validity studies, also called LSAT Correlation Studies, since the test was first administered.

Contextual Information for Holistic Evaluation in Law School Admission

The figure below shows the progression of undergraduate yearly grade-point average (GPA) for law school applicants across 4 years of undergraduate study divided into quintiles based on individual Law School Admission Test (LSAT) scores.

The goal of the Law School Admission Council (LSAC) Skills Analysis Study is to identify the skills that law school faculty consider important for success in required law school courses. If certain tasks are required of all or most law school required courses, the skills involved in those tasks can be inferred to be essential to success in law school.

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group.

The aim of this study was twofold: First, we investigated whether scores on an admission test administered in proctored and unproctored environments led to similar predictions of future academic success. Second, we explored how Bayesian modeling can be of help in interpreting admission-testing data. Results showed that the two modes of administering an admission test did not require the use of different models for predicting academic success, and that Bayesian modeling provides a very useful and easy-to-interpret framework for predicting future academic success.

This study employed a data simulation to evaluate the impact of a strategy to reduce test length by including only high-quality test questions, where quality was defined by a statistical indicator of the degree to which a question distinguishes between more and less able test takers. The impact of this strategy on the rank ordering of simulated test takers according to their total test score was evaluated, as was the predictive validity and classification accuracy of scores based on the shorter tests.

With computerized testing, it is possible to record not only the responses of test takers to test questions but also other details about the test taker’s activity, such as the amount of time spent responding to each question. These details comprise a new type of data called process data. This report proposes a new approach to modeling responses, response times, and other process data: Test-taker data that naturally belong together are grouped in a cross-classification structure. Five examples of models applying this approach are illustrated.