Research Library

A Foundation of Validity

Beginning with the very first notion of a standardized test for admission to law school, validity was a primary focus. In his May 17, 1945 letter to the College Entrance Examination Board (CEEB) suggesting the development of such a test, Frank H. Bowles, Director of Admissions at Columbia University, stated the seven criteria listed below.

Since the inception of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) has sought to evaluate and ensure its validity for use in the law school admission process. As predictive validity is an important component in the overall evaluation of test validity, LSAC has carried out predictive validity studies, also called LSAT Correlation Studies, since the test was first administered.

Contextual Information for Holistic Evaluation in Law School Admission

The figure below shows the progression of undergraduate yearly grade-point average (GPA) for law school applicants across 4 years of undergraduate study divided into quintiles based on individual Law School Admission Test (LSAT) scores.

The goal of the Law School Admission Council (LSAC) Skills Analysis Study is to identify the skills that law school faculty consider important for success in required law school courses. If certain tasks are required of all or most law school required courses, the skills involved in those tasks can be inferred to be essential to success in law school.

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group.

The aim of this study was twofold: First, we investigated whether scores on an admission test administered in proctored and unproctored environments led to similar predictions of future academic success. Second, we explored how Bayesian modeling can be of help in interpreting admission-testing data. Results showed that the two modes of administering an admission test did not require the use of different models for predicting academic success, and that Bayesian modeling provides a very useful and easy-to-interpret framework for predicting future academic success.

This study employed a data simulation to evaluate the impact of a strategy to reduce test length by including only high-quality test questions, where quality was defined by a statistical indicator of the degree to which a question distinguishes between more and less able test takers. The impact of this strategy on the rank ordering of simulated test takers according to their total test score was evaluated, as was the predictive validity and classification accuracy of scores based on the shorter tests.

With computerized testing, it is possible to record not only the responses of test takers to test questions but also other details about the test taker’s activity, such as the amount of time spent responding to each question. These details comprise a new type of data called process data. This report proposes a new approach to modeling responses, response times, and other process data: Test-taker data that naturally belong together are grouped in a cross-classification structure. Five examples of models applying this approach are illustrated.

A new statistical model is proposed to study the effects of various testing conditions on a population of test takers. This flexible model allows for numerous effects to be considered simultaneously. A Bayesian approach is employed, taking prior information into consideration. An empirical example demonstrates the utility of the suggested model to test the influence of item presentation formats on the performance of test takers. This research could be of practical value in a potential transition of the Law School Admission Test (LSAT) from a paper-and-pencil format to a digital mode.

Many standardized tests are now administered via computer rather than paper-and-pencil format. The computer-based delivery mode brings with it certain advantages, one of which is the ability to record not only the test taker’s response to each item (i.e., question), but also the amount of time the test taker spends considering and answering each item. Research on how to represent and utilize response time data has proliferated, but most of the research is based on the assumption of constant working speed in relation to a certain accuracy level.

Since the inception of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) has sought to evaluate and ensure its validity for use in the law school admission process. As predictive validity is an important component in the overall evaluation of test validity, LSAC has carried out annual predictive validity studies, also called LSAT Correlation Studies, since the test was first administered.

Test theory typically deals with categorical responses to test questions (items), for instance, correct/incorrect responses or responses that represent a choice from a finite number of alternatives. Whenever technically possible, it is attractive to collect information on continuous response variables that accompany these responses as a covariate. One obvious example is response time; other examples are information on cursor movement in computer-based testing, eye-tracking information, or physiological information.

In high-stakes testing, it is important to verify the validity of individual test scores. Although a test, in general, results in valid test scores for most test takers, there may be individual test takers with unusual answer patterns for whom test score validity is questionable. One example of such aberrance is a test taker who guesses on a large number of questions or one who has preknowledge of the answers to some questions. An effective statistical technique (developed for a single test) was extended for tests that consist of multiple subtests, as does the Law School Admission Test.

Several statistics used to detect inconsistent patterns of correct/incorrect answers to test questions (items) were evaluated based on data from one Analytical Reasoning (AR) and one Logical Reasoning (LR) section of the Law School Admission Test. Item score patterns were also evaluated based on gender and racial/ethnic subgroups. We showed that test takers who were consistently flagged by all statistics evaluated and for both the AR and the LR sections had relatively low scores, which may have been the result of extensive guessing.