Research Library

A Foundation of Validity

Beginning with the very first notion of a standardized test for admission to law school, validity was a primary focus. In his May 17, 1945 letter to the College Entrance Examination Board (CEEB) suggesting the development of such a test, Frank H. Bowles, Director of Admissions at Columbia University, stated the seven criteria listed below.

Since the inception of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) has sought to evaluate and ensure its validity for use in the law school admission process. As predictive validity is an important component in the overall evaluation of test validity, LSAC has carried out predictive validity studies, also called LSAT Correlation Studies, since the test was first administered.

Contextual Information for Holistic Evaluation in Law School Admission

The figure below shows the progression of undergraduate yearly grade-point average (GPA) for law school applicants across 4 years of undergraduate study divided into quintiles based on individual Law School Admission Test (LSAT) scores.

The goal of the Law School Admission Council (LSAC) Skills Analysis Study is to identify the skills that law school faculty consider important for success in required law school courses. If certain tasks are required of all or most law school required courses, the skills involved in those tasks can be inferred to be essential to success in law school.

Test collusion (TC) is the sharing of test materials or answers to test questions (items) before or during a test. Because of the potentially large advantages for the test takers involved, TC poses a serious threat to the validity of score interpretations. The proposed approach applies graph theory methodology to response similarity analyses to identify groups involved in TC while minimizing the false-positive detection rate. The new approach is illustrated and compared with a recently published method using real and simulated data.

This study employed a data simulation to evaluate the impact of a strategy to reduce test length by including only high-quality test questions, where quality was defined by a statistical indicator of the degree to which a question distinguishes between more and less able test takers. The impact of this strategy on the rank ordering of simulated test takers according to their total test score was evaluated, as was the predictive validity and classification accuracy of scores based on the shorter tests.

This report addresses a general type of cluster aberrancy in which a subgroup of test takers has an unfair advantage on some subset of administered items. Examples of cluster aberrancy include item preknowledge and test collusion. In general, cluster aberrancy is hard to detect due to the multiple unknowns involved: Unknown subgroups of test takers have an unfair advantage on unknown subsets of items. The issue of multiple unknowns makes the detection of cluster aberrancy a challenging problem from the standpoint of applied mathematics.

Most high-stakes testing programs apply methods to identify unlikely patterns of correct/incorrect responses to test questions. Some examples of why such patterns may occur include misinterpretation of questions, question preknowledge, answer copying, or guessing behavior. This report provides an overview of existing approaches to identifying atypical response patterns that fall into a class of analyses known as nonparametric statistics. Results of a simulation study comparing the different approaches, along with guidelines for applying these indices in practice, are also presented.

Since the inception of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) has sought to evaluate and ensure its validity for use in the law school admission process. As predictive validity is an important component in the overall evaluation of test validity, LSAC has carried out annual predictive validity studies, also called LSAT Correlation Studies, since the test was first administered.

This report presents a new algorithm for detecting groups of test takers (aberrant groups) who had access to subsets of test questions (aberrant subsets) prior to an exam. This method is in line with the development of statistical methods for detecting test collusion, a new research direction in test security. Test collusion may be described as the large-scale sharing of test materials, including answers to test questions. The algorithm employs several new statistics to perform a sequence of statistical tests to identify aberrant groups.

The Law School Admission Council (LSAC) has carried out annual predictive validity studies, also called LSAT Correlation Studies, since the Law School Admission Test (LSAT) was first administered. These studies are geared toward evaluating and ensuring the effectiveness and validity of LSAT scores for use in the law school admission process. In conjunction with these predictive validity studies, LSAC also conducts differential validity and differential prediction studies on the LSAT to ensure that the test is fair across gender subgroups.

The Law School Admission Council (LSAC) has carried out annual predictive validity studies, also called LSAT Correlation Studies, since the Law School Admission Test (LSAT) was first administered. These studies are geared toward evaluating and ensuring the effectiveness and validity of LSAT scores for use in the law school admission process. In conjunction with these predictive validity studies, LSAC also conducts differential validity and differential prediction studies on the LSAT to ensure that the test is fair across racial/ethnic subgroups.

Many standardized tests are now administered via computer rather than paper-and-pencil format. In a computer-based testing environment, it is possible to record not only the test taker’s response to each question (item), but also the amount of time spent by the test taker in considering and answering each item. Response times (RTs) provide information not only about the test taker’s ability and response behavior but also about item and test characteristics. The current study focuses on the use of RTs to detect aberrant test-taker responses.

In standardized testing, test takers may change their answer choices for various reasons. The statistical analysis of answer changes (ACs) has uncovered multiple testing irregularities on large-scale assessments and is now routinely performed at some testing organizations. Research on answer-changing behavior has recently branched off in several directions, including modeling of ACs and addressing scanning errors.

While an admission test may strongly predict success in university or law school programs for most test takers, there may be some test takers who are mismeasured. To address this issue, a class of statistics called person-fit statistics is used to check the validity of individual test scores. However, most person-fit statistics are designed for a single test, and not much is known about the performance of these statistics for admission tests consisting of multiple highly correlated subtests.

In standardized multiple-choice testing, test takers often change their answers for various reasons. The statistical analysis of answer changes (ACs) has uncovered multiple testing irregularities on large-scale assessments and is now routinely performed at some testing organizations. This report presents two new approaches to analyzing ACs at the individual test-taker level. The information about all previous answers is used only to partition the data into two disjoint subsets: responses where an AC occurred and responses where an AC did not occur.