Research Reports

Unraveling the Relationship Between Testlet Features and Item Parameters: An Empirical Example (RR 12-06)

A mathematical model called item response theory (IRT) is often applied to high-stakes tests to determine the characteristics of test questions (i.e., items), such as difficulty, discrimination, and susceptibility to guessing. Note that in this context, the term “discrimination” refers to how well an item distinguishes between higher- and lower-ability test takers. Often, these tests contain subsets of items grouped around a common stimulus (testlet). This grouping often leads to items within one group (testlet) being more strongly correlated among themselves than among items from other groups, which can result in moderate to strong testlet effects.

Recently, it was shown that stimulus features could be used to predict the size of the testlet effect. Furthermore, a strong relationship was found between average item difficulty and the magnitude of the testlet effect. This study explores the relationship between stimulus features and the IRT parameters of difficulty, discrimination, and guessing. It was found that stimuli associated with easy items consisted of many different (but commonly used) words as well as an intermediate proportion of negative words. Relatively short stimuli containing many different words were found to have a high information density, and thus are considered to be very useful for distinguishing between test takers of different ability levels. No useful predictions could be made with regard to susceptibility to guessing, since that parameter did not vary much from one item to the next. It was concluded that stimulus features can be used to manipulate passage texts in such a way that they will have “favorable” properties in terms of testlet effect and average item parameters.

Request the full report

Additional reports in this collection

researchers study paperwork and examine charts and figures on a tablet

Evidence to Support Validity Claims for Using LSAT Scores...

Law School Admission Test (LSAT) scores provide a standard measure of an applicant’s proficiency in a well-defined set of important skills associated with success in law school coursework. LSAT scores are also a strong predictor of first-year grades (FYG) and cumulative grade point average (CGPA) in law school. The most recent correlational study of LSAT results (2019) shows that LSAT scores are far superior to undergraduate grade point average (UGPA) in predicting FYG...

Understanding and Interpreting Law School Enrollment Data...

The Law School Admission Council (LSAC) has a long-standing commitment to diversity, equity, and inclusion in legal education and in the legal profession. In line with its mission to promote quality, access, and equity in legal education, LSAC is providing this report, Understanding and Interpreting Law School Enrollment Data: A Focus on Race and Ethnicity, to help law schools, admission professionals, and other legal education stakeholders understand how we are measuring who is the pipeline.