Detection of Answer Copying via Kullback–Leibler Divergence and K-Index (RR 09-01)
In every high-stakes test, there will be test takers who try to gain an advantage by copying answers from another test taker. All testing organizations take steps to prevent such behavior, but complete isolation of each test taker from any other test taker is not possible. A testing location may have more than 100 test takers, and exams are often administered in one large room. Despite the efforts of exam proctors, there is always the potential for answer copying to occur. That being said, a balance must be obtained between test security and practical considerations. The purpose of this report is to present a two-stage approach to automatically detect pairs of test takers who have a high probability of involvement in copying.
The Law School Admission Test (LSAT) consists of an operational part with four sections and a variable part with a single section. Typically, two test takers in close proximity will have the same items on the operational part and different items on the variable part. Test takers put the same effort into responding to all sections because it is unknown which section is variable. This format for the LSAT allows the construction of an automatic method to identify copying. Significant differences in performance level on the variable and operational parts are first recognized. Anyone in this category is placed on a list of potential subjects. This means that the response patterns of the test takers on the list are suspicious. Further checks are necessary to detect if the subject copied from a source. All test takers in the vicinity of the subject are potential sources. A statistical test involving the matching of responses between each subject–source pair takes place to identify copying.
In this study, the detection process entailed two automatic stages. In the first stage, we created a list of potential subjects by comparing performance differences on the variable and operational parts. In the second stage, we performed a statistical test to detect copying. The copying-detection methods used were Kullback–Leibler (KL) divergence and the K-Index statistic. In the first stage, we used KL divergence to recognize performance differences between the operational and variable parts. Test takers with large KL divergence values showed aberrant behavior and were placed on the list of potential subjects. In the second stage, we used the K-Index statistic to identify copying. This statistic is based on the matching of incorrect responses between pairs of test takers and has been used at Educational Testing Service (ETS) for several decades. Using this two-stage approach, we were able to automatically identify subject-source pairs. This is demonstrated in the results section of the report.
A third nonautomatic stage could be added to this two-stage approach. This last stage would involve inspecting seating charts, interviewing proctors, and deciding on the actions to be taken.