As sponsor of the Law School Admission Test (LSAT), the Law School Admission Council (LSAC) is committed to assuring that every LSAT form meets rigorous standards of quality and fairness. In order to achieve this goal, each test question is subjected to a multilevel review process before it is ever used on an LSAT form. LSAT questions are written by test-question-writing experts both outside LSAC and in LSAC Test Development. Each new question undergoes a careful review by LSAC staff and by independent experts outside LSAC to ensure that it
- is clear and unambiguous,
- has one and only one best answer, where the best answer is the one among the choices provided that most accurately and most completely answers the question that is posed, and
- meets the LSAC standards for fairness and sensitivity.
Questions that meet the strenuous LSAC review criteria are assembled into pretest sections. Pretest sections are administered to a sample of test takers from the LSAT test-taking population. Results from the pretest provide test development staff with statistical information about each question, and with information about possibly ambiguous or misleading information in the question or in one or more of the answer choices. If problems are identified, either the question is discarded or it is revised and pretested again. All questions that pass the quality standards of a pretest administration are placed in the LSAT test question item bank. New test sections are assembled by selecting questions from this LSAT item bank. Each fully assembled test section is administered on one or more separate occasions for the purpose of pre-equating the new form. Pre-equating is a statistical method used to adjust for minor fluctuations in the difficulty of different test forms so that a test taker is neither advantaged nor disadvantaged by the particular form that is given. Following each pre-equating administration, the statistical information about each question is reviewed to ensure that the question is of appropriate difficulty, discriminates higher-ability test takers from lower-ability test takers, is unambiguous, and has a single best answer. When the test is given at a regular LSAT administration, but before final scoring is completed, statistical analysis is conducted one last time. Each question is evaluated using the same criteria that were applied following the pretesting and pre-equating administrations. If a problem is found, the question is eliminated from the test before final scoring and reporting are accomplished.
Despite these precautions, on rare occasions, an error or ambiguity may be found in a test question by a test taker. If the test taker demonstrates that the test question does not have one and only one best answer among the choices provided, corrective action is taken. The LSAC Board of Trustees has adopted the following test question challenge policy in order to provide test takers the opportunity to inquire about or to challenge the scoring of test questions.
A candidate who has taken the LSAT and whose answer to a question has not been scored as the "credited response" (i.e., has been scored incorrect) is entitled to have the question reviewed pursuant to the following policy if, within 90 days after a score report is sent to the candidate, he or she files with LSAC an initial written inquiry about, or challenge to, the scoring of the question, stating and supporting the reasons why the credited response is not the one and only best answer to the question.
First Review Level
Upon receipt of an initial written inquiry about, or challenge to, the scoring of a test question, and explanation of why the credited response is not the one and only best answer, the LSAC test development staff will conduct an expeditious and thorough review of the inquiry or challenge, and, upon completion of the review, will respond in writing to the candidate. The written response will state the decision reached by LSAC concerning the candidate's inquiry or challenge, and explain the reasons for that decision.
Second Review Level
If the foregoing First Review sustains the scoring of the question about which the candidate inquired, or which the candidate challenged, the candidate may request a second review of the question's scoring. The request must be in writing and filed with LSAC within 30 days after the written response to the initial inquiry or challenge is sent to the candidate by LSAC and must be supported by the reason or reasons why the initial LSAC response was incorrect or inadequate.
Upon receipt of such a request, the LSAC Director of Test Development will initiate a review of the scoring of the question by a panel of three independent external experts. The Director will select, from a list of subject matter and testing experts, the most appropriate reviewers for the particular test question, taking into consideration the subject matter of the question and the nature of the inquiry or challenge. No expert whose name is included on this list may be employed by, or associated with, the LSAC Board of Trustees or its committees, LSAC, or its item writing or test assembly contractors, other than as a question challenge reviewer under the Second Review Level.
The Director of Test Development will forward to this panel of experts all documents, whether produced by the candidate or by LSAC, related to the question inquiry or challenge. The Director's transmittal letter may respond to arguments raised in the candidate's request for panel review. The panel's decision concerning the scoring of the question about which the candidate inquired, or which the candidate challenged, will be that of a majority of panel members, and must be filed in writing with the Director within 30 days after the last member of the panel is named. The panel's written communication to the Director must state its decision concerning the scoring of the question, and explain the reasons for the decision. Within 10 days of receipt of the panel's written decision, the Director will forward a copy of the decision to the candidate.
Third Review Level
If the decision of the Second Review Level panel is not unanimous, either LSAC or the candidate may request a further review by arbitration of the question about which the candidate inquired or which the candidate challenged. Such a request must be filed within 30 days after a copy of the written decision of the Second Review Level panel is sent to the candidate. Arbitration will not be available until the Second Review Level panel has filed its written decision concerning the test question inquiry or challenge with the Director of Test Development.
Arbitration proceedings shall be conducted in Philadelphia, Pennsylvania, pursuant to the current arbitration rules of the American Arbitration Association. Hearings shall be conducted before a board of three arbitrators to be drawn from lists provided by the American Arbitration Association.
If LSAC requests arbitration, the reasonable compensation and expenses of all three arbitrators, and the cost of a transcript, if any, of the arbitration proceedings shall be paid by LSAC. If the candidate requests arbitration, the reasonable compensation and expenses of the three arbitrators, and the cost of a transcript of the arbitration proceedings, if any, shall be divided equally between LSAC and the candidate. If the candidate requests arbitration and the candidate's inquiry about, or challenge to, the scoring of the test question is sustained by the arbitrators, however, LSAC will reimburse the candidate for the candidate's share of the reasonable compensation and expenses of the arbitrators, and of the cost of a transcript of the arbitration proceedings, if any. If LSAC or the candidate requests additional copies of the transcript of the arbitration proceedings, costs shall be imposed according to the rules of the American Arbitration Association.
Arbitration shall be the final and conclusive review of inquiries about, or challenges to, the scoring of a test question. Unless LSAC or the candidate requests a hearing before the arbitrators, the decision of the arbitrators shall be based on documents, whether produced by the candidate or by LSAC, related to the scoring of the test question about which the candidate inquired, or which the candidate challenged.
The sole purpose of arbitration will be to determine whether the question, the scoring of which the candidate has challenged, has one and only one best answer, where the best answer is the one, among the choices provided, that most accurately and most completely answers the question that is posed. Arbitrators acting according to these procedures shall not be empowered to address any other issues. The party that requests arbitration shall have the burden of proving that the decision of the panel of three experts who reviewed the scoring of the question at the Second Review Level is incorrect. Any decision of a majority of the arbitrators within the scope of the standard stated in the first sentence of this paragraph shall be final and binding on the candidate and LSAC and judgment upon the decision rendered by the arbitrators may be entered in any court having jurisdiction thereof.