When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards

The advent of transformer-based Large Language Models (LLMs) (OpenAI, 2023; Deepmind, 2023; Anthropic, 2023; Anil et al., 2023; Touvron et al., 2023) has led to a generational leap in generative models, enabling interaction with computing devices through natural language. This advancement encompasses improvements that have rendered many earlier benchmarks and leaderboards obsolete (Laskar et al., 2023; Shen et al., 2023), leading to the compilation of more challenging and comprehensive tests. However, the current generation of leaderboards still does not satisfy many of the requirements of researchers and practitioners looking to build on LLMs Ethayarajh and Jurafsky (2021); Dehghani et al. (2021). Since LLMs are extremely expensive to both train and inference, selecting the LLM (or LLM training recipe) is often the most costly decision for the entire project. Stable leaderboards are critical to making the right decision.

Leaderboards based on multiple choice questions (MCQ) for evaluation (Wang et al., 2018, 2019; Nie et al., 2019; Zhong et al., 2023; Hendrycks et al., 2020) present both convenience and significant limitations (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023). While MCQs offer an automated and quantifiable means to assess certain aspects of model ability (e.g., knowledge), they fall short as a stable means to measure performance. Figure 1 demonstrates the instability of the leaderboard ranking of one popular benchmark, Massive Multitask Language Understanding (MMLU) Hendrycks et al. (2020), under small perturbations.

Moreover, the reliance on MCQs raises concerns about the models being overfit to these benchmarks, potentially excelling in structured tests while lacking real-world applicability. This discrepancy highlights the need for more holistic and diverse evaluation methods that transcend the simplicity of MCQs Liang et al. (2023). It also prompts critical reflection on how these models might inadvertently be trained to achieve high scores through spurious correlations, pattern recognition, and optimization for specific question formats rather than genuine language comprehension or knowledge. As LLMs continue to evolve, it is imperative to develop evaluation frameworks that can more accurately assess their abilities in a way that mirrors the complexity of real-world use.

Despite being widely used, benchmarking with MCQs has turned out to be anything but simple. It requires the full synchronization of evaluation frameworks and results often vary wildly due to nuanced differences. For example, minor changes in prompting and scoring can produce invalid results for particular LLMs¹¹1https://huggingface.co/blog/evaluating-mmlu-leaderboard. Recent studies have investigated the issue of sensitivity in evaluating LLMs, with some demonstrating that LLMs are susceptible to the ordering of answer choices and bias towards specific tokens/symbols (Zheng et al., 2023; Pezeshkpour and Hruschka, 2023; Lu et al., 2022), and others exploring different prompt modifications and their effects in benchmarking LLMs (Mizrahi et al., 2024; Sclar et al., 2023; Weber et al., 2023; Zhao et al., 2021).

In this work, we conduct a broad range of minor perturbation experiments to MCQ benchmarks and observe the disruption it causes to model rankings on leaderboards. We also take additional steps to precisely identify the limitations of LLMs on this measurement approach.

The contributions of this paper can be summarized as follows:

1. 1.

   Existing model rankings on popular benchmarks break down under slight perturbations, particularly in the medium to small model sizes.

2. 2.

   This behavior can be explained by the susceptibility of all tested LLMs to various forms of bias in MCQ.

3. 3.

   Some families of LLMs have an over-reliance on format, pointing to potential benchmark leakage.

4. 4.

   We find that LLMs also exhibit bias to the scoring method for answer choices in MCQ.

5. 5.

   We demonstrate that some categories of modifications do not affect the benchmark rankings.

Evaluating LLMs with MCQs has rapidly become a standard for measuring the knowledge and reasoning capabilities of the model OpenAI (2023); Anil et al. (2023); Deepmind (2023); Jiang et al. (2023). Many such MCQ benchmarks have been used to measure LLMs, including Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020), Ai2 Reasoning Challenge (ARC) (Clark et al., 2018a), and Common-sense Question Answers (CSQA) (Saha et al., 2018).

Mechanically, testing LLMs with MCQs is accomplished by presenting the question along with the answer choices to the model and selecting the choice deemed most probable by the model. Although this setup appears straightforward, LLMs react in unpredictable ways to formatting and other minor changes to the questions or the answers. LLM performance on an MCQ test can change with the introduction of an extra space (e.g., between the question and answer) or adding an additional instructional phrase (e.g., "Choices:"). In addition to this brittleness, Pezeshkpour and Hruschka (2023) found changes to the order in which answer choices are presented to GPT4 and instructGPT can change the model’s prediction.

These findings lead us to take a deeper look at how MCQ-based benchmark results are affected by small perturbations to question formats, LLM prompts, presentation of few-shot examples, and other dimensions. In particular, we introduce variations in three categories:

• •

  Answer choice format and ordering: testing the limits of LLM sensitivity to ordering and formatting (Section 3.1).

• •

  Prompt and scoring modifications: changing text included in the prompt and analyzing different scoring schemes (Section 3.2).

• •

  In-context knowledge manipulation: inserting relevant/irrelevant information in the prompt and/or few-shot examples (Section 3.3).

Our main aim is to quantify how these small perturbations/variations change the rankings of a set of models on a particular benchmark. As MCQ benchmarks-based leaderboards are often used to compare models and guide model selection, we investigate the robustness of benchmarks for this purpose. Figure 1 demonstrates how existing benchmarks exhibit significant undesirable shifts in rankings under small perturbations.

In this section, we describe and justify the perturbations we apply in each category. We note that some MCQ test changes, like modifying the order of answer choices, can change performance even for humans but the effect is typically not pronounced (Lions et al., 2021). In general, our modifications are designed to be small perturbations to the MCQ and prompts that should not affect performance. The exception to this is some of the in-context knowledge manipulations described in Section 3.3, which are designed to improve or degrade performance drastically.

In the bulk of our experiments, we focus on the MMLU benchmark due to the extensive nature of our experiments (11 models, 22+ settings), and extend some experiments to ARC-challenge to show generalizability.

MMLU Hendrycks et al. (2020) is a commonly used benchmark for comparing LLMs, consisting of 57 subjects spanning four domains: humanities, STEM, social sciences, and others. Each subject includes at least 100 multiple-choice questions with 4 answer choices. The entire benchmark contains 14,042 questions (Tables  A.29 and  A.30 have a breakdown of the MMLU subjects and their distributions).

Ai2 Reasoning Challenge Clark et al. (2018b) is a benchmark consisting of 7787 grade school science questions. The benchmark is split into two sets: Easy and Challenge. We conduct experiments on the Challenge set (ARC-C) which is proven to contain harder questions for existing models. The questions in ARC-C have 3-5 answer choices.

Unless otherwise stated, the reported score for each experiment/model combination on MMLU is the mean accuracy across all 14,042 questions. All tested model tokenizers encode the multiple-choice answers as single tokens. Hence, the accuracy is equivalent to the normalized accuracy. All baseline and modified MMLU benchmarks were performed using the LM Evaluation Harness Gao et al. (2023) library. Their implementation of MMLU measures the log-likelihood of each of the answer tokens [’A’, ’B’, ’C’, ’D’] after the input prompt and chooses the letter with the highest probability as the model’s answer.

Some of our experiments require permuting the answer choice order (Table 1, A.8, A.9, and  A.10), however, this can be confusing for questions where the answer choices are dependent on their position, such as “D. All of the above.”, or reference other choices, such as “C. Both A and B.”. To circumvent this dependency, we manually inspected and modified the questions from three subjects to ensure their answers are permutation-independent for a subset of our experiments. The modified subjects are college chemistry, college mathematics, and global facts.

For each variation introduced to the MCQ benchmarks, we calculate the change in accuracy ($\Delta$Acc) and recall standard deviation (RStd) for each model. RStd measures the bias of a model to a particular answer choice by computing the standard deviation of recalls for each answer choice Zheng et al. (2023). This metric quantifies how much the model favors particular positions for the correct answer choice. We typically observe whether RStd changes ($\Delta$RStd) are significant across experimental settings.

To measure the change in ranking induced by an applied perturbation to a benchmark, we measure the normalized Kendall’s $\tau$ distance between two rankings of $n$ models Kendall (1938). Kendall’s $\tau$ computes the number of swapped pairs between two rankings normalized by the total number of pairs $\frac{n(n-1)}{2}$. We report $k_{\tau}=\frac{\tau+1}{2}$, where $k_{\tau}=1.0$ indicates total agreement between rankings, and $k_{\tau}=0$ indicates complete disagreement by reversing the original rankings.

In this section, we highlight the major findings of our work and combine the results of multiple lines of experimentation (detailed in Section 3) into concise observations. Additional observations and complete experimental results can be found in the appendix (Section A.1).

Benchmarks for the evaluation of LLMs Chang et al. (2023) such as MMLU Hendrycks et al. (2020), HELM Liang et al. (2023), and BigBench Suzgun et al. (2022) have seen widespread adoption recently. Depending on the ability that is being assessed (e.g., language generation, knowledge understanding, complex reasoning) some benchmarks are designed in the form of close-ended problems like MCQs. To facilitate comparisons among LLMs, a number of leaderboards aggregating these benchmarks have been established, such as the OpenLLM Leaderboard Beeching et al. (2023) and OpenCompass Contributors (2023).

However, issues with the leaderboards and the underlying benchmarks have emerged. In a case study, Deng et al. (2023) discovered contamination/leakage of the MMLU benchmark in the training sets of multiple models. A significant portion of models memorized benchmark questions and was able to perfectly reconstruct the removed part of some benchmark questions or answers. For instance, GPT-4 correctly completed the questions in 29% of the prompts with URL hinting.

Even under the assumption of uncontaminated data, the performance of models on the underlying benchmarks are not robust to minor perturbations. Pezeshkpour and Hruschka (2023) showed that specific orderings of MMLU answer choices resulted in up to $\pm 30\%$ deviations in GPT-4 performance on various subjects. Similarly, Zheng et al. (2023) demonstrate that models are biased to certain answer letters. On llama-30B, they showed a 27% difference in MMLU accuracy by forcing all correct answers to either position A or D. As well, Robinson et al. (2023) find that the accuracy of LLMs improve (without regard to bias) when evaluating using a pure multiple choice question style vs a cloze question answering style.

While prior work has highlighted weaknesses in LLMs themselves Zheng et al. (2023); Pezeshkpour and Hruschka (2023), evaluation method Robinson et al. (2023), or the contents of benchmarks Dehghani et al. (2021) in our work we thoroughly study the effects these factors have on existing leaderboards and demonstrate where leaderboards lack robustness.

Building robust leaderboards is a major challenge for the community, as leaderboards help practitioners select the best methods and models for continued research. Given this importance, it is critical to address the breakdown of existing leaderboards to the slight perturbations we demonstrated in our work. In addition to building our understanding of the causes of this sensitivity (e.g., bias in LLMs and bias in scoring methods), future work should aim to adopt and design benchmark practices that avoid these pitfalls.

The limitations of our work fall into two main categories: (i) understanding the causes of LLM bias and (ii) our limited success at overcoming leaderboard sensitivity.

To explain LLM bias, we attempted to design experiments that isolate each source of bias under MCQ but were unable to quantify the relative effects of bias or conclude why they occur. This was further complicated by our inability to access the pretraining datasets of the LLMs to rule out benchmark contamination. Future work in this direction will most likely require tools from interpretability research (e.g. mechanistic interpretability).

One of our main contributions was to highlight where MCQ-based leaderboards fail to deliver stable rankings. Although we succeeded in showing this, we were unable to demonstrate a robust solution to this problem. Our recommendation to, for example, use hybrid scoring methods is still not completely robust to perturbations.

In this work, we do not present a new leaderboard. There is a risk that Figure 1 is interpreted as a leaderboard or used for model selection. Our intention was to demonstrate how minor changes can affect model rankings.