LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?):
A Comprehensive Evaluation, Framework, and Benchmarks

To scale SecLLMHolmes for any chat-based LLM, we have established three configurable user inputs:

LLM responses are significantly impacted by two parameters which control token sampling: (1) temperature controls the determinism of an LLM’s output. A higher value of temperature generates more ‘creative’/‘random’ text by adding noise to potential token scores (2) top_p controls the nucleus sampling, where the LLM considers the results of the tokens with top_p probability mass. We discuss how we select the values of these parameters in Section 4.

SecLLMHolmes explores four techniques for LLM prompting: (1) zero-shot task-oriented (ZS - TO), (2) zero-shot role-oriented (ZS - RO), (3) few-shot task-oriented (FS - TO), and (4) few-shot role-oriented (FS - RO). Moreover, we divide our set of prompts in the following three categories (all prompts are described in Table III):

We design 228 code scenarios (48 hand-crafted, 30 real-world, and 150 with code augmentations) to test various aspects of the capabilities of LLMs to detect software vulnerabilities in code. We use these scenarios to craft prompts by including code, examples, definitions, and step-by-step reasoning as shown in Table III. In the following, we describe how we built our code scenarios in detail.

Similar to previous work [8], we create six code scenarios (three pairs of vulnerable and patched scenarios) for each CWE. Moreover, we design our code scenarios with three difficulty levels, (1) easy, (2) medium, (3) hard. Code scenario ‘$2_{v}$’ in a specific CWE represents the vulnerable scenario with ‘medium’ difficulty level and ‘$2_{p}$’ its patch. The difficulty levels assess how LLMs interact with code of increasing complexity. Easy scenarios consist of simple programs containing only one function and less than 30 lines of code. Medium level scenarios increase the complexity by making the program longer, using different library functions, and adding more than one user input. Hard level introduces scenarios with multiple functions in which functions can be safe on an individual level but when they work together they make the program vulnerable.²²2Appendix A shows examples of all difficulty levels for ‘CWE-22’.

As the length of the code increases significantly for the real-world code scenarios, it can exceed the maximum number of tokens a model can take as an input. To solve this problem, we shorten code files by removing comments and functions that are neither called by nor call the vulnerable (or patched) function. Also to maintain fairness among the LLMs, we make sure that all CVEs after truncation have a number of tokens less than or equal to 6,144, which is the maximum token limit supported by Palm2, and the lowest among all LLMs in our study.

1. 1.

   Trivial Augmentations. This class of code augmentations measure the robustness of LLMs to random noise. We select 12 CWE scenarios from two classes, i.e., CWE-787 (C) (#1 MITRE) and CWE-89 (Py) (#3 MITRE) of our hand-crafted dataset, and apply seven trivial augmentations (Table VI) on them and create a total of 84 different code scenarios (12 per augmentation). We choose these two CWEs as (1) they can lead to the most catastrophic impacts like root privilege escalation, and data loss, etc, (2) they belong to two completely different levels of abstractions i.e., “lower-level” (C) and “higher-level” (Python) languages, and (3) the presence of their instances can be determined directly from code without any need for external information.

2. 2.

   Non-Trivial Augmentations. We design non-trivial code augmentations to perform stress-tests on LLMs to measure their robustness and bias towards semantics of function or variable names, specific library functions, or code security practices. We use combinations of all CWEs defined in Table IV and the six non-trivial code augmentations from Table VI and design 66 code scenarios (12 per NT1-NT4 and 9 per NT5 and NT6) ³³3See Appendix LABEL:sec:app-robust for details on creation of non-trivial augmented code scenarios.

In addition to ground truth labels indicating if a code snippet contains a vulnerability, we also need explanations for these vulnerabilities, as we aim to evaluate the reasoning capabilities of LLMs and assess whether they can justify their decisions. To this end, we randomly sample 48 code scenarios out of the 228 total scenarios, and have three security experts, including the first author of the paper equally divide the sampled code scenarios amongst each other and create a 100-word ground truth reason for each scenario using MITRE’s official CWE documentation [41] as a guide. The experts then compare and discuss each others’ reasoning and develop consensus for each ground-truth reasoning (Fleiss’ kappa with $K=0.93$, meaning almost perfect agreement [46]). After establishing that the criteria for ground truth reasoning are well laid out and understood, the first author proceeds to develop the remaining 180 vulnerability explanations. This ground truth reasoning $G_{r}$ is then used by the ‘Evaluator’ module in the next step.

The output of an LLM for a specific test is passed to the Evaluator (see Figure 2). As SecLLMHolmes is fully automated, we leverage GPT-4 to analyze the response. First, the response is passed to GPT-4 ⁴⁴4Our manual evaluation shows that GPT-4 performs the best for this extraction process., with an additional role-based instruction prompt $P_{e}$ (shown in Figure 7 in the Appendix) in the ‘system’ input to extract two pieces of information from the raw response. The first one is the binary answer, which is “yes/no” based on whether the LLM found a vulnerability in the given code or not. We find that in some cases the LLM does not provide a definite answer, therefore we include a third verdict, i.e., “n/a.”⁵⁵5For example, in some cases LLMs provide some variation of the following response: “As an AI model I cannot answer this question.” The second part of the information extracted is the textual reasoning provided by the LLM ($P_{r}$). To extract it, we ask GPT-4 to summarize the reason described by the model output on why a vulnerability is present in the code or not, in 100 words. We extract the summary of root cause of vulnerability from raw responses of LLMs to maintain consistency for further evaluation methods, and to avoid contents like suggestions, fixed code, etc. to be analyzed. In the rest of this section we provide more details on the evaluation metrics used to evaluate the final answer, and its reasoning provided by LLM for the vulnerability detection task.

1. 1.

   Rouge [47] score is traditionally used in NLP to measure the similarity between a machine-translated summary and reference summaries using overlapping n-grams. In our case, we use it to measure similarity between the summaries $P_{r}$ and $G_{r}$. We first sample $50$ pairs of $P_{r}$ and their corresponding $G_{r}$, as our reasoning score validation set ($R_{val}$), and manually check the consistency and alignment of their reasonings. We find that at the optimal threshold $Rouge_{thres}$ of $0.34$, $43$ out of $50$ $P_{r}$ are consistent with $G_{r}$. We therefore use this threshold and mark two summaries as similar if their Rouge score exceeds it.

2. 2.

   Cosine Similarity is a metric commonly used in NLP to measure how similar two documents are irrespective of their sizes. We first convert the summaries $P_{r}$ and $G_{r}$ into fixed length vectors using OpenAI’s embedding model ‘text-similarity-davinci-001’ and calculate the cosine similarity between them. Similar to $Rouge_{thres}$, we find that the optimal threshold $Cos_{thres}$ is $0.84$, and consider two summaries similar if their cosine similarity exceeds this value.

3. 3.

   GPT-4 is prompted to evaluate if the reasoning in $P_{r}$ and $G_{r}$ align. If the reasonings are similar and they align with each other, GPT-4 responds ‘yes’ and we assign a reasoning score of $1$, otherwise we assign $0$. We find that GPT-4 successfully classifies $48$ out of $50$ $P_{r}$ correctly to their corresponding $G_{r}$, when validated on $R_{val}$.

To perform a rigorous comparison between LLMs and assess their capabilities, it is of critical importance that their responses are consistent, meaning that running the same test multiple times under identical parameters should provide the same final verdict. Therefore, we first investigate whether this consistency is achievable at all, and what LLM parameters deliver the most consistent results. OpenAI’s documentation [48] recommends a temperature of $0.2$ and ‘top_p’ of $0.1$ to achieve the most deterministic output for code related tasks. Similarly, the recommended ‘temperature’ value for all LLMs in our evaluation is $0.2$. When experimenting with modifying these values, both the OpenAI documentation [49] and previous research [7] recommend to keep the value of ‘top_p’ constant and modify the value of ‘temperature.’ We therefore fix ‘top_p’ to default value specific to an LLM, and perform experiments using two different ‘temperature’ values: $0.2$ (‘default’) and $0.0$. We perform this experiment on two vulnerable and two patched medium code difficulty level scenarios ($2_{v}$ and $2_{p}$) from two distinct vulnerabilities, “out-of-bound write” (CWE-787) in C and “SQL injection” (CWE-89) in Python (for the same reasons as discussed in Section V). For the input prompts we select the set of Standard prompts (see Section 3.3). We run each experiment ten times, and record how many times the model provides the same answer. We consider a model to be consistent if it always provides the same binary answer, irregardless of whether it is correct.