Feature Engineering-Based Detection of Buffer Overflow Vulnerability in Source Code

Using Neural Networks

Mst Shapna Akter1, Hossain Shahriar2, Juan Rodriguez Cardenas3, Sheikh Iqbal Ahamed4, and Alfredo Cuzzocrea5
1
Department of Computer Science, Kennesaw State University, USA
2
Department of Information Technology, Kennesaw State University, USA
3
Department of Information Technology, Kennesaw State University, USA
4
Department of Computer Science, Marquette University, USA
5
iDEA Lab, University of Calabria, Rende, Italy

Abstract—One of the most significant challenges in the field represents the greatest proportion of software vulnerabilities in
of software code auditing is the presence of vulnerabilities in at least three years, accounting for 41.8% [3]. Furthermore,
software source code. Every year, more and more software flaws according to a Frost and Sullivan analysis released in 2018,
are discovered, either internally in proprietary code or publicly
disclosed. These flaws are highly likely to be exploited and can severe and high severity vulnerabilities increased from 693 in
lead to system compromise, data leakage, or denial of service. 2016 to 929 in 2017, with Google Project Zero coming in
To create a large-scale machine learning system for function-level second place in terms of disclosing such flaws. On August 14,
vulnerability identification, we utilized a sizable dataset of C and 2019, Intel issued a warning about a high-severity vulnerability
C++ open-source code containing millions of functions with poten- in the software it uses to identify the specifications of Intel
tial buffer overflow exploits. We have developed an efficient and
scalable vulnerability detection method based on neural network processors in Windows PCs [4]. The paper claims that these
models that learn features extracted from the source codes. The defects, including information leaking and denial of service
source code is first converted into an intermediate representation assaults, might substantially affect software systems. Although
to remove unnecessary components and shorten dependencies. We the company issued an update to remedy the problems, attack-
maintain the semantic and syntactic information using state-of- ers can still use these vulnerabilities to escalate their privileges
the-art word embedding algorithms such as GloVe and fastText.
The embedded vectors are subsequently fed into neural networks on a machine that has already been compromised. In June
such as LSTM, BiLSTM, LSTM-Autoencoder, word2vec, BERT, 2021, a vulnerability in the Windows Print Spooler service
and GPT-2 to classify the possible vulnerabilities. Furthermore, was discovered that allowed attackers to execute code remotely.
we have proposed a neural network model that can overcome The vulnerability, known as PrintNightmare, was caused by a
issues associated with traditional neural networks. We have used buffer overflow and affected multiple versions of Windows in
evaluation metrics such as F1 score, precision, recall, accuracy,
and total execution time to measure the performance. We have 2021 [5]. Microsoft released a patch to address the issue, but
conducted a comparative analysis between results derived from reports later emerged that the patch was incomplete and still
features containing a minimal text representation and semantic left systems vulnerable.
and syntactic information. We have found that all neural network
models provide higher accuracy when we use semantic and syntac- To reduce losses, early vulnerability detection is a good
tic information as features. However, this approach requires more technique. The proliferation of open-source software and code
execution time due to the added complexity of the word embed- reuse makes these vulnerabilities susceptible to rapid propaga-
ding algorithm. Moreover, our proposed model provides higher tion. Source code analysis tools are already available; however,
accuracy than LSTM, BiLSTM, LSTM-Autoencoder, word2vec
and BERT models, and the same accuracy as the GPT-2 model
they often only identify a small subset of potential problems
with greater efficiency. based on pre-established rules. Software vulnerabilities can be
found using a technique called vulnerability detection. Con-
Keywords: Cyber Security; Vulnerability Detection; Neural ventional vulnerability detection employs static and dynamic
Networks; Feature Extraction; techniques [6]. Static approaches evaluate source code or exe-
cutable code without launching any programs, such as data flow
I. INTRODUCTION analysis, symbol execution [7], and theorem proving [8]. Static
Security in the digital realm is becoming increasingly im- approaches can be used early in software development and have
portant, but there is a significant threat to cyberspace from excellent coverage rates, but they have a significant false pos-
invasion. Attackers can breach systems and applications due to itive rate. By executing the program, dynamic approaches like
security vulnerabilities caused by hidden software defects. In- fuzzy testing and dynamic symbol execution can confirm or
ternally, proprietary programming contains thousands of these ascertain the nature of the software. Dynamic methods depend
flaws each year [1]. For example, the ransomware Wannacry on the coverage of test cases, which results in a low recall
swept the globe by using a flaw in the Windows server mes- despite their low false positive rate and ease of implementation.
sage block protocol [2]. According to the Microsoft Security The advancement of machine learning technology incorporates
Response Center, there was an industry-wide surge in high- new approaches to address the limitations of conventional
severity vulnerabilities of 41.7% in the first half of 2015. This approaches. One of the key research directions is to develop
intelligent source code-based vulnerability detection systems. It successfully used for classifying software security activities
can be divided into three categories: using software engineering like malware, ransomware, and network intrusion detection.
metrics, anomaly detection, and weak pattern learning [9]. We have examined machine learning-related papers that have
Initially, software engineering measures, including software been applied to the software security domain. Previously,
complexity [10], developer activity [11], and code commits Zeng et al. [16] reviewed software vulnerability analysis and
[12] were investigated to train a machine learning model. This discovery using deep learning techniques. They found four
strategy was motivated by the idea that software becomes game-changing methods that contributed most to software
more susceptible as it becomes more complicated, but accuracy vulnerability detection using deep learning techniques. These
and recall need to be improved. Allamanis et al. [13] have concepts are automatic semantic feature extraction using deep
shown that the syntactic and semantic information in the learning models, end-to-end solutions for detecting buffer over-
code increases the detection accuracy in anomaly detection. flow vulnerabilities, applying a bidirectional Long Short-Term
Moreover, one work has shown the detection of the anomaly Memory (BiLSTM) model for vulnerability detection, and deep
using fully-fledged codes [14]. It reveals previously uniden- learning-based vulnerability detectors for binary code. Zhou et
tified weaknesses, but false positive and false negative rates al. [17] proposed a method called graph neural network for
are high. Another work has shown an approach with clean vulnerability identification with function-level granularity to
and vulnerable samples to learn vulnerable patterns [15]. This address the issue of information loss during the representation
method performs very well but relies on the quality of the learning process. They transformed the samples into a code
dataset. In our work, we propose a solution for detecting property graph format. Then, a graph neural network made
software buffer overflow vulnerability using neural networks up of a convolutional layer and a gated graph recurrent layer
such as Simple RNN, LSTM, BilSTM, word2vec, BERT, learned the vulnerable programming pattern. This method im-
GPT2, and LSTM-Autoencoder. We first transform source code proves the detection of intra-procedural vulnerabilities. How-
samples into the minimum intermediate representations through ever, they did not address inter-procedural vulnerabilities. Iorga
a tokenizer provided by the Keras library. Later, we extract et al. [18] demonstrated a process for early detection of
semantic features using word embedding algorithms such as cyber vulnerabilities from Twitter, building a corpus of 650
GloVe and fastText. After finishing the data preprocessing annotated tweets related to cybersecurity articles. They used
stage, we feed the input representation to the neural networks the BERT model and transfer learning model for identifying
for classification. Moreover, we develop a neural network that cyber vulnerabilities from the articles. The BERT model shows
works best among all the models. All the models have been 91% accuracy, which they found adequate for identifying
evaluated using evaluation metrics such as f1 score, precision, relevant posts or news articles. Sauerwein et al. [19] presented
recall, accuracy, and total execution time. The following is a an approach for automated classification of attackers’ TTPs
summary of our contributions: by combining NLP with ML techniques. They extracted the
1. Extracting semantic and syntactic features using GloVe attackers’ TTPs from unstructured text. To extract the TTPs,
and fastText. 2. Vulnerability Detection in Source Code using they used a combination of NLP with ML techniques. They
LSTM, BilSTM, LSTM-Autoencoder, word2vec, BERT, and assessed all potential combinations of the specified NLP and
GPT-2 with an minimal intermediate feature representation of ML approaches with 156 processing pipelines and an automati-
the texts. 3. Vulnerability Detection in Source Code using cally generated training set. They found that tokenization, POS
LSTM, BiLSTM, LSTM-Autoencoder, word2vec, BERT, and tagging, IoC replacement, lemmatization, one-hot encoding,
GPT-2 with semantic and syntactic features. 4. Proposal of binary relevance, and support vector machine performed best
a neural network that outperforms the results derived from for the classification of techniques and tactics. Harer et al.
existing models. Comparison between results derived from [20] created a dataset composed of millions of open-source
neural networks trained with a minimal intermediate feature functions annotated with results from static analysis. The
representation of the texts and semantic and syntactic features. performance of source-based models is then compared against
The rest of the paper is organized as follows: we provide a approaches applied to artifacts extracted from the build process,
brief background study on software vulnerability detection in with source-based methods coming out on top. The best
section 2. Then we explain the methods we followed for our performance is found when combining characteristics learned
experimental research in section 3. The results derived from by deep models with tree-based models. They evaluated the use
the experiment are demonstrated in Section 4. Finally, section of deep neural network models alongside more conventional
5 concludes the paper. models like random forests. Finally, their best model achieved
an area under the ROC curve of 0.87 and an area under
II. LITERATURE REVIEW the precision-recall curve of 0.49. Pistoia et al. [21] surveyed
Researchers are interested in the recently developed ma- static analysis methods for identifying security vulnerabilities
chine learning strategy for identifying and preventing soft- in software systems. They discussed three topics that have
ware and cybersecurity vulnerabilities in order to address been linked to security vulnerability sources: application pro-
the shortcomings of conventional static and dynamic code gramming interface conformance, information flow, and access
analysis techniques. Various machine learning techniques, in- control. They addressed static analysis methods for stack-based
cluding naive bayes, logistic regression, recurrent neural net- access control and role-based access control separately since
works (RNN), decision trees, and support vector machines are access control systems can be divided into these two main
types. They reviewed some effective static analysis techniques, understanding of what kind of important features the dataset
including the Mandatory Access Rights Certification of Objects might have.
(MARCO) algorithm, the Enterprise Security Policy Evaluation
(ESPE) algorithm, the Static Analysis for Validation of Enter- TABLE I: Most common words and their frequencies
prise Security (SAVES) algorithm, and Hammer, Krinke, and index Common words Count
Snelting’s algorithm. However, static analysis produces false
0 = 505570
positive results and relies on predefined rules. For new errors,
1 if 151663
the static analysis method is unsuitable, as it cannot recognize
and detect them. 2 {\n 113301
3 == 92654
III. METHODOLOGY 4 return 77438
From the standpoint of source code, the majority of flaws 5 * 71897
originate in critical processes that pose security risks, such as 6 the 71595
functions, assignments, or control statements. Adversaries can 7 }\n 63182
directly or indirectly affect these crucial operations by manip- 9 int 53673
ulating factors or circumstances. To successfully understand
10 /* 51910
patterns of security vulnerabilities from code, neural network
models must be trained on a large number of instances. In this 11 ¡ 43703
study, we analyze the lowest level of codes in software package 12 */\n 43591
functions, capable of capturing vulnerable flows. We utilized a 13 + 41855
sizable dataset containing millions of function-level examples 14 to 39072
of C and C++ code from the SATE IV Juliet Test Suite, the 15 && 36180
Debian Linux distribution, and open-source Git repositories
16 for 35849
on GitHub, as mentioned in Russell’s work [22]. Our project
17 }\n\n 34017
employs the CWE-119 vulnerability feature, which indicates
issues related to buffer overflow vulnerability. Buffer overflow 18 char 33334
occurs when data written to a buffer exceeds its length, 19 else 31358
overwriting storage units outside the buffer. According to a
2019 Common Weakness Enumeration report, buffer overflow 1) Data Preprocessing: In this study, we conducted a series
vulnerability has become the most adversely affected issue. of data preprocessing techniques to prepare our dataset for
Although we focus on buffer overflow, our method can identify the neural networks. The data preprocessing steps we em-
other vulnerabilities. Figure 1 illustrates an intra-procedural ployed include tokenization, stop word removal, stemming,
buffer overflow vulnerability. Our dataset is divided into three lemmatization, and the use of pre-trained embeddings. Initially,
subfolders—train, validation, and test—each containing a CSV we performed tokenization, which is the process of breaking
file with 100,000, 20,000, and 10,000 data instances, respec- down the source code into smaller units called tokens. Tokens
tively. The CSV files store text data and corresponding labels, represent the basic units of analysis for computational purposes
allowing systematic evaluation of the model’s performance and in natural language processing tasks. For this process, we
adaptability throughout the learning process. utilized the Keras tokenizer, which provides methods such as
tokenize() and detokenize() to process plain text and separate
words [23]. Following tokenization, we applied stop word
removal, stemming, and lemmatization techniques to further
preprocess the tokens. Stop word removal eliminates common
words that do not provide significant information, while stem-
ming and lemmatization normalize the tokens by reducing them
to their root form. These techniques help in reducing noise and
improving the efficiency of the neural networks.
We first converted the tokens into numerical representations
using minimal intermediate representation with the Keras to-
kenizer. The Keras tokenizer assigns a unique integer index
to each token in the vocabulary and represents the source
code as a sequence of these integer indices. This represen-
Fig. 1: An example of buffer overflow vulnerability. tation is more efficient than one-hot encoding, as it does not
involve creating large, sparse vectors. However, it still lacks
semantic information about the tokens. To further enhance the
We analyzed the dataset and found some common words representation of the source code tokens and better capture
(shown in Table 1) with their corresponding counts. The visual- semantic and syntactic information, we utilized pre-trained
ization of common words in the dataset provides a preliminary embeddings, namely GloVe and fastText. We stacked GloVe
and fastText embeddings together for extracting the semantic gradient problem of traditional RNNs.It was first proposed
and syntactic information from the source code. Both of these by Hochreiter and Schmidhuber [27]. Using this model for
embeddings have demonstrated strong performance in various sequential datasets is effective, as it can handle single data
NLP tasks and can effectively capture the relationships between points. It follows the Simple RNN model’s design and is an
words in the source code. GloVe is an unsupervised learning extended version of that model [28, 29]. Our LSTM model
algorithm that generates vector representations of words based consists of an input layer that determines the dimensionality
on global word-word co-occurrence statistics from a corpus of the input data features. We incorporated three hidden layers,
[24]. FastText, an extension of the skip-gram method, generates each containing 128 memory cells that can capture long-term
character n-grams of varying lengths for each word and learns dependencies in the input sequence. The output of each LSTM
weights for each n-gram, as well as the entire word token, layer is fed into a dropout layer with a dropout rate of 0.2 to
allowing the model to capture the meaning of suffixes, prefixes, prevent overfitting. The final output of the last LSTM layer is
and short words [25]. We separately fed the minimal inter- fed into a dense layer with two units and a sigmoid activation
mediate representation with Keras tokenizer and the semantic function to produce the final binary classification output. The
and syntactic representations derived from GloVe and fastText LSTM cell comprises three gates: the input gate, forget gate,
into our neural network models. This approach allowed us to and output gate, which regulate the flow of information into
compare the performance of the models when using different and out of the cell. To introduce non-linearity into the model,
input representations, helping us identify the most effective we use the hyperbolic tangent (tanh) activation function in
method for detecting security vulnerabilities in the source code. the LSTM cell. Furthermore, we utilize the Rectified Linear
Unit (ReLU) activation function in the output layer to generate
A. Classification Models
non-negative predictions. We optimize the LSTM model using
In this section, we discuss various classification models that the Binary Cross-Entropy loss function and Adam optimization
were utilized in our study. These models include Simple RNN, algorithm. The model’s hyperparameters include a learning rate
LSTM, BiLSTM, LSTM-Autoencoder, Word2vec, BERT, and of 0.001, batch size of 32, and 50 training epochs.
GPT-2. These models are designed to work with different types
of data, such as text, time series, and sequences, and have D. Bidirectional Long short-term memory (BiLSTM)
been widely employed in natural language processing and other The Bidirectional Long Short-Term Memory (BiLSTM) is a
related tasks. type of recurrent neural network that enhances the capabilities
B. Simple Recurrent Neural Network (RNN) of the traditional LSTM by introducing bidirectional processing
The Simple Recurrent Neural Network (RNN) is a type of of the input sequence. It was first proposed by Graves [30].
artificial neural network that can model sequential data by This idea sets it apart from the LSTM model, which can learn
utilizing a directed graph and temporally dynamic behavior. patterns from the past to the future [31] .Our BiLSTM model
RNNs consist of an input layer, a hidden layer, and an output comprises an input layer that determines the dimensionality
layer [26]. These networks have a memory state added to each of the input data features. We have incorporated three hidden
neuron, allowing them to capture temporal dependencies in the layers, each containing 128 memory cells that can capture long-
data. The dimensionality of the input layer in our Simple Re- term dependencies in the input sequence. The output of each
current Neural Network (RNN) model is determined based on BiLSTM layer is fed into a dropout layer with a dropout rate of
the input data features. The hidden layer consists of 256 units, 0.2 to prevent overfitting. The final output of the last BiLSTM
which use memory states to capture temporal dependencies layer is fed into a dense layer with two units and a sigmoid
in the data. We use the hyperbolic tangent (tanh) activation activation function to produce the final binary classification
function in the hidden layer to introduce non-linearity into output. The BiLSTM cell has two sets of three gates, namely
the model. We chose this activation function due to its ability the input gate, forget gate, and output gate, one set that
to handle vanishing gradients more effectively compared to processes the input sequence in the forward direction and
other activation functions like sigmoid. The output layer of another set that processes the input sequence in the backward
the Simple RNN model is designed to generate predictions direction. This bidirectional processing allows the model to
based on the processed input data. The number of units in capture dependencies in both the past and future context of
the output layer corresponds to the number of classes, which the input sequence. To introduce non-linearity into the model,
is two. We use an appropriate activation function, such as we use the hyperbolic tangent (tanh) activation function in the
sigmoid for binary classification, in the output layer to generate BiLSTM cell. Furthermore, we utilize the Rectified Linear Unit
probability scores for each class. To optimize the model, we (ReLU) activation function in the output layer to generate non-
choose the Binary Cross entropy loss function and employ the negative predictions. We optimize the BiLSTM model using
Adam optimization algorithm. We set hyperparameters such as the Binary Cross-Entropy loss function and Adam optimization
learning rate to 0.001, batch size to 32, and the number of algorithm. The model’s hyperparameters include a learning rate
training epochs to 50. of 0.001, batch size of 32, and 50 training epochs.

C. Long short-term memory (LSTM) E. LSTM-Autoencoder

The Long Short-Term Memory (LSTM) is a type of recurrent The LSTM-Autoencoder is a variant of the Long Short-
neural network designed to solve the vanishing and exploding Term Memory (LSTM) model that utilizes an autoencoder
architecture. The LSTM-Autoencoder is designed to read input We set the batch size to 128 and the number of training epochs
sequences, encode sequences, decode sequences, and recon- to 5.
struct sequences for a given sequential dataset, referred to
G. BERT
as encoder-decoder [32]. Its performance is estimated based
on how well the model can recreate the sequence. LSTM BERT (Bidirectional Encoder Representations from Trans-
autoencoder can be used on video, text, audio, and time-series formers) is a state-of-the-art pre-trained language model de-
sequence data. The model accepts a series of various lengths veloped by Google. BERT is a bidirectional transformer-based
of inputs and outputs for various purposes, such as translating architecture that can capture the context of a word in a sentence
from one language to another. The series is transformed into by looking at the surrounding words [34]. The BERT model
a vector representation by the encoder, and the vector is consists of 12 transformer blocks for the base version and
transformed back into a sequence of outputs or texts by the 24 transformer blocks for the large version. Each transformer
decoder. The meaning of the outputs is maintained in the block has a multi-head attention mechanism and a feed-forward
vector representation. In this model, we have an input layer neural network, making it capable of modeling long-term
that determines the dimensionality of the input data features. dependencies in the input sequence. In our implementation of
The LSTM encoder layer contains 128 memory cells that can BERT, we utilized the pre-trained BERT model and fine-tuned
capture long-term dependencies in the input sequence. The it on our specific NLP task. We utilized the pre-trained BERT
LSTM decoder layer has the same number of memory cells model with 12 transformer blocks, 12 attention heads, and 110
as the encoder layer, which allows the model to reconstruct million parameters. We added a dense layer with 2 units and
the input sequence. To introduce non-linearity into the model, a sigmoid activation function to perform binary classification.
we use the hyperbolic tangent (tanh) activation function in the We utilized the Binary Cross-Entropy loss function and Adam
LSTM cells. Additionally, we utilize the Mean Squared Error optimization algorithm to optimize the model. We set the
(MSE) loss function to calculate the reconstruction loss of the learning rate to 2e-5 and the batch size to 32. To fine-tune the
autoencoder. The model’s hyperparameters include a learning pre-trained BERT model, we trained it on our specific NLP
rate of 0.001, batch size of 32, and 50 training epochs. To eval- task using a training set of 100,000 instances and a validation
uate the performance of the LSTM-Autoencoder, we calculate set of 20,000 instances. We trained the model for 3 epochs and
the reconstruction error between the input and reconstructed evaluated its performance on a separate test set, which constists
sequence. The lower the reconstruction error, the better the of 10,000 instances.
model’s ability to capture the input sequence’s structure. H. GPT-2
GPT-2 (Generative Pre-trained Transformer 2) is a state-
F. Word2vec of-the-art language model developed by OpenAI. It is a
transformer-based language model that can generate coherent
Word2vec is a word embedding model specifically designed and fluent text in a wide range of styles and topics [35]. GPT-
for working with textual data. Word embedding is a tech- 2 has a large number of parameters, with the base version
nique for representing words that allows computer programs having 117 million parameters, and the largest version having
to understand words with similar meanings. By employing 1.5 billion parameters. In our implementation of GPT-2, we
a neural network model to map words into vectors of real utilized the pre-trained GPT-2 model to generate text for our
numbers, word2vec is capable of capturing significant accurate specific NLP task. We fine-tuned the pre-trained GPT-2 model
syntactic and semantic word relationships. After training, the on a large corpus of text relevant to our task to improve its
two-layer neural network can recognize synonymous terms and performance. We used the GPT-2 model with 117 million
suggest new words for incomplete phrases [33]. Our Word2vec parameters for our task. To fine-tune the pre-trained GPT-2
model comprises an input layer that takes in the one-hot model, we used a training set of 100,000 instances and a
encoded words and a single hidden layer containing a specified validation set of 20,000 instances. We fine-tuned the model
number of neurons, which represent the latent dimensions of for 3 epochs and evaluated its performance on a separate test
the word embeddings. We utilize the Skip-gram architecture set. We used the perplexity metric to evaluate the performance
with negative sampling to train the Word2vec model. In this of the model. We utilized the Adam optimization algorithm
architecture, the model predicts the surrounding words given with a learning rate of 1e-5 and a batch size of 32 to optimize
a target word or predicts the target word given surrounding the model.
words. The negative sampling technique helps to efficiently
train the model by reducing the computation required to update I. Proposed Model
the weights of the model. The output layer is not used in the We propose a stacking ensemble learning approach to im-
Word2vec model, and the trained weights of the hidden layer prove the performance of our system. Stacking ensemble is an
represent the learned word embeddings. These embeddings advanced machine learning technique that combines multiple
can be used in various downstream NLP tasks such as text heterogeneous weak learners (base models) to form a single
classification, sentiment analysis, and machine translation. To stronger model (meta-learner). In this approach, the base mod-
optimize the model, we use the Stochastic Gradient Descent els’ predictions are used as input to the meta-learner, which
(SGD) optimization algorithm with an initial learning rate of ultimately makes the final prediction. The meta-learner used
0.025 and decrease the learning rate linearly over time to 0.001. in this case is a logistic regression model, while the base
models consist of Simple RNN, LSTM, BiLSTM, word2vec, prediction yfinal
i for the i-th sample can be computed using
and LSTM-Autoencoder. These models are trained with one- the logistic function:
dimensional data as input.
Since the predicted dataset from Level 0 already contains
the expected values’ probabilities, the meta-learner can provide
accurate probabilities from Level 0. To mitigate overfitting, the
meta-learner is trained using both the validation dataset and the
outputs. The final result is the level-1 prediction.

1
yifinal = weighted
)
1+ e−(p i

Fig. 2: Proposed stacking ensemble learning architecture.

The architecture is divided into two levels, Level 0 and Level

1, as shown in Figure 2. Level 0 consists of Simple RNN,
LSTM, BiLSTM, word2vec, and LSTM-Autoencoder models.
After learning the data patterns, each of the base models
generates predictions simultaneously. All models in Level 0
contribute equally to the overall model performance.
Level 1, also referred to as the meta-learner, is built using
logistic regression. The meta-learner at Level 1 is fed the Level
0 predicted outputs as input. Based on the Level 0 predictions,
the meta-learner calculates the best weighted outputs. A “meta-
learner” is a model that can quickly learn a pattern or adapt to
different datasets with a small amount of training data. It learns
patterns from the outputs generated by the five base models. As By using a diverse set of base models, we can mitigate
a result, the model can effectively learn completely new data the limitations of traditional stacking ensemble approaches that
and produce acceptable output. The meta-learner’s parameters employ similar base models, leading to similar predictions. If
are a combination of the parameters of the five neural networks a single base model performs poorly on the dataset, there is
in the base models. a high likelihood that the final output will also be inferior.
Mathematically, the stacking ensemble learning approach Conversely, with a diverse set of base models, the strengths
can be represented as follows: and weaknesses of individual models complement each other,
Let M be the number of base models, pm be the probability which results in a more robust and accurate overall model.
i
This is because each base model is able to capture different
of the positive class for the i-th sample predicted by the m-th aspects or patterns in the data, thereby reducing the reliance on
base model, and wm be the weight assigned to the m-th base
any single model’s performance. Additionally, the meta-learner
model. The weighted probability pi weighted for the i-th sample
can effectively combine these diverse predictions to generate
can be computed as:
a more accurate and stable final prediction, minimizing the
M
impact of individual model biases or errors. In conclusion,
X
pweighted
i
= wm · pm
i
the utilization of heterogeneous base models in a stacking
m=1
ensemble approach provides a more resilient and powerful
predictive model, capable of handling various types of data
The weights wm are determined by the meta-learner using and delivering superior performance compared to traditional
the Level 0 predictions and the validation data. The final ensemble methods.
Algorithm 1 Proposed Stacking Ensemble Learning Algo- metrics contributes to a comprehensive understanding of the
rithm. model’s effectiveness, generalization, and efficiency [36–38].
Function stacking_ensemble(data, train ratio, Below, we provide a brief description of each evaluation metric:
val ratio, test ratio)
// Initialize Level 0 base models A. Precision
simple rnn ← SimpleRNN(); Precision is a measure of the accuracy of the positive
lstm ← LSTM(); predictions made by the model. It is calculated as the ratio of
bi lstm ← BiLSTM(); true positive predictions to the sum of true positive and false
lstm autoencoder ← LSTM Autoencoder(); positive predictions. In other words, it quantifies the proportion
word2vec model ← Word2Vec(); of correct positive predictions among all the instances predicted
models ← [simple rnn, lstm, bi lstm, as positive. A higher precision value indicates that the model
lstm autoencoder, word2vec model]; is better at identifying relevant instances and minimizing false
// Initialize Level 1 meta-learner positive predictions.
meta learner ← LogisticRegression()
True Positives
Precision = (1)
// Split the data into training, True Positives + False Positives
validation, and testing sets B. Recall
X train, X val, X test, y train, y val, y test ← Recall, also known as sensitivity or true positive rate,
data split(data, train ratio, val ratio, test ratio) measures the proportion of actual positive instances that are
correctly identified by the model. It is calculated as the ratio of
// Train Level 0 base models true positive predictions to the sum of true positive and false
foreach model in models do negative predictions. A higher recall value indicates that the
model.fit(X train, y train) model is better at detecting positive instances and minimizing
// Make predictions with Level 0 base false negative predictions.
models
True Positives
Level0 outputs ← list() Recall = (2)
foreach model in models do True Positives + False Negatives
pred ← model.predict(X val) C. F1-score
Level0 outputs.append(pred)
F1-score is the harmonic mean of precision and recall, and it
// Concatenate Level 0 outputs provides a balanced measure of both metrics. It is particularly
Level0 outputs combined ← concate- useful when dealing with imbalanced datasets, where one class
nate(Level0 outputs) is significantly more prevalent than the other. The F1-score
ranges from 0 to 1, with a higher value indicating better overall
// Train Level 1 meta-learner performance of the model in terms of both precision and recall.
meta learner.fit(Level0 outputs combined, y val)
Precision · Recall
F1-score = 2 · (3)
// Make final predictions with Level 1 Precision + Recall
meta-learner D. Accuracy
Level0 test outputs ← list() Accuracy is a widely-used metric that quantifies the propor-
foreach model in models do
tion of correct predictions made by the model, both positive and
test pred ← model.predict(X test)
negative, relative to the total number of instances. It provides
Level0 test outputs.append(test pred)
an overall indication of the model’s performance, but it may
// Concatenate Level 0 test outputs not be a reliable metric when dealing with imbalanced datasets,
Level0 test outputs combined ← concate- as it can be biased towards the majority class.
nate(Level0 test outputs)
True Positives + True Negatives
Accuracy = (4)
// Generate Level 1 final predictions Total Instances
final predictions ← meta learner.predict(Level0 test outputs)E. Execution Time
return final predictions Execution time is a measure of the computational efficiency
of the model. It refers to the amount of time required to
train the model and make predictions. A shorter execution
IV. EVALUATION METRICS time indicates that the model is more efficient, which can be
In order to assess the performance of the Neural Networks particularly important in real-world applications where time
and our proposed stacking ensemble model, we have employed constraints are critical. By evaluating the execution time, we
a range of evaluation metrics that provide insight into various can assess the trade-offs between model performance and
aspects of model performance. These metrics include precision, computational resources. These evaluation metrics provide a
recall, F1-score, accuracy, and execution time. Each of these comprehensive and robust assessment of our neural network
and proposed model’s performance. By considering multiple 0.92, 0.92, 0.93, and 0.94. The LSTM Autoencoder model’s
aspects of performance, we can ensure that our model is not performance slightly decreased with an accuracy of 0.90. The
only accurate but also efficient, generalizable, and reliable BERT, GPT-2, and proposed models maintain their superior
across various datasets and application scenarios. performance, with accuracies of 0.94, 0.95, and 0.95, re-
spectively. The execution times for all models vary, with the
V. RESULT AND DISCUSSION proposed model having a runtime of 2 hours and 46 minutes.
In this study, we investigated the role of semantic and syntac- Figure 3 shows the performance metrics for different neural
tic features in vulnerability prediction for CWE-119, focusing network models on vulnerable source code without using any
on buffer overflow vulnerabilities. We began by converting word embedding algorithms. The models considered are Sim-
the text dataset into a minimal intermediate representation ple RNN, LSTM, BiLSTM, Word2vec, LSTMAutoencoder,
using a tokenizer provided by the Keras library. This basic BERT, GPT-2, and a proposed model. The metrics considered
representation assigns a numerical value to each word with- are accuracy, precision, recall, and F1 score. The results
out considering semantic information. Since the meaning of demonstrate that the proposed model outperforms all other
code is often better captured by considering the context of models in terms of accuracy and F1 score, achieving an
multiple words, we employed state-of-the-art word embedding accuracy of 0.94 and an F1 score of 0.99. The execution time
algorithms—GloVe and fastText—to extract semantic features of the proposed model is also relatively fast compared to other
from function-level codes. These features were then fed into models, taking only 2 hours and 31 minutes.
neural network models for vulnerability prediction. We used Figure 4 presents the classification results of the same neural
100,000 instances for training, 20,000 for validation, and network models on vulnerable source code using GloVe and
10,000 for testing. Our evaluation metrics included accuracy, fastText word embedding algorithms. The results demonstrate
precision, recall, and F1 score, with a focus on minimizing false that all models achieved higher accuracy and F1 score com-
positives and false negatives. We trained seven neural network pared to the results in Figure 3. The proposed model continues
models (Simple RNN, LSTM, BiLSTM, word2vec, BERT, to perform the best with an accuracy of 0.95 and an F1 score
GPT-2, and LSTM-Autoencoder) and our proposed stacking of 0.99. However, the execution time of the proposed model is
ensemble neural network model. Our ensemble learning model longer compared to Figure 3, taking 2 hours and 46 minutes.
outperformed single models, achieving the highest accuracy in These figures provide a clear comparison of the performance
vulnerability prediction. of different neural network models and highlight the effective-
Table 2 presents the results of vulnerable source code ness of using word embedding algorithms for improving the
classification using different neural network models without classification results of vulnerable source code. The proposed
word embedding algorithms. The Simple RNN model achieves model performs well in both scenarios, showing its potential
an accuracy of 0.89, precision of 0.88, recall of 0.88, and F1 as a reliable classification model.
score of 0.92, with an execution time of 42 minutes and 8 In Table 4, we present a comparison analysis between
seconds. The LSTM model has slightly better performance with our proposed model and previous works in the domain of
an accuracy of 0.90, precision of 0.90, recall of 0.90, and F1 vulnerability detection. The table highlights the differences in
score of 0.92, and takes 29 minutes and 48 seconds to run. terms of the purpose of each study, the data used, whether
The BiLSTM model shows further improvement, obtaining an semantic or syntactic feature extraction was performed, the
accuracy of 0.91, precision of 0.93, recall of 0.90, and F1 score highest performance achieved, and if efficiency measurements
of 0.87, but requires 2 hours and 5 minutes for execution. were conducted.
The Word2vec model yields an accuracy of 0.89, precision Lorga et al. [18] aimed at vulnerability detection using
of 0.92, recall of 0.95, and F1 score of 0.93, with a runtime of Twitter text data, but they did not perform semantic or syn-
40 minutes and 2 seconds. The LSTM Autoencoder model has tactic feature extraction. Their model achieved an accuracy of
an accuracy of 0.91, precision of 0.93, recall of 0.94, and F1 94.96%, and they did not provide any efficiency measurements.
score of 0.94, taking 53 minutes and 13 seconds for execution. Similarly, Foret et al. [39] worked on vulnerability detection
The BERT model performs better with an accuracy of 0.92, using news articles without incorporating semantic or syntactic
precision of 0.93, recall of 0.93, and F1 score of 0.95, but features, resulting in an 87% accuracy. No efficiency measure-
requires 2 hours and 38 minutes to run. The GPT-2 model has ment analysis was conducted in their work as well. Harer et
an accuracy of 0.92, precision of 0.97, recall of 0.98, and F1 al. [20] and Russell et al. [22] both focused on vulnerability
score of 0.97, with a considerably longer execution time of 7 detection in source code but did not consider semantic or
hours and 48 minutes. Lastly, the proposed model outperforms syntactic feature extraction. Their models achieved F1-scores
the other models with an accuracy of 0.94, precision of 0.99, of 49.99% and 56.6%, respectively, without any efficiency
recall of 0.98, and F1 score of 0.99, and takes 2 hours and 31 measurement analysis. Behzadan et al. [40] also worked on vul-
minutes to execute. nerability detection in source code without extracting semantic
Table 3 shows the results when using GloVe and Fast- or syntactic features. They reported an accuracy of 94.72%,
Text embeddings. In general, the performance of all models but no efficiency measurement analysis was performed.
improved when using these embeddings. The Simple RNN, Our proposed model targets vulnerability detection in source
LSTM, BiLSTM, and Word2vec models show a similar trend code and incorporates semantic and syntactic feature extraction
in improvement, with their respective accuracies increasing to using GloVe and fastText embeddings. As a result, our model
TABLE II: Vulnerable Source code Classification results using different Neural network models with no word embedding
algorithms
Models Accuracy precision Recall F1 Execution
Score Time
Simple RNN 0.89 0.88 0.88 0.92 42min 8s
LSTM 0.90 0.90 0.90 0.92 29min 48s
BiLSTM 0.91 0.93 0.90 0.87 2h 5min
Word2vec 0.89 0.92 0.95 0.93 40min 2s
LSTMAutoencoder 0.91 0.93 0.94 0.94 53min 13s
BERT 0.92 0.93 0.93 0.95 2h 38min
Gpt2 0.92 0.97 0.98 0.97 7h 48min
Proposed Model 0.94 0.99 0.98 0.99 2h 31min

TABLE III: Vulnerable Source code Classification results using different Neural network models with embedding algorithms
GloVe + fastText
Models Accuracy precision Recall F1 Execution
Score time
Simple RNN 0.92 0.93 0.93 0.97 42min 8s
LSTM 0.92 0.93 0.95 0.97 33min 13s
BiLSTM 0.93 0.96 0.96 0.99 45min 3s
Word2vec 0.94 1.00 0.98 0.99 42min 56s
LSTMAutoencoder 0.90 0.93 0.94 0.95 59min 53s
BERT 0.94 0.95 0.95 0.99 5h 16min
Gpt2 0.95 0.97 0.98 0.99 8h 33min
Proposed Model 0.95 0.97 0.98 0.99 2h 46min

TABLE IV: Comparative analysis with previous work

Previous authors Purpose Data Semantic or Syn- Highest percent- Efficiency Mea-
tactic feature ex- age surement?
traction?
Lorga et al. [18] Vulnerability detection Twitter text No 94.96% No
data (Accuracy)
Foret et al. [39] Vulnerability detection News No 87% (Accuracy) No
Articles
Harer et al. [20] Vulnerability detection Source code No 49.99% (F1- No
score)
Russell et al. [22] Vulnerability detection Source code No 56.6% (F1-score) No
Behzadan et al. Vulnerability detection Source code No 94.72% No
[40] (Accuracy)
Our Proposed Vulnerability detection Source code Yes 95% (Accuracy) Yes
Model

achieves the highest accuracy of 95% compared to the previ- VI. CONCLUSION
ous works. Moreover, we contribute to efficient measurement Our research aims to detect implementation vulnerabilities
analysis and perform an in-depth analysis of the features that early in the development cycle by leveraging the power of
were not considered in previous studies. This comprehensive
neural networks. We have collected a large dataset of open-
approach allows us to better understand the factors influencing source C and C++ code and developed a scalable and efficient
the performance of vulnerability detection models and develop
vulnerability detection method based on various neural network
more effective methods for detecting security vulnerabilities in models. We compared the performance of different models,
source code. including Simple RNN, LSTM, BiLSTM, LSTM-Autoencoder,
Fig. 3: Performance Metrics for Different Neural Network Models on Vulnerable Source Code without Word Embedding
Algorithms

Word2Vec, BERT, and GPT-2, and found that models with ceedings of the 28th international conference on Software
semantic and syntactic information extracted using state-of-the- engineering, pp. 492–501, 2006.
art word embedding algorithms such as GloVe and FastText [2] T. Manikandan, B. Balamurugan, C. Senthilkumar,
outperform those with a minimal text representation. Our R. R. A. Harinarayan, and R. R. Subramanian, “Cyberwar
proposed neural network model has shown to provide higher is coming,” Cyber Security in Parallel and Distributed
accuracy with greater efficiency than the other models evalu- Computing: Concepts, Techniques, Applications and Case
ated. We have also analyzed the execution time of the models Studies, pp. 79–89, 2019.
and proposed a trade-off between accuracy and efficiency. [3] A. Arora and R. Telang, “Economics of software vulnera-
Overall, our research contributes to the development of large- bility disclosure,” IEEE security & privacy, vol. 3, no. 1,
scale machine learning systems for function-level vulnerability pp. 20–25, 2005.
identification in source code auditing. [4] K. Jochem, “It security matters,”
[5] “cisa.” https://www.cisa.gov/
ACKNOWLEDGEMENT
news-events/alerts/2021/06/30/
The work is supported by the National Science Founda- printnightmare-critical-windows-print-spooler-vulnerability,
tion under NSF Award #2209638, #2100115, #2209637, 2022. Accessed April 26, 2023.
#2100134, #1663350. Any opinions, findings, recommenda- [6] T. N. Brooks, “Survey of automated vulnerability de-
tions, expressed in this material are those of the authors and tection and exploit generation techniques in cyber rea-
do not necessarily reflect the views of the National Science soning systems,” in Science and Information Conference,
Foundation. pp. 1083–1102, Springer, 2018.
REFERENCES [7] C. Cadar, D. Dunbar, D. R. Engler, et al., “Klee: unas-
[1] T. D. LaToza, G. Venolia, and R. DeLine, “Maintaining sisted and automatic generation of high-coverage tests for
mental models: a study of developer work habits,” in Pro- complex systems programs.,” in OSDI, vol. 8, pp. 209–
Fig. 4: Performance Metrics for Different Neural Network Models on Vulnerable Source Code without Word Embedding
Algorithms

224, 2008. code audits,” in Proceedings of the 22nd ACM SIGSAC

[8] T. A. Henzinger, R. Jhala, R. Majumdar, and G. Sutre, Conference on Computer and Communications Security,
“Software verification with blast,” in International SPIN pp. 426–437, 2015.
Workshop on Model Checking of Software, pp. 235–239, [13] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A
Springer, 2003. survey of machine learning for big code and naturalness,”
[9] S. M. Ghaffarian and H. R. Shahriari, “Software vul- ACM Computing Surveys (CSUR), vol. 51, no. 4, pp. 1–
nerability analysis and discovery using machine-learning 37, 2018.
and data-mining techniques: A survey,” ACM Computing [14] S. Wang, D. Chollak, D. Movshovitz-Attias, and L. Tan,
Surveys (CSUR), vol. 50, no. 4, pp. 1–36, 2017. “Bugram: bug detection with n-gram language models,” in
[10] A. Younis, Y. Malaiya, C. Anderson, and I. Ray, “To fear Proceedings of the 31st IEEE/ACM International Confer-
or not to fear that is the question: Code characteristics ence on Automated Software Engineering, pp. 708–719,
of a vulnerable functionwith an existing exploit,” in 2016.
Proceedings of the sixth ACM conference on data and [15] G. Grieco, G. L. Grinblat, L. Uzal, S. Rawat, J. Feist, and
application security and privacy, pp. 97–104, 2016. L. Mounier, “Toward large-scale vulnerability discovery
[11] Y. Shin, A. Meneely, L. Williams, and J. A. Osborne, using machine learning,” in Proceedings of the Sixth
“Evaluating complexity, code churn, and developer ac- ACM Conference on Data and Application Security and
tivity metrics as indicators of software vulnerabilities,” Privacy, pp. 85–96, 2016.
IEEE transactions on software engineering, vol. 37, no. 6, [16] P. Zeng, G. Lin, L. Pan, Y. Tai, and J. Zhang, “Software
pp. 772–787, 2010. vulnerability analysis and discovery using deep learning
[12] H. Perl, S. Dechand, M. Smith, D. Arp, F. Yamaguchi, techniques: A survey,” IEEE Access, vol. 8, pp. 197158–
K. Rieck, S. Fahl, and Y. Acar, “Vccfinder: Finding 197172, 2020.
potential vulnerabilities in open-source projects to assist [17] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Ef-
 fective vulnerability identification by learning comprehen- pp. 83–102, Springer, 2023.
sive program semantics via graph neural networks,” Ad- [30] Y. Wang, M. Huang, X. Zhu, and L. Zhao, “Attention-
vances in neural information processing systems, vol. 32, based lstm for aspect-level sentiment classification,” in
2019. Proceedings of the 2016 conference on empirical methods
[18] D. Iorga, D.-G. Corlatescu, O. Grigorescu, C. Sandescu, in natural language processing, pp. 606–615, 2016.
M. Dascalu, and R. Rughinis, “Yggdrasil—early detec- [31] M. S. Akter, H. Shahriar, A. Cuzzocrea, N. Ahmed, and
tion of cybernetic vulnerabilities from twitter,” in 2021 C. Leung, “Handwritten word recognition using deep
23rd International Conference on Control Systems and learning approach: A novel way of generating handwritten
Computer Science (CSCS), pp. 463–468, IEEE, 2021. words,” in 2022 IEEE International Conference on Big
[19] C. Sauerwein and A. Pfohl, “Towards automated clas- Data (Big Data), pp. 5414–5423, 2022.
sification of attackers’ ttps by combining nlp with ml [32] H. Nguyen, K. P. Tran, S. Thomassey, and M. Hamad,
techniques,” arXiv preprint arXiv:2207.08478, 2022. “Forecasting and anomaly detection approaches using
[20] J. A. Harer, L. Y. Kim, R. L. Russell, O. Ozdemir, L. R. lstm and lstm autoencoder techniques with the applica-
Kosta, A. Rangamani, L. H. Hamilton, G. I. Centeno, tions in supply chain management,” International Journal
J. R. Key, P. M. Ellingwood, et al., “Automated software of Information Management, vol. 57, p. 102282, 2021.
vulnerability detection with machine learning,” arXiv [33] K. W. Church, “Word2vec,” Natural Language Engineer-
preprint arXiv:1803.04497, 2018. ing, vol. 23, no. 1, pp. 155–162, 2017.
[21] M. Pistoia, S. Chandra, S. J. Fink, and E. Yahav, “A [34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert:
survey of static analysis methods for identifying security Pre-training of deep bidirectional transformers for lan-
vulnerabilities in software systems,” IBM systems journal, guage understanding,” arXiv preprint arXiv:1810.04805,
vol. 46, no. 2, pp. 265–288, 2007. 2018.
[22] R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, [35] R. Dale, “Gpt-3: What’s it good for?,” Natural Language
O. Ozdemir, P. Ellingwood, and M. McConley, “Auto- Engineering, vol. 27, no. 1, pp. 113–118, 2021.
mated vulnerability detection in source code using deep [36] D. S. Depto, S. Rahman, M. M. Hosen, M. S. Akter,
representation learning,” in 2018 17th IEEE interna- T. R. Reme, A. Rahman, H. Zunair, M. S. Rahman, and
tional conference on machine learning and applications M. Mahdy, “Automatic segmentation of blood cells from
(ICMLA), pp. 757–762, IEEE, 2018. microscopic slides: a comparative analysis,” Tissue and
[23] A. Ahmed and M. A. Yousuf, “Sentiment analysis on Cell, vol. 73, p. 101653, 2021.
bangla text using long short-term memory (lstm) recur- [37] M. S. Akter, M. J. H. Faruk, N. Anjum, M. Masum,
rent neural network,” in Proceedings of International H. Shahriar, N. Sakib, A. Rahman, F. Wu, and A. Cuz-
Conference on Trends in Computational and Cognitive zocrea, “Software supply chain vulnerabilities detection
Engineering, pp. 181–192, Springer, 2021. in source code: Performance comparison between tradi-
[24] J. Pennington, R. Socher, and C. D. Manning, “Glove: tional and quantum machine learning algorithms,” in 2022
Global vectors for word representation,” in Proceedings IEEE International Conference on Big Data (Big Data),
of the 2014 conference on empirical methods in natural pp. 5639–5645, IEEE, 2022.
language processing (EMNLP), pp. 1532–1543, 2014. [38] M. S. Akter, H. Shahriar, S. Sneha, and A. Cuz-
[25] V. Gaikwad and Y. Haribhakta, “Adaptive glove and fast- zocrea, “Multi-class skin cancer classification architecture
text model for hindi word embeddings,” in Proceedings based on deep convolutional neural network,” in 2022
of the 7th ACM IKDD CoDS and 25th COMAD, pp. 175– IEEE International Conference on Big Data (Big Data),
179, 2020. pp. 5404–5413, IEEE, 2022.
[26] M. S. Akter, H. Shahriar, R. Chowdhury, and M. Mahdy, [39] P. F. de la Foret, S. Ruseti, C. Sandescu, M. Dascalu, and
“Forecasting the risk factor of frontier markets: A novel S. Travadel, “Interpretable identification of cybersecurity
stacking ensemble of neural network approach,” Future vulnerabilities from news articles,” in Proceedings of the
Internet, vol. 14, no. 9, p. 252, 2022. International Conference on Recent Advances in Natural
[27] S. Hochreiter and J. Schmidhuber, “Long short-term Language Processing (RANLP 2021), pp. 428–436, 2021.
memory,” Neural computation, vol. 9, no. 8, pp. 1735– [40] V. Behzadan, C. Aguirre, A. Bose, and W. Hsu, “Corpus
1780, 1997. and deep learning classifier for collection of cyber threat
[28] M. S. Akter, H. Shahriar, N. Ahmed, and A. Cuzzocrea, indicators in twitter stream,” in 2018 IEEE International
“Deep learning approach for classifying the aggressive Conference on Big Data (Big Data), pp. 5002–5007,
comments on social media: Machine translated data vs IEEE, 2018.
real life data,” in 2022 IEEE International Conference on
Big Data (Big Data), pp. 5646–5655, 2022.
[29] M. S. Akter, H. Shahriar, and Z. A. Bhuiya, “Automated
vulnerability detection in source code using quantum
natural language processing,” in Ubiquitous Security: Sec-
ond International Conference, UbiSec 2022, Zhangjiajie,
China, December 28–31, 2022, Revised Selected Papers,