Software defect prediction using hybrid model (CBIL) of convolutional neural network (CNN) and bidirectional long short-term memory (Bi-LSTM)

CNN

CNN (Li et al., 2019a) is a particular kind of neural network that is used in the areas such as Natural Language Processing (NLP), speech recognition, and text classification. CNN (Meilong et al., 2020) is categorized into one-dimensional (1D) CNN for NLP and two-dimensional (2D) CNN for image recognition.

Simple CNN architecture (Dahou et al., 2019) is shown in Fig. 3. The number of convolutional filters is defined in the convolutional layer. It is applied to input data for producing feature maps. Then, CNN uses the pooling layer to minimize the dimensionality of the output. Max pooling is mostly used as it accomplishes better effectiveness compared to both Min pooling and Average pooling techniques (Goncalves dos Santos, 2020). Then, flatten layer transforms a 2D matrix into a vector that can be used as input to a fully connected (FC) layer. At last, FC layer and sigmoid represent the output layer, it generates the results of the classification based on previous layers.

CNN has two key features (Sheng, Lu & Lin, 2020): sparse connectivity and shared weights. These features minimize the capacity of the model and catch global manners rather than local ones.

RNN

RNN (Sherstinsky, 2018), it is a sequential model, a generalization of feedforward neural network with internal memory to save the states of every input in the network. In a feedforward neural network, the information goes through only one direction, while RNN can store information over time. The loops in RNN called recurrent because the information is passed internally from a one-time step to the next time step. RNN (Deng, Lu & Qiu, 2020) has been achieved a good progress in many fields, including sequence recognition, sequence reproduction, and temporal association. RNN (Zhang, Chen & Huang, 2018) takes input vector as X_t, Y_t represents the output then the calculations are executed to update an internal state h_t. The following equation is executed: (1)${\displaystyle h_t=fw\left(h_{t-1},X_t\right)}$ Where fw is the activation function with a set of w weights, h_t−1 represents the old state, and X_t is the input vector at time step t. In RNN, the same procedures are used at every step.

To update the internal state by applying activation function (tanh), the following equation is executed: (2)${\displaystyle h_t=tanh\left(w_{hh}{h}_{t-1}{+w}_{xh}{X}_t\right)}$ Where w_hh and w_xh weights of internal state and input.

The output will be calculated as the following equation: (3)${\displaystyle Y_t=w_{hy}{h}_t}$ Where w_hy the weight of the output layer.

A common problem in RNN is the “vanishing gradient” problem (Gao et al., 2020), where the gradients become extremely small, and it might be impossible to make any optimization. Long Short-Term Memory (LSTM) (Ralf & Rothstein Morris, 2019) is a type of RNN that is used to solve the vanishing gradient. LSTM contains a “memory cell” that can track and maintain information of long-term dependencies in memory for very long periods of time. LSTM units (Wu et al., 2020) consist of three gates: input gate, forget gate, and output gate. Figure 4 shows the framework of standard LSTM units, where σ is a logistic sigmoid function, Tanh is the activation function, X_t represents new input, h_t−1 represents output from previous timestep, h_t is the output, old cell state C_t−1, new cell state C_t. Also, F_t, I_t, O_t symbolize forget gate, input gate, and output gate respectively.

The gates enable the information to be added or removed at a cell state. The data will be passed through a sigmoid function. The sigmoid function is forcing the input to be between 0 and 1. The information will be passed if the value equals 1 only. Forget gate performs computations to store relevant history of new information to cell state and discards the irrelevant history from the cell state. Then input gate controls if the cell state is updated or not. Also, the output gate handles the value in the cell state. Then the activation of LSTM unit will be calculated. The following equations are used to compute h_t: (4)${\displaystyle F_t=\sigma \left(W_f{X}_t+U_f{h}_{t-1}+b_f\right)}$ (5)${\displaystyle I_t=\sigma \left(W_i{X}_t+U_i{h}_{t-1}+b_i\right)}$ (6)${\displaystyle O_t=\sigma \left(W_o{X}_t+U_o{h}_{t-1}+b_o\right)}$ (7)${\displaystyle C_t^{\wedge }=tanh\left(W_c{X}_t+U_c{h}_{t-1}+b_c\right)}$ (8)${\displaystyle C_t=ft\Theta C_{t-1}+I_t\Theta C_{\mathrm{t}}^{\wedge }}$ (9)${\displaystyle h_t=O_t\Theta tanh\left(c_t\right)}$ Where b is the bias, W and U are the weights of the three gates. C^∧_t is the candidate value. Θ is the multiplication operation; it determines if the information will pass through the gates or not.

Bi-directional LSTM (Bi-LSTM), It combines two independent LSTM together. It is useful for sentiment classification. It enables the information to be passed through forward and backward directions.

Parsing source code into AST

There are two main steps to use the source code of the files in the model:

First, Abstract Syntax Tree (AST) is built for the selected Java files included in PROMISE dataset.

Second, the following kinds of AST nodes are selected to be generated as tokens: (1). nodes of control flow such as if/while/do statements which are recorded as the types of the nodes. (2). nodes of class instance creations and method invocations which are recorded as the names of classes or methods without Parenthesis (3). nodes of declarations such as method/class declarations which are recorded as their names. Other AST nodes are removed as it may affect the importance of the selected AST nodes. The selected AST nodes are shown in Fig. 6. We apply the Python package “javalang” to parse the source code into AST.

Mapping tokens

When the source code is parsed, text tokens are extracted for every file. But it is difficult to use them directly as input into DL model. So, it is needed to convert the text tokens to an integer vector. The mapping between the tokens and the integers is built with the range of one to the total number of tokens. So, a unique integer will represent each token. “Keras Text tokenization utility” class is applied to convert all text tokens to a sequence of integers. All integer vectors should have fixed length. Identified length will be selected. So, if the vector length is smaller than the identified length, it is completed by 0. And, If the vector length is bigger than the identified length, the extra length will be removed.

Class imbalance

The data are imbalanced most of the time, as instances of defective files are less than the number of instances of clean files. Therefore, the prediction results will be classified as clean as the instances of clean files represents the majority class. To resolve the class imbalance issue (Eivazpour & Keyvanpour, 2021), there are two common techniques to be used: oversampling and undersampling. In oversampling, the instances are duplicated in the minority class. In undersampling, the instances will be deleted from the majority class. To ensure the completeness of data, oversampling technique is applied by using “imblearn RandomOverSampler” method. It will duplicate the instances of minority class randomly over time.

Embedding layer

Integer vectors cannot carry the context information of AST tokens. Therefore, word embedding (Yildiz & Tezgider, 2021)technique is used to turn each integer vector to a dense vector. In traditional embeddings, large sparse vectors are used to represent each word within a vector to represent an input sequence. However, this representation is sparse because the input sequences are huge. For example, AST tokens will be preserved while the relationship between them will be ignored. Instead, in word embedding, words are represented using dense vectors with a fixed length where a vector represents the word’s projection into a continuous vector space. For example, a “For Statement” node is embedded to dense vector [0.25,0.1]. Word embedding has two main benefits. Firstly, the embedded vector has lower dimensions than the sparse vector. Secondly, AST nodes that share identical contexts are located near each other in the vector space.

CNN layers

It includes two layers: convolutional layer and max pooling layer. In convolutional layer, it is applied to automatically learn the features of software defects. It extracts the key semantics based on number of filters and filter length. We analyze the values of filter length and number of filters to choose the optimal values. The experiments have been conducted on project Jedit, Lucene, Poi and Xalan as a sample of the selected projects. Filter length is set to 10, and number of filters to 20. The details of the optimal values are discussed in ‘Results’. Then, max pooling is applied to reduce the dimensionality of the output. It is used to gather the extracted information. The results of max pooling are entered as input to Bi-LSTM layer to filer the useful information.

Bi-LSTM layer

It preserves the sequential order between the data. The implementation of Bi-LSTM helps to detect the information of long-term dependencies which can successfully capture necessary features. It runs two LSTMs to detect both of forward and backward information. We analyze the value of LSTM units to choose the optimal value. The experiments have been conducted on project Jedit, Lucene, Poi and Xalan as a sample of the selected projects. The number of LSTM units is set to 32. The detail of the optimal value is discussed in ‘Results’.

Dense layer

Finally, the Bi-LSTM layer is connected to the dense layer to obtain the results of the prediction. CBIL model is trained by using a training set based on their labels (i.e., defective, or clean). Then, Logistic Regression classifier is applied on the test set to generate the probability of the file being defective.