Application of data mining technology in detecting network intrusion and security maintenance

1 Introduction

With the development of the Internet, the Internet has become an important part of human work and life. Especially after the realization of globalization and informatization, many enterprises, government agencies, and individuals carry out various businesses and operations on the open Internet, such as Financial companies carry out online banking, etc. [1]. People use the Internet to facilitate work and life, and at the same time, change people’s work lifestyle and meet the needs of modern people for being fast, efficiency, and convenience. People use the Internet to obtain the information they need, as we all know The Internet is open and shared. These two characteristics make the resources integrated into the Internet rich, but at the same time it also brings hidden dangers to the security of the Internet itself [2]. Intrusion Detection System (IDS) is one of the technologies to improve network security. It collects information on certain key points in the network for technical analysis to find abnormalities or intrusion attacks. Dynamic security technology is for network detection, alarm, and response. The main function of an IDS is to detect intrusive behaviors and identify intrusive events. In essence, it is to classify network behavior data [3]. The network data are divided into normal and abnormal data, and the judgment of abnormal behavior is intrusion detection. The IDS can monitor the hosts, running applications, and their status in the system in a comprehensive and real-time manner. It can also detect intrusion behaviors of internal and external attacks in the system in real time, actively identify intrusions, issue an alarm, etc. The above characteristics make it play an important role in network security. Figure 1 illustrates the architecture of a basic intrusion detection system.

Figure 1

Architecture of a basic intrusion detection system.

Data mining technology is useful in finding the patterns for a large dataset using various multidisciplinary techniques like machine learning (ML), statistics, and artificial intelligence. Data mining is basically about discovering the hidden and unpredictable relationships among the data by the detection of data patterns, knowledge extraction, and revealing the unknown information. The insight of these data mining strategies can be used to evaluate the probability of future events which can be used in various fields of marketing, scientific discovery, fraud and intrusion detection, etc. The usage of data mining techniques for intrusion detection is significantly gaining importance in the current scenario for predictive analysis of suspicious intrusions. The data mining-based IDSs are effective in identifying the data of user’s interest and can better predict the outcomes for future security maintenance [4,5]. Intrusion detection was initially introduced in the year 1980 by Anderson [6] and later, the concept of data mining advanced the detection process. This advancement involves a great deal of consideration in the IT sector for the discovery of knowledge in the society [7,8]. Data mining can easily access the useful information from the large volume of data [9]. In Figure 1, the intrusion model has been centrally placed in order to capture all the data packets and transmit them over a network. The data collected from the database is also preprocessed in order to make it free from the redundancy and noise [10]. These preprocessed data are then further analyzed and categorized according to the severity measures, and based on the data state, the alarm is raised to the security maintenance administrator to handle the situation. The literature related to IDS provides various alternatives for addressing this problem; however, the security maintained is still challenging due to complex data classification and data labeling tasks.

The shortcoming of intrusion detection technology has been addressed in this work, to facilitate the performance of computer and network security systems. This work proposes an improved k-means algorithm and an improved Apriori algorithm which are applied in the data mining technology for the detection of network intrusion and security maintenance. The classical dataset has been exploited in this work by performing the experimentation with the improved algorithms. The contribution of improved algorithms has been tested experimentally in terms of detection rate and false alarm rate. Significant improvement has been noticed in the outcomes obtained and the improved algorithm advances the detection efficiency and accuracy using the designed detection model. When the improved and tested detection model is then applied to a new IDS, a significant improvement is seen in the detection accuracy while simultaneous reduction in the false alarm rate.

The remaining paper is organized as follows: Section 2 presents the literature review of the state of the art work in the field of intrusion detection. The dataset description and data preprocessing are detailed in Section 3 followed by the experimental results and analysis in Section 4. Section 5 presents the conclusion of the entire research work followed by the future research scope.

2 Literature review

Deng and others analyzed the characteristics of network security and security issues and discussed the system framework of Internet security and some key security technologies, including key management, authentication and access control, routing security, privacy protection, intrusion detection, and Fault tolerance and this article introduces the current problems of the Internet of Things in network security and points out the necessity of intrusion detection. Several intrusion detection technologies are discussed, and their applications in the Internet of Things architecture are analyzed. We compared the applications of different intrusion detection technologies and looked forward to the next phase of research. Using data mining and ML methods to study network intrusion technology has become a hot issue. It is difficult to improve the detection rate of network intrusion detection by a single type of feature or detection model. The performance of the proposed model is verified by a public database [11]. Zhang J., with IDS as the research object, established an IDS model based on data mining, obtained experimental results, and reached relevant experimental conclusions. At the same time, it was compared with traditional IDS, and six experiments were carried out. The output results of the detection rate, false negative rate, and false positive rate of two different methods in six experiments were obtained. The experiment draws a conclusion: the IDS using data mining has better network protection and security performance, and the detection ability of network vulnerability intrusion is stronger. This research provides a new way for the detection and research of network protection security loopholes [12]. Bagaa M. et al. proposed a novel ML-based security framework that can automatically deal with extended security aspects related to the IoT domain. The framework utilizes software-defined networking and network function virtualization enablers to mitigate different threats. The AI framework combines monitoring agents and ML-based AI-based reaction agents. These ML-Models are divided into network pattern analysis and anomaly-based intrusion detection in IoT systems. The framework uses supervised learning, distributed data mining systems, and neural networks to achieve its goals. The experimental results prove the effectiveness of the scheme. In particular, the use of data mining methods to distribute attacks is very successful in detecting attacks with high performance and low cost. Regarding an anomaly-based IDS for the Internet of Things, the experiment has been evaluated in a real smart building scenario using a type of SVM. The detection accuracy of anomalies reaches 99.71%. A feasibility study was conducted to determine the current potential solutions to be adopted and to promote the development of the study in the direction of open challenges [13].

Ektefa et al. [14] evaluated the IDSs using the machine leaning platforms, and among C4.5 and Support Vector Machine (SVM) methods, C4.5 performs better. A hybrid Particle Swarm Optimization (PSO) was proposed by Holden and Freitas [15] to deal with the nominal attributes for intrusion detection. Ardjani et al. [16] applied the combination of PSO and SVM to optimize the ML algorithm performance and 10-fold cross validation is done for accuracy estimation. Multidimensional dataset was dealt with to remove the redundancy and inconsistency in the feature vectors and provide better classification outcomes [17]. Two-class classification method was proposed by Wael et al. [18] to classify the normal situation from the attack class. This method was effective in providing the least error and better detection rate comparative to its counterparts. New form of attack patterns was captured by Petrussenko [19] for the detection of attacks in the network packets and it was found that higher detection rate was achieved by utilizing a preprocessor module in between [20,21]. Improving IDS by ML needs large sets of labeled data for attack scenarios and normal scenarios. Acquiring such sets is costing time and effort. Also, it can be done by experts. A large set of pre-labelled training data was utilized by an unsupervised method to produce less accuracy; therefore, the authors dealt with the semi-supervised algorithm to provide better accuracy detection rates. A fuzzy clustering-based approach was proposed by the authors in ref. [22], while evaluating both the Euclidean distance and statistical cluster properties to identify the possibility of intrusion. Wang and Megalooikonomou [23] presented a co-training framework to improve the intrusion detection in case of unlabeled data. This method provides the lower error rate, while incorporating an active learning technique to improve the system performance. Higher detection rates were achieved by utilization of a semi-supervised learning mechanism which yields comparatively reduced false alarm rate [24]. In order to improve the accuracy rate of intrusion detection, a tri-training SVM algorithm has been used [25]. A tree-based clustering technique was proposed by Li et al. [26], which finds the clusters among the intrusion detection dataset. This method works efficiently faster comparative to other algorithms. A combination of misuse and anomaly detection was utilized by the authors in ref. [27] for determining the cost function of the semi-supervised strategy involving SVM classifiers. For addressing the distributed denial of service attacks, various advents have been offered [28]. Fu and Papatriantafilou [29] proposed a perspective proactive algorithm to divide the network into set of clusters and the intrusion is detected using these clusters. Fu et al. [30] presented a synchronous communication-based method for intrusion detection and reducing the false alarming rates.

The usage of data mining combined with the ML methods has been made for intrusion technology in various related works. All the proposed models in the literature try to fit into the prescribed dataset and reduce the administrator workload by improving the detection of attacks. However, from this literature survey, it was found that it is challenging to improve the detection rate of network intrusion detection by a single type of feature or detection model, as used in the current existing methods. These methods are still needed to be improved and there are manifolds possibilities of improvement in IDSs.

3 Introduction to KDD cup99 data set

KDDcup99 is a data set specially used to evaluate the situation of research IDSs. It is a data set collected by Lincoln Laboratory during intrusion detection and evaluation of DARPA networks in 1998. It contains various types of data containing different intrusion categories. The data set contains a total of 4.9 million data; here, only 10% of the data are taken from “kddcup.data_10. percent” [31]; that is, 396,473 normal behavior data and abnormal behavior data. Since all the data in the KDDcup99 data set are the original network data packets obtained by Tcpdump, the data packets are converted into ASCII format.

Intrusion detection evaluation KDDcup99 data set mainly has the following types of attacks:

1. Denial Service (Denial Service);

2. Illegal remote host access R2L attack (Remote to Local);

3. Local unprivileged users conduct unauthorized access U2R attacks (User to Root) of local super users;

4. Reconnaissance and detection of probing attacks;

5. Data transmission data attack.

The data provide 41 characteristics, divided into three different categories: traffic characteristics, content characteristics, and other characteristics.

3.2 Data preprocessing

Preprocessing is an important data preparation work for the data mining module before mining. According to the requirements of the system algorithm, the data are processed in advance to make the data more in line with the data required by the system mining. Data preprocessing must complete the following tasks:

1. Data extraction: It refers to the integration of operational databases oriented to transaction processing to databases oriented to data mining.

2. Data cleaning: Including some inconsistent data, empty data, data cleaning is to deal with these data, including inconsistent data transformation, vacant data supplement, and so on.

In this work, the two algorithms are used, one is the FP-growth algorithm depicted in Figure 2 and the Apriori algorithm whose flowchart is shown in Figure 3.

Figure 2

Flowchart of FP-growth algorithm.

Figure 3

Flowchart of Apriori algorithm.

The FP-growth algorithm undergoes a process of preparing the clusters and the number of k-clusters are computed followed by the centroid computation. The grouping of different data groups is done depending upon the grouping criteria and the model generation is done using the association rule. Using those association rules from the database, the intrusion scenario is detected.

For the Apriori algorithm, the supports of each item are calculated, and depending upon the support condition, the items are removed or inserted into the frequent item set. The confidence for every non-empty subset is computed and the selection of the subset is dependent upon the confidence condition. The intrusion detection is accomplished in this algorithm on the basis of the addition of strong rules.

In this experiment, the two algorithms depicted in Figures 2 and 3 based on association rules and clustering analysis are used for experimentation with the subset data kddcup.data_10. percent of KDDcup99. The data are divided into training data and detection data. The training data are used to train the data and build a detection model, and the detection data are used to detect the data and analyze the results of the detection performance [32,33]. When constructing the detection model, it is necessary to filter the abnormal data in the training data, filter out most of the intrusion data and leave only a little intrusion data, and extract the normal data to reach 95%. As long as the detection data are directly extracted as it is, 2,000 network data set records are extracted. The value of the holding degree is set from 0.35 to 8%. The data of 20 mining experiments based on the FP-growth algorithm and the Apriori algorithm are shown in Table 2.

Table 2

Average mining time for different minimum support values

Minimum support (%)	0.35	1.75	3.5	5.25	6.3	7	8
Average time of FP-growth algorithm	4.97	4.77	4.67	4.56	4.44	4.25	4.29
Average time of Apriori algorithm	8.42	8	7.8	7.6	7.4	7.2	7

In order to more intuitively reflect the mining effect of the experiment, the conversion is shown in Figure 4.

Figure 4

Average mining time of the two algorithms.

From the above experiment, it can be found that when the value of the minimum support is smaller, the mining efficiency of the FP-growth algorithm is higher than that of Apriori, and when the value of the minimum support is changed, the mining time is more stable [34]. This is because the FP-growth algorithm does not generate candidate item sets during mining, while the Apriori algorithm generates a large number of candidate item sets, which affects the mining efficiency.

The k-means algorithm first needs to input the number of clusters. This number has a greater impact on the results of clustering. Below, we will test the effect of clustering through different numbers of clusters. Select the cluster centers as 10, 12, 16, 18, 20, and 28, respectively, for testing. The training sample data are 8,000 records, and to make the proportion of normal data larger, only 400 intrusion data are left. Reduce 41 attributes to only 13 data field attributes. The two parameters of the improved algorithm are the minimum clustering radius r and exponential factor m. After experiment and observation, the values of the two parameters are r = 0.3 and m = 0.5, respectively. When r = 0.3 and m = 0.5, the detection rate and false alarm rate can be selected in a reasonable range. The data of the experimental results are shown in Table 3.

Table 3

Performance table after algorithm improvement

Number of cluster centers	k-means algorithm		An improved k-means algorithm
	Detection rate (%)	False alarm rate (%)	Detection rate (%)	False alarm rate (%)
8	43.7	1.3	53.6	1.1
10	52.5	2.2	62.2	2.0
12	68.3	3.7	76.4	2.1
14	77.2	6.5	83.1	3.9
16	82.3	8.6	87.6	5.7

In order to understand the improved performance of the k-means algorithm more intuitively, the experimental results are plotted as shown in Figure 5.

Figure 5

False alarm rate of the two algorithms.

From Figure 5, we can see that the improved k-means algorithm improves the detection rate and reduces the false alarm rate. In short, the improved detection performance of k-means algorithm has been improved.

Table 4

Nample.txt data table

Property name	Average value	Mean absolute error
ip_llen	30560.3	18537.5
ip – tt1	78.03	26.632
tcp-win	0.0023	29431.22
ip_options – len	8.08	0.00268
tcp_options – len	532.8	7.753
dsize	32776.26	576.6
udp – len		13768.6

4 Simulation experimental results and analysis

5 Conclusion

k-means algorithm is an initial value-sensitive algorithm in which different k values produce different clustering results. In view of the shortcomings of k-means algorithm, an improved k-means algorithm is proposed in this work. The k value does not need to be determined in advance, and the data set automatically produces an optimal value through clustering. The clustering center does not need to be changed after determination and the whole data set only needs to be scanned once. The efficiency of both the improved k-means algorithm and the clustering effect has improved greatly.

1. The improved k-means algorithm is verified through experiments. The experimental data show that the improved k-means algorithm has detection efficiency and detection ability. Compared with the previous algorithm, the algorithm improves the detection efficiency and detection ability of the system. A maximum improvement in the detection accuracy of 90.57% can be seen for new attack type intrusion using the proposed algorithm.

2. Apply the improved algorithm to the IDS and design an IDS based on data mining, which mainly includes pre-detection modules, feature extraction modules, etc. and conducts intrusion detection experiments on the new IDS. The experimental results show that the proposed system improves the detection efficiency and reduces the false detection rate.

The future work in this field will address the shortcomings in the research of IDS based on data mining technology, such as the high cost of the improved Apriori algorithm in space.