Provenance-based Intrusion Detection Systems: A Survey

The Event Tracing for Windows (ETW) is a built-in device driver for logging kernel or application-defined events and consists of three main components: (1) the controller that starts and stops event tracing sessions; (2) the provider which creates events and publishes them in a session; and (3) the consumer that consumes events from a session [77]. Sysmon logs security-related system activities by accessing and aggregating multiple log sources of ETW. It logs various events such as network connections, pipes, registry-, file-, and image operations, and a rule set defines the scope of events monitored [78].

Lineage File System is one of the first DPC systems proposed to track, store, and query lineage information of files [92]. Linux has a built-in auditing system that monitors security and non-security-related events by intercepting syscalls in the user-space. It’s shortcoming is the high runtime overhead, which can be as high as 43% [73, 75, 115]. RecProv addresses the runtime overhead issue by applying a record and replay method from software debugging to generate system-level data provenance. It has a secure data provenance store to protect the data integrity [59].

Early system-level DPC systems for Linux cannot monitor kernel-initiated actions and thus, could miss essential events for intrusion detection and forensic analysis. Hi-Fi is the first system-level DPC system that intercepts syscalls in the kernel-space and observes kernel-initiated actions to collect whole-system data provenance [91]. Provmon is a port of Hi-Fi that adds additional semantic information to system entities such as versions to files and remote IP addresses, and ports to network events. Hi-Fi and Provmon were designed for now outdated kernel releases and cannot run on current Linux systems. CamFlow, a provenance capture system in the kernel space, uses self-contained easily maintainable Linux Security Modules (LSM) and NetFilters to collect whole-system data provenance. It utilizes the latest kernel features to increase the efficiency to collect whole-system data provenance. CamFlow significantly reduces the runtime overhead compared to previous approaches [88].

Data provenance contains OS specific types and properties of nodes and edges. As a result, data provenance from various Operating Systems (OSs) cannot be simply merged. To address this, multiple cross-platform system-level DPC systems have been proposed.

Dtrace is an early cross-platform system-level DPC systems developed by Sun Microsystems, which can be deployed on various OSs such as Windows, Linux, and MacOS [24, 107]. Nevertheless, Dtrace does not support the aggregation of heterogeneous system-level data provenance. Provenance-aware storage systems (PASS) address this through a provenance-aware filesystem that provides a disclosed provenance Application Programming Interface (API) to interface between the layers and naming conversions of various OSs [84, 85]. Another cross-platform system-level DPC system is Support for Provenance Auditing in Distributed Environments (SPADE) that aggregates heterogeneous system-level data provenance to support distributed debugging and causality analysis across multiple OSs [32].

A major problem of system-level DPC systems is that the resulting provenance graph can be too coarse, especially for long-running processes, leading to a dependency explosion problem [64, 74].

The runtime overhead is a major evaluation criterion of DPC systems. Table 2 shows that the data provenance’s granularity is closely related to the runtime overhead of a DPC system. The finer the granularity of the data provenance, the higher the runtime overhead and impact on the overall system performance. Provenance capture systems which capture provenance data on instruction-level [6, 96, 97, 113] can pose a runtime overhead higher than 100%. Researchers tackled this problem by finding a trade-off between high-fidelity data provenance and low runtime overhead. E.g., RAIN [58] continuously captures system-level data provenance but can replay a scenario to apply instruction-level data provenance capture for further investigation.

The runtime overhead values in Table 2 can only be compared to a limited extent. First, researchers have used different system behavior as benchmarks for their evaluations [6]. Second, the experimental setup of the evaluation was significantly different, e.g., PASS was evaluated on a system with a single 500MHZ Intel Pentium 3 CPU, whereas Linux Provenance Modules (LPM) was evaluated on a system with a dual Intel Xeon CPU [88]. Third, the number of applications running on a system has increased significantly over the years. Therefore, provenance capture systems that were proposed ten years ago had significantly fewer syscalls to monitor.

Most DPC systems monitor syscalls in the kernel-space and, therefore, are implemented as kernel modules. Due to rapid changes in the kernel versions, researchers could not keep up with maintaining their source code. As a result, many DPC systems proposed in the literature are outdated due to lack of maintenance [88]. Therefore, researchers are falling back to built-in DPC systems that are provided by the OS [78, 98] or maintained through open-source research projects [24, 32, 88]. The results of the usage analysis of DPC systems in graph summarization and intrusion detection research are shown in Figure 5.

DPC systems must tolerate failures such as full disk, network outages, reboots, or system overloads caused by both malicious and non-malicious faults. In such cases, the provenance data’s integrity and completeness have to be ensured [57].

The goal of attackers is to stay undetected while pursuing their goal, e.g., to compromise the system. To do so, attackers may try to compromise the DPC system itself, delete data provenance that includes their malicious behavior or inject additional provenance data to veil their traces. Hence, PIDS must consider the trustworthiness of the data provenance captured [57]. Many researchers assume that the data provenance is trustworthy.

Data provenance contains privacy-sensitive information of users such as websites visited or files accessed. While general privacy issues of IDS have been discussed [72, 87], to the best of our knowledge, there is no research on privacy issues of DPC systems of PIDS nor proposed privacy-preserving mechanisms.

Reviewed DPC systems collect data provenance based on system calls or CPU instructions. Multiple other existing data sources can be incorporated to enrich a provenance graph’s contextual information. Possible data sources include network, application and database logs. For example, a system call that creates a network connection can be linked to a network packet in the network logs. While adding more contextual information would likely improve the detection accuracy, it would significantly increase the storage overhead, and negatively affect the detection time.

Reference [71] proposed a taxonomy for graph summarization techniques. We extend their taxonomy for PIDS data summarization techniques. We categorize the approaches into bit compression-based reduction, simplification-based reduction, policy-based reduction, and grouping-based reduction methods, which are defined as follows:

Online graph summarization approaches can compress real-time data provenance without using historical data provenance. Offline graph summarization methods query the data provenance from the persistent storage, compress it, and then push it back to the persistent storage. PIDS profit from online graph summarization approaches because they reduce the sheer size of data before their transmission over the network. This reduces the complexity of the data provenance and the Create, Read, Update, and Delete (CRUD) operations made to the persistent storage [7, 48, 51, 65, 73, 75, 93, 99, 112].

The reduction rate is a commonly used evaluation metric of graph summarization techniques for PIDS and indicates how many nodes and edges in a provenance graph are removed by a technique to improve storage and analysis efficiency. We have used this evaluation metric to compare the reviewed techniques.

However, the reduction rate does not reflect the security value of these techniques. To address this issue, metrics to measure the security value of a particular graph summarization approach under different attack models were proposed in [76]. Lossless forensics defines the percentage of the number of removed edges compared to the number of edges in the original provenance graph. Causality-preserving forensics defines the percentage in which the graph summary approach preserves the information flow and causal relationships of the original provenance data. Attack-preserving forensics defines the percentage in which the graph summarization approach can remove benign information flows while preserving all malicious information flows. Since these metrics have only been proposed recently, none of the authors have evaluated their graph summarization technique by these metrics. We suggest authors of upcoming graph summary techniques to use these metrics for their evaluation.

The remainder of this section reviews graph summarization techniques for PIDS and is structured based on their categories.

The majority of the proposed approaches apply lossy graph summarization techniques to remove events that are not malicious or events that do not have a notable effect on intrusion detection performance. For instance, the data provenance of temporary files [7, 48, 51, 65, 75, 99, 112], or of read-only libraries or of dead files [75] can be removed without affecting the intrusion detection performance. In general, graph summarization of data provenance is a trade-off between compression rate and preserving enough semantic information required for sufficient intrusion detection and forward- and backward tracing in forensic analysis.

The scalability of a graph summarization technique describes the property to handle a growing number of hosts. Notably, large enterprise networks, where sophisticated attacks such as APT are most common, can contain several hundred computers, and thus, graph summarization techniques need to be scalable. Most existing approaches focus only on the graph summarization rate itself. They are not considering any other factors, such as the number of hosts the data provenance is collected from or the number of resources available for graph summarization. Furthermore, the datasets used for evaluation are collected from a limited number of hosts and not replayed in real-time to apply and evaluate a graph summarization technique in a realistic setup. Thus, more research is required on scalable graph summarization techniques in a realistic setup.

The most important evaluation criteria of data compressions approaches are their compression rates. Table 3 shows that all proposed techniques can reduce the space overhead of data provenance significantly. However, with the following differences: Firstly, some techniques require the partitioning of long-running processes into units [65, 73, 75], which initially generates additional logs to be included in the evaluation dataset. Secondly, most graph summarization techniques have been proposed and optimized for a specific scenario. For example, Winnower [48] compresses data provenance of cluster environments that run multiple instances of the same service. Since each instance of one service generally generates semantically similar data provenance, it can be significantly compressed by first generalizing it and then finding consensus with other instances of the same service. Another example is LogGC [65], which achieved better compression rates in a server environment than a client environment.

A common issue in evaluating graph summarization approaches for data provenance is the benchmark dataset used. As highlighted in Table 3, most of the datasets were created by the authors of the approach itself, and the results are not reproducible because they have not published their datasets. Consequently, there is a high demand for a publicly available benchmark dataset to evaluate graph summarization approaches for PIDS.

Another important evaluation criterion is the runtime overhead generated by graph summarization techniques. This measure is crucial when the graph summarization technique is executed on a device with limited resources. Table 3 gives an overview of the runtime overhead for each of the previously reviewed approaches, and the differences are in their details. First, ProTracer [75], LogGC [65], ProvWalls [7], and KCAL [73] are graph summarization techniques with an integrated data provenance capture system. Their stated runtime overhead includes both the runtime overhead of the graph summarization and the data provenance capture system. Second, the runtime overhead can vary in different execution scenarios of the graph summarization techniques. For example, KCAL generates only 1% of runtime overhead on clients, but up to 10% runtime overhead on servers. The runtime overhead on a server is significantly higher than on a client due to the enormous number of clients that a server application must serve, resulting in many dependencies [7].

The proposed approaches can be categorized into anomaly-based, rule-based and, tag-propagation-based intrusion detection techniques, which we define as follows:

Traditional IDS used either signature- or anomaly-based intrusion detection techniques. Signature-based techniques have been only partly used for PIDS due to the following reasons. First, PIDS have been introduced to overcome the challenges of traditional IDS to detect more sophisticated attack types such as APT. Second, a provenance graph enables fine-grained event correlation, and thus, signatures in nodes or edges can be chained to a rule, which can describe a cyberattack more comprehensively.

A sequence learning method builds a model based on the causal order of benign events in a provenance graph. This model is then used to predict the next event for a given sequence of events. A deviation of the predicted event from the actual event indicates a potentially malicious event.

DeepLog [25] utilizes a multi-class classifier to predict the next event based on a sequence of past events. It also applies stacked Long Short-Term Memory (LSTM) to detect anomalies in the sequence of event properties.

Tiresias [94] is a security event predictor that uses an LSTM to learn from a Endpoint Detection and Response (EDR) dataset of past security events to then predict an attacker’s next attack steps with up to 93% accuracy.

CamQuery [89] provides a vertex-centric API to implement provenance queries as value propagation applications. An example propagation application generates a feature vector for each vertex in the provenance graph, trains a Replicator Neural Network (RNN) and the resulting model is used to detect anomalies in streaming data provenance.

NeedleHunter [114] generates version-based provenance graphs to detect state changes of objects over time. It uses rules derived from the Tactics, Techniques, and Procedures (TTP) defined in the MITRE ATT&CK Matrix for Enterprises to taint malicious objects and then analyze the sequential dependencies between malicious tainted objects to detect APT [82].

ADSAGE [31] uses a Recurrent Neural Network (ReNN) to model the sequences of application logs and uses this model to predict future events and a Feed-Forward Neural Network (FFNN) to model the validity of the events to predict the anomaly score of future events.

Sequence learning methods can detect malicious behavior with high accuracy and are suitable for real-time detection. However, they might suffer from false alarms if the learned benign behavior changes due to evolution.

A graph embedding technique transforms nodes, edges, and their properties into vectors while conserving properties such as the structure of the provenance graph. Next, traditional anomaly detection methods for vector spaces can be applied to detect malicious behaviors.

SIGL [45] transforms provenance graphs into the vector space by using a component-based node embedding technique for graphs. It applies a LSTM to extract the graph features, calculates the anomaly scores for new software installations, and classifies them as benign or malicious based on a predefined threshold.

ProvDetector [106] selects the k rarest paths of a provenance graph and uses word2vector [79] to protect the paths to the vector space. The Local Outlier Factor (LOF) algorithm is used for each path of a new provenance graph to predict if it is malicious. A threshold is then used to judge if the whole provenance graph is considered malicious.

Another approach uses generic and provenance-specific network metrics to summarize the meaningful information and knowledge of a large and complex provenance graph as a vector. Generic network metrics include the number of nodes and edges, graph diameter, associative coefficient, average clustering coefficient, and degree distribution. Provenance-specific network metrics incorporate the number of node and event types, Maximum Finite Distance (MFD), MFD of derivations, and average clustering coefficient by node type. Then, the authors evaluated their proposed metrics by training a decision tree classifier to detect malicious provenance graphs [52].

Reference [4] suggested Node2Vec [40] to learn continuous feature representations for each node in a provenance graph and then applies Adaptive Online Metric Learning (AOML) to minimize the separation between malicious nodes and maximize the separation between malicious and benign nodes.

Reference [5] proposed a supervised learning approach to detect APT infections of a system. Each node of a provenance graph is considered an instance labeled by a continuous labeling algorithm. For each instance, the algorithm calculates a probability of being malicious based on its distance and path to a known bad instance. Then, for each instance, a feature vector is produced by combining the provenance graph context, APT inception features and accounting for agents, processes, and file names. Finally, a random forest classifier is trained, which is then used to classify further instances as benign or malicious. The classifier was able to detect malicious instances with an accuracy of 50% and False Positive Rate (FPR) under 4%.

Various graph embedding techniques have been proposed to transform provenance graphs into the vector space while preserving important information such as the graph structure. The evaluation results have demonstrated that the combination of graph embedding techniques and traditional anomaly detection methods can effectively detect malicious behavior. However, graph embedding techniques are computationally expensive and, thus, not always applicable for real-time detection.

Clustering-based approaches cluster nodes of a provenance graph into benign or malicious clusters by detecting behavior deviations of a system over time.

Clustering-based approaches in the literature utilize the Louvain method [90], Kullback-Leibler Distance (KLD) [43], or pair-wise similarity comparison of nodes [69] to cluster nodes in provenance graphs.

One common issue of clustering-based approaches is that they suffer from a high false alarm rate because it is challenging to distinguish between unknown benign system behavior, unknown malicious system behavior, and system errors [43].

Regular grammar-based approaches describe benign provenance graphs with regular grammars and then classify new provenance graphs, which cannot be described with these grammars as malicious.

Reference [66] proposed Directed Acyclic Graph regular grammars (DAGr) to model benign provenance graphs with regular grammars and deterministic Directed Acyclic Graph automata (dDAGa)s to represent those regular grammars as DFAs. New provenance graphs which cannot be modeled with these Deterministic Finite State Automatas (DFAs) are considered malicious. Regular grammar-based approaches suffer from a high false alarm rate if the normal behavior changes and cannot be described with the initially created grammar.

Rule learning-based approaches automatically learn rules from benign provenance graphs and then classify new provenance graphs, which these rules cannot describe as malicious.

PIDAS [109] and Pagoda [108] built rules based on the causalities between nodes in benign provenance graphs of the training dataset. For a new provenance graph, the provenance graph is classified as malicious if the ratio between matching and non-matching rules exceeds a predefined threshold. However, PIDAS and Pagoda cannot detect variances in benign or malicious provenance graphs that are not in the training dataset.

To overcome this shortcoming, P-Gaussian [111] uses a Gaussian distribution detection schema that determines the similarity between two provenance graph paths while considering event orders and the number of events to detect variances of provenance graph paths. Like regular grammar-based approaches, rule learning-based approaches suffer from high false alarm rates if the normal behavior changes.

Anomaly Score-based approaches calculate an anomaly score for a provenance graph and judge it as malicious if the score exceeds a predefined threshold.

Reference [9] evaluated the following generic unsupervised learning algorithms to assess their effectiveness of detecting APT: (1) Frequent Pattern Outlier Factor (FPOF) determines frequent patterns of the properties across the objects in the dataset based on a predefined threshold. Then, the anomaly score of an object is calculated by the number of frequent patterns it includes. Objects with lower anomaly scores would be considered possible anomalies. (2) Outlier Degree (OD) determines first, the frequent patterns of the properties across the objects in the dataset and second, high-confidence rules, which describe the associations of different patterns. Such rules describe, for example, if object o1 contains the pattern p1, then it must also contain the pattern p2. Then, each object is scored by applying the high-confidence rules to it. High scores correspond to high anomaly possibility. (3) Attribute Value Frequency (AVF) determines the frequency numbers of the individual properties and then scores each object in the dataset by adding up the probabilities of the actual properties in that specific object. So lower scores refer to rare properties which imply a possible anomaly. (4) One Class Classification by Compression (OC3) compresses objects in the dataset by using the compression algorithm Krimp. Krimp identifies common properties, stores them in a table, and uses them to compress the objects. Therefore, low compression rates of objects refer to a possible anomaly. (5) Similar to OC3, CompreX compresses the objects in the dataset, and the compression rates refer to the anomaly score. However, a more sophisticated compression strategy is used, which stores the identified patterns in multiple tables. This allows better exploitation of correlations between groups of patterns and improves the runtime overhead of compression.

The results show that FPOF and OD were not competitive on any of their evaluation dataset. AVF and OC3 achieved high normalized Discounted Cumulative Gain (nDCG) scores across all datasets and completed within three minutes. CompreX archived the highest nDCG score for the datasets, which include the smallest amount of object properties but fails or could not be completed within three hours for the datasets with more object properties. A downside of FPOF, OD, and OC3 is that they require parameter tuning, which is not always ideal for an unsupervised learning setting. Also, only AVF can be applied in a streaming anomaly detection environment.

Further, the same authors suggested in [8] score aggregation techniques to improve the detection performance of the generic unsupervised learning algorithms AVF and OC3. Different aggregation techniques such as sum, average, median, geometric mean, and minimum have been evaluated in different contextual settings. The evaluation results show that aggregating anomaly scores can improve detection performance. However, it is not predictable which aggregation technique will deliver the best anomaly detection performance for a specific context.

NoDoze [47] is a ranking system for security alerts generated by an IDS to reduce FPR. NoDoze creates a scenario graph for each security alert, assigns anomaly scores to the edges based on their frequency of occurrence, and then determines the overall anomaly score by aggregating the anomaly score of each edge.

However, NoDoze lacks in the reasoning about the causal dependencies between such security alerts. RapSheet [46] addresses this issue by annotating edges in the scenario graph with attack context, in particular TTP of the MITRE ATT&CK Matrix for Enterprises. Then, it calculates the anomaly score of a scenario graph based on the causal order of security alerts.

One major issue of anomaly score-based approaches is that they rely heavily on a predefined threshold. This threshold is defined in advance and optimized based on contextual information about the current scenario. Consequently, the anomaly score of a malicious provenance graph can fall below this threshold.

We analyzed the reviewed intrusion detection approaches for correlations between their category, attack type, and performance.

The majority of the approaches aimed to detect APT, most likely to overcome the low performance of traditional intrusion detection techniques for this attack type. These approaches cover all of the categories introduced in the taxonomy (see Figure 3). It is noticeable that tag-propagation-based approaches show outstanding performance with low false-positive rates, high true-positive rates, and fast detection times [49, 50].

The remaining approaches are all anomaly-based and aim to detect insider threats, SQL injection, data exfiltration, and Denial of Service (DoS) attacks. There is no clear correlation between their category, attack type, and performance. Notably, most of the approaches have been evaluated with a self-made and not publicly available dataset and thus, crucial correlations between the category, attack type, and performance can be hidden.

A majority of intrusion detection approaches for PIDS require a predefined threshold to distinguish malicious from benign events (see Table 4). Most researchers evaluate the impact of the selected threshold on the detection accuracy and false alarm rate. The final evaluation results are obtained from the threshold with the best trade-off between detection accuracy and false alarm rate. As a result, the selected threshold might work exceptionally well for the scenarios in the benchmark dataset used for the evaluation but might fail for other scenarios. Therefore, more research on adaptive or contextual threshold selection is required to make intrusion detection approaches more robust for real-world environments.

Up to date, only a few researchers have explored additional contextual sources to enhance the detection performance and reduce the false alarm rate of PIDS. Explored sources are security alarms gathered from commercial EDR [46, 47, 94] and attack context such as the TTP of the MITRE ATT&CK Matrix for Enterprises [46, 94, 114]. In future work, other potential contextual sources should be considered which include the system’s properties and user behaviors. The former could be used to select a threshold based on the system’s properties. For example, benign data provenance on a server environment shows significantly different characteristics as on a client environment. An optimized threshold for each environment is likely to increase the detection performance and reduce false alarms. The latter one could be used to better distinguish between unknown benign behavior, unknown malicious behavior, and system errors, a known issue in PIDS [43].

The performance of rule-based intrusion detection techniques highly relies on predefined rules composed of chained signatures defined by security experts [8]. We see a high potential to automate rules and signatures generation by using provenance graphs. A likely approach could be as follows: A malicious file is executed in a sandbox environment with a data provenance capture system. The captured data provenance is transformed into a provenance graph. Signatures are derived from the nodes and edges in the provenance graph. Finally, the derived signatures are chained to create a rule. As a result, rules and signatures can be generated in a more timely manner so that other affected systems can be identified more quickly.

Like the graph summarization approaches, most intrusion detection approaches have been evaluated on a self-made dataset by the authors (see Table 4). As a result, there is a high demand for a publicly available benchmark dataset to evaluate data analysis approaches for PIDS.

DARPA conducted five engagements as part of their transparent computing program, intending to develop technologies and prototypes utilizing data provenance for real-time detection and forensic analysis of APT. Each engagement had specific objectives, involved simulated APTs and benign background activity on realistic server infrastructure, and generated a public benchmark dataset as a result. Multiple data provenance capturing systems such as ETW for Windows or SPADE for Linux have been utilized to collect data provenance [22]. Nevertheless, the benign background activity has not been published as part of the datasets [42].

To determine the scalability of the TC program, DARPA created the OpTC dataset [21]. A two-week experiment was conducted on a test network with 1,000 hosts running Windows 10. An APT was simulated over a three-day time window. System-level data provenance and network logs from Zeek sensors [101] were collected, and the resulting OpTC dataset contains over 17 billion events, 0.3 million of them being malicious ones (0.0016%) [2]. Drawbacks of the dataset are the lack of documentation, information about the generation of the benign data, and that it is limited to one APT scenario.

In recent years DARPA has put in a significant effort to create benchmark datasets for PIDS. Their two latest datasets, the TC Engagement 5 and OpTC, have not been used yet by researchers: one potential reason could be the lack of documentation [2].

The CERT Insider Threat Test Dataset contains synthetic logon events, email traffic, web browsing traces, file access logs, removable media usage, and LDAP information describing organizational hierarchy and user roles [68]. The dataset contains a total of 135,117,169 operations of 4,000 users during 516 days, whereby six users have 470 malicious operations resulting from six attack scenarios. Nonetheless, researchers utilizing this dataset stated an extreme imbalance problem [31, 69].

LANL’s comprehensive cybersecurity event dataset contains 58 days of de-identified events collected from five data sources within LANL ’s corporate computer network. The sources are logon events, process events, DNS lookups, network flow events, and simulated red team events. In total, the dataset includes 1,648,275,307 events of 12,425 users on 17,684 computers. Well-known event properties such as system users (e.g., SYSTEM) or network ports (e.g., 80) have not been re-identified [61].

The ADFA IDS datasets consist of three datasets, the ADFA Linux Dataset (ADFA-LD) [19], the ADFA Windows Dataset (ADFA-WD), and the ADFA Windows Dataset: Stealth Attacks Addendum (ADFA-WD:SAA) [18].

ADFA-LD contains system-level data provenance from one ubuntu host collected by the Linux Audit Framework. Benign data provenance contains activities such as web browsing and latex document preparation. Malicious data provenance contains events from six simulated general attacks [19]. Despite all that, the dataset only contains system call numbers, and thus, this dataset cannot be used to evaluate approaches that need system call properties [70].

The ADFA-WD contains DLL traces of processes from a Windows XP host collected by Procmon. The datasets include 356 normal-, 1,828 validation-, and 5,773 attack traces. The attack traces are generated by exploiting twelve zero-day attacks on the host [41].

ADFA-WD:SAA: The ADFA-WD:SAA extends the ADFA-WD by 863 additional attack traces produced by three stealthy attacks, namely Doppelganger, Chimera, and Chameleon. The objective of these traces is to validate the resistivity of future HIDS [41]. Similar to ADFA-LD, ADFA-WD, and ADFA-WD:SAA are incomplete as they only contain DLL traces of processes, and the attack traces are only based on a few vulnerabilities [14].

To address the issues of the ADFA IDS datasets, the authors of [14] published the AWSCTD dataset. The objectives of this dataset are to use malware from a public repository to renew the attack traces quickly, utilize a wide selection of malware to generate attack traces, and contain complete system call traces and their properties. The dataset contains a total of 112.56 million attack traces generated from 10,276 malware. For comparison, the ADFA IDS datasets contain a total of 6,636 attack traces generated from 15 malware. However, the AWSCTD dataset does not contain benign traces nor multi-stage attacks such as APT.

It is challenging to create a benchmark dataset that mimics a real-world environment. The rapid changes in attack and defense techniques lead to publicly available datasets quickly become outdated and no longer represent the latest attack patterns [17, 19, 50, 67, 109].

The ratio between benign and malicious events in benchmark datasets does not reflect the ratio in real-world environments. Benchmark datasets tend to have a higher percentage of malicious events compared to real-world data [86].

Most benchmark datasets contain synthetic benign data that is easier to distinguish from malicious behavior than real-world benign data. Intrusion detection approaches could archive a high detection performance on this data but could fail in real-world environments [86].

Benign data represents the most significant portion of a benchmark dataset. A majority of the benchmark datasets contain synthetic benign data because data provenance contains privacy-sensitive information of users such as websites visited, files created, and applications used. Hence, collecting real-world benign data is not practical due to privacy issues which would not only expose the users but also the network topology of an organization [2, 17, 95].

Researchers tried to overcome the privacy issue by anonymizing and re-identifying the real-world benign data, e.g., LANL’s Dataset [61] contains re-identified real-world benign data. However, researchers have stated that heavily anonymized real-world benign data reduces the data quality and amount of semantic information [95].

There are several ways to generate synthetic benign data. Most often, an autonomous agent randomly performs a predefined set of actions on a system. One drawback is the predictability of future events due to the limited number of actions an agent performs.

It is challenging to utilize recent benchmark datasets due to a lack of documentation and tools. E.g., DARPA OpTC [21] contains billions of events resulting in Terabyte (TB)s of data. Even though this dataset contains high-quality data, researchers struggle to use this dataset because detailed explanations of the attack scenarios and information and tools on how to use the data are missing. The authors of [2] published additional documentation and analysis results of the DARPA OpTC [21] to make it easier for other researchers to use this dataset.

Due to the lack of real-world benchmark datasets, lack of high-quality benign data, and lack of documentation and tools, many researchers created a self-made benchmark dataset to evaluate their approaches. Even though these datasets contain the latest attack scenarios and high-quality, real-world benign data, researchers cannot share these datasets due to privacy issues. This is a significant drawback because evaluation results cannot be reproduced, so researchers cannot compare their approaches. Another drawback is that self-made benchmark datasets may favor the approach of authors [17, 50].

Benign data can be generated either through real-world user studies or through simulations of user behavior. While the former suffers from privacy issues, the latter suffers from correctly mimicking reality. Little research on data obfuscation techniques for data provenance has been conducted. One major challenge is to obfuscate privacy-sensitive information while keeping as much semantic information as possible [95]. Multiple data obfuscation techniques have been proposed for NIDS, such as homomorphic encryption, hash functions, bloom filters, and more [87]. There is a need to evaluate the effectiveness of these techniques on data provenance.

Since many researchers have used self-made benchmark datasets, metrics are needed to describe the data quality of these datasets. Researchers could then publish these metrics to allow other researchers to generate comparable benchmark datasets that replicate these metrics making the release of benign data redundant.

Malicious data can be generated by red teams trying to exploit vulnerable targets or by simulations, which mimic realistic APTs. Simulations can be described by a configuration file, which enables the reproducibility of the scenarios at any time and is hence, the preferred method to generate malicious data.

There are various frameworks to simulate APTs, an overview is given in Table 7. Especially Xanthus [42], Caldera [81], Atomic Red Team [36], Splunk Attack Range [38], and PruleSharp [35] are very promising, since they allow one to configure simulations by describing cyberattacks with the TTP defined in the MITRE ATT&CK matrix [82]. Besides, the malicious data generated by these frameworks can be labeled more precisely by using TTP identifiers.

Previous benchmark datasets for PIDS have been published in heterogeneous data formats, which makes it challenging to evaluate an approach with multiple benchmark datasets. In addition, there is a lack of tools to parse data provenance for PIDS from one format to another. Table 8 gives an overview of data formats utilized in existing benchmark datasets for PIDS. Especially eCAR [34] and CDM [22] provide not only promising data models to represent data provenance for PIDS but also support various OS. Despite this, these data models lack tools to parse data provenance captured by data provenance capturing systems.

Ideally, benchmark datasets for PIDS should be easily customizable to add and publish new attack scenarios for other researchers quickly. Not only attack trends but also software evolve rapidly.

The trend towards frequently changing attack surfaces creates increased vulnerabilities for attackers. To adapt intrusion detection approaches to the latest attack trends, benchmark datasets need to include these trends. Frequent releases of new software versions including OS, data provenance capture system, and other application updates may lead to changes in the data provenance that need to be reflected in benchmark dataset.