SoftED: Metrics for Soft Evaluation of Time Series Event Detection

This section gives an example of the problem of evaluating inaccurate event detections and defines the problem in the paper concerning the demand for adequate detection performance metrics. Consider, for example, a time series $X$ containing an event at time $t$, represented in Figure 1. Since detectors A and B are applied to $X$, a user must select one of them as the most adequate for the underlying application. Detector A detects an event at time $t+k_{1}$, while Detector B detects an event at time $t-k_{2}$ ($k_{2}>k_{1}$). As none of the methods could correctly detect the event at time $t$, based on the usual detection accuracy evaluation, the user would deem both inaccurate and disposable.

However, inaccuracy in detecting an event can often result from its preceding or lingering effects. Take the adoption of a new policy in a business. While a domain specialist may consider the moment of policy enforcement as a company event, its effects on profit may only be detectable a few months later. On the other hand, preparations for policy adoption may be detectable in the antecedent months. Moreover, when accurate detections are not achievable, which is common, detection applications demand events to be identified as soon as possible [37], or early enough to allow necessary actions to be taken, mitigating possible critical system failures or help mitigate urban problems resulting from extreme weather events, for example. In this context, the results of Methods A and B would be valuable to the user. Note that while Detector B seems to anticipate the event, its application to $X$ and resulting detection is made after the event’s occurrence. On the other hand, the detection of Detector A came temporally closer to the event, possibly more representative of its effects.

In this context, evaluating event detection is particularly challenging, and the detection accuracy metrics usually adopted are insufficient and inadequate for the task [59]. Standard classification metrics do not consider the concept of time, which is fundamental in the context of time series analysis, and do not reward early detection [1], for example, or any relevant neighboring detections. For the remainder of this paper, neighboring or close detections refer to detections whose temporal distance to events is within a desired threshold. Current metrics only reward true positives (exact matches in event detection). All other results are “harshly” and equally discredited.

In this case, there is a demand to soften the usual concept of detection accuracy and evaluate the methods while considering neighboring detections. However, state-of-the-art metrics designed for scoring anomaly detection, such as the NAB score [37], are still limited [59], while also being biased towards results preceding events. To the best of our knowledge, no metrics available in the literature consider the concept of time and tolerance for detections sufficiently close to time series events. This paper focuses on addressing this demand.

This paper introduces the SoftED metrics, a new set of metrics for evaluating event detectors regarding their detection accuracy and ability to produce neighboring detections to events. Inspired by fuzzy sets, SoftED metrics assess the degree to which a detection represents a particular event. It is an innovative approach since, while fuzzification is commonly applied to time series observations [33], in SoftED, the fuzzification occurs in the time dimension [72]. Hence, they incorporated time and temporal tolerance in evaluating the accuracy of event detection. These are the scenarios that domain specialists and users often face. Up until now, there have been no standard or adequate evaluation metrics for event detection in such scenarios. SoftED metrics soften the standard classification metrics, which are considered in this paper as hard metrics, to support the decision-making regarding the most appropriate method for a given application.

This paper also contributes by introducing a new general protocol for evaluating the performance of metrics in time series event detection. This protocol is designed to evaluate metrics that incorporate temporal tolerance into detection evaluation. It is inspired by a strategy commonly adopted in ontology design. It is based on using competency questions [64, 45].

Computational experiments were conducted to analyze the contribution of the developed metrics against the usual hard and state-of-the-art metrics [37]. The results illustrate that SoftED can improve event detection evaluation by associating events and their representative (neighboring) detections, incorporating temporal tolerance in over 36% of the conducted experiments compared to usual hard metrics. More importantly, following the proposed evaluation protocol, we evaluated the SoftED metrics based on the devised competency questions. Domain specialists confirmed the contribution of SoftED metrics to the problem of detector evaluation.

The remainder of this paper is organized as follows. Section 2 provides concepts on time series event detection and reviews the literature on detection performance metrics for detection evaluation. Section 3 formalizes the developed SoftED metrics. Section 4 presents a quantitative and qualitative experimental evaluation of the developed metrics and their empirical results. Finally, conclusions are made in Section 5.

Events correspond to a phenomenon, generally pre-defined in a particular domain, with an inconstant or irregular occurrence relevant to an application. In time series, events represent significant changes in expected behavior at a certain time or interval [30]. In general, instant events of a given time series $X$ = <$x_{1},x_{2},x_{3},\cdots,x_{n}$>, can be identified in a simplified way by $e(X,k,\sigma)$ using the Equation 1, where $k$ represents the length of nearby observations¹¹1Due to limited space, the general formalization of event intervals lie outside the scope of this paper.. A temporal component (TC) for an observation $x_{t}$ is expressed as $tc(x_{t})$. The TC can refer to the observation itself or its instant trend, respectively represented as $x_{t}$ and $tr(x_{t})$.

Considering a typical autoregressive behavior [27], one can expect a TC for $x_{t}$ to be related to previous observations. Let $ep$ be the expected TC for $x_{t}$ based on the previous $k$ observations, where $ep(tc(x_{t}),k)=E(tc(x_{t})~{}|~{}tc(x_{t-k}),\dots,tc(x_{t-1}))$. Analogously, one can also expect TC for $x_{t}$ to be explained from the following observations [40]. So let $ef$ be the expected TC for $x_{t}$ based on the following $k$ observations, where $ef(tc(x_{t}),k)=E(tc(x_{t})~{}|~{}tc(x_{t+1}),\dots,tc(x_{t+k}))$. If a TC for $x_{t}$ escapes the expected value above a threshold $\sigma$ based on previous or following $k$ observations [27, 40], it can be considered an event. Equation 1 also considers an event if the expected TC from previous and following $k$ observations are different above the threshold $\sigma$.

Event detection is the process of identifying the occurrence of such events based on data analysis. It is recognized as a basic function in surveillance and monitoring systems. Moreover, it becomes even more relevant for applications based on time series and sensor data analysis [48]. Event detectors found in the literature are usually based on model deviation analysis, classification-based analysis, clustering-based analysis, domain-based analysis, or statistical techniques [12, 31, 48, 3]. Regardless of the adopted detection strategy, an important aspect of any event detector is how the events are reported. Typically, the outputs produced by event detectors are either scores or labels. Scoring detectors assign an anomaly score to each instance in the data depending on the degree to which that instance is considered an anomaly. On the other hand, labeling detectors assign a label (normal or anomalous) to each data instance. Such methods are the most commonly found in the literature [12].

Detectors might vary performance under different time series [23]. Therefore, there is a demand to compare the results they provide. Such a process aims to guide the choice of suitable methods for detecting events of a time series in a particular application. For comparing event detectors, standard classification metrics, such as F1, Precision, and Recall, are usually adopted [31, 37, 62].

As usual, the standard classification metrics depend on measures of true positives ($TP$), true negatives ($TN$), false positives ($FP$), and false negatives ($FN$) [31]. In event detection, the $TP$ refers to the number of events correctly detected (labeled) by the method. Analogously, $TN$ is the number of observations that are correctly not detected. On the other hand, the measure $FP$ is the number of detections that did not match any event, that is, false alarms. Analogously, $FN$ is the number of undetected events. Among the standard classification metrics, Precision and Recall are widely adopted. Precision reflects the percentage of detections corresponding to time series events (exactness), whereas Recall reflects the percentage of correctly detected events (completeness). Precision and Recall are combined in the F_β metrics [31]. The F1 metric is also widely used to help gauge the quality of event detection balancing Precision and Recall [62].

As discussed in Section 1.1, event detection is particularly challenging to evaluate, event detection is particularly challenging to evaluate. In this context, the Numenta Anomaly Benchmark (NAB) provided a common scoring algorithm for evaluating and comparing the efficacy of anomaly detectors [37]. The NAB score metric is computed based on anomaly windows of observations centered around each event in a time series. Given an anomaly window, NAB uses the sigmoidal scoring function to compute the weights of each anomaly detection. It rewards earlier detections within a given window and penalizes $FP$s. Also, NAB allows the definition of application profiles: standard, reward low $FP$s, and reward low $FN$s. Based on the window size, the standard profile gives relative weights to $TP$s, $FP$s, and $FN$s.

Nonetheless, the NAB scoring system presents challenges for its usage in real-world applications. For example, the anomaly window size is automatically defined as 10% of the time series size, divided by the number of events it contains, and values generally not known in advance, especially in streaming environments. Furthermore, Singh and Olinsky [59] pointed out the scoring equations’ poor definitions and arbitrary constants. Finally, score values increase with the number of events and detections. Every user can tweak the weights in application profiles, making it difficult to interpret and benchmark results from other users or setups.

In addition to NAB [37], this section presents other works related to the problem of analyzing and comparing event detection performance [12]. Recent works focus on the development of benchmarks to evaluate univariate time series anomaly detectors [36, 8, 70]. Jacob et al. [36] provide a comprehensive benchmark for explainable anomaly detection over high-dimensional time series. In contrast, the benchmark developed by Boniol et al. [8] allows the user to assess the advantages and limitations of anomaly detectors and detection accuracy metrics.

Standard classification metrics are generally used for evaluating the ability of an algorithm to distinguish normal from abnormal data samples [12, 62]. Aminikhanghahi and Cook [4] review traditional metrics for change point detection evaluation, such as Sensitivity, G-mean, F-Measure, ROC, PR-Curve, and MSE. Detection evaluation measures have also been investigated in the areas of sequence data anomaly detection [13], time series mining and representation [19], and sensor-based human activity learning [17, 68].

Metrics found in the literature are mainly designed to evaluate the detection of instant anomalies. However, many real-world event occurrences extend over an interval (range-based). Motivated by this, Tatbul et al. [62] and Paparrizos et al. [47] extend the well-known Precision and Recall metrics, and the AUC-based metrics, respectively, to measure the accuracy of detection algorithms over range-based anomalies. Other recent metrics developed for detecting range-based time series anomalies are also included in the benchmark of Boniol et al. [8]. In addition, Wenig et al. [70] published a benchmarking toolkit for algorithms designed for detecting anomalous subsequences in time series [7, 9].

Few works opt to evaluate event detection algorithms based on metrics other than traditional ones. For example, Wang, Vuran, and Goddard [67] calculate the delay until an individual node and the delivery delay in a transmission network detect an event. The work presents a framework for capturing delays in detecting events in large-scale WSN networks with a time-space simulation. Conversely, Tatbul et al. [62] also observe the neighborhood of event detections, not to calculate detection delays, but to evaluate positional tendency in anomaly ranges. Our previous work uses the delay measure to evaluate the bias of algorithms to detect real events in time series [22]. It furthers a qualitative analysis of the tendency of algorithms to detect before or after the occurrence of an event.

Under these circumstances, there is still a demand for event detection performance metrics that incorporate both the concept of time and tolerance for detections sufficiently close to time series events. Therefore, this paper contributes by introducing new metrics for evaluating methods regarding their detection accuracy and considering neighboring detections, incorporating temporal tolerance for inaccuracy in event detection.

We incorporate a distance-based temporal tolerance for events. It is done by defining the relevance of a particular detection to an event. This section formalizes the proposed approach. Table 1 defines the main variables used in the formalization of SoftED. Given a time series $X$ of length $|X|$ containing a set of $m$ events, $E=\{e_{1},e_{2},\dots,e_{m}\}$, where $e_{j}$, $j=1,\dots,m$, is the j–th event in $E$ occurring at time point $t_{e_{j}}$. A particular detector applied to $X$ produces a set of $n$ detections, $D=\{d_{1},d_{2},\dots,d_{n}\}$, where $d_{i}$, $i=1,\dots,n$, is the i–th detection in $D$ indicating the time point $t_{d_{i}}$ as a detection occurrence.

The degree to which a detection $d_{i}$ is relevant to a particular event $e_{j}$ is given by an event membership function $\mu_{e_{j}}(t)$ as defined in Equation 4 and illustrated in Figure 3(a). The $\mu_{e_{j}}(t)$ represents a Euclidean distance function, which is of simple computation and interpretation. Moreover, this solution was inspired by Fuzzy sets, where we innovate by fuzzifying the time dimension rather than the time series observations [72]. The definition of $\mu_{e_{j}}(t)$ considers the acceptable tolerance for inaccuracy in event detection for a particular domain application. The acceptable time range in which an event detection is relevant for allowing an adequate response reaction to a domain event is given by the constant $k$. For example, a meteorologist needs to provide alerts of extreme weather events at least 6 hours in advance, or an engineer needs to intervene in the operation of an overheated piece of machinery within 5 minutes before critical system failure. In this case, they might set $k$ to 6 hours or 5 minutes, respectively. However, this approach defines a domain-agnostic default value of $k$, set to $15$, defining a tolerance window of $30$ observations enough to hold the central limit theorem.

Figure 3(b) represents the evaluation of $\mu_{e_{j}}(t)$ for two detections, $d_{1}$ and $d_{2}$, produced by a particular detector. In this context, $\mu_{e_{j}}(t_{d_{i}})$ gives the extent to which a detection $d_{i}$ represents event $e_{j}$, or, in other words, its temporal closeness to a hard true positive (TP) regarding $e_{j}$. In that case, detection $d_{1}$ is closer to a TP, and $d_{2}$ lies outside the tolerance range given by $k$ and could be considered a false positive.

We are interested in the SoftED metrics, which are still preserving concepts applicable to traditional (hard) metrics. In particular, the SoftED metrics are designed to express the same properties as their hard correspondents. Moreover, they are designed to maintain the reference to the perfect detection performance (score of $1$ as in hard metrics) and indicate how close a detector came to it. To achieve this goal, this approach defines constraints necessary for maintaining integrity concerning the standard hard metrics:

1. 1.

   A given detection $d_{i}$ must have only one associated score.

2. 2.

   The total score associated with a given event $e_{j}$ must not surpass $1$.

The first constraint comes from the idea that the detection $d_{i}$ should not be rewarded more than once. It avoids the possibility of the total score for $d_{i}$ surpassing the perfect reference score of $1$. Take, for example, the first scenario, presented in Figure 3(c), in which we have one detection and many close events. The detection $d_{1}$ is evaluated for events $e_{1}$, $e_{2}$, and $e_{3}$ resulting in three different membership evaluation of $\mu_{e_{1}}(t_{d_{1}})$, $\mu_{e_{2}}(t_{d_{1}})$, and $\mu_{e_{3}}(t_{d_{1}})$, respectively. Nevertheless, to maintain integrity with hard metrics, a given detection $d_{1}$ must not have more than one score. Otherwise, $d_{1}$ would be rewarded three times, and its total score could surpass the score of a perfect match, which would be $1$.

To address this issue, we devise a strategy for attributing each detection $d_{i}$ to a particular event $e_{j}$. The adopted attribution strategy is based on the temporal distance between $d_{i}$ and $e_{j}$. It facilitates interpretation and avoids the need to solve an optimization problem for each detection. In that case, we attribute $d_{i}$ to the event $e_{j}$ that maximizes the membership evaluation $\mu_{e_{j}}(t_{d_{i}})$. This attribution is given by $E_{d_{i}}$ defined in Equation 5. According to Figure 3(c), $d_{1}$ is attributed to event $e_{1}$ given the maximum membership evaluation of $\mu_{e_{1}}(t_{d_{1}})$. If there is a tie for the maximum membership evaluation of two or more events, $E_{d_{i}}$ represents the set of events to which $d_{i}$ is attributed. Consequently, we can also derive the set of detections attributed to each event $e_{j}$, $D_{e_{j}}$, defined by Equation 6. The addition of a detection $d_{i}$ to the set $D_{e_{j}}$ is further conditioned by the tolerance range, that is, a membership evaluation greater than $0$ ($\mu_{e_{j}}(t_{d_{i}})>0$).

The second constraint defined by this approach comes from the idea that a particular detector should not be rewarded more than once for detecting the same event $e_{j}$. It assures that the total score of detections for event $e_{j}$ does not surpass the perfect reference score of $1$. Take, for example, the second scenario, presented in Figure 3(d), in which many detections are attributed to the same event. The event $e_{1}$ is present in the sets $E_{d_{1}}$, $E_{d_{2}}$, $E_{d_{3}}$ and $E_{d_{4}}$. Moreover, $D_{e_{1}}$ contains $d_{1}$, $d_{2}$, and $d_{3}$. But to maintain integrity with hard metrics, the total score for $D_{e_{1}}$ ($\mu_{e_{1}}(t_{d_{1}})+\mu_{e_{1}}(t_{d_{2}})+\mu_{e_{1}}(t_{d_{3}})$) must not surpass the score of a perfect match.

To address this issue, we devise an analogous distance-based strategy for attributing a representative detection $d_{i}$ to each event $e_{j}$. In that case, we attribute to event $e_{j}$ the detection $d_{i}$, contained in $D_{e_{j}}$, that maximizes the membership evaluation $\mu_{e_{j}}(t_{d_{i}})$. This attribution is given by $\hat{d}_{e_{j}}$ defined in Equation 7. According to Figure 3(d), $e_{1}$ is best represented by detection $d_{1}$ given the maximum membership evaluation of $\mu_{e_{1}}(t_{d_{1}})$. Consequently, we can compute the associated score for each event $e_{j}$ as $es(e_{j})$, defined by Equation 8.

Finally, each detection $d_{i}$ produced by a particular detector is scored by $ds(d_{i})$ defined in Equation 9. Representative detections ($d_{i}=\hat{d}_{e_{j}}$) are scored based on $es(e_{j})$. All other detections are scored $0$. This definition ensures the total score for detections of a particular method does not surpass the number of real events $m$ contained in the time series $X$. The Equation 10 holds and maintains the reference to the score of a perfect detection Recall according to usual hard metrics. Furthermore, it penalizes false positives and multiple detections for the same event $e_{j}$.

The pseudocode for calculating $ds(d_{i})$ for each detection $d_{i}$ is given by Algorithm 1, which defines the function $soft\_scores$. This function takes three arguments: $detections$, $events$, and $k$. The $detections$ argument is a logical vector of the same length as a time series $X$ ($|X|$), indicating the detections by a given method, and $events$ is a logical vector of the same length, indicating the true events in $X$. The $k$ argument is an optional parameter (default is set to $15$) that sets the temporal tolerance for a detection to be considered a match with a true event.

The function $soft\_scores$ in Algorithm 1 first takes the vectors $E$ and $D$ containing the indices of all events and detections in the input vectors $events$ and $detections$, respectively. Then, for each $i$-th detection, it finds the indices of all matching events within a distance of $k$ (line $7$) and stores them in $E_{match}$. If no matching events exist, $ds(d_{i})$ is $0$. Otherwise, it calculates the membership scores between each matching event in $E_{match}$ and the $i$-th detection (line $11$) according to Equation 4. Finally, the soft score for each detection ($ds(d_{i})$) is the maximum membership score among the matching events (line $12$), indicating the best match between the current detection and the events and respecting the constraints necessary for maintaining integrity with hard metrics defined by Equations 5 to 10. Lastly, a vector of soft scores for each detection ($ds$) is returned.

The scores computed for each detection $d_{i}$, $ds(d_{i})$, are used to create soft versions of the hard metrics TP, FP, TN, and FN, as formalized in Table 2. In particular, while the value of TP gives the number of detections that perfectly matched an event (score of $1$), the sum of $ds(d_{i})$ scores indicate the degree to which the detections of a method approximate the $m$ events contained in time series $X$ given the temporal tolerance of $k$ observations. Hence, the soft version of the TP metric, $\text{TP}_{s}$ is given by $\sum_{i=1}^{n}ds(d_{i})$. Conversely, the soft version of FN, $\text{FN}_{s}$, indicates the degree to which a detector could not approximate the $m$ events in an acceptable time range. The $\text{FN}_{s}$ can then be defined as the difference between $\text{TP}_{s}$ and the perfect recall score ($m-\text{TP}_{s}$).

On the other hand, while the value of FP gives the number of detections that did not match an event (score of $0$), its soft version, $\text{FP}_{s}$, indicates how far the detections of a method came to the events contained in time series $X$ given the temporal tolerance of $k$ observations. In that sense, $\text{FP}_{s}$ is the complement of $\text{TP}_{s}$ and can be defined by $\sum_{i=1}^{n}\left(1-ds(d_{i})\right)$. Finally, the soft version of TN, $\text{TN}_{s}$, indicates the degree to which a detector could avoid nonevent observations of $X$ ($|X|-m$). The $\text{TN}_{s}$ is given by the difference between $\text{FP}_{s}$ and the perfect specificity score ($(|X|-m)-\text{FP}_{s}$).

Due to the imposed constraints described in Section 3.2, the defined SoftED metrics $\text{TP}_{s}$, $\text{FP}_{s}$, $\text{TN}_{s}$, and $\text{FN}_{s}$, hold the same properties and the same scale as traditional hard metrics. Consequently, using the same characteristic formulas, they can derive soft versions of traditional scoring methods, such as Sensitivity, Specificity, Precision, Recall, and F1. Moreover, SoftED scoring methods still provide the same interpretation while including temporal tolerance for inaccuracy, which is pervasive in time series event detection applications. An implementation of SoftED metrics in R is made publicly available at Github²²2SoftED implementation, datasets and experiment codes: https://github.com/cefet-rj-dal/softed.

This section presents the datasets selected for evaluating the SoftED metrics. The selected datasets are widely available in the literature and are composed of simulated and real-world time series regarding several different domain applications such as water quality monitoring (GECCO) [51]³³3The GECCO dataset is provided by the R-package EventDetectR [42]., network service traffic (Yahoo) [69], social media (NAB) [1], oil well exploration (3W) [65], and public health (NMR) ⁴⁴4The NMR dataset was produced by Fiocruz and comprised data on neonatal mortality in Brazilian health facilities from 2005 to 2017. It is publicly available at https://doi.org/10.7303/syn23651701., among others. The selected datasets present over $6$ hundred representative time series containing different events. In particular, GECCO, Yahoo, and NAB contain mostly anomalies, while 3W and NMR contain mostly change points. Moreover, the datasets present different nonstationarity and statistical properties to provide a more thorough discussion of the effects of the incorporated temporal tolerance on event detection evaluation on diverse datasets.

For evaluating SoftED metrics, a set of up to $12$ different event detectors were applied to all time series in the adopted datasets, totalizing $4,026$ event detection experiments. Each experiment comprised an offline detection application, where the methods had access to the entire time series given as input. The applied detectors are implemented and publicly available in the Harbinger framework [55]. It integrates and enables the benchmarking of different state-of-the-art event detectors. These methods encompass searching for anomalies and change points using statistical, volatility, proximity, and machine learning methods. The adopted methods are described in detail by Escobar et al. [22], namely: the Forward and Backward Inertial Anomaly Detector (FBIAD) [40], K-Nearest Neighbors (KNN-CAD) [25], anomalize (based on time series decomposition [21, 58]) [29, 18], and GARCH [11], for anomaly detection; the Exponentially Weighted Moving Average (EWMA) [50], seminal method of detecting change points (SCP) [30], and ChangeFinder (CF) [60], for change point detection; and the machine learning methods based on the use Feed-Forward Neural Network (NNET) [52], Convolutional Neural Networks (CNN) [28, 39], Support Vector Machine (SVM) [14, 49], Extreme Learning Machine (ELM) [34, 61], and K-MEANS [44], for general purpose event detection.

In each experiment, detectors were evaluated using the hard metrics Precision, Recall, and F1. Among them, the F1 was the main metric used for comparison. The NAB score was also computed with the standard application profile for each detector result. The NAB scoring algorithm is implemented and publicly available in the R-package otsad [35]. Anomaly window sizes were automatically set. During the computation of the NAB score, confusion matrix metrics are built. Based on these metrics, the F1 metric of the NAB scoring approach was also computed. Finally, the SoftED metrics were computed for soft evaluation of the applied event detectors. In particular, this experimental evaluation sets the constant of temporal tolerance, $k$, to its default value ($15$) unless stated otherwise. Nevertheless, $k$ also experimented with values in $\{30,45,60\}$ for sensitivity analysis. The datasets and codes used in this experimental evaluation were available for reproducibility2.

This section presents the quantitative analysis of the SoftED metrics. The main goal of this analysis is to assess the effects of temporal tolerance incorporated by SoftED in event detection evaluation. In particular, this section intends to assess (i) whether the proposed metrics can incorporate temporal tolerance to event detection evaluation and, if so, (ii) whether the incorporated temporal tolerance affects the selection of detectors. Answering both questions demands an experimental evaluation to compare the proposed metrics against other baseline and state-of-the-art metrics.

Although the analysis of Section 4.3 is meant to evaluate the quantitative effects of temporal tolerance, it is not enough to answer whether these effects benefit the evaluation of event detection performance. Hence, there is a demand for a qualitative analysis of the contribution of the temporal tolerance incorporated by the proposed metrics under different scenarios. In this context, a given tolerant metric contributes if its adoption leads to selecting the ideal event detector for a given detection scenario. For that, we established a protocol (Section 4.4.1) and conducted the qualitative analysis (Section 4.4.2).