From Detection to Action: a Human-in-the-loop Toolkit for Anomaly Reasoning and Management

xStream consists of three main steps, which we review briefly for the paper to be self-contained, and refer to [25] for details.

2.1.2 Step 2. Denstiy Estimation with Half-space Chains. Anomaly detection relies on density estimation at multiple scales via a set of so-called Half-space Chains (HC), a data structure akin to multi-granular subspace histograms. Each HC has a length L, along which the (projected) feature space $\mathcal {F}_p$ is recursively halved on a randomly sampled (with replacement) feature, where f_l ∈ {1, …, K} denotes the feature at level l = 1, …, L.

Given a sketch s, the goal is to efficiently identify the bin it falls into at each level. Let $\boldsymbol {\Delta }\in \mathbb {R}^K$ be the vector of initial bin widths, equal to half the range of the projected data along each dimension $f \in \mathcal {F}_p$. Let $\bar{\mathbf {z}}_l \in \mathbb {Z}^K$ denote the bin identifier of s at level l, initially all zeros. At level 1, bin-id is updated as $\bar{\mathbf {z}}_1[f_1] = \lfloor \mathbf {s}[f_1]/\boldsymbol {\Delta }[f_1] \rfloor$. At consecutive levels, it can be computed incrementally, as

(3)

where o(f_l, l) denotes the number of times feature f_l = {1, …, K} has been sampled in the chain until and including level l. Then, level-wise (multi-scale) densities are estimated by counting the number of points with the same bin-id per level.

Overall, xStream is an ensemble of M HCs, $\mathcal {H}= \lbrace HC^{(m)} := (\boldsymbol {\Delta }, \mathbf {f}^{(m)}, \mathcal {C}^{(m)})\rbrace _{m=1}^M$ where each HC is associated with (i) bin-width per feature $\boldsymbol {\Delta }\in \mathbb {R}^K$, (ii) sampled feature per level $\mathbf {f}^{(m)} \in \mathbb {Z}^L$, and (iii) counting data structure per level $\mathcal {C}^{(m)} = \lbrace C_l^{(m)}\rbrace _{l=1}^L$.

To estimate the weight of a feature for a high-score anomaly, we follow a simple procedure that leverages the ensemble nature of xStream. In a nutshell, it identifies the half-space chains in the ensemble that “use” the feature in binning the feature space, and (re)calculates the the anomaly score of the point only based on this set of chains. The higher it is, the more important the feature is in assigning a high score to the (anomalous) point.

Specifically, recall from Sec. 2.1.2 that $\mathbf {f}^{(m)} \in \mathbb {Z}^L$ denotes the sequence of features used in halving the feature space by chain m. Given M chains $\mathcal {H}= \lbrace HC^{(m)}\rbrace _{m=1}^M$, and a feature f to estimate its importance for a (projected) point s, we partition the chains into two groups: those that do and do not “use” f in f^(m).

The definition of “use” needs care here, due to how the anomaly score of a point is estimated by a chain. Note in Eq. (4) that the level l at which the extrapolated count is the minimum provides the score; in effect, only the features up to l contribute to a point's score. Therefore, a feature is considered “used” by a chain if it is a halving feature from the top down to this scoring level only.

Let l_s denote the level at which a point s is scored by a chain. A feature f is used by chain m if f ∈ f^(m)[1]…f^(m)[l_s]. Let $\mathcal {M}_u^{(f)}$ denote the chain indices that use feature f. Then, the importance weight of f for point s is given as:

We note that several alternative importance measures did not perform well, such as the difference between scores from the chains that do and do not use f, or the drop in the anomaly score when the chains that use f are removed. The reason is multicollinearity; when chains that did not use an important feature f used correlated features instead, they continued to yield a high anomaly score.

Recall from Sec. 2.1.1 that xStream creates projection features k = {1…K} to sketch high-dimensional and/or mixed-type data. The chains are built using the projected features, thus, the estimated importances above are for those “compound” features.

To this end, the relations between the projected and original features can be captured as a sparse bipartite graph. Nodes k = {1…K} on one side depict the projected features with pre-computed feature importances (node weights) as described in Sec. 3.1. Nodes $F=\lbrace 1\ldots |\mathcal {F}_r \cup \mathcal {F}_c|\rbrace$ on the other side depict the original features. Note that this graph is built separately for each (anomalous) point x to be explained. Thanks to the binary hash functions, there exists an edge (k, F) only when h_k(F) ≠ 0 (or for categorical F, when h_k(F⊕x[F]) ≠ 0), with expected density 1/3.

To attribute importances from projection features to the original features, a simple approach could sum the importances of the projection features whose compound an original feature participates in. However, this may attribute spurious importance from a neighbor that is important due to a different feature in its compound. It would also fail to tease apart additive feature attributions [23] in the presence of multicollinearity. As an intermediate solution, we go beyond the direct neighbors and diffuse in the graph the initial projection feature weights via random walk with restart [10].

Let $\mathbf {A}\in \lbrace 0,1\rbrace ^{(K\times |\mathcal {F}_r \cup \mathcal {F}_c|)}$ denote the adjacency matrix of the bipartite graph, π = π_o‖π_p denote the concatenated ‘topic-sensitive’ Pagerank vector for the original and projected features, respectively, and w_p be the vector of projected feature importance weights based on Eq. (5). w_p is normalized to capture the fly-back probabilities, and π is initialized randomly and normalized over iterations. Then,

To assess the performance of our feature explanations, we require data containing anomalies with ground-truth feature importance weights, which (to our knowledge) does not publicly exist. To this end, we create a new simulator synthesizing anomalies in subspaces along with feature importances. A basic simulator could generate data from a predefined distribution, altering subset of features to create anomalies. However, it may produce unrealistic data. Also, consider the case where features A and B are expanded by different factors (5 times and 10 times respectively). It is not clear which feature, A or B, is more responsible for outlierness.

To overcome the above difficulties, we propose using generative models with real-world data to synthesize anomalous points and feature importances. We use the variational auto-encoder (VAE) [16] to capture complex data distributions with both real-valued and categorical features. A VAE embeds a training point x into a lower-dim. z. At training stage, the encoder minimizes the distance of a surrogate posterior q_ϕ(x|z) to the true p_θ(z), while the decoder maximizes p_θ(x|z). The likelihood of anomaly can be determined by VAE's reconstruction probability p_θ(x|z), which we also use to calculate feature importances, by comparing the reconstruction probability of a feature is altered vs. not altered.

Given a dataset, we feed all the normal points into the VAE and generate m normal points $\lbrace {\bf x}_{nor}^i\rbrace _{i=1}^{m}$ with $\lbrace {\bf z}^i\rbrace _{i=1}^{m}$ hidden variables. Then, we set a threshold τ for anomalous points as:

Data:  We evaluate our anomaly explanation-by-feature importances approach on three real-valued and three mixed-type datasets, which are commonly used in anomaly detection literature for tabular data. Table 1 lists the dataset names and descriptions. All data are publicly available at the UCI machine learning repository [6].¹

For each dataset, we remove data points with missing values and apply normal points to our simulator to generate 5000 normal points and 500 anomalies. We randomly inflate 1/3 features for each anomaly. Real-valued features are inflated to yield either global or local anomalies, as described in Sec. 4.1. Categorical features are inflated by replacing value with that of lowest probability.

Baselines:  We evaluate our method along with two popular explanation methods: SHapley Additive exPlanations (SHAP) [23] and Depth-based Isolation Forest Feature Importance (DIFFI) [4]. SHAP is a model-free method for interpreting the output of any machine learning model. SHAP's feature importance can be computed using the predictions of both xStream as well as Isolation Forest (IF) [21] algorithm—one of the state-of-the-art anomaly detection methods for tabular data [7]. In contrast, DIFFI is a feature importance method that is specifically based on using IF as the backbone anomaly detection method.

In addition, we compare feature importances by xStream without as well as with feature projection to varying dimensions for K. We set the other hyperparameter values sufficiently large as suggested in [25] so as to obtain good detection performance.

Metrics:  The main metric for evaluation is ranking based, quantifying how well we rank the features by importance; namely Normalized Discounted Cumulative Gain (NDCG) [13]. NDCG sums the relevance-weighted scores of the items in the predicted ranking to an ideal ranking. We prefer NDCG as it i) gives more weight to the top anomalous features (we apply log 2 as the base of the discount factor), and ii) can use ground-truth feature importances as the relevance score. We also measure the time for xStream and other comparison methods to acquire feature importance explanations.

Table 2 displays the NDCG ranking quality w.r.t. to the synthesized ground-truth feature importances, as well as the approximate computational time required, comparing xStream (without projection) and various baseline methods. The highest NDCG scores are achieved with the combination xStream +SHAP (for detection+explanation, respectively) for almost all datasets. However, it is computationally quite demanding, taking more than 130 hours. The IF+SHAP combination provides a faster solution, delivering results in about 5 minutes, as it uses sped-up computations of SHAP [22] for tree-based methods like IF. xStream and DIFFI, two model-specific explanation methods, are even faster. xStream (w/out projection) is comparable to state-of-the-art explanation models or often the runner-up for many of the datasets. Importantly, xStream can be applied to distributed and/or streaming data, which makes it more appealing and practical for large data real-world systems, in comparison to DIFFI and IF+SHAP.

Table 3 shows the NDCG scores of xStream with different number of projections. The usage of projection diminishes xStream’s capability to detect and subsequently explain the anomalies. In other words, when projection is used there is a noticeable decrease in both AUROC and NDCG. The decline may be driven by two factors. First, when xStream is used with projection, its detection accuracy decreases which associates with lesser quality chains, making it difficult to obtain an explanation. Second, the graph propagation-based attribution is a heuristic and may not be accurate in fully capturing the direct feature effects. Nevertheless, the use of projection allows xStream to explain feature-evolving streaming data without requiring a complete retraining of the algorithm, making it more suitable for real-time applications.

Besides quantitative comparison, we also note disagreement among the explanations themselves, where different methods yield feature importances with significant variations [18]. This suggests that no explanation can be considered the definitive truth for end-users, and highlights the importance of the human in the loop: rather than blindly accepting the feature importances produced by any specific algorithm, analysts should be able to actively participate in the anomaly mining, reasoning, and management cycle.

Given a list of anomalies to be inspected, the summary view clusters them by similarity. Similarity is based on the feature importances as estimated by anomaly explanation (Sec. 3), rather than feature values in the original space. That is, we use $\text{sim}(\boldsymbol {\pi }_{o,i}, \boldsymbol {\pi }_{o,i^{\prime }})$ that helps group the points that stand out as anomalous in similar subspaces.

As Fig. 2 illustrates, anomalies are presented in a 2-dimensional MDS embedding space [19] for visualization, which preserves the aforementioned pairwise similarities as best as possible. The analyst can choose the number of clusters, where different symbols depict the cluster membership of the anomalies. The color of the points reflect their anomaly score from xStream. Clicking a point opens up a window that shows its feature importances in horizontal bars.

While the anomalies to be inspected could be the top k highest scoring points from xStream, ALARM also allows the analyst to import points of their own interest to inspect; e.g. (labeled) anomalies obtained via external reporting. In that case, the analyst data is passed onto xStream, which provide anomaly scores and explanations for the labeled points. Summary View displays only these labeled anomalies of interest, with explanations from xStream that are used toward clustering.

Fig. 3 illustrates the tools that ALARM offers for data exploration, consisting of four main components that can aid sense-making and verification: Histogram, Density plot and Parallel plot display the comparison of anomalies vs. inliers with a single, two and multiple dimensions respectively. For all three components, the analyst is able to select which features to display. Finally, Lookout [8] presents a few scatter plots which are automatically selected feature pairs that maximally-explain (“maxplain”) the anomalies in two dimensions, i.e. wherein the anomalies stand out the most. Analyst can choose the budget interactively, adjusting the number of plots that can be used to maxplain all the anomalies.

Given a group of anomalies, x-PACS algorithm [24] generates concise rules, with a small set of predicates, that characterize the anomalous pattern. It estimates the univariate kernel and histogram density of the anomalous points, respectively for each numerical and nominal feature, to identify the intervals or values of significant peaks. It then combines these from selected features, where the feature intervals that define the peaks are presented as the predicates.

Fig. 4 shows a screenshot of our Rule Candidates View, which displays up to three rules that satisfy two user-specified thresholds: coverage (C) and purity (P). C is the fraction of anomalies in the group that comply with the rule, and P is the fraction of inlier points that do not pass the rule. As such, the higher both C and P are, the better, as they associate with high recall but low false alarm rates.

Fig. 5 illustrates a screenshot of ALARM’s interface toward facilitating the design of a new rule with high coverage and purity. The analyst can select a candidate rule to revise or design a rule from scratch by adding or deleting features, and adjusting their values. Real-valued feature predicates are adjusted by sliders that allow specifying intervals, and categorical features can be assigned a value by scrolling through a drop-down list. “Calculate Scores” button displays the coverage and purity of the latest set of predicates. Upon completion, the rule can be saved locally or in a rule database used for supervised anomaly detection.

Participants:  We recruited three professional fraud analysts from Capital One bank to participate in our user study. The analysts had years of experience respectively, with card fraud, bank fraud, and AML (anti-money laundering) as part of their job.

Data:  We conducted the user studies on two separate datasets with ground-truth anomalies. First dataset Czech is based on our simulator. From the 1999 Czech Financial Dataset², We applied our simulator (see Sec.4.1) and simulated 1,000 normal points and 3 anomalous clusters with 20 anomalies each, based on different inflated feature subspaces. Data contains 2 numerical and 4 categorical features, as well as the inlier or anomaly cluster labels.

Second dataset Card contains a random sample of Capital One credit card transactions for a specific vendor over a period of time when they experienced a high (attempted) fraud rate. The credit card data has been anonymized with features renamed, numeric values renormalized, and categorical values hash-encoded. The dataset consists of 374 transactions, 82 of which are fraudulent or attempted fraudulent transactions. Each transaction has 3 numeric and 4 categorical features, and a label indicating whether the transaction is normal or a fraudulent one.

Procedure:  We quantitatively evaluate the effectiveness and time-efficiency that ALARM provides via case studies. We also conduct an interview study and report qualitative feedback from the analysts.

Case Studies: We conducted three case studies on Czech and two case studies on Card. Each study is associated with a specific task. To avoid leakage or prior familiarity between tasks, we used a separate one of three anomalous clusters in Czech for each task. Card came equipped with an existing domain-rule from earlier investigation, which helped prevent this issue. We describe the different tasks as follows.

• Task 1: On Czech, we first ask the analyst to write a rule 1.a) using self-tools³, and then 1.b) using ALARM. We measure and compare i) quality (coverage C and purity P) of rules as well as ii) time it takes to write them (denoted time-to-rule). On Card, we skip step 1.a) as the dataset readily came with an associated domain rule, developed by earlier investigators. This study is to quantify the added benefit of ALARM vs. ad-hoc tools that analysts otherwise use.
  
• Task 2: On Czech, we ask the analysts to explore the automatically generated rules and improve one candidate rule using the exploration (Expl) and the rule design interface (RDI). We measure how quickly and how much they can improve the rules, in terms of average C and P. On Card, the rule to be improved is the readily available domain-rule. This study is to quantify the added benefit of  Expl and RDI in improving a potentially suboptimal rule.
  
• Task 3: on Czech, we ask the analyst to write rules solely using Expl and RDI. We measure i) proximity of analyst-generated rules to the ground-truth, and ii) time it takes to write them (compared to avg. analyst time-to-rule via self-tools). We do not provide the analysts with any candidate rules. This study is to measure ALARM’s role in helping the analyst get to the ideal rule.
  

Metrics:  We measure the coverage (C) and purity (P) of the rules designed as well as the duration or time-to-rule in all case studies. Interview Study: We followed the studies with a list of interview questions to which the analysts responded with short answers. The interview probed for their feedback regarding the usability and functionality of ALARM.

Fig. 6 for Task 1 (ad-hoc tools vs. ALARM) demonstrates that the analysts using ALARM generally produced comparable rules to those using ad-hoc tools or to the pre-existing domain rule. On Czech, analysts have generated higher coverage rules using ad-hoc tools, only by having a disjunctive “OR” clause that treats the anomalies as two groups (while the latest version of ALARM now supports “OR” conditions). On Card, all analysts consistently produced rules with higher coverage than that of the domain-rule, sacrificing purity slightly. Since the analysts target financial fraud, they typically prioritize high coverage to avoid potentially large monetary losses from false negatives.

Furthermore, ALARM is more efficient. Different from ad-hoc tools, ALARM’s putting “the components in one place” provides an advantage. Average time-to-rule on Czech using ALARM (around 6 mins) is shorter than the time of ad-hoc tools (8 mins). On Card, all analysts were quick (also 6 mins) to produce rules comparable to the domain-rule without any training, whereas the domain rules are created with significantly longer time (“about 10-30 mins”) and require specialized domain knowledge.

Fig. 7 for Task 2 (initial-rule vs. improve-w/ALARM) shows that ALARM’s interactive exploration and rule design can assist in improving existing rules. On Czech, all analysts selected the same initial rule from among the candidates due to its simplicity (single predicate), and two of the three were able to improve to higher coverage without changing purity, within 2 mins on average. On Card, all three analysts have improved the purity of the domain-rule, with only one having to sacrifice coverage slightly, in about 6 mins on average. The improvements on Card are particularly notable, since the domain-rule for Card is already a carefully-crafted deployed rule.

On Task 3 (ground-truth vs. ALARM-w/out-Cand) we find that the ALARM-based rules explored by the analysts and the (hidden) ground-truth rule are consistent with each other, regarding their common usage of a predicate that aligns with the crucial ground-truth predicate (balance between 60000 and 75000).

As shown in Fig. 8, while one analyst built a very similar rule to the ground-truth w.r.t. coverage and purity, the others traded those in opposite directions⁴ by choosing different predicates besides balance. The study showcased the space of alternative rules that ALARM allowed the analysts to explore and choose from based on other hard-to-quantify metrics such as policy, ethics, and deployment cost.

We compile a list of learned lessons based on the analysts’ interviews. Overall, the analysts found ALARM to be “valuable for exploring the data and anomalies”, “useful for comparing and contrasting”, and helpful in identifying “specific pockets of risk”.

On efficiency:  Analysts agreed that ALARM is “a large time saver”, and enjoyed that it allowed them to “instantly generate rule candidates”, “quickly adjust thresholds and calculate how these adjustments affected the coverage and purity”, as well as “quickly getting a sense of where the anomalies are and how they're spread/clustered”.

On automation & interaction: While some analysts found Cand, i.e. “automatic rule mining component to be the most useful”, others perceived it as “unable to generate optimal rules” which made them “reluctant to trust” it “in favor of writing [their] own”. RDI was unanimously valued both in terms of the efficacy and efficiency it provided over manual practice: “attempt to iterate... was a massive value proposition as compared to doing this manually.”

On complexity:  All analysts consistently took most advantage of the simple histogram and density plots, and some also the “string” (i.e. parallel) plot to “quickly and easily identify where anomalies were located”. However, MDS based summary viz. (esp. the axes) and the LookOut were deemed “too complex to get into”; suggesting that “individuals less familiar with ML may need robust setup instructions in order to understand and use [those]”.

Other desired functionalities:  Several commented that the “ability of the user to choose their own clusters” (also split or merge existing clusters) would be “a powerful part of this tool”. While designing their rule, they liked to observe “highlights on visuals of the regions covered by rule”. One analyst suggested to add the flexibility to import additional data (e.g. last month's records from a specific vendor) while another suggested the ability to add new, analyst-crafted features. Broadly, all analysts were eager to “experiment with the tool using live data in [their] respective field” based on which they could “make more specific suggestions for improvement”.