Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Enhanced Federated Anomaly Detection Through Autoencoders Using Summary Statistics-Based Thresholding

Sofiane Laridi L3S Research Center, Faculty of Electrical Engineering and Computer Science, Leibniz University Hannover, 30167, Germany laridi@l3s.de Gregory Palmer L3S Research Center, Faculty of Electrical Engineering and Computer Science, Leibniz University Hannover, 30167, Germany Kam-Ming Mark Tam Department of Architecture, University of Hong Kong, Knowles Building, Pokfulam Road, Hong Kong SAR
Abstract

In Federated Learning (FL), anomaly detection (AD) is a challenging task due to the decentralized nature of data and the presence of non-IID data distributions. This study introduces a novel federated threshold calculation method that leverages summary statistics from both normal and anomalous data to improve the accuracy and robustness of anomaly detection using autoencoders (AE) in a federated setting. Our approach aggregates local summary statistics across clients to compute a global threshold that optimally separates anomalies from normal data while ensuring privacy preservation. We conducted extensive experiments using publicly available datasets, including Credit Card Fraud Detection, Shuttle, and Covertype, under various data distribution scenarios. The results demonstrate that our method consistently outperforms existing federated and local threshold calculation techniques, particularly in handling non-IID data distributions. This study also explores the impact of different data distribution scenarios and the number of clients on the performance of federated anomaly detection. Our findings highlight the potential of using summary statistics for threshold calculation in improving the scalability and accuracy of federated anomaly detection systems.

keywords:
Federated Learning, Anomaly Detection, Auto-Encoder

Introduction

In Federated Learning (FL), the effectiveness of Anomaly Detection (AD) is crucial due to the decentralized nature of data across multiple clients. Federated Autoencoders (FAE) have emerged as a popular approach for detecting anomalies in such settings, as they allow each client to train local models on their data and collaboratively learn a global model without sharing raw data. However, determining an optimal threshold for AD within this federated framework remains a significant challenge.

Traditional AD methods often rely on thresholds determined from local validation data by using the trained AE to predict the validation data and thus the reconstruction errors of this validation data, then calculating a threshold that separates anomalies from normal data. These methods fail to capture the global data distribution in federated environments, leading to sub-optimal performance (McMahan et al.[1]). They rely on local thresholds that may not adequately reflect the variations in data distributions across FL clients, especially when the data is non-IID (Kairouz et al.[2]). Additionally, many existing methods focus solely on normal data to establish the threshold, neglecting the characteristics of anomalies, compromising their effectiveness in certain scenarios.

To overcome these limitations, this study introduces an FL threshold calculation method that integrates summary statistics from both normal and anomalous data across local validation datasets of all clients, producing a federated threshold that learns the decision boundary more accurately and robustly. Research in this direction has been limited. This study demonstrates that incorporating both summary statistics and anomalies in the threshold determination process enhances the accuracy and robustness of AD in FL (Yang et al.[3]). The research compares the performance of the proposed FL threshold calculation technique against conventional local and federated methods across various FL and data distribution scenarios, examining different numbers of clients, degrees of data distribution skewness, and anomaly rates. The findings also provide insights into the method’s effectiveness across diverse FL environments and data distribution conditions.

In summary, this paper makes the following contributions:

  • Proposes a novel approach for calculating the FL threshold for AD using FL clients’ local summary statistics and aggregated global statistics, while preserving privacy.

  • Collects and compares several state-of-the-art threshold calculation techniques, thoroughly analyzing their performance under different FL and data distribution scenarios.

  • Offers insights into whether a client should adopt the proposed FL threshold or rely on a locally calculated threshold based on the comparison of local and global summary statistics.

Literature Review

FL is widely used in AD due to its effectiveness in handling distributed data while maintaining privacy. Various techniques have been integrated into FL frameworks to detect anomalies. On the other hand, AEs have been effective in detecting malicious network activities (Li et al.[4]). In IoT networks, FL helps collaboratively train models for AD, with additional security provided by blockchain integration (Ali et al.[5]). Multi-task learning within FL, discussed by Smith et al.[6], addresses task heterogeneity across devices by leveraging diverse data sources. The FedGroup model, for example, computes learning updates from a group of devices, demonstrating its effectiveness in anomaly detection in IoT environments (Li et al.[4]).

However, federated AD faces several challenges. A major challenge is handling non-IID data across clients, which can negatively impact the model’s performance (Bonawitz et al.[7]). Traditional methods often assume IID data, but newer approaches, such as graph-based methods by Liu et al.[8], consider data relationships for better detection. Zhao et al.[9] have focused on optimizing FL systems to deal with non-IID data and improve communication efficiency. Generative Adversarial Networks are used to identify anomalies through reconstruction errors or discriminator scores (Schlegl et al.[10]). Other methods, such as softmax scores from neural network classifiers by Hendrycks & Gimpel[11] or outlier exposure by Liang et al.[12], help improve AD. Combining generative and discriminative models by Nalisnick et al.[13] has also been shown to enhance accuracy. Other advanced neural architectures, like attention-based models by Vaswani et al.[14] and LSTM networks for temporal data by Zhu et al.[15], are used.

In addition, AEs are effective tools for AD, leveraging their ability to reconstruct input data and highlight deviations as anomalies (Xu et al.[16]). By learning compact representations, AEs excel in identifying unusual patterns across various domains, including network security and IoT systems (Li et al.[4]). More complex versions of AEs, such as Variational Autoencoders, enhance this process by modeling probabilistic distributions, improving the detection of subtle anomalies (An & Cho[17]). Different adaptations of AEs address specific data types: convolutional autoencoders for spatial anomalies (Baur et al.[18]) and hybrid models combining AEs with LSTM networks for temporal data (Malhotra et al.[19]). Attention mechanisms integrated into AEs by Zong et al.[20] further refine AD by focusing on the most relevant data features. Additionally, combining AEs with Generative Adversarial Networks (GANs) has shown robust performance in AD through enhanced reconstruction quality (Zenati et al.[21]). Despite all the mentioned AE variations, fully connected AEs are preferred for handling large-scale data due to their simplicity, interpretability, and efficiency (Hinton & Salakhutdinov[22]).

Given their reliance on reconstruction error, effective AD with AEs requires appropriate threshold calculation techniques, which are crucial aspects of AD. In FL, this task is particularly challenging due to the distributed nature of data across multiple clients. Therefore, different threshold calculation methods for AEs have been explored. The majority of methods we have found calculate the threshold locally by focusing on client-specific data without federated aggregation. For instance, the Local Kernel Quantile Estimator (KQE) (Huong et al.[23]) sets thresholds based on reconstruction error quantiles using a kernel estimator. The Local IQR Range method determines thresholds using the interquartile range of reconstruction errors. Similarly, the Local Percentile method (Percentile) uses a specific percentile of the error distribution to establish thresholds (Novoa-Paradela et al.[24]). Other local methods include the Local Largest MSE (Largest-MSE) (Sáez-de-Cámara et al.[25]), which bases thresholds on the highest observed MSE. The Local Peak Over Threshold (POT) method identifies anomalies by focusing on errors that exceed a high quantile (Kea et al.[26]). Schlegl et al.[27] calculate a threshold by generating multiple thresholds between the minimum and maximum reconstruction errors of the validation data (Local-MinMax), then selecting the threshold with the highest F1 score.

Federated threshold calculation techniques have been explored as well. One common approach is where Wang et al.[28] set a federated threshold by averaging the mean-squared error plus the standard deviation across clients (Fed-MSE-StD). While straightforward, this method can be less effective since it does not consider anomalies directly in the threshold calculation process. Sánchez et al.[29] calculate the mean and standard deviation of the local thresholds and filter out thresholds with a z-score greater than 1.5 (Fed-Filtered). The global threshold is then set as the maximum of the remaining filtered thresholds. One of the limitations of these federated techniques is the neglect of actual anomalies in the federated threshold calculation. An alternative federated approach by Pourahmadi et al.[30] involves generating candidate thresholds between the global minimum and maximum reconstruction errors (Fed-MinMax), with clients selecting the optimal threshold based on F1 scores. However, this interval between the global minimum and maximum can be vague, especially in non-IID data distributions, and manually setting the number of thresholds might not capture the optimal threshold. Additionally, this method does not suggest a fair aggregation of the threshold candidates’ F1 scores, which gives equal influence to clients with large validation data and clients with minimal validation data. This can lead to a sub-optimal global threshold.

The limitations of these methods underscore the need for a more automated and equitable threshold calculation method that can account for the diverse data distributions across clients and provide a balanced approach in FL. We assume that summary statistics, such as mean, variance, skewness, and kurtosis, can offer valuable insights into the distribution of the clients’ validation data, helping to develop thresholds that are more representative of the overall data landscape across FL clients. One of the significant advantages of using summary statistics in FL is their privacy-preserving nature. Unlike raw data, which contains detailed information about individual data points, summary statistics aggregate this information into a form that reveals general trends without exposing specific data values (McMahan et al.[1]).

We have listed in Table 1 the different state-of-the-art methods mentioned, along with their characteristics and limitations, for better clarification.

Thresholding Method Federated Anomalies Statistics Used Local Data Distribution Consideration
Our Method Mean, Variance, etc.
Fed-MinMax Min/Max
Fed-MSE-StD Mean, StD
Fed-Filtered Mean, StD
Local-MinMax MSE, Percentile
KOE
IQR IQR
Percentile Percentile
Largest-MSE MSE
POT High Quantile
Local-MSE-Std Mean, StD
Table 1: SOTA Threshold Calculation Approaches

Problem Definition

Federated AD using FAE aims to collaboratively identify anomalies in distributed data without sharing raw data between clients. The key challenge is to determine an optimal global threshold θglobalsubscript𝜃global\theta_{\text{global}}italic_θ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT for AD, leveraging each client’s local validation data while preserving privacy.

Notations

  • Dtrain,isubscript𝐷train𝑖D_{\text{train},i}italic_D start_POSTSUBSCRIPT train , italic_i end_POSTSUBSCRIPT: Training dataset for client i𝑖iitalic_i

  • Mtsubscript𝑀𝑡M_{t}italic_M start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT: Global FAE model at round t𝑡titalic_t

  • Mt,isubscript𝑀𝑡𝑖M_{t,i}italic_M start_POSTSUBSCRIPT italic_t , italic_i end_POSTSUBSCRIPT: Local model for client i𝑖iitalic_i at round t𝑡titalic_t

  • Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT: Validation dataset for client i𝑖iitalic_i containing both normal and anomalous samples

  • D^val,isubscript^𝐷val𝑖\hat{D}_{\text{val},i}over^ start_ARG italic_D end_ARG start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT: Reconstructed validation dataset for client i𝑖iitalic_i

  • Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Array of reconstruction errors for client i𝑖iitalic_i, where Ei={ei,1,ei,2,,ei,n}subscript𝐸𝑖subscript𝑒𝑖1subscript𝑒𝑖2subscript𝑒𝑖𝑛E_{i}=\{e_{i,1},e_{i,2},\ldots,e_{i,n}\}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = { italic_e start_POSTSUBSCRIPT italic_i , 1 end_POSTSUBSCRIPT , italic_e start_POSTSUBSCRIPT italic_i , 2 end_POSTSUBSCRIPT , … , italic_e start_POSTSUBSCRIPT italic_i , italic_n end_POSTSUBSCRIPT }

  • μisubscript𝜇𝑖\mu_{i}italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Mean of the reconstruction errors for client i𝑖iitalic_i

  • σi2superscriptsubscript𝜎𝑖2\sigma_{i}^{2}italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT: Variance of the reconstruction errors for client i𝑖iitalic_i

  • Sisubscript𝑆𝑖S_{i}italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Skewness of the reconstruction errors for client i𝑖iitalic_i

  • Kisubscript𝐾𝑖K_{i}italic_K start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Kurtosis of the reconstruction errors for client i𝑖iitalic_i

  • Nisubscript𝑁𝑖N_{i}italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Number of samples in the validation dataset for client i𝑖iitalic_i

  • μglobalsubscript𝜇global\mu_{\text{global}}italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global mean of the reconstruction errors

  • σglobal2subscriptsuperscript𝜎2global\sigma^{2}_{\text{global}}italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global variance of the reconstruction errors

  • Sglobalsubscript𝑆globalS_{\text{global}}italic_S start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global skewness of the reconstruction errors

  • Kglobalsubscript𝐾globalK_{\text{global}}italic_K start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global kurtosis of the reconstruction errors

  • T𝑇Titalic_T: Array of n𝑛nitalic_n thresholds, T={μ1,μ2,,μn}𝑇subscript𝜇1subscript𝜇2subscript𝜇𝑛T=\{\mu_{1},\mu_{2},\ldots,\mu_{n}\}italic_T = { italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_μ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } determined based on the overlap region

  • Fisubscript𝐹𝑖F_{i}italic_F start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: Array of F1 scores for client i𝑖iitalic_i corresponding to thresholds T𝑇Titalic_T

  • Favgsubscript𝐹avgF_{\text{avg}}italic_F start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT: Array of average F1 scores for each threshold across all clients

  • θglobalsubscript𝜃global\theta_{\text{global}}italic_θ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global threshold with the highest average F1 score

Given

  1. 1.

    A FAE model M𝑀Mitalic_M trained on distributed training data Dtrainsubscript𝐷trainD_{\text{train}}italic_D start_POSTSUBSCRIPT train end_POSTSUBSCRIPT.

  2. 2.

    Each client’s local validation data Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT with corresponding reconstruction errors Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT.

Objective

To compute a global threshold θglobalsubscript𝜃global\theta_{\text{global}}italic_θ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT that optimally separates anomalies from normal data across all clients while addressing the following challenges:

  • Data Privacy: Each client’s validation data Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT is private and cannot be shared with other clients or the server.

  • Data Distribution: Each client may have different data distributions, leading to variations in reconstruction errors Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT.

Method

Our methodology for federated anomaly detection using autoencoders is divided into two main steps: Federated Autoencoder Training and Federated Threshold Calculation. The detailed steps of our methodology are outlined below:

Federated Auto-Encoder Training

The first step involves training a FAE model using the Federated Averaging (FedAvg) algorithm to aggregate the clients’ model weights, as illustrated in Algorithm 1. Model synchronization between clients and the server occurs at the beginning of each round, where the server broadcasts the global model to all clients, and at the end of each round, clients send their updated weights back to the server for aggregation, following the FedAvg paradigm. We selected a fully connected AE due to its proven effectiveness in AD, particularly for high-dimensional datasets such as Shuttle, Covertype, and Credit Card Fraud Detection (Sakurada & Yairi[31]).

Algorithm 1 Federated Auto-Encoder Training with FedAvg Aggregation
1:Input: Dtrain,isubscript𝐷train𝑖D_{\text{train},i}italic_D start_POSTSUBSCRIPT train , italic_i end_POSTSUBSCRIPT: Training dataset for client i𝑖iitalic_i, N𝑁Nitalic_N: Number of clients, E𝐸Eitalic_E: Local epochs, η𝜂\etaitalic_η: Learning rate, R𝑅Ritalic_R: Rounds
2:Output: M𝑀Mitalic_M: Trained global autoencoder model
3:Initialize global model M0subscript𝑀0M_{0}italic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT
4:for each round t=1,,R𝑡1𝑅t=1,\ldots,Ritalic_t = 1 , … , italic_R do
5:     Server sends global model Mt1subscript𝑀𝑡1M_{t-1}italic_M start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT to all clients
6:     for each client i=1,,N𝑖1𝑁i=1,\ldots,Nitalic_i = 1 , … , italic_N in parallel do
7:         Initialize local model Mt,isubscript𝑀𝑡𝑖M_{t,i}italic_M start_POSTSUBSCRIPT italic_t , italic_i end_POSTSUBSCRIPT with Mt1subscript𝑀𝑡1M_{t-1}italic_M start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT weights
8:         Train Mt,isubscript𝑀𝑡𝑖M_{t,i}italic_M start_POSTSUBSCRIPT italic_t , italic_i end_POSTSUBSCRIPT on Dtrain,isubscript𝐷train𝑖D_{\text{train},i}italic_D start_POSTSUBSCRIPT train , italic_i end_POSTSUBSCRIPT for E𝐸Eitalic_E epochs using learning rate η𝜂\etaitalic_η
9:         Send local updates ΔMt,iΔsubscript𝑀𝑡𝑖\Delta M_{t,i}roman_Δ italic_M start_POSTSUBSCRIPT italic_t , italic_i end_POSTSUBSCRIPT to server
10:     end for
11:     Server aggregates updates to form global model Mtsubscript𝑀𝑡M_{t}italic_M start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT
12:     Update global model: MtMt1+Mtsubscript𝑀𝑡subscript𝑀𝑡1subscript𝑀𝑡M_{t}\leftarrow M_{t-1}+M_{t}italic_M start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ← italic_M start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT + italic_M start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT
13:end for
14:Return: Trained global model MTsubscript𝑀𝑇M_{T}italic_M start_POSTSUBSCRIPT italic_T end_POSTSUBSCRIPT
Refer to caption
Figure 1: Federated Threshold Calculation using Summary Statistics

Summary Statistics-Based Threshold Selection

Figure 1 illustrates the workflow of our federated AD approach using FAE, emphasizing how thresholds are calculated using aggregated summary statistics. The process is divided into several key steps:

Prediction and Extraction of Summary Statistics

Each client employs its local Autoencoder (AE) model M𝑀Mitalic_M to compute reconstruction errors Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT on its local validation dataset Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT. Based on these errors, the client calculates summary statistics SSi𝑆subscript𝑆𝑖SS_{i}italic_S italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, including the mean (μisubscript𝜇𝑖\mu_{i}italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT), variance (σi2superscriptsubscript𝜎𝑖2\sigma_{i}^{2}italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT), skewness (Sisubscript𝑆𝑖S_{i}italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT), and kurtosis (Kisubscript𝐾𝑖K_{i}italic_K start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT).

Weighted Aggregation of Summary Statistics and Threshold Selection

After aggregating the summary statistics from all clients, the server identifies the overlap region where the distributions of normal and anomalous reconstruction errors intersect. This overlap region aids in isolating reconstruction error outliers by focusing on thresholds generated within this specific range.

The initial overlap region is determined using the global means (μnormalsubscript𝜇normal\mu_{\text{normal}}italic_μ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT, μanomalysubscript𝜇anomaly\mu_{\text{anomaly}}italic_μ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT) and standard deviations (σnormalsubscript𝜎normal\sigma_{\text{normal}}italic_σ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT, σanomalysubscript𝜎anomaly\sigma_{\text{anomaly}}italic_σ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT) of the normal and anomalous distributions, calculated via the upper and lower bounds. These bounds are subsequently fine-tuned by adjusting for skewness and kurtosis to ensure that the region accurately reflects the data’s distributional shape. Skewness compensates for asymmetry in the data, while kurtosis adjusts the bounds to account for tail behavior.

Threshold Candidates Generation

Within the refined overlap region, the server generates a set of candidate thresholds. These candidates are typically spaced evenly across the overlap range, with the number of candidates determined by a predefined parameter (e.g., 1000 candidates). The generated thresholds are then distributed to all clients for evaluation.

F1 Score Calculation and Aggregation

Each client evaluates these candidate thresholds using its local validation data, calculating the F1 score for each threshold. The server aggregates these F1 scores across all clients to select the global threshold θglobalsubscript𝜃global\theta_{\text{global}}italic_θ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT, which maximizes the overall F1 score.

The detailed steps of this method are illustrated in Algorithm 2.

Algorithm 2 Federated Threshold with Weighted Aggregation
1:Input: Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT: Validation dataset for client i𝑖iitalic_i
2:Output: μglobalsubscript𝜇global\mu_{\text{global}}italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT: Global threshold
3:for each client i𝑖iitalic_i do
4:     Use the trained model M𝑀Mitalic_M to reconstruct Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT.
5:     Compute reconstruction errors Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT for each sample in Dval,isubscript𝐷val𝑖D_{\text{val},i}italic_D start_POSTSUBSCRIPT val , italic_i end_POSTSUBSCRIPT.
6:     Calculate summary statistics for Eisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT: mean μisubscript𝜇𝑖\mu_{i}italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, variance σi2superscriptsubscript𝜎𝑖2\sigma_{i}^{2}italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT, skewness Sisubscript𝑆𝑖S_{i}italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, kurtosis Kisubscript𝐾𝑖K_{i}italic_K start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and count Nisubscript𝑁𝑖N_{i}italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT.
7:end for
8:for each client i𝑖iitalic_i do
9:     Send summary statistics μisubscript𝜇𝑖\mu_{i}italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, σi2superscriptsubscript𝜎𝑖2\sigma_{i}^{2}italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT, Sisubscript𝑆𝑖S_{i}italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, Kisubscript𝐾𝑖K_{i}italic_K start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and Nisubscript𝑁𝑖N_{i}italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT to the server.
10:end for
11:Server computes the global summary statistics of both normal and anomaly using weighted aggregation:
12:      μglobal=i=1kNiμii=1kNisubscript𝜇globalsuperscriptsubscript𝑖1𝑘subscript𝑁𝑖subscript𝜇𝑖superscriptsubscript𝑖1𝑘subscript𝑁𝑖\mu_{\text{global}}=\frac{\sum_{i=1}^{k}N_{i}\mu_{i}}{\sum_{i=1}^{k}N_{i}}italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT = divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG
13:      σglobal2=i=1kNi(σi2+(μiμglobal)2)i=1kNisubscriptsuperscript𝜎2globalsuperscriptsubscript𝑖1𝑘subscript𝑁𝑖superscriptsubscript𝜎𝑖2superscriptsubscript𝜇𝑖subscript𝜇global2superscriptsubscript𝑖1𝑘subscript𝑁𝑖\sigma^{2}_{\text{global}}=\frac{\sum_{i=1}^{k}N_{i}\left(\sigma_{i}^{2}+(\mu_% {i}-\mu_{\text{global}})^{2}\right)}{\sum_{i=1}^{k}N_{i}}italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT global end_POSTSUBSCRIPT = divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG
14:      Sglobal=i=1kNiSiNi(σglobalσi)3i=1kNisubscript𝑆globalsuperscriptsubscript𝑖1𝑘subscript𝑁𝑖subscript𝑆𝑖subscript𝑁𝑖superscriptsubscript𝜎globalsubscript𝜎𝑖3superscriptsubscript𝑖1𝑘subscript𝑁𝑖S_{\text{global}}=\frac{\sum_{i=1}^{k}N_{i}S_{i}\cdot\sqrt{N_{i}}\cdot\left(% \frac{\sigma_{\text{global}}}{\sigma_{i}}\right)^{3}}{\sum_{i=1}^{k}N_{i}}italic_S start_POSTSUBSCRIPT global end_POSTSUBSCRIPT = divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⋅ square-root start_ARG italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ⋅ ( divide start_ARG italic_σ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG
15:      Kglobal=i=1kNiKiNi(σglobalσi)4i=1kNisubscript𝐾globalsuperscriptsubscript𝑖1𝑘subscript𝑁𝑖subscript𝐾𝑖subscript𝑁𝑖superscriptsubscript𝜎globalsubscript𝜎𝑖4superscriptsubscript𝑖1𝑘subscript𝑁𝑖K_{\text{global}}=\frac{\sum_{i=1}^{k}N_{i}K_{i}\cdot N_{i}\cdot\left(\frac{% \sigma_{\text{global}}}{\sigma_{i}}\right)^{4}}{\sum_{i=1}^{k}N_{i}}italic_K start_POSTSUBSCRIPT global end_POSTSUBSCRIPT = divide start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_K start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⋅ italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⋅ ( divide start_ARG italic_σ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG
16:Server determines the overlap region:
17:      Lower Bound=max(μnormal3σnormal,μanomaly3σanomaly)Lower Boundsubscript𝜇normal3subscript𝜎normalsubscript𝜇anomaly3subscript𝜎anomaly\text{Lower Bound}=\max(\mu_{\text{normal}}-3\sigma_{\text{normal}},\mu_{\text% {anomaly}}-3\sigma_{\text{anomaly}})Lower Bound = roman_max ( italic_μ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT - 3 italic_σ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT , italic_μ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT - 3 italic_σ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT )
18:      Upper Bound=min(μnormal+3σnormal,μanomaly+3σanomaly)Upper Boundsubscript𝜇normal3subscript𝜎normalsubscript𝜇anomaly3subscript𝜎anomaly\text{Upper Bound}=\min(\mu_{\text{normal}}+3\sigma_{\text{normal}},\mu_{\text% {anomaly}}+3\sigma_{\text{anomaly}})Upper Bound = roman_min ( italic_μ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT + 3 italic_σ start_POSTSUBSCRIPT normal end_POSTSUBSCRIPT , italic_μ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT + 3 italic_σ start_POSTSUBSCRIPT anomaly end_POSTSUBSCRIPT )
19:Server generates an array of n𝑛nitalic_n thresholds T={μ1,μ2,,μn}𝑇subscript𝜇1subscript𝜇2subscript𝜇𝑛T=\{\mu_{1},\mu_{2},\ldots,\mu_{n}\}italic_T = { italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_μ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } within the overlap region.
20:Server sends the threshold array T𝑇Titalic_T to each client.
21:for each client i𝑖iitalic_i do
22:     for each threshold μjsubscript𝜇𝑗\mu_{j}italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT in T𝑇Titalic_T do
23:         Calculate F1 scores Fisubscript𝐹𝑖F_{i}italic_F start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT for each threshold μjsubscript𝜇𝑗\mu_{j}italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT.
24:     end for
25:end for
26:for each client i𝑖iitalic_i do
27:     Send F1 score array Fisubscript𝐹𝑖F_{i}italic_F start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT to the server.
28:end for
29:Server calculates average F1 scores Favgsubscript𝐹avgF_{\text{avg}}italic_F start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT for each threshold.
30:Server identifies the threshold μglobalsubscript𝜇global\mu_{\text{global}}italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT with the highest average F1 score in Favgsubscript𝐹avgF_{\text{avg}}italic_F start_POSTSUBSCRIPT avg end_POSTSUBSCRIPT.
31:Server sends the global threshold μglobalsubscript𝜇global\mu_{\text{global}}italic_μ start_POSTSUBSCRIPT global end_POSTSUBSCRIPT to all clients.

Results

In this study, we evaluate our federated thresholding method using three publicly available datasets: Credit Card Fraud Detection (284,807 samples, 492 anomalies, 29 dimensions), Shuttle (49,097 samples, 3,511 anomalies, 9 dimensions), and Covertype (581,012 samples, 2,747 anomalies, 10 dimensions). The datasets were sourced from UCL and Kaggle. The following steps outline our data preparation and distribution for simulating a federated learning environment.

  • Normalization and Scaling: All datasets were normalized and scaled to ensure consistency across features. This preprocessing step is crucial for the performance of the AE, enabling it to effectively reconstruct normal data for AD across different clients.

  • Train-Validation-Test Splitting: Each dataset was split into training, validation, and test sets. The training data consisted solely of normal samples, as the AE is trained to model normal data behavior. Both validation and test sets included a mixture of normal and anomalous data, which were used to assess the thresholding methods. For scalability experiments, we varied the number of clients from 2 to 50, distributing the data evenly across clients to study the effect of client count on model performance (Bonawitz et al.[7]).

  • Federated Learning Splitting on Clients: To explore various federated learning scenarios:

    • Evenly Distributed Data: The training, validation, and test data were uniformly distributed across all clients. This uniform distribution served as a baseline to assess our method under ideal and balanced conditions, which is a standard assumption in many federated learning experiments (Li et al.[4]).

    • Non-IID Data: For the Shuttle and Covertype datasets, which include multiple classes, we designated one class as anomalous while treating the remaining classes as normal data. The normal data was distributed across clients, with each client receiving data from different classes. The anomalous data was divided among clients using the k-means clustering algorithm to ensure a diverse distribution of anomalies. Since these datasets include seven classes, we assigned six clients for normal data and distributed the anomalous data among them. To maintain consistency, the Credit Card Fraud Detection dataset, which is binary, was also split among six clients by applying the k-means clustering algorithm (Ahmed et al.[32]) to both normal and anomalous data, thus simulating a highly non-IID scenario similar to that of the other datasets (Zhao et al.[9]).

Method Shuttle Credit Card Cover
Our Method 0.9873 0.8985 0.8440
Fed Threshold 0.9861 0.8972 0.8395
Fed Mean MSE + StD 0.9844 0.8959 0.4003
Fed Filtered Threshold 0.9865 0.8881 0.3637
Local Iterative 0.9836 0.8878 0.8228
Local Inter Quantile Range 0.9773 0.8862 0.7318
Local Percentile 0.9657 0.8865 0.5273
Local Kernel Quantile Estimator 0.9775 0.8768 0.4512
Local Max MSE 0.9849 0.8594 0.3584
Local Mean MSE + Std 0.9619 0.8817 0.4828
Local Peak Over Threshold 0.9802 0.8125 0.4551
Table 2: Average F1 Scores Across Different Methods Using Evenly Distributed Data
Method Shuttle Credit Card Cover
Our Method 0.9251 0.8725 0.8351
Fed Threshold 0.9033 0.8705 0.8383
Fed Mean MSE + StD 0.3841 0.8556 0.7121
Fed Filtered Threshold 0.4231 0.8460 0.7542
Local Iterative 0.9120 0.8712 0.8558
Local Inter Quantile Range 0.4628 0.8416 0.7605
Local Percentile 0.4742 0.8419 0.7890
Local Kernel Quantile Estimator 0.4781 0.8347 0.8027
Local Max MSE 0.3546 0.8143 0.7803
Local Mean MSE + Std 0.4614 0.8363 0.8088
Local Peak Over Threshold 0.3615 0.7938 0.7822
Table 3: Average F1 Scores Across Different Methods Using Non-IID Data
Refer to caption
Figure 2: Comparison of F1 Scores Across Different Threshold Calculation Methods Using Randomly Distributed Data

Data Distribution

Even Distribution

The results demonstrate that our method for generating thresholds is effective across various data distribution scenarios. In the evenly distributed data setting (Table 3), our method consistently achieves the highest F1 scores across all datasets, indicating superior performance in AD compared to other methods. Additionally, our method remains highly reliable, maintaining strong performance even as the number of clients increases. The Fed Mean MSE + StD method also performs well but is typically outperformed by our approach. In contrast, local methods such as Local Inter Quantile Range and Local Percentile exhibit greater variability in their F1 scores, rendering them less consistent and reliable in distributed data scenarios. Overall, these results confirm that federated approaches, particularly our proposed method, are highly effective in managing evenly distributed data across multiple clients.

Non-IID Distribution

In the Non-IID data scenario (Table 3), where clients have heterogeneous data distributions, our method consistently outperforms other approaches, achieving the highest F1 scores across the Shuttle (0.9251), Credit Card (0.8725), and Covertype (0.8351) datasets. Notably, the Local-MinMax method remains highly competitive, particularly in the Credit Card dataset where it attains an F1 score of 0.8712, nearly matching the performance of our method and surpassing other federated approaches. Similarly, in the Covertype dataset, the Local-MinMax method achieves an F1 score of 0.8558, outperforming all other methods, including federated ones.

These findings suggest that while the federated approach effectively aggregates information across clients to establish a robust global threshold, certain local methods, particularly Local-MinMax Thresholding, can more effectively address client-specific data variations. This is especially evident in datasets like Credit Card and Covertype, where data distributions vary significantly across clients.

Given these insights, it is evident that there are scenarios where a client may benefit more from utilizing a local threshold rather than a federated one. This observation underscores the importance of investigating and understanding the conditions under which a client should prefer the federated threshold over its own local threshold. Future analyses will focus on the summary and aggregated statistics utilized in our federated threshold calculation to determine the optimal thresholding strategy for different client scenarios.

Random Distribution

To evaluate the robustness of our method under varying data distribution conditions, we employed diverse random distributions with varying numbers of samples per client. This approach allowed us to simulate a wide range of scenarios that may occur in federated learning environments. Figure 2 presents the boxplots generated from these experiments, illustrating the F1 scores obtained across different random setups.

The width and spread of the boxplots provide insights into the consistency and robustness of each threshold calculation method. Our proposed method is represented by consistently narrower boxplots, indicating lower variance in F1 scores across different random distribution scenarios. This narrow spread suggests that our method is both robust and reliable, maintaining high performance regardless of variability in data distribution among clients.

In contrast, the wider boxplots associated with some of the other methods indicate greater variability in performance. This variability suggests that these methods are more sensitive to changes in data distribution, resulting in less consistent outcomes. Additionally, the presence of outliers in these boxplots further highlights the instability of these methods under certain random distribution setups.

Overall, the relatively narrow boxplots of our method demonstrate its superior robustness, as it maintains high and stable F1 scores across a diverse set of random distribution scenarios. This underscores the adaptability and effectiveness of our federated thresholding approach, even under challenging and unpredictable data conditions.

Global Anomaly and Normal Reconstruction Data Overlap Analysis

Refer to caption
Figure 3: Visualization of Error Distribution with Overlap Regions Defined by Upper and Lower Bounds Across Different Datasets and Data Distributions

In addition to performance evaluation, the global reconstruction errors, as depicted in Figure 3, provide valuable insights into the functioning of our method under different data distribution scenarios. The visualizations reveal that in evenly distributed data scenarios (top row), the overlap region between normal and anomalous reconstruction errors is smaller compared to Non-IID scenarios (bottom row). This smaller overlap suggests that in evenly distributed data, fewer threshold candidates are needed to accurately distinguish between normal and anomalous data points.

Conversely, the larger overlap observed in Non-IID scenarios indicates that a higher number of threshold candidates may be necessary to achieve similar accuracy. This difference highlights the necessity of adapting the threshold generation process based on the data distribution. Specifically, more candidates are likely required in Non-IID settings to manage the increased overlap between normal and anomalous errors effectively.

These findings emphasize the importance of considering data distribution characteristics when designing thresholding methods in federated learning. By accounting for the extent of overlap between normal and anomalous reconstruction errors, our method can dynamically adjust the threshold selection process to maintain high detection accuracy across diverse and complex data environments.

Scalability

Number of Clients

Refer to caption
Figure 4: Impact of Number of Clients on F1 Scores Across Different Threshold Calculation Methods and Datasets

The results from the Shuttle, Covertype, and Credit Card Fraud Detection datasets, as shown in Figure 4, demonstrate the effectiveness of various thresholding techniques. Our Fed Threshold method consistently achieves the highest F1 scores across all datasets and varying numbers of clients, underscoring its robustness and reliability. Specifically, for the Shuttle dataset, the Fed Threshold method maintains a high F1 score close to 0.99 regardless of the number of clients. The Fed Mean MSE + StD method also performs comparably well, suggesting that these federated approaches effectively aggregate client data to sustain performance.

In contrast, other methods such as Local Inter Quantile Range, Local Percentile, Local Kernel Quantile Estimator, and Local Peak Over Threshold generally exhibit lower F1 scores. These methods show some improvement in the Credit Card Fraud Detection and Shuttle datasets as the number of clients increases but fail to match the consistency and high performance of the federated approaches. Conversely, in the Shuttle dataset, local methods—especially Local-MSE + StD—show a slight decrease in F1 scores as the number of clients increases. This decline occurs because, as the data is partitioned among more clients and given the extreme imbalance of the Shuttle dataset, each client receives fewer and less representative data points to calculate reliable thresholds.

For the Covertype dataset, our Fed Threshold and Fed Mean MSE + StD methods maintain high F1 scores near 0.9, irrespective of the number of clients. Other methods, including Local Kernel Quantile Estimator and Local Peak Over Threshold, exhibit significant fluctuations and lower F1 scores. The performance of these alternative methods does not consistently improve with an increasing number of clients, highlighting the stability and effectiveness of the federated approaches.

The Credit Card Fraud Detection dataset results further reinforce these observations. The Fed Threshold and Fed Mean MSE + StD methods consistently demonstrate high F1 scores around 0.9, with minimal performance degradation as the number of clients increases. In contrast, methods such as Local Max MSE and Local Peak Over Threshold display considerable variability and generally lower F1 scores, emphasizing their instability and less effective performance in a federated setting.

Overall, the Fed Threshold method consistently outperforms other thresholding techniques across all datasets and client numbers. Its ability to maintain high performance and scalability makes it highly suitable for real-world scenarios where anomaly detection must be aggregated from multiple clients. The Fed Mean MSE + StD method also exhibits strong performance, albeit slightly less consistent than our Fed Threshold method.

When evaluating the performance of our Fed Threshold approach with varying numbers of clients, it remains highly effective, exhibiting minimal degradation in F1 scores as the number of clients increases. This demonstrates the robustness of the Fed Threshold method in maintaining accuracy, even as the client count grows, making it well-suited for real-world federated learning scenarios involving numerous clients.

Execution Time

Refer to caption
Figure 5: Execution time of different threshold calculation methods

The execution time results across the Credit Card Fraud Detection, Covertype, and Shuttle datasets, as shown in Figure 5, reveal important trade-offs between accuracy and computational cost. The Fed Threshold method (solid blue line) exhibits the highest execution time, increasing significantly as the number of clients grows, reaching approximately 3 seconds for 50 clients. This increase reflects the computational complexity associated with aggregating and processing data across clients to calculate a global threshold. Although more resource-intensive, this method provides superior accuracy.

The Fed Mean MSE + StD method (solid red line) also shows increased execution time as the number of clients rises, though it remains slightly lower than the Fed Threshold method. This approach balances relatively high accuracy with more moderate computational demands. Similarly, the Fed Filtered Threshold method (solid green line) follows a comparable trend, indicating similar efficiency but with slightly lower accuracy.

In contrast, local methods such as Local Iterative, Local IQR, and Local Percentile (various dashed lines) maintain significantly lower and more stable execution times as the number of clients increases. These methods are computationally efficient because they operate on local data without the need for extensive aggregation, but they do so at the expense of lower accuracy.

Overall, federated methods such as Fed Threshold and Fed Mean MSE + StD are more computationally demanding but offer higher accuracy, making them suitable for scenarios where precision is critical. Conversely, local methods provide faster execution times, which may be preferable in resource-constrained environments or situations requiring quick decisions.

While the federated methods demonstrate robustness in maintaining performance as the number of clients increases, we also observed a significant increase in execution time, particularly in the Credit Card Fraud Detection dataset, where execution time grew by 460% from 0.5 seconds to 2.8 seconds as the number of clients increased from 10 to 50. This highlights a trade-off between computational cost and accuracy, indicating that while these methods are scalable in terms of performance, their scalability in terms of computational efficiency may present challenges in resource-constrained environments.

Robustness against Noise

Refer to caption
Figure 6: Impact of Corrupt Data on F1 Scores Across Different Threshold Calculation Methods with Up to 30 Corrupt Clients

The robustness of the thresholding methods against noise is illustrated in Figure 6, where the x-axis represents the increasing number of clients with corrupted data, up to 30 clients.

For the Covertype dataset, Our Method consistently maintains a high F1 score, experiencing only a gradual decline as the number of corrupted clients increases. This demonstrates strong resilience to noise. In contrast, the Fed-MinMax and Fed-Filtered methods exhibit significantly lower F1 scores from the outset, indicating their reduced robustness to noisy data.

In the Shuttle dataset, Our Method again demonstrates superior robustness, maintaining a relatively high F1 score that decreases more slowly compared to other methods. Notably, the Fed-MSE-Std method shows a sharp decline in performance as the number of corrupted clients increases, highlighting its vulnerability to noise.

For the Credit Card Fraud Detection dataset, a similar trend is observed. Our Method maintains the highest F1 scores across all levels of noise, while the performance of the Fed-Filtered and Fed-MSE-Std methods decreases more rapidly. This further underscores the superior robustness of Our Method.

Overall, these results emphasize the robustness of Our Method against noise, making it a reliable choice in scenarios where data corruption is a significant concern. Other methods, while effective in certain contexts, tend to exhibit more substantial performance degradation as the number of corrupted clients increases.

Follow-Up Analysis: Predicting the Benefit of Federated vs. Local Threshold

Based on observations from our previous experiments, we identified that certain clients may benefit more from using their locally calculated thresholds rather than a federated threshold. This insight prompted us to investigate whether valuable information could be extracted from the summary statistics—both local and federated—collected during those experiments. The objective of this study is to explore whether these summary statistics can be leveraged to predict when a client would benefit more from a federated threshold versus a local one.

We conducted experiments across various data distributions, varying numbers of clients, and differing levels of non-IIDness to simulate a wide range of real-world scenarios. This comprehensive approach provided a robust dataset for analysis. In each use case, we employed the Federated Autoencoder model and collected validation data to extract summary statistics. These statistics, along with the corresponding F1 scores for both local and federated thresholds, formed the foundation of our analysis.

The dataset’s features include locally calculated summary statistics from both normal and anomalous validation data, as well as aggregated summary statistics. Specifically, the local statistics comprise measures such as mean, variance, skewness, kurtosis, and count for both normal and anomalous data. The aggregated statistics are derived by combining these local statistics across all clients, producing features such as aggregated mean, variance, skewness, kurtosis, and proportional counts for both normal and anomalous data. In total, the dataset comprises 20 features.

For labeling, we focused on the difference in F1 scores between the best-performing local threshold and the federated threshold. Based on our earlier results, the Local Iterative Threshold was identified as the most effective local method, while our Federated Threshold emerged as the best federated method. Consequently, the label for each client’s data point represents the difference in F1 scores achieved by the federated threshold and the Local Iterative Threshold on the client’s local test data.

To evaluate the benefits of federated versus local thresholds, we employed two modeling approaches: binary classification and regression. The binary classification model aimed to predict whether a client would benefit from using the federated threshold, while the regression model predicted the actual difference in F1 scores between the federated and local thresholds.

For the binary classification task, we employed a Support Vector Machine (SVM) model [33] and achieved an accuracy of 89% when the data was randomly split into training, validation, and test sets across all three datasets. However, when we tested a more realistic scenario—training and validating on data from the Shuttle and Covertype datasets while testing on the Credit Card Fraud Detection dataset—the SVM’s performance dropped to an accuracy of 68%. This suggests that while the model performs well under random splits, its ability to generalize across different datasets is limited.

For the regression task, we employed a Random Forest model [34]. In the random split scenario, the Random Forest regressor achieved a moderate R² score of 0.67, indicating some predictive capability but also challenges in accurately predicting the differences in F1 scores. When tested under the more realistic split, the model’s performance further decreased, attaining an R² score of only 0.59. This highlights the difficulty in generalizing the regression model to new, unseen datasets.

Refer to caption
Figure 7: Correlation Matrix of Summary Statistics and F1 Score Differences in Predicting the Benefit of Federated Thresholds

The correlation matrix presented in Figure 7 offers further insights into the relationships between various summary statistics and the F1 score difference (f1_difference). Analysis of the matrix reveals that certain features, such as normal_aggr_mean and anomaly_aggr_mean, exhibit a relatively strong correlation with f1_difference. This suggests that these features are effective predictors of whether the federated threshold will outperform the local threshold. Conversely, other features, including normal_proportional_count and anomaly_proportional_count, display little to no correlation with f1_difference. This indicates that these features may be less useful in predicting the advantage of employing a federated threshold.

These observations highlight opportunities for enhancing our feature selection process. By refining our feature engineering strategies and potentially incorporating more relevant features, we can improve the predictive power of our models. Additionally, the weak correlation observed for some features underscores the necessity to explore alternative statistical measures or advanced modeling techniques that can more effectively capture the complexities inherent in the data.

Our experiments emphasize the critical role of appropriate feature selection in enhancing the generalization capabilities of models within federated learning environments. The insights derived from the correlation analysis will inform future endeavors aimed at developing more adaptive and robust thresholding mechanisms, thereby advancing the effectiveness of federated anomaly detection.

Conclusion & Future Work

In this paper, we introduced an innovative federated thresholding method for anomaly detection (AD) using autoencoders (AEs), leveraging summary statistics to improve robustness and accuracy across multiple clients. Our approach consistently outperformed traditional local and federated thresholding techniques in both IID and non-IID data scenarios. Its scalability was evident, with minimal performance degradation as the number of clients increased, making it highly suitable for real-world federated learning applications. While federated methods generally provided superior results, certain local methods remained effective in cases with highly varied data distributions.

Future research could focus on further enhancing the thresholding process by incorporating advanced statistical measures, such as entropy or mutual information, to capture more complex patterns and improve detection accuracy. Adaptive thresholding mechanisms that respond in real-time to changes in client data distributions could also increase the flexibility and responsiveness of anomaly detection systems. Additionally, integrating differential privacy techniques may further enhance data protection and ensure compliance with privacy regulations. Adaptive federated learning models that dynamically adjust to varying data distributions could further improve the robustness and scalability of federated anomaly detection systems.

Data Availability

The datasets analyzed during the current study are publicly available as follows:

  • Credit Card Fraud Detection: Provided by Worldline and the Machine Learning Group of ULB, 2014. This dataset can be accessed at Kaggle.

  • Shuttle Dataset: Available from the UCI Machine Learning Repository. This dataset can be accessed at UCI Shuttle Dataset.

  • Covertype Dataset: Available from the UCI Machine Learning Repository. This dataset can be accessed at UCI Covertype Dataset.

References

  • [1] McMahan, H. B., Moore, E., Ramage, D., Hampson, S. et al. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282 (2017).
  • [2] Kairouz, P. et al. Advances and open problems in federated learning. \JournalTitleFoundations and trends in machine learning 14, 1–210 (2021).
  • [3] Yang, Q., Liu, Y., Chen, T. & Tong, Y. Federated machine learning: Concept and applications. \JournalTitleACM Transactions on Intelligent Systems and Technology (TIST) 10, 1–19 (2019).
  • [4] Li, T., Sahu, A. K., Talwalkar, A. & Smith, V. Federated learning: Challenges, methods, and future directions. \JournalTitleIEEE Signal Processing Magazine 37, 50–60 (2020).
  • [5] Ali, M., Karimipour, H. & Tariq, M. Integration of blockchain and federated learning for internet of things: Recent advances and future challenges. \JournalTitleComputers & Security 108, 102355 (2021).
  • [6] Smith, V., Chiang, C.-K., Sanjabi, M. & Talwalkar, A. S. Federated multi-task learning. \JournalTitleAdvances in Neural Information Processing Systems (2017).
  • [7] Bonawitz, K. et al. Towards federated learning at scale: System design. \JournalTitleProceedings of machine learning and systems 1, 374–388 (2019).
  • [8] Liu, F. T., Ting, K. M. & Zhou, Z.-H. Isolation forest. In 2008 eighth ieee international conference on data mining, 413–422 (IEEE, 2008).
  • [9] Zhao, Y. et al. Federated learning with non-iid data. \JournalTitlearXiv preprint arXiv:1806.00582 (2018).
  • [10] Schlegl, T., Seeböck, P., Waldstein, S. M., Langs, G. & Schmidt-Erfurth, U. f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. \JournalTitleMedical image analysis 54, 30–44 (2019).
  • [11] Hendrycks, D. & Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. \JournalTitlearXiv preprint arXiv:1610.02136 (2016).
  • [12] Liang, S., Li, Y. & Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. \JournalTitlearXiv preprint arXiv:1706.02690 (2017).
  • [13] Nalisnick, E., Matsukawa, A., Teh, Y. W. & Lakshminarayanan, B. Detecting out-of-distribution inputs to deep generative models using typicality. \JournalTitlearXiv preprint arXiv:1906.02994 (2019).
  • [14] Vaswani, A. et al. Attention is all you need. \JournalTitleCoRR abs/1706.03762 (2017). 1706.03762.
  • [15] Zhu, H. et al. Long short term memory networks based anomaly detection for kpis. \JournalTitleComputers, Materials & Continua 61 (2019).
  • [16] Xu, H. et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 world wide web conference, 187–196 (2018).
  • [17] An, J. & Cho, S. Variational autoencoder based anomaly detection using reconstruction probability. \JournalTitleSpecial lecture on IE 2, 1–18 (2015).
  • [18] Baur, C., Denner, S., Wiestler, B., Navab, N. & Albarqouni, S. Autoencoders for unsupervised anomaly segmentation in brain mr images: a comparative study. \JournalTitleMedical Image Analysis 69, 101952 (2021).
  • [19] Malhotra, P., Vig, L., Shroff, G., Agarwal, P. et al. Long short term memory networks for anomaly detection in time series. In Esann, vol. 2015, 89 (2015).
  • [20] Zong, B. et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations (2018).
  • [21] Zenati, H., Foo, C. S., Lecouat, B., Manek, G. & Chandrasekhar, V. R. Efficient gan-based anomaly detection. \JournalTitlearXiv preprint arXiv:1802.06222 (2018).
  • [22] Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. \JournalTitlescience 313, 504–507 (2006).
  • [23] Huong, T. T. et al. Detecting cyberattacks using anomaly detection in industrial control systems: A federated learning approach. \JournalTitleComputers in Industry 132, 103509 (2021).
  • [24] Novoa-Paradela, D., Fontenla-Romero, O. & Guijarro-Berdiñas, B. Fast deep autoencoder for federated learning. \JournalTitlePattern Recognition 143, 109805 (2023).
  • [25] Sáez-de Cámara, X., Flores, J. L., Arellano, C., Urbieta, A. & Zurutuza, U. Clustered federated learning architecture for network anomaly detection in large scale heterogeneous iot networks. \JournalTitleComputers & Security 131, 103299 (2023).
  • [26] Kea, K., Han, Y. & Kim, T.-K. Enhancing anomaly detection in distributed power systems using autoencoder-based federated learning. \JournalTitlePlos one 18, e0290337 (2023).
  • [27] Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U. & Langs, G. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, 146–157 (Springer, 2017).
  • [28] Wang, X. et al. Federated deep learning for anomaly detection in the internet of things. \JournalTitleComputers and Electrical Engineering 108, 108651 (2023).
  • [29] Sánchez, P. M. S. et al. Studying the robustness of anti-adversarial federated learning models detecting cyberattacks in iot spectrum sensors. \JournalTitleIEEE Transactions on Dependable and Secure Computing 21, 573–584 (2022).
  • [30] Pourahmadi, V., Alameddine, H. A., Salahuddin, M. A. & Boutaba, R. Spotting anomalies at the edge: Outlier exposure-based cross-silo federated learning for ddos detection. \JournalTitleIEEE Transactions on Dependable and Secure Computing 20, 4002–4015 (2022).
  • [31] Sakurada, M. & Yairi, T. Anomaly detection using autoencoders with nonlinear dimensionality reduction. \JournalTitleMLSDA 2 (2014).
  • [32] Ahmed, M., Seraj, R. & Islam, S. M. S. The k-means algorithm: A comprehensive survey and performance evaluation. \JournalTitleElectronics 9, 1295 (2020).
  • [33] Vishwanathan, S. & Murty, M. N. Ssvm: a simple svm algorithm. In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN’02 (Cat. No. 02CH37290), vol. 3, 2393–2398 (IEEE, 2002).
  • [34] Rigatti, S. J. Random forest. \JournalTitleJournal of Insurance Medicine 47, 31–39 (2017).