PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection

Existing reconstruction based methods (Xu et al., 2022; Zhou et al., 2023b; Wu et al., 2022) generally neglect the anomalies in the training data, which may disrupt mask reconstruction. For instance, if normal points are masked while anomalous points are utilized as observations to reconstruct the masked time series fragments, it may result in large reconstruction errors (Xu et al., 2024). To address this issue, we propose the anomaly-driven mask selection strategy to first identify the anomalies, and then assign them larger weights to be chosen for masking. The module combines the analysis on intra- and inter-patch variability to calculate the anomaly score of patches, capturing both patch-specific deviations and the contextual evolution of patterns over time.

Intra-patch Decomposition. To capture the intrinsic characteristics of the $i$-th patch (denoted as $\bm{{P}}_{i}$), we utilize time series decomposition technique (Hassani, 2007). Specifically, we decompose each patch into $M$ components, as formulated in Eq. (3), and extract residual components to calculate the intra-anomaly score of patches.

where $\bm{g}_{j}$ denotes the $j$-th component, $a_{j}$ is the coefficient for $j$-th component, and $\varepsilon$ denotes the noise term.

Specifically, we use Singular Spectrum Analysis (SSA) (Hassani, 2007) to decompose patches. In SSA, patch $\bm{{P}}_{i}$ is first transformed into a Hankel matrix $\bm{\mathcal{P}}_{i}$ through embedding, and then Singular Value Decomposition (SVD) is applied to the matrix, decomposing $\bm{\mathcal{P}}_{i}$ into the product of three matrices: $\bm{\mathcal{P}}_{i}=\bm{U}\bm{\Sigma}\bm{V}^{T}$, where $\bm{U}$ and $\bm{V}$ denote the left and right singular vector matrices, respectively, and $\bm{\Sigma}$ denotes the diagonal matrix of singular values. Then, the original patch is reconstructed by

where $K$ denotes number of non-zero eigenvalues of $\bm{{\mathcal{P}}}_{i}$. $\bm{\sigma}_{k}$ is the $k$-th singular value, $\bm{u}_{k}$ is the $k$-th left singular vector, and $\bm{v}_{k}$ is the $k$-th right singular vector.

Matrix $\bm{{\mathcal{P}}}_{i}$ constitutes the main structure of the original patches. For instance, the trend, seasonal, and residual components correspond to the low, mid, and high frequency components of matrix $\bm{{\mathcal{P}}}_{i}$. We can obtain these components by filtering. Residuals often contain anomalies in the time series (Schmidl et al., 2022). Therefore, we extract the residual component after decomposition, and calculate the mean of the residual components as the residual value $\mathcal{R}_{i}$, as formulated in Eq. (5). A higher residual value indicates a larger likelihood to be an anomaly. We then normalize $\mathcal{R}_{i}$ to calculate the anomaly score $\mathcal{R}_{i}^{{}^{\prime}}$ for the $i$-th patch.

where subscript $k$ denotes the $k$-th value of the matrix, and $K_{N}$ denotes the set of singular values associated with residual components obtained by filtering.

Inter-patch Similarity Assessment. The inter-patch similarity assessment provides insights into the dynamic evolution of patterns patches. Assuming $\bm{\mathcal{A}}_{i}$ is the vector of patch $i$, we calculate the cosine similarity between the $i$-th and ($i$-$1$)-th patches.

The cosine similarity ranges from -1 to 1, and a larger value indicates a higher similarity between patches. Patches with lower similarity to the previous patches are more likely to be anomalous, so we alter the monotonicity and normalize $C_{i}$ to calculate the anomaly score $C_{i}^{{}^{\prime}}$ for the $i$-th patch.

Anomaly Score of Patches. We synthesize the intra-patch time series decomposition and the inter-patch similarity assessment to obtain a final anomaly score for patch $i$ as follows:

The patches whose anomaly scores surpass a predefined threshold are considered as anomalies and are assigned larger weights to be chosen for masking. Since the masked patches are more emphasized by the model, the anomaly-driven mask selection strategy can enhances the model’s adaptability towards change points, thus improving the robustness of anomaly detection.

In federated learning, clients may have different data distributions and features, posing a data heterogeneity challenge that makes the generalization of the aggregated model difficult. To address this issue, we propose a privacy-preserving shared dataset synthesis scheme coupled with knowledge distillation.

Privacy-Preserving Shared Dataset Synthesis. Recent works have demonstrated that reducing mutual information can facilitate privacy protection in dataset generating (Yang et al., 2023). Inspired by this idea, we employ a constrained mutual information approach to obtain synthetic data for preserving the privacy of local data. Specifically, Client $i$ trains a variational autoencoder (VAE) model to synthesize time series $\mathcal{T}_{s,i}$ from the local time series $\mathcal{T}_{i}$. The mutual information $I(\mathcal{T}_{i};\mathcal{T}_{s,i})$ measures the extent to which $\mathcal{T}_{s,i}$ reveals $\mathcal{T}_{i}$. Through constraining $I(\mathcal{T}_{i};\mathcal{T}_{s,i})$, the likelihood of inferring $\mathcal{T}_{i}$ from $\mathcal{T}_{s,i}$ has been reduced, thereby better protecting data privacy and facilitating the synthesis of privacy-preserving time series.

where $p(x,y)$ denotes the joint probability distribution, with $p(x)$ and $p(y)$ as the marginal probabilities of $x$ and $y$, respectively.

In order to ensure the validity of the synthesized time series, we introduce a constraint to maintain the distribution similarity between the synthesized and the original time series. We use Wasserstein distance to quantify this distribution similarity (Rüschendorf, 1985). A smaller Wasserstein distance indicates a lower cost of transforming from one distribution to another, implying that the two distributions are more similar. Given two time series $X$ = $\langle\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{m}\rangle$ and $Y$ = $\langle\bm{y}_{1},\bm{y}_{2},\ldots,\bm{y}_{n}\rangle$, and their cumulative distribution functions $F_{X}$ and $F_{Y}$, the Wasserstein distance can be obtained as follows,

where $\gamma$ denotes the joint distributions between $F_{X}$ and $F_{Y}$, and $\Gamma(F_{X},F_{Y})$ denotes the set of all joint distributions with the marginal distributions $F_{X}$ and $F_{Y}$.

We use VAE to synthesize time series, which consists of an encoder and a decoder. The encoder first encodes the input time series as a feature representation, and the decoder then attempts to generate a synthesized time series based on the representation. The raw data privacy and the synthesized data validity are guaranteed by constraining mutual information and Wasserstein distance, respectively. The loss function for VAE is given by

where $\mathcal{L}_{vae}$ denotes the base loss function of VAE. $\bm{x}$ and $\bm{z}$ denote the input and latent vectors, respectively. $q(\bm{z}|\bm{x})$ and $p(\bm{x}|\bm{z})$ denote the output distributions of the encoder and decoder, respectively. $KL(\cdot)$ denotes the Kullback-Leibler divergence  (Van Erven and Harremos, 2014), which can be calculated as follows:

where both $q(\bm{z}|\bm{x})$ and $p(\bm{z})$ are assumed to follow multivariate Gaussian distributions. $\mu_{i}$ and $\sigma_{i}$ are the mean and standard deviation of the Gaussian distribution.

Then, the server integrates the synthesized time series from clients to form a shared dataset $\mathcal{D}_{sh}$. Note that time series synthesis is a one-time offline process before local training.

Knowledge Distillation. We further perform knowledge distillation from the global model to the client models using the shared dataset to reduce the data heterogeneity across clients. Specifically, we first obtain the learned representations of the local and global models on the shared dataset separately, and then calculate the difference between the two representations. We use the consistency loss to measure this difference. Through reducing this discrepancy, the model can achieve more consistent client updates, thereby improving the performance and stability of the aggregated global model. The consistency loss is introduced as a regularization term to the local loss function as follows,

where $\hat{T}^{i}_{j}$ and $T^{i}_{j}$ denote the reconstructed and real values of $j$-th time series of client $i$, respectively. $\theta_{i}$ and $\theta_{g}$ represent the parameters of the $i$-th local and global model, respectively. $\lambda$ is a parameter to trade off the two loss terms.

We conduct experiments on four real-world time series anomaly detection datasets: SMD, PSM, SWaT, and MSL. The 4 datasets are widely used by existing studies and are collected from various real-world domains, covering Internet data, server operational data, critical infrastructure system data, and spacecraft monitoring system events.

• •

  SMD. Server Machine Dataset (SMD) (Su et al., 2019) is a 5-week-long dataset collected from a large Internet company with 38 feature dimensions.

• •

  PSM. Pooled Server Metrics (PSM) dataset (Abdulaal et al., 2021) is collected from multiple application servers at eBay with 25 feature dimensions.

• •

  SWaT. Secure Water Treatment (SWaT) dataset (Mathur and Tippenhauer, 2016) is obtained from 51 sensors of the critical infrastructure system under continuous operations.

• •

  MSL. Mars Science Laboratory rover (MSL) dataset (Hundman et al., 2018) contains the telemetry anomaly data derived from the incident surprise anomaly reports of spacecraft monitoring systems with 55 feature dimensions.

We compare PeFAD with the following 12 baselines including classical methods: OCSVM (Tax and Duin, 2004), Isolation Forest (IF) (Liu et al., 2008) LOF (Breunig et al., 2000), GANF (Dai and Chen, 2022), MTGFLOW (Zhou et al., 2023a), centralized reconstruction-based methods: Anomaly Transformer (AT) (Xu et al., 2022), TimesNet (Wu et al., 2022), and FPT (Zhou et al., 2023b), centralized prediction-based methods: Autoformer (Wu et al., 2021), Informer (Zhou et al., 2021), and FEDformer (Zhou et al., 2022). In addition, we transform centralized methods with FedAvg (McMahan et al., 2017) into their federated version: AT_fl, Autoformer_fl, Informer (Zhou et al., 2021), and FEDformer (Zhou et al., 2022), TimesNet_fl, and FPT_fl. We also compare PeFAD with the best performing model (i.e., DeepSVDD) in FedTADBench (Liu et al., 2022).

Precision (P), Recall (R), F1-Score (F1), and AUC-ROC (AUC, the Area Under the Receiver Operating Characteristic curve) are adopted as the evaluation metrics. A higher value of the metrics means a better performance.

We implement our model with the PyTorch framework on NVIDIA RTX 3090 GPU. The pre-trained language models (i.e., GPT2, BERT, ALBERT, RoBERTa, DeBERTa, DistillBERT, and Electra) are downloaded from Huggingface. We first split the time series into consecutive non-overlapping segments by sliding window (Shen et al., 2020). The patch length and batch size are set to 10 and 32, respectively. Adam is adopted for optimization. We adopt the widely-used point adjustment strategy (Shen et al., 2020; Su et al., 2019; Xu et al., 2018). We employ GPT2 as the PLM, where the first eight layers of GPT2 are used for training. $\lambda$ is set to $1e1$, $2e0$, $2e3$, and $15e4$ for SMD, PSM, SWaT, and MSL, respectively. The threshold $r$ for SMD, MSL, PSM, and SWaT is set to $0.5$, $2$, $1$, and $1$, respectively.

To test the effect of different tuning strategies of PLM, we compare PeFAD with strategies of fine-tuning different numbers of PLM layers, including no fine-tuning (w/o_ft), tuning the last one to seven layers of PLM (PeFAD_t1l - PeFAD_t7l), and fully fine-tuning (PeFAD_fft). The result is shown in Table LABEL:tab:tuning. We use GPT2-based FPT_fl as a reference. One can observe that freezing the first layers while fine-tuning the last few layers is a reasonable tuning strategy. By freezing the first layers, the model retains the ability to understand generalized knowledge, and fine-tuning the last few layers facilitates the model’s adaptation to downstream tasks, enabling the transfer of domain-specific knowledge from the pre-trained model to the time series anomaly detection task. Specifically, for the SMD dataset with more training data, PeFAD remains relatively stable with different tuning layers, and achieves optimal performance when tuning the last 3 and 4 layers. For the smaller MSL dataset, the model performance decreases with the increase of tuning layers, reaching optimal performance when tuning the last layer. The experiments on other datasets are provided in the appendix due to space limitation. In PeFAD, we choose to fine-tune the last layer for MSL and fine-tune the last three layers for the other datasets.

The result shows that our approach consistently outperforms FPT regardless of the number of tuning layers. Compared with FPT, PeFAD achieves the performance improvement of 9.85% and 7.09% in terms of F1-Score on SMD and MSL, respectively. PeFAD reduces the communication cost by 41.2% and 94.9%, which shows the efficiency of PeFAD and the effectiveness of the proposed parameter-efficient federated training module. Furthermore, PeFAD without fine-tuning (w/o_ft) outperforms all federated baselines on both datasets, which demonstrates the superior cross-modality knowledge transfer ability of PLM. PeFAD_fft does not achieve the best performance on both datasets while tuning less, especially last few layers, works better. This is because the initial layers of PLM contain generic knowledge and the last layers are better suited to learn task-specific information. However, due to the scarcity of anomalous data, fully fine-tuning may increase the risk of overfitting, leading to performance degradation.

Next, we study the effect of using different PLMs on the model performance. We compare seven mainstream pre-trained models, i.e., BERT, ALBERT, RoBERTa, DeBERTa, DistilBERT, and Electra. The results are presented in Figure 4. One can see that GPT2 achieves the best performance followed by DeBERTa. Compared to other PLMs, GPT2 improves the performance by up to 6.22% and 5.06% on F1-Score and AUC metrics on SMD, respectively. On the MSL dataset, the F1-Score and AUC values are improved by up to 8.84% and 6.99%, respectively. This is because GPT2 has been exposed to a broader range of contexts during pre-training, enabling it to learn from time series more effectively.

We next study the sensitivity of the model to the mask ratio $r_{m}$ and patch length $l_{p}$, We only give the result of F1-Score on SMD as an example due to space limitation, as shown in Figure 5(5(a)). One can observe that the incorporation of masking or patching mechanisms can improve the model performance, demonstrating the effectiveness of these two mechanisms. As the $r_{m}$ and $l_{p}$ increase, the model performance first improves and then declines. The optimal model performance is achieved when $r_{m}$ is 20% and $l_{p}$ is 10.

We next investigate the effect of synthetic data length on model performance, and the result is shown in Figure 5(5(b)). Specifically, we vary the length of the synthetic time series for each client on the SMD dataset. We observe that the F1-Score curve first increases and then drops slightly. Generally, the result demonstrates that the model obtains the best performance when the length of the synthetic time series is set to 100. With the increase of length from 20 to 100, the synthetic time series may bring more useful information, which facilitates the model with more effective representation learning. However, a too large length value will lead to performance decline. This is because longer synthetic time series may bring redundant or noisy information, which degrades the model performance.