Multi-Patch Prediction: Adapting LLMs for Time Series
Representation Learning

The field of time series representation learning has witnessed increasing interest in recent years, with self-supervised learning methods playing a pivotal role. These methods generally fall into two categories:

Contrastive Learning. This category encompasses methods designed to refine the representation space by leveraging various forms of consistency, such as subseries consistency (Franceschi et al., 2019a; Fortuin et al., 2019), temporal consistency (Tonekaboni et al., 2021; Woo et al., 2022a; Yue et al., 2022), transformation consistency (Zhang et al., 2022c; Yang & Hong, 2022), and contextual consistency (Eldele et al., 2021). The goal is to ensure that representations of positive pairs are closely aligned, whereas those of negative pairs are distinctly separated. Despite their strengths, contrastive learning methods often struggle with aligning to low-level tasks such as forecasting, primarily due to their focus on high-level information (Xie et al., 2023).

Masked Modeling. PatchTST (Nie et al., 2023) pioneers the utilization of patches as the basic unit for processing time series, advocating for the prediction of masked subseries-level patches to grasp local semantic information while minimizing memory consumption. However, this approach tends to compromise temporal dependencies and fails to guarantee adequate representation learning of temporal dynamics, as the masked values are often predictably inferred from adjacent contexts (Li et al., 2023).

The adaptation of pre-trained LLMs for time series analysis has garnered significant attention, exploiting their exceptional ability in sequence representation learning. This body of work can be categorized into three primary approaches:

• •

  Zero-shot Adaptation. Studies such as those by Gruver et al. (2023); Xue & Salim (2023); Yu et al. (2023) leverage the inherent sequence representation capabilities of frozen LLMs to facilitate time series prediction without any task-specific training.

• •

  Prompt Optimization. Works by Jin et al. (2023); Cao et al. (2023); Sun et al. (2023) employ reprogrammed input time series in conjunction with the innate sequence modeling prowess of pre-trained LLMs, aiming for enhanced forecasting accuracy.

• •

  Limited Fine-tuning. Research efforts like those of Zhou et al. (2023); Liu et al. (2023); Chang et al. (2023) apply fine-tuning to selected components of LLMs to improve performance in time series analysis tasks.

While these approaches yield encouraging outcomes, they predominantly focus on distinct TSA tasks, instead of achieving a holistic time-series representation.

Time-Series Forecasting (TSF) is the fundamental challenge in time series analysis (Ma et al., 2023), aiming to analyze the dynamics and correlations among historical time-series data to predict future behavior, formulated as:

where $L$ is the look-back window size and $x_{i}^{t}$ is the value of the $i_{th}$ variate at the $t_{th}$ time step, and the modeling target is to learn the unknown distribution of the $H$ future values.

Casual Language Model Pre-training. Current LLMs mostly belong to casual language models (OpenAI, 2023; Radford et al., 2019). They utilize a diagonal masking matrix, ensuring that each token can only access information from previous tokens. The training objective is to predict the next token based on the history information, defined as:

where $N$ is the number of tokens, $\mathbf{x}_{i}$ denotes the $i$-th token.

We explain the motivation of our proposed aLLM4TS solution by raising and answering the following two questions.

How can we effectively adapt LLMs in time series modality? Traditional contrastive learning $\mathcal{L}_{CL}$ and mask-and-reconstruction $\mathcal{L}_{MR}$ serve as a training loss, defined as:

where $N$ denotes the number of training samples (pairs), $B-1$ is the number of negative samples, $\mathbf{x}_{i}$ denotes the $i$-th sample, $\mathbf{x}_{i}^{p}$ is the only positive sample, $\mathbf{x}_{i}^{j}$ is the $j$-th negative sample of $\mathbf{x}_{i}$, $\widehat{\mathbf{x}}_{i}$ is the corresponding masked sample, $f(\cdot)$ is the forward pass. Both are inappropriate for adapting LLMs into time series since the non-negligible misalignment between their modeling objective with time series modeling and casual sequence pre-training processes of LLMs (Xie et al., 2023; Dong et al., 2023). However, during the pre-training stages in Equation 2, casual LLMs undergo a similar training process as TSF in Equation 1 on massive tokens (Brown et al., 2020; Radford et al., 2019), where $x_{i}^{t}$ is the $t_{th}$ token of the $i_{th}$ sentences. This drives us to reformulate time-series forecasting as a self-supervised multi-patch prediction task. This offers several benefits: (1) Guarantee modeling consistency between high-level representation optimization and downstream low-level tasks, where contrastive learning fails, fostering a versatile representation with robust predictive capabilities excelling in diverse TSA tasks. (2) Avoid the temporal dependencies disruption caused by random masking in masked modeling. (3) Align with LLMs’ pre-training where each token is casually predicted, facilitating the seamless adaptation of LLM to the temporal domain. Thus, we devise a forecasting-based self-supervised training strategy, as in Figure 2 (a) and (b), to naturally sync LLMs’ excellent representation learning capabilities with time-series variations, including a casual next-patch continual pre-training and a fine-tuning for multi-patch prediction in the target time-series context.

Is the sequence-level decoder suitable for patch-based time series representation? Current TSA models based on LLMs (Jin et al., 2023; Zhou et al., 2023) follow the traditional patch-based framework in Figure 1 ❷. Given patch-based time series representation $\{{\bm{p}}_{1},\cdots,{\bm{p}}_{L_{p}}\},{\bm{p}}_{i}\in\mathbb{R}^{D}$ across $L_{p}$ patches with dimension $D$, they concatenate and map the patch sequence to prediction horizon $H_{p}$, through a sequence-level decoder $\mathbf{W}_{s}\in\mathbb{R}^{(L_{p}\cdot D)\times H_{p}}$, which can be particularly oversized if either one or all of these values are large, causing severe downstream task overfitting. Instead of mapping at the sequence level, we disentangle the encoding and decoding within our framework through a patch-wise decoder in Figure 1 ❸, which is involved throughout our pipeline (Stage 1 and Stage 2). This empowers the LLM backbone and patch-wise decoder to excel in its designated role: encoding each patch for better representation and decoding each patch independently to the temporal domain.

In this section, we propose to conduct casual next-patch continual pre-training, to sync pre-trained LLMs sequence modeling capabilities with time-series modalities on diverse time series datasets(e.g., Weather, Traffic), as in Figure 2 (a).

Forward Process. Given time series from various datasets, we first flatten them into $M$ univariate sequences. We denote the $i$-th univariate series of look-back window size $L$ starting at time index $t$ as ${\bm{x}}^{(i)}_{t:t+L-1}=\{x_{t}^{(i)},...,x_{t+L-1}^{(i)}\}\in\mathbb{R}^{1\times L}$ where $i=1,...,M$. Then each of them is first divided into patch sequence ${\bm{p}}^{(i)}_{t_{p}:t_{p}+L_{p}-1}=\{{\bm{p}}_{t_{p}}^{(i)},...,{\bm{p}}_{t_{p}+L_{p}-1}^{(i)}\}\in\mathbb{R}^{L_{p}\times P}$ where $t_{p}=\lfloor\frac{(t-P)}{S}\rfloor+1$ is starting patch index, $L_{p}=\lfloor\frac{(L-P)}{S}\rfloor+1$ is the number of patches, $P$ is the patch length, and $S$ is the sliding stride. Finally, each sequence is fed independently into the casual LLM backbone, such as GPT2 (Radford et al., 2019) for the channel-independence setting. Then we get the casual next-patch prediction $\hat{{\bm{p}}}^{(i)}_{t_{p}+1:t_{p}+L_{p}}=\{\hat{{\bm{p}}}^{(i)}_{t_{p}+1},...,\hat{{\bm{p}}}^{(i)}_{t_{p}+L_{p}}\}\in\mathbb{R}^{L_{p}\times D}$.

Loss Function. We choose to use the MSE loss to guide the representation alignment at the patch level. The loss in each time series is gathered and averaged over $M$ time series to get the overall objective loss: $\mathcal{L}_{p}=\mathbb{E}_{{\bm{p}}}\frac{1}{M}\sum_{i=1}^{M}\|\hat{{\bm{p}}}^{(i)}_{t_{p}+1:t_{p}+L_{p}}-{\bm{p}}^{(i)}_{t_{p}+1:t_{p}+L_{p}}\|_{2}^{2}$ , which is employed to guide the casual next-patch time-series representation optimization to uncover the hidden temporal dynamics while aligning with LLMs sequential modeling abilities.

In this section, we fine-tune for the self-supervised, multi-patch prediction task, further refining patch representations to align with the target time series temporal contexts based on the casual next-patch pre-training model in subsection 4.1.

Forward Process. As illustrated in Figure 2 (b), given the $i$-th univariate time series ${\bm{x}}^{(i)}_{t:t+L-1}=\{x_{t}^{(i)},...,x_{t+L-1}^{(i)}\}\in\mathbb{R}^{1\times L}$ of look-back window size $L$ starting at time index $t$, the prediction horizon $H$ and the time-series-aligned LLM $f_{\theta}(\cdot)$ trained in subsection 4.1, we firstly prepare prediction anchors $\dot{{\bm{x}}}^{(i)}_{t+L:t+L+H-1}=\{\dot{x}_{t+L}^{(i)},...,\dot{x}_{t+L+H-1}^{(i)}\}\in\mathbb{R}^{1\times H}$ through non-parametric methods (Recent history ${\bm{x}}^{(i)}_{t+L-H:t+L-1}$or Discrete fourier prediction). Then the look-back window input and anchors are divided into patch sequence ${\bm{p}}^{(i)}_{t_{p}:t_{p}+L_{p}-1}=\{{\bm{p}}_{t_{p}}^{(i)},...,{\bm{p}}_{t_{p}+L_{p}-1}^{(i)}\}\in\mathbb{R}^{L_{p}\times P}$ and $\dot{{\bm{p}}}^{(i)}_{t_{p}+L_{p}:t_{p}+L_{p}+H_{p}-1}=\{\dot{{\bm{p}}}_{t_{p}+L_{p}}^{(i)},...,\dot{{\bm{p}}}_{t_{p}+L_{p}+H_{p}-1}^{(i)}\}\in\mathbb{R}^{H_{p}\times P}$ where $t_{p}=\lfloor\frac{(t-P)}{S}\rfloor+1$, $L_{p}=\lfloor\frac{(L-P)}{S}\rfloor+1$, $H_{p}=\lfloor\frac{(H-P)}{S}\rfloor+1$, $P$ is the patch length, and $S$ is the sliding stride. Next, these two patch sequences are concatenated and fed into the time-series-aligned LLM backbone $f_{\theta}(\cdot)$ trained in stage 1 with a position-aware attention mask $\mathbf{A}_{p}$, which enhances the temporal relationships among patches (Each future anchor can only see all accurate history patches and itself.). Finally, we employ the patch-wise projection layer $\mathbf{W}_{p}\in\mathbb{R}^{D\times P}$ to independently decode optimized anchors $\dot{{\bm{p}}}^{(i)_{o}}_{t_{p}+L_{p}:t_{p}+L_{p}+H_{p}-1}\in\mathbb{R}^{H_{p}\times D}$ into temporal patches $\hat{{\bm{p}}}^{(i)}_{t_{p}+L_{p}:t_{p}+L_{p}+H_{p}-1}\in\mathbb{R}^{H_{p}\times P}$, formulated as $\hat{{\bm{p}}}_{t_{p}+L_{p}+k}^{(i)}=\dot{{\bm{p}}}_{t_{p}+L_{p}+k}^{(i)_{o}}\mathbf{W}_{p},k\in[0,H_{p}-1]$.

Loss Function. We flatten the predicted patches from $\hat{{\bm{p}}}^{(i)}_{t_{p}+L_{p}:t_{p}+L_{p}+H_{p}-1}$ to $\hat{{\bm{x}}}^{(i)}_{t+L:t+L+H-1}$, gather and average the loss in $M$ time series: $\mathcal{L}_{s}=\mathbb{E}_{{\bm{x}}}\frac{1}{M}\sum_{i=1}^{M}\|\hat{{\bm{x}}}^{(i)}_{t+L:t+L+H-1}-{\bm{x}}^{(i)}_{t+L:t+L+H-1}\|_{2}^{2}.$ Notably, during multi-patch (anchor) representation optimization, most parameters in the time-series-aligned LLM $f_{\theta}(\cdot)$ trained in subsection 4.1 are frozen except for the Position Embedding and Layer Normalization Layer (Less than $0.01\%$ of the overall parameters) to make better adaptions in target time series. Also, once finish a single stage 2 adaption in a target time series dataset, history $L$, horizon $H$, we can perform any other forecasting tasks with other input/out length $\hat{L}/\hat{H}$ without any re-training since our patch-wise representation decoding is independent of input/out length.

Datasets. For long-term forecasting, few-shot forecasting, and representation learning, we evaluate our proposed aLLM4TS on $8$ popular datasets, including Weather, Traffic, Electricity, ILI, and $4$ ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2). These datasets have been extensively used and are publicly available in  Wu et al. (2021). For short-term forecasting, we evaluate models on widely used marketing dataset M4 (Makridakis et al., 2018). For anomaly detection, we evaluate models on five widely employed datasets: SMD (Su et al., 2019), MSL (Hundman et al., 2018), SMAP (Hundman et al., 2018), SWaT (Mathur & Tippenhauer, 2016), and PSM (Abdulaal et al., 2021).

Baselines. For time series forecasting and anomaly detection task, we compare $11$ baseline methods: the SOTA LLM-based model GPT4TS (Zhou et al., 2023), nine Transformer-based models, including PatchTST (Nie et al., 2023), FEDformer (Zhou et al., 2022), Autoformer (Wu et al., 2021), Non-Stationary Transformer (Liu et al., 2022), ESTformer (Woo et al., 2022b), LightTS (Zhang et al., 2022b), Pyraformer (Liu et al., 2021), Reformer (Kitaev et al., 2020), Informer (Zhou et al., 2021), and the SOTA MLP-based model DLinear (Zeng et al., 2023). And we also add a strong text-guided LLM baseline Time-LLM (Jin et al., 2023) for long-term forecasting. Besides, N-HiTS (Challu et al., 2022) and N-BEATS (Oreshkin et al., 2019) are added for comprehensive short-term forecasting performance comparison. For representation learning, we compare aLLM4TS with $5$ baseline methods: the SOTA masking-based representation learning method PatchTST (Nie et al., 2023), and four contrastive learning methods for time series, BTSF (Yang & Hong, 2022), TS2Vec (Yue et al., 2022), TNC (Tonekaboni et al., 2021), and TS-TCC (Eldele et al., 2021).

Experiment Settings. For the time series forecasting task, all models follow the same experimental setup with prediction length $H\in\{24,36,48,60\}$ for the ILI dataset and $H\in\{96,192,336,720\}$ for other datasets. We use the default look-back window $L=336$ for all baseline models and our proposed framework aLLM4TS.

Metrics. We use the following metrics in the experiment comparison: Mean Square Error (MSE), Mean Absolute Error (MAE), Symmetric Mean Absolute Percentage Error (SMAPE), and F1-score (Grishman & Sundheim, 1996).

As shown in the table 1, aLLM4TS outperforms all baseline methods in most cases. Specifically, our informative two-stage forecasting-based, self-supervised representation optimization for LLMs and patch-wise decoding lead to an average performance improvement of $\mathbf{9.71\%}$ over GPT4TS, the current SOTA LLM-based method which directly employs the sequence modeling capabilities in LLMs without any representation adaption. Compared to Time-LLM, which combines textual prompts with the sequence modeling capabilities of frozen LLMs, aLLM4TS achieves over $\mathbf{9.40\%}$ performance improvement. Compared with the SOTA Transformer model PatchTST, aLLM4TS realizes an average MSE reduction of $2.03\%$. Our improvements are also noteworthy compared with the other model classes, e.g., DLinear or TimesNet, exceeding $\mathbf{19.3\%}$. Notably, in stage 1, we conduct a shared casual next-patch pre-training with training sets of Weather, Traffic, Electricity, ILI, and 4 ETT datasets. In stage 2, for $4$ horizons $H\in\{96,192,336,720\}$ in a target dataset, aLLM4TS performs only one training for $H=720$, then it can forecast arbitrary horizons due to its patch-wise decoder that is independent with horizon, while other baselines have to fine-tune for each horizon length.

Table 2 shows the results of short-term forecasting in the M4 benchmark, which contains marketing data in different frequencies. Specifically, we use the same backbone as subsection 5.2 which undergoes a casual next-patch pre-training on training sets of Weather, Traffic, Electricity, ILI, and 4 ETT datasets. Then we employ the same model configuration as GPT4TS to fine-tune for each frequency. We achieve a competitive performance close to the current SOTA TimesNet, whose CNN-based structure is usually considered to perform better in datasets characterized by diverse variations but limited volume (Wu et al., 2022), such as M4. The overall 0.472 SMAPE reduction compared with SOTA LLM-based GPT4TS is attributed to the importance of syncing LLM capabilities with the temporal dynamics. This also verifies the excellent transferability of our forecasting-aligned representation across time series from different domains.

LLMs have demonstrated remarkable performance in few-shot learning (Liu et al., 2023; Brown et al., 2020) due to their ability to obtain strong general representations. In this section, we evaluate the few-shot forecasting ability of our time-series-aligned LLM in ETT datasets. To avoid data leakage, we conduct stage 1 on ETT1 datasets and perform multi-patch prediction with only $\textbf{5}\%$ training data in ETT2, and vice versa. The few-shot forecasting results are shown in Table 3. aLLM4TS remarkably excels over all baseline methods, and we attribute this to the successful representation syncing in our two-stage representation adaption. Notably, both our aLLM4TS and GPT4TS consistently outperform other competitive baselines, further verifying the potential of LLMs as effective time series machines. Significantly, aLLM4TS achieves an average MSE reduction of $\textbf{8.6}\%$ compared to SOTA LLM-based GPT4TS, indicating the benefits of our forecasting-based adaption and patch-wise decoding. In comparison to convolution-based TimesNet and MLP-based DLinear models that are usually considered more data-efficient for training and suitable for few-shot learning methods, aLLM4TS still demonstrates an average MSE reduction of $\textbf{30.6}\%$ and $\textbf{28.5}\%$ respectively.

Time series anomaly detection has various industrial applications, such as health monitoring or finance evaluation. Similar to short-term forecasting, we use the same backbone as subsection 5.2 which undergoes a casual next-patch pre-training on training sets of Weather, Traffic, Electricity, ILI, and 4 ETT datasets. Then we employ the same model configuration as GPT4TS to fine-tune in each anomaly dataset. Results in Table 4 demonstrate that aLLM4TS achieves the SOTA performance with the averaged F1-score $\textbf{87.51}\%$. We attribute this better capability of detecting infrequent anomalies within time series to the forecasting-aligned representation learned in stage 1 casual next-patch pre-training on diverse time series. The aligning process syncs the LLM sequence modeling abilities with time series and further enhances the representation’s transferability and generality across various time series domains and downstream tasks.

In addition to aLLM4TS’s outstanding performance across various downstream tasks, we further explore its superiority in adapting LLMs for time-series representation learning in ETTh1 forecasting. This is achieved through comparisons with state-of-the-art representation learning methods, and various comparative experiments for our two-stage forecasting-based aLLM4TS. Detailed results are in Table 5.

${\text{Sta}_{1}}$: Casual Next-patch Pre-training. Comparing the column ${\text{Sta}_{1}}$+${\text{Sta}_{2}}$ with column Masking+${\text{Sta}_{2}}$³³3Explanation of column names can be found in table caption., we observe a distinct average performance decline over $\textbf{13.5}\%$, whether in a pre-trained LLM or a vanilla PatchTST. We attribute it to the challenge that masking-based patch pre-training struggles to model the vital temporal variations due to its random masking and reconstruction training paradigm. The results of other contrastive-learning-based methods also lead to a significant performance decline, as a result of the non-negligible non-alignment between their high-level objective with the downstream time series analysis tasks.

${\text{Sta}_{2}}$: Multi-patch Prediction Fine-tuning. Previous patch-based models (Nie et al., 2023; Zhou et al., 2023) all choose to concatenate and project in sequence-level as in Figure 1 ❷. Comparing column Masking+SD and Masking+${\text{Sta}_{2}}$, more than $\textbf{13.4}\%$ deterioration occurs, strongly indicating the great risks of overfitting in the huge sequence-wise decoder.

In this section, we conduct several ablations on framework design and the effectiveness of our two-stage forecasting-based self-supervised training. Brief results are in Table 6.

Casual Next-Patch Continual Pre-training. Comparing row A.1 and B.1 in Table 6, an average MSE increase of $\textbf{8.80}\%$ is observed, indicating that ablating casual next-patch continual pre-training significantly harms the sequence pattern recognition and forecasting modeling of the LLM for effective time series analysis. We attribute it to the inadequate adaption to apply pre-trained LLMs in time series without alignment that fits the temporal dynamics, forecasting modeling, and the casual pre-training of LLMs.

LLM Pre-trained Weight. We designed two sets of ablation experiments with different model sizes to avoid the mismatch between training data and model parameter quantity. We discard the pre-trained weights of the LLMs and train from scratch the first $6$ layers (B.2) and the first $3$ layers (B.3) of GPT-2. Ablating the LLM pre-trained weights directly results in the loss of the learned sequential representation capabilities from massive sequential text data (Zhou et al., 2023; Gruver et al., 2023). Consequently, it becomes difficult to learn the temporal representation from scratch within the LLM architecture, leading to the degradation in performance of $\textbf{5.15}\%$ and $\textbf{7.91}\%$, respectively.

Patch-level Decoder. In ablation experiment C.1, we employed the conventional sequence-level decoder, resulting in an average performance loss exceeding $\textbf{8.54}\%$. Despite using a decoder over $100$ times larger and can train specifically for each input/output length, a substantial performance loss occurred. This is attributed to the potential downstream task overfitting of the huge sequence-level head and the incapability to disentangle the patch representation encoding and decoding process, leading to inadequate patch representation optimization in the LLM backbone.

Position-aware Attention Mask. In aLLM4TS, we transform the forecasting into multi-patch representation optimization based on well-aligned patch-based time series knowledge. Position-aware attention mask is designed to further enhance the optimization process by removing the unwanted confusion brought by other being-optimized anchors during the optimization. Ablation of this component (C.2) results in over $\textbf{10.01}\%$ performance deterioration.

We conducted a case study on the Traffic dataset to illustrate the evolution of attention weights from the prediction horizon patches to look-back window patches at four stages in Figure 3. The 4 subplots detail the attention weights optimization process from randomly-initialized (Stage ❶), through LLMs-pretrained (Stage ❷), casually next-patch continual pre-trained (Stage ❸) to multi-patch prediction adaption (Stage ❹). Our observations are as follows: Obs.① After stage ❹, aLLM4TS adeptly captures the complex multi-periodic properties of time series and a discernible trend of increasing information importance along the temporal dimension. In Figure 3 (d), look-back window patches closest to the prediction horizon exhibit similar patterns from prediction horizon patches at time steps $t$, $t+3$, $\cdots$. With a patch size of 16 and a stride of 8, sampling hourly, this corresponds to local day cycles. Additionally, there exist 20-patch cycles (equivalent to 168 hours), indicating weekly cycles. Furthermore, look-back window patches closer to the predicted horizon receive increasing attention due to their temporal proximity, indicating their greater informational significance. Obs.② After stage ❸, aLLM4TS learns universal single-period features (e.g., day) and showcases a noticeable trend of increasing attention along the time dimension in Figure 3 (c), stemming from the process of casually predicting the next patch. Obs.③ Pre-trained LLM parameters capture fundamental time-series cycle attributes in Figure 3 (a) and (b), serving as a robust optimization anchor for time-series representation learning when compared with random initialization.