Multiple-Resolution Tokenization
for Time Series Forecasting
with an Application to Pricing

Components of the original transformer architecture (Vaswani et al., 2017) powers the state of the art for large language models and has seen more computational resource investment than any other deep learning architecture. Empirically, the transformer is outstanding for language modelling and benefits from scaling laws. Due to the success of transformers with the sequential task of language there has been considerable interest in applying transformers to time series. Academic research has produced mixed results and tends to be limited in the scope of study. In fact there are strong arguments to be made against using transformers for time series with plenty of vocal opponents.

Transformers have architectural limitations when applied to time series (Zeng et al., 2023). Transformers are permutation invariant and thus inherently do not create an inductive bias which reflects that observations happen sequentially (typically in a highly auto-correlated manner). Transformers for language use two core architectural adjustments for dealing with the lack of sequential inductive bias. Positional encoding adds a bias to the latent matrix representation of a language sequence which is designed or learnt to reflect sequentiality. Decoder architectures mask out the attention matrix so that for each unit of latent representation (token) can only observe the tokens that occurred before it in the sequence. The effectiveness of either of these techniques for forecasting is still an open question. The evidence so far is mixed and simple models have been shown to outperform very complex transformer models on academic benchmarks (Zeng et al., 2023).

At a higher level, the concept of a token in language¹¹1Note that tokenisation in LLMs is typically not trained end-to-end and is fixed, transferred into the model from a different pretraining task. This is often a cause of problems. does not trivially map to time series. Time series are processes which have regular or irregular observations, but single observations are not necessarily meaningful units in terms of the dynamics of a time series. An analogous reasoning from classical time-series approaches is the concept of the order of integration which is the number of times a time series has to be differenced (how many adjacent points need to be considered) to become stationary. Time series are often decomposed into trend and seasonality or frequencies, both of which are transformations of more than just a single observation. A notable transformer time series architecture uses frequency decomposition to tokenize a time series (Zhou et al., 2022). Alternatively, a popular and increasingly standard method (Das et al., 2023) for tokenisation is patching (Nie et al., 2023): it splits a series up into fixed length sequences which are then mapped into the embedding space using a shared learnable linear transformation. The original PatchTST uses patches of length 16 with a stride of 8 (allowing for overlapping patches). This can be interpreted as the resolution at which a time series is processed. As opposed to previous by observation period encoder-decoder designs (Lim et al., 2021) patching loses the 1-to-1 correspondence between tokens and forecast horizon periods. Instead, the matrix output from the transformer blocks is flattened and passed to a large linear layer (presenting a potential bottleneck) which outputs the forecast vector. The implicit assumption is that meaningful information about the time series occurs at resolutions denser than the base time series. A linear model over the whole series assumes that we need to look at the highest density; the whole series. Our work builds on this assumption by extending patching to multiple resolutions.

There have been many attempts to design transformer and transformer like architectures for time series, particularly for long forecast horizons due to a set of popular testing benchmarks. The vast majority of innovation appears either in the design of the tokenisation or the architecture of the token processing engine. This repository is an excellent overview of recent developments across a range of modelling approaches.

We would like to outline the differentiating contributions of our work in contrast with other recent approaches which employ the idea of utilising multiple resolution modelling in transformer architectures. We provide a brief description of each model and point out the key difference in capabilities and architecture. Our model is unique in that it uses the tokens from multiple resolutions in the same attention mechanism allowing for explicit pairwise modelling at multiple resolutions across multiple data types. In addition we provide a scheme for how to tokenize time-varying known data at multiple resolutions including for the time steps which occur in the future.

The multiple resolution patching is defined by a resolution set $K={k_{1},...,k_{r}}$ takes an input of the form $[B,C,T=l]$ and uses a set of $|K|=r$ linear transformations to create an output of the form $[B,C,T,L]$. Denote the size of the latent dimension of the model as $d_{m}$. We extend patching to multiple resolutions by using several divider blocks. A divider block splits a time series into $k_{i}$ roughly equal parts. For a series of length $h$ the base patch length is set as $b_{i}=\lfloor h/k_{i}\rfloor$. If $k$ does not divide $h$ we increase the length of the first $h-b_{i}k$ patches by 1. In our application $h$ is either $l$ (past data) or $l+h$ (time-varying known). For each resolution $k_{i}$ we learn a linear projection into the latent space $\mathbb{R}^{b+1}\mapsto\mathbb{R}^{d_{m}}$. Weights are shared for all patches within a resolution. We left pad patches that are $b$ long with a 0. The core operation is $[B,C,T=b_{i}+1]\rightarrow[B,C,L=d_{m}]$ which is repeated $\sum_{k_{i}\in K}k_{i}$ times ($k_{i}$ times for each resolution) and stacked along a patch/token dimension into $[B,C,T=n_{MRP},L]$. The created dimension $T$ contains $\sum_{k_{i}\in K}k_{i}=n_{MRP}$ tokens obtained from multiple resolution patching. An illustration of multiple resolution patching is presented in figure 1.

A key advantage of deep learning models over classical models is that they can accommodate arbitrary forms of input enabling us to jointly model all available information for a time series. Auxiliary information is not trivially useful for time series forecasting. Most recent state of the art papers exclusively focus on past/observed data. The inclusion of auxiliary data often leads to issues with overfitting, explainable by the noise in time series data and small datasets. In existing work which includes auxiliary information it is typically mixed with past data to form tokens representing all information at each time step or resolution. We take a different approach and design a separate tokenisation scheme for different types of auxiliary data. This enables us to explicitly learn contextual representations of the time series that are purely auxiliary and not directly dependent on the noise in the time series, which should help reduce overfitting.

The overfitting problem has also been empirically observed for the inclusion of cross-series information, which has motivated the use of channel independence Han et al. (2024). We propose a channel mixer module which employs a mixer architecture to generate a set of cross-series tokens (CST) from the tensor containing all other tokens.

We call the module responsible for extracting cross-series information the channel mixer. As input the channel mixer takes the tensor of all existing tokens MRP, ST, and TVKT. Define the number of these tokens as $n_{B}=n_{MRP}+n_{S}+n_{TVK}$. The input tensor takes the form $[B,C,T=n_{B},L]$. We employ a mixer architecture inspired by Chen et al. (2023) to extract cross series tokens. Mixer architectures have been shown to perform well on time series forecasting while extracting cross-series representations in every block. We roughly illustrate the module in figure 3. The operating tensor is normalised before each mixing step and a skip connection is employed across each mixing step. The first mixing step is across the token dimension $T$, applying a linear transformation $[B,C,L,T=n_{B}]\rightarrow[B,C,L,T=n_{B}]$ followed by an activation function and dropout. The number of tokens is then squeezed down to a predefined number $n_{CST}$ with another linear layer $[B,C,L,T=n_{B}]\rightarrow[B,C,L,T=n_{CST}]$ (we use $A$ here to signal that the tokens have been mixed into an auxiliary information representation). The second mixing step is across the channel dimension $C$. Two linear layers are employed, the first projects the channel dimension of size $d_{c}$ into the latent dimension of the cross series module $d_{cross}$ (adjustable hyperparameter), so $[B,A,L,C=d_{c}]\rightarrow[B,A,L,C=d_{cross}]$. After an activation function and a dropout layer the second linear layer projects back into the original channel dimension $[B,A,L,C=d_{cross}]\rightarrow[B,A,L,C=d_{c}]$. After the end of the second mixing another linear layer squeezes the channel dimension to one leaving us with $[B,C=1,T,L]$. The reason this is done is so that the cross-series tokens are the same across channels. To align it with the dimensions of our collection of tokens we repeat the tensor $|C|=d_{c}$ times to obtain $[B,C=d_{c},T=n_{CST},L]$. We append it to all other existing tokens to obtain the final input matrix $A$ with size $n_{MRT}=n_{B}+n_{CST}$ which is passed to the transformer module $[B,C,T=n_{MRT},L]$.

The tensor of all tokens $A$ is processed by a sequence of transformer blocks after which it is passed to the reverse splitter output head. We use multi-head self-attention in the transformer block. The learning in self attention can be interpreted as learning four different mixing rules across the $L$ dimension to infer good pairwise comparisons. Since the focus of this work is on tokenisation strategies we avoid extensively investigating positional encoding or masking strategies.

Multiple resolution patching and associated auxiliary variable tokenisation adds many more hyperparameters in the form of the resolution set. The space of resolution sets is combinatorial making hyperparameter optimisation a challenging task. We somewhat constrain the space by not allowing overlaps, thus avoiding the need for a stride hyperparameter associated with every resolution. Our tokenisation modules increase the number of learnable parameters in the embedding especially for small $k_{i}$ which makes scaling to large latent sizes more difficult. The CST are sequentially computed downstream of all other tokens limiting the degree of parallelisation in tokenisation. Since all tokenisation is done end-to-end a degree of overfitting is expected. Computationally the main difficulty is the increased number of tokens which scales computation quadratically. This constrains how high $n_{MRP}$ can get limiting dense resolution sets which may be useful for longer context windows with long lookback horizons. Exploiting and encouraging sparsity in self-attention may be one possibility to deal with the larger context window.

From a model architecture standpoint this dataset poses two key problems: irregularly spaced sampling, and the varying length of the series.

The experiments were defined by the range of year-weeks they contained. The experiments are limited to just one week due to memory constraints. Each experiment contains the markdown data for one week across all shops and all items. For any experiment the series corresponding to the last $20\%$ (rounded up) of the due-dates were held out as the testing set, the $15\%$ (rounded up) of due-dates immediately preceding were used as the validation set and the remaining as the training set.

The experiments are somewhat limited in scope. This is mainly due to computational constraints and the delayed timing of the project. More experimentation is needed to weigh the evidence on this architecture. With more compute the experiments could be extended to much larger dataset ranges and latent spaces. Scaling properties are clearly of interest. Ablation would demonstrate what relative benefit there is for each of our tokenisation procedures. Hyperparameter exploration in the space of resolution sets would help determine heuristics on what resolution combinations are performant and uncover good heuristics like perhaps including multiple copies of the same resolution if it is highly involved in attention patterns.

Due to commercial sensitivity the dataset cannot be shared²²2We provide the model code here.. We acknowledge that this is a serious limitation from a reproducibility standpoint, but believe that the experiments performed here are a substantial contribution to research on transformers for forecasting. This is because most existing architectures have been empirically evaluated on a narrow set of datasets which are typically small scale such as ETT (Zhou et al., 2021). What is most likely dataset bias has caused a shift of interest away from using auxiliary or cross-series information. In contrast we demonstrate that our transformer architecture outperforms currently used in-house methods, DLinear, and PatchTST on a messy real-world problem. The work presents evidence that for certain forecasting problems modelling auxiliary data and cross series information adds to generalisation. More ablation is needed to understand what aspects of our tokenisation strategies are the most important in supporting this claim.