FastEE: Fast Ensembles of Elastic Distances for time series classification

Abstract

In recent years, many new ensemble-based time series classification (TSC) algorithms have been proposed. Each of them is significantly more accurate than their predecessors. The Hierarchical Vote Collective of Transformation-based Ensembles (HIVE-COTE) is currently the most accurate TSC algorithm when assessed on the UCR repository. It is a meta-ensemble of 5 state-of-the-art ensemble-based classifiers. The time complexity of HIVE-COTE—particularly for training—is prohibitive for most datasets. There is thus a critical need to speed up the classifiers that compose HIVE-COTE. This paper focuses on speeding up one of its components: Ensembles of Elastic Distances (EE), which is the classifier that leverages on the decades of research into the development of time-dedicated measures. Training EE can be prohibitive for many datasets. For example, it takes a month on the ElectricDevices dataset with 9000 instances. This is because EE needs to cross-validate the hyper-parameters used for the 11 similarity measures it encompasses. In this work, Fast Ensembles of Elastic Distances is proposed to train EE faster. There are two versions to it. The exact version makes it possible to train EE 10 times faster. The approximate version is 40 times faster than EE without significantly impacting the classification accuracy. This translates to being able to train EE on ElectricDevices in 13 h.

Appendix A: Existing lower bounds for elastic distances

1.1 A.1 DTW lower bounds

Being the most popular elastic distance, lower bound for DTW has been widely studied (Yi et al. 1998; Kim et al. 2001; Keogh and Ratanamahatana 2005; Lemire 2009; Shen et al. 2018). Note that DDTW is a variant of DTW, so the lower bounds for DTW are directly applicable to DDTW.

Fig. 16

Illustration of aKim and bKeogh lower bound

The simplest and loosest DTW lower bound is the Kim lower bound (LB_Kim) described in Eq. 14 (Kim et al. 2001). LB_Kim uses the maximum differences of the maximum, minimum, first and last points of Q and C as the lower bound for DTW. With initialisation, LB_Kim can be computed very quickly with O(1) time. Although a looser lower bound, it is still effective in filtering out the obvious unpromising candidates. Figure 16a illustrates this lower bound.

$$\begin{aligned} \textsc {LB\_Kim}(Q,C) = \max {\left\{ \begin{array}{ll} |q_1 - c_1| \\ |q_L - c_L| \\ |\max (Q)- \max (C)| \\ |\min (Q) - \min (C)| \end{array}\right. } \end{aligned}$$

(14)

The Keogh lower bound (LB_Keogh) (Keogh and Ratanamahatana 2005) is arguably one of the most used lower bound for DTW due to its simplicity and medium-high tightness. First, it creates two envelopes encapsulating the candidate time series. The upper envelope (UE) is built by finding the maximum within a warping window $r$ range, and the lower envelope (LE) is by finding the minimum, as shown in Eq. 15.

$$\begin{aligned} \begin{matrix} UE_i = \max (c_{i-r}:c_{i+r}) \\ LE_i = \min (c_{i-r}:c_{i+r}) \end{matrix} \end{aligned}$$

(15)

Then LB_Keogh distance of Q and C is the Euclidean distance of all points in Q that are outside of the envelope to the envelopes UE and LE, as described in Eq. 16. Figure 16b illustrates LB_Keogh, where the sum of the length of the green lines is the LB_Keogh distance.

$$\begin{aligned} \textsc {LB\_Keogh}{}(Q,C) = \sqrt{\sum _{i=1}^{L}{ {\left\{ \begin{array}{ll} (q_i-UE_i)^2 &{} \quad \text {if } q_i > UE_i\\ (q_i-LE_i)^2 &{}\quad \text {if } q_i < LE_i\\ 0 &{}\quad \text {otherwise} \end{array}\right. }}} \end{aligned}$$

(16)

There are more sophisticated lower bounds that are tighter than LB_Keogh but has higher computation overheads. The Improved lower bound (LB_Improved) (Lemire 2009) performs LB_Keogh in 2 passes. The first pass computes standard LB_Keogh(Q, C) on the query and the second pass computes LB_Keogh$(Q',C)$ on the projection of the query $Q'$ onto the envelopes. The New lower bound (LB_New) (Shen et al. 2018) takes advantages of the boundary and continuity conditions for DTW warping path to create a tighter lower bound. The boundary condition requires that every warping path contains $(q_1,c_1)$ and $(q_L,c_L)$. The continuity condition ensures that every $q_i$ is paired with at least one $c_j'$, where $j\in \lbrace \max (1,i-r)\ldots \min (L,i+r)\rbrace$. The authors (Shen et al. 2018) sorts the points in $c_j'$ and do a binary search if $q_i$ is within the maximum and minimum of $c_j'$.

1.2 A.2 ERP lower bounds

DTW lower bounds can be adapted for the ERP distance by taking into account the ERP’s penalty parameter g (Chen and Ng 2004). Equation 17 describes LB_Kim for ERP by considering that the first and last point may be a gap, where $q'_1=q_1$ or g, $q'_L=q_L$ or g, $Q_{\max }'=\max (Q_{\max },g)$, $Q_{\min }'=\min (Q_{\min },g)$. The same applies the candidate time series C.

$$\begin{aligned} \textsc {LB\_Kim}_{\textsc {ERP}{}}(Q,C) = \max {\left\{ \begin{array}{ll} |q_1' - c_1'| \\ |q_L' - c_L'| \\ |Q_{\max }'- C_{\max }'| \\ |Q_{\min }' - C_{\min }'| \end{array}\right. } \end{aligned}$$

(17)

Similarly to compute LB_Keogh for ERP ($\textsc {LB\_Keogh}_\textsc {ERP}$), the envelopes need to be adjusted for g where the maximum and minimum values have to include the g parameter. Equation 18 describes these new envelopes. Note that $\texttt {bandsize}$ is used instead of $r$. Then LB_Keogh for ERP is computed exactly the same way as LB_Keogh for DTW using Eq. 16 by substituting with the ERP envelopes.

$$\begin{aligned} \begin{matrix} UE'_i = \max (g, \max (c_{i-\texttt {bandsize}}:c_{i+\texttt {bandsize}})) \\ LE'_i = \min (g, \min (c_{i-\texttt {bandsize}}:c_{i+\texttt {bandsize}})) \end{matrix} \end{aligned}$$

(18)

All the previous lower bounds were developed specifically for DTW. Thus, the authors (Chen and Ng 2004) develop LB_ERP, a new lower bound specifically for ERP. By setting $g=0$, LB_ERP is defined in Eq. 19 as the absolute difference of the sum of both time series. The authors showed that LB_ERP has better pruning power than LB_Keogh$_{\textsc {ERP}{}}$. Currently LB_ERP is only defined for $g=0$ and there are no further proofs for $g\ne 0$. Therefore, we will only be using the LB_Keogh version for ERP in our work.

$$\begin{aligned} \textsc {LB\_ERP}(Q,C)=\left| {\sum Q - \sum C}\right| \end{aligned}$$

(19)

1.3 A.3 LCSS lower bound

The core of LCSS is based on the length of the longest common subsequence between two time series. Then the distance is the percentage of points that are not a match—having distance larger than $\varepsilon$. Recall that LCSS also uses a local constraint $\varDelta$, using similar idea as LB_Keogh by constructing an envelope around the candidate time series C, a lower bound function for LCSS distance has been proposed in (Vlachos et al. 2003). The envelope for Q is constructed using $\varepsilon$ and $\varDelta$ as described in Eq. 20.

$$\begin{aligned} \begin{array}{l} \mathbb {UE}_i = \max (c_{i-\varDelta }:c_{i+\varDelta }) + \varepsilon \\ \mathbb {LE}_i = \min (c_{i-\varDelta }:c_{i+\varDelta }) + \varepsilon \end{array} \end{aligned}$$

(20)

The sum of all $q_i\in Q$ within the envelope creates an upper bound (UB) to the longest common subsequence. Then the lower bound distance for LCSS (LB_LCSS) is $1-\text {UB}$, defined in Eq. 21, as the percentage of points that are not within the envelope.

$$\begin{aligned} \textsc {LB\_LCSS}{}(Q,C) = 1 - \frac{1}{L}\sum _{i=1}^{L}{ {\left\{ \begin{array}{ll} 1 &{} \quad \text {if } \mathbb {LE}_i \le q_i \le \mathbb {UE}_i\\ 0 &{}\quad \text {otherwise} \end{array}\right. }} \end{aligned}$$

(21)