Comparing dimensionality reduction techniques for visual analysis of the LSTM hidden activity on multi-dimensional time series modeling

Abstract

Long short-term memory (LSTM) network is widely applied to multi-dimensional time series modeling to solve many real-world problems, and visual analytics plays a crucial role in improving its interpretability. To understand the high-dimensional activations in the hidden layer of the model, the application of dimensionality reduction (DR) techniques is essential. However, the diversity of DR techniques dramatically increases the difficulty of selecting one among them. In this paper, aiming at the applicability of DR techniques for visual analysis of LSTM hidden activity on multi-dimensional time series modeling, we select four representative DR techniques as the comparison objects, including principal component analysis (PCA), multi-dimensional scaling (MDS), t-distributed stochastic neighbor embedding (t-SNE), and uniform manifold approximation and projection (UMAP). The original continuous modeling data and the symbolically processed discrete data are used as knowledge of model learning, which are associated with LSTM hidden layer activity, and the ability of DR techniques to maintain high-dimensional information of the hidden layer activation is compared. According to the model structure of LSTM and the characteristics of modeling data, the controlled experiments were carried out in five typical tasks, namely the quality evaluation of DR, the abstract representation of high and low hidden layers, the association analysis between model and output variable, the importance analysis of input features and the exploration of temporal regularity. Through the complete experimental process and detailed result analysis, we distilled a systematic guidance for analysts to select appropriate and effective DR techniques for visual analytics of LSTM.

Appendices

Appendix A: Model description

In this section, we introduce the working mechanism of the simple recurrent neural network (SRN) [44] and the basic long short-term memory (LSTM).

The SRN represents the most basic form of recurrent neural networks (RNNs) and serves as a fundamental building block for more advanced variants. In the forward propagation process, the hidden state at t time step $h_t$ is calculated by the hidden state at $t-1$ time step $h_{t-1}$ the input at t time step $x_t$ and learning parameters (U, W, b), as follows:

$$\begin{aligned} h_t=\sigma (Ux_t+Wh_{t-1}+b). \end{aligned}$$

(6)

When the time step length is too long, SRN will suffer from the vanishing or exploding gradient issue, which makes learning SRN using gradient descent very difficult. Hochreiter and Schmidhuber [1] proposed LSTM to solve this problem, enabling RNNs to be applied to long sequence time series. There are three gate controllers on each LSTM hidden layer neuron, namely input gate $i_t$, forget gate $f_t$, and output gate $o_t$. The core component of LSTM is the memory cell, which possesses an internal state, referred to as the cell state $c_t$, for storing and transmitting information. The input gate controls the input of information, the forget gate resets the retention of historical state information of hidden units, and the output gate controls the output of information. The process can be expressed as

$$\begin{aligned} \left\{ \begin{array}{l} i_t=\sigma (U_\textrm{i}x_t+W_\textrm{i}h_{t-1}+V_\textrm{i}c_{t-1}+b_\textrm{i}) \\ \\ f_t=\sigma (U_\textrm{f}x_t+W_\textrm{f}h_{t-1}+V_\textrm{f}c_{t-1}+b_\textrm{f}) \\ \\ c_t=f_\textrm{t}\cdot c_{t-1}+i_\textrm{t}\cdot \mathrm{{tan}}h(U_\textrm{c}x_t+W_\textrm{c}h_{t-1}+b_\textrm{c}) \\ \\ o_t=\sigma (U_\textrm{o}x_t+W_\textrm{o}h_{t-1}+V_\textrm{o}c_{t-1}+b_\textrm{o}) \\ \\ h_t=o_t\cdot tanh(c_t) \end{array}\right. \end{aligned}$$

(7)

where W and U are weights, and b is bias. h is the activation value of the hidden layer output.

Appendix B: Dataset description

Fig. 10

Temporal fluctuation curves of the normalized predicted variables in the ETT dataset and HST dataset

The electricity transformer temperature (ETT) is a crucial indicator in the electric power long-term deployment. The ETT dataset introduced by Zhou et al. [45] is a popular dataset for time series forecast (TSF) tasks, which is publicly available and can be acquired at https://paperswithcode. com/dataset/ett. This dataset consists of 2 years data from two separated counties in China and the sampling interval is 1 min. Each data point consists of the target value “oil temperature” and 6 power load features. Figure 10 shows the temporal fluctuation curves of the normalized predicted variables in the ETT dataset and HST dataset. A total of 18 000 sets of data are taken as the training dataset, 1 000 sets as the validation dataset, and 1 000 sets as the test dataset. We selected this dataset for new multi-dimensional time series modeling, aiming to compare the applicability of dimensionality reduction (DR) techniques in various analysis tasks.

Fig. 11

Association changes between the model’s hidden layers and output variable

Appendix C: Additional case study with the ETT dataset and the basic LSTM

Here, we use the basic LSTM for ETT prediction and extract the activations of the hidden layers. The selected LSTM architecture consists of two hidden layers, each comprising 100 neurons. Similarly, we conducted comparative experiments on DR techniques in the previously proposed visual analysis tasks. All projections presented were created from activations of a test set subset and inspecting a training set subset can also provide similar insights.

Fig. 12

Association between activation state and symbolic output variables (“A,” “C,” and “E”)

Table 5 Statistical results of feature importance

The abstract representation of the high and low hidden layers. As shown in Fig. 11, t-SNE and UMAP projecting the two hidden layers is not conducive to identifying the abstract ability of the hidden layers, and the difference in the activation shape of the projection view of the two hidden layers is small. However, after PCA processing, the projection points of the low hidden layer show a uniformly distributed circular shape, which cannot separate different types of samples well. While the projection points of the high hidden layer show a shape distribution with obvious angles and edges, showing more complex and nonlinear inter-layer abstract representations. Unlike previous experiments, we do not observe clustered projection scatters of PCA and MDS in the low hidden layer, possibly due to the time series fluctuation of the predicted variable values of the ETT dataset being more intense than that of the HST dataset, as shown in Fig. 10.

The association analysis between model and output variable. As shown in Fig. 11, when the associated object of activations is continuous, PCA and MDS produce smoother and more continuous scatter distributions than other DR techniques. During the training process, the spatial projection views of the four DR techniques can reflect the separated projection results of the sample points gradually become better. However, only the projection transformation of PCA matches the magnitude of actual activation changes, which also reflects the advantages of the linear DR techniques. Additionally, we use the same method to obtain discrete symbol datasets and project various classes of samples. The results depicted in Fig. 12 demonstrate that t-SNE and UMAP yield superior separation effects in this context.

The importance analysis of input features. As shown in Table 6, we calculate the correlation coefficients between input and predicted variables. $X_2$ has the strongest correlation with the predicted parameter, followed by $X_4$ or $X_6$. Table 5 shows the statistical results of the feature importance by DR techniques with different parameter settings. We find that the stronger the correlation between the input variables and the predicted variable, the greater their feature importance, especially in DR techniques with a strong ability to preserve the global neighborhood. Therefore, consistent with the previous experiments, PCA and MDS perform better in this task.

Table 6 Statistical results of the correlation coefficient between the input variables and the predicted variable

The exploration of temporal regularity. After observing the temporal projection views of numerous samples, we noticed that PCA and MDS exhibit more low-quality inflection points, making their projection trajectories less smooth compared to t-SNE and UMAP. Figure 13 also confirms that the appearance of inflection points is associated with the local neighborhood preservation ability of these techniques, which highlights the advantage of t-SNE and UMAP in this task.

Fig. 13

Comparison result of the deviation degree of nodes with different local errors in the temporal trajectory

Since we selected a new dataset and model structure in the experiment, the activation pattern characteristics of the hidden layers we observed have also changed to a certain extent. Nevertheless, the DR techniques still show similar performance to previous experiments in LSTM visual analysis tasks, which reinforces the generalizability of the guidelines proposed in this paper.