Clustering Mixed Data Based on Density Peaks and Stacked Denoising Autoencoders

Abstract

With the universal existence of mixed data with numerical and categorical attributes in real world, a variety of clustering algorithms have been developed to discover the potential information hidden in mixed data. Most existing clustering algorithms often compute the distances or similarities between data objects based on original data, which may cause the instability of clustering results because of noise. In this paper, a clustering framework is proposed to explore the grouping structure of the mixed data. First, the transformed categorical attributes by one-hot encoding technique and normalized numerical attributes are input to a stacked denoising autoencoders to learn the internal feature representations. Secondly, based on these feature representations, all the distances between data objects in feature space can be calculated and the local density and relative distance of each data object can be also computed. Thirdly, the density peaks clustering algorithm is improved and employed to allocate all the data objects into different clusters. Finally, experiments conducted on some UCI datasets have demonstrated that our proposed algorithm for clustering mixed data outperforms three baseline algorithms in terms of the clustering accuracy and the rand index.

1. Introduction

Clustering analysis, an important unsupervised data analysis and data mining approaches, has been studied extensively and applied successfully to various domains, such as gene expression analysis [1,2], fraud detection [3], imagine segmentation [4], and document mining [5,6]. The basic clustering algorithms mainly group data objects into different clusters based on the similarities between objects in original data space, making objects in the same cluster are more similar while those in different clusters are more dissimilar. Many traditional clustering approaches, such as k-means [7], DBSCAN [8], spectral clustering [9,10], can only handle datasets with numerical attributes. However, the partition of mixed data objects with both categorical attributes and numerical attributes is indispensable in various fields, such as economics, finance, politics, and clinical medicine. In these cases, we need to consider simultaneously numerical attributes and categorical attributes, which provide more useful information and are helpful to find the potential grouping structure. For example, Italian local labor market areas can be identified by employing a min-cut approach based on iterative partitioning of the transitions graph [11]. Patients with heart disease can be recognized by analyzing various features consisting of categorical attributes and numerical attributes. Bank credit officers can decide to approve or reject the credit card applications depending on the mixed attribute values of credit card applicants. In these practical applications, it is crucial that how to transform categorical attributes into numerical values with keeping useful information as much as possible. In order to conduct clustering the mixed data better, more and more clustering algorithms have been proposed, such as k-prototypes [12], OCIL [13], DP-MD-FN [14], and so on.

A simple approach to clustering mixed data is transformation-based method, which converts one type of attribute values to other type ones, then employs corresponding clustering algorithms for single type of attributes, such as conceptual k-means [15] and DSqueezer [16]. The most popular and classical partition-based clustering algorithm for mixed data was k-prototypes proposed by Huang [12], which was the integration of k-means and k-modes [17]. The variants of k-prototypes, such as fuzzy k-prototype [18] and KL-FCM-GM [19], were presented to improve k-prototypes by addressing the uncertainty of mixed data. Density peaks clustering (referred to as DPC) [20] is an effective clustering algorithm based on local density and relative distance. In order to make DPC algorithm handle mixed data, some algorithms were proposed based on different types of unified similarity or dissimilarity metrics for numerical and categorical attributes, such as DP-MD-FN [14], DPC-MD [21], and DPC-M [22]. In recent years, deep learning has attracted more attention in various fields and many methods have been investigated by integrating deep neural networks and clustering algorithms on numerical data, such as DEC [23], DBC [24], and DNC [25].

However, two main limitations exist in the clustering algorithms above:

(1)

DPC-based clustering algorithms for mixed data, such as DP-MD-FN [14], DPC-MD [21], and DPC-M [22], need to select the number of cluster centers by decision graph manually as DPC.

(2)

Most of deep learning-based clustering algorithms, such as DEC [23], DBC [24], and DNC [25], are only suitable for clustering numerical data with spherical distribution due to integration of the deep learning and k-means algorithm.

Due to the sparsity and discontinuity of the original mixed data space, many traditional clustering approaches for mixed data based on the distance or similarity in original data space are hard to find the true grouping structure of mixed data objects, especially in context with noise. The aim of this paper is to construct a continuous feature space for mixed data and present a clustering framework named DPC-SDAE based on this feature space. Firstly, one-hot encoding technique is employed to transform categorical attribute values into numerical ones with keeping useful information as much as possible. Secondly, based on the transformed categorical attribute values and normalized numerical attribute values, stacked denoising autoencoders are used to extract feature representations robust to noise for mixed data to construct a continuous feature space. Finally, the improved density peaks clustering algorithm is employed to perform clustering mixed data objects based on the feature space, which is suitable for clustering data objects with nonspherical distribution effectively.

The main contributions of this paper can be summarized as follows.

(1)

The DPC algorithm is improved by employing L-method to determine the number of cluster centers automatically, which overcomes the drawback of selecting the number of cluster centers by decision graph manually in DPC-based algorithms.

(2)

Based on numeralization of categorical attributes by the one-hot encoding technique, we propose a framework for clustering mixed data by integrating stacked denoising autoencoders and density peaks clustering algorithm, which can utilize the robust feature representations learned from SDAE to enhance the clustering quality and be also suitable for clustering data with nonspherical distribution based on density peaks clustering algorithm.

(3)

By conducting experiments on six datasets from UCI machine learning repository, we observed that our proposed clustering algorithm outperforms three baseline algorithms in terms of clustering accuracy and rand index, which also demonstrates that stacked denoising autoencoders can be applied to clustering small sample datasets effectively besides clustering large scale datasets.

The rest of this paper is organized as follows. In Section 2, some related works are reviewed. Section 3 describes our proposed methodology. The details of our experiments results and discussions are presented in Section 4, followed by a conclusion in Section 5.

2. Related Works

In this section, we mainly review related works on (1) clustering approaches to mixed data and (2) clustering algorithms based on deep learning. Based on analyzing the limitations of related works in detail, we present our DPC-SDAE clustering algorithm to address these limitations.

2.1. Clustering Approaches to Mixed Data

Most existing methods for clustering mixed data can be divided into three subsets—transformation-based clustering, partition-based clustering, and density-based clustering.

The transformation-based clustering method is the earliest type of clustering mixed data, which usually converts one type of attribute values to the other so that corresponding algorithms for clustering only single type data can be adopted. For example, H. Ralarbondrainy [15] encoded each categorical attribute value with a set of binary numerical values, then employed k-means algorithms to perform clustering analysis for mixed data on transformed categorical attributes and numerical attributes. To overcome the difficulty of measuring similarity degree of the above binary encoding, Hsu and Huang [26] proposed an incremental clustering method for mixed data based on distance hierarchy from domain knowledge. In addition, Zhang et al. [27] presented a method to transform categorical attributes into numerical representations through multiple transitive distance learning and embedding. In contrast, He et al. [16] presented the DSqueezer algorithm based on the discretization of all the numerical attributes, which could be utilized to cluster mixed data. Moreover, David and Averbuch [28] converted input data into categorical values according to the Calinski–Harabasz index and then reduced their dimensionalities in SpectralCAT, followed by spectral clustering algorithm conducted on. In general, either encoding categorical attributes into numerical ones or discretizing numerical attributes to categorical values may destroy the structure of original data space, inevitably causing loss of useful information, for instance, the difference between the values of each numerical attribute [29].

Different from transformation-based clustering algorithms, partition-based clustering algorithms attempt to group all the data objects into k clusters according to the unified distance of categorical and numerical attributes between each data object and each initialized cluster center firstly, where k is the preassigned number of clusters, and then they reallocate each data object to the cluster with its center nearest to the corresponding data object by an iterative way, until the partition remains unchanged or reaches the predefined maximum number of iterations. The most popular and classical partition-based clustering algorithm for mixed data is k-prototypes [12], presented by Huang in 1997, which is the integration of k-means for clustering numerical data and k-modes [17] for clustering categorical data. It mixes the means of numeric attributes and the modes of categorical attributes as the prototypes, then obtains the clustering results by alternatively updating the prototypes and group allocations of each data object based on a unified distance. This unified distance is a weighted distance of Euclidean distance for numerical attributes and Hamming distance for categorical attributes. To take into account the uncertainty of mixed data, Ji et al. proposed a fuzzy k-prototype clustering algorithm [18], which represented the prototype of a cluster with the integration of mean and fuzzy centroid, and improved the k-prototype algorithm based on a dissimilarity measure considering the significance of each attribute. To overcome convergence to the local optimum of k-prototype clustering algorithm, an evolutionary k-prototype (EKP) algorithm [30] was put forward to conduct clustering for mixed data with a framework of integrating evolutionary algorithm with k-prototype algorithm. Extensive experiments performed with the k-prototype algorithm and its variants have demonstrated that clustering results are determined greatly on the parameter γ, which is utilized to balance the importance degree of numerical attributes and categorical attributes. Nevertheless, determining an appropriate value of the parameter γ is a very tricky task for a common user. Instead of adjusting the attribute importance of different types by a parameter, Cheung and Jia [13] developed a parameter-free iterative clustering algorithm based on the concept of object-cluster similarity (OCIL), which adopted a unified similarity metric to conduct the clustering of mixed data. The KL-FCM-GM algorithm [19] provided a fuzzy c-means (FCM) type algorithm for clustering mixed data by constructing a fully probabilistic dissimilarity functional. In general, most of partition-based clustering algorithms are sensitive to the initial centroids or prototypes and only suitable for clustering mixed data with spherical distribution.

To circumvent the above problems of partition-based clustering algorithms, a wide variety of density-based algorithms are proposed to handle the clustering analysis of mixed data with nonspherical distribution. For example, EPDCA [31] extended DBSCAN [8], a well-known density-based clustering algorithm for numerical data, to cluster mixed data with integration of entropy and probability distribution. To address the difficulty of parameter specification in density-based clustering algorithms, Behzadi et al. [32] proposed a parameter-free modified DBSCAN clustering algorithm (MDBSCAN) for mixed data, which employed distance hierarchy and minimum description length. Density peaks clustering (referred to as DPC) [20] proposed by Rodriguez and Laio is an effective clustering algorithm based on local density and relative distance. The main idea of DPC algorithm is that the data objects with higher local density and being farther from other objects with a higher local density are chosen as cluster centers, followed by assigning each remaining data object to the same cluster as its nearest cluster center. To make DPC algorithm be applicable for clustering mixed data, DP-MD-FN [14] combined an entropy-based strategy with DPC and employed fuzzy neighborhood to calculate the local density-based on a uniform similarity measure. In addition, some researchers extended DPC by defining other different types of unified similarity or dissimilarity metrics to handle numerical and categorical attributes simultaneously, such as DPC-MD [21] and DPC-M [22]. However, any unified similarity or dissimilarity metric may not be effective for all mixed datasets, and defining a proper unified similarity or dissimilarity metric for a given dataset is not an easy task.

2.2. Clustering Algorithm Based on Deep Learning

With the development of deep learning technology, a variety of deep neural networks are applied to clustering analysis so as to improve the effect of clustering. For example, deep embedded clustering (DEC) [23] employed stacked denoising autoencoders [33] to extract clustering-oriented representations and iteratively refined clusters with an auxiliary target distribution, optimizing both feature representations and cluster assignments simultaneously. Discriminatively boosted clustering (DBC) [24] used a fully convolutional autoencoder to extract coarse image features, and then conducted a discriminatively boosted clustering with learned features and soft k-means. Clustering convolutional neural network (CCNN) [34] leveraged a convolutional neural network (CNN) to perform large-scale image clustering by optimizing cluster centroid updating and representation learning iteratively with stochastic gradient descent approach. Deep nonparametric clustering (DNC) [25] proposed a unified framework of learning unsupervised features by employing deep belief network (DBN) and nonparametric clustering with maximum margin. In recent years, deep generative models and their applications have become a hot topic in machine learning and pattern recognition, attracting the extensive interest of researchers. Among of them, variational autoencoder (VAE) [35] and generative adversarial network (GAN) [36] are the two most prominent models for classification tasks. Variational deep embedding (VaDE) [37] generalized VAE to perform clustering tasks by a mixture of Gaussians prior instead of the single Gaussian prior. Information maximizing generative adversarial network (InfoGAN) [38] was presented to cluster data based on adding an information-theoretic regularization to the optimizing objective of standard GAN. An unsupervised feature learning (UFL) with fuzzy ART (UFLA) [39] was proposed to perform clustering on the mixed data. A high-order k-means algorithm was proposed based on dropout deep learning models for heterogeneous data clustering in cyber–physical–social systems [40]. To save space of this paper, more details related to clustering algorithms based on deep learning can be found in [41,42]. Although clustering methods based on deep generative models can achieve more satisfactory effect by generating more samples to extract robust representations, they are not suitable for clustering large-scale data due to their high computation complexity. In general, most of existing clustering approaches based on deep learning are designed for clustering numerical data by employing partition-based clustering algorithms, which are only suitable for data with spherical distribution.

To overcome the limitations of clustering algorithms above, this paper propose an algorithm framework named DPC-SDAE for clustering mixed data by integrating stacked denoising autoencoders with density peaks clustering, which can utilize the robust feature representations learned from SDAE and the capability of clustering data with any distribution of density peaks clustering to enhance the clustering quality.

3. Methodology

In this section, we introduce our proposed algorithm for clustering mixed data in detail. First of all, our overall framework of DPC-SDAE is shown in Figure 1, then, the data preprocessing procedure is introduced, followed by extracting features by a stacked denoising autoencoders (SDAE). In addition, we improve density peaks clustering by utilizing an L-method to determine the number of cluster centers automatically. Finally, the proposed DPC-SDAE algorithm is summarized and its time complexity is analyzed.

As illustrated in Figure 1, the categorical attributes of mixed data are encoded into binary values by one-hot encoding technique, and the numerical attributes of mixed data are transformed to the normalized values using a min–max normalization technique. Then, the transformed values are input to the stacked denoising autoencoders to learn useful and robust feature representations. Based on these feature representations, the original mixed data can be clustered by our improved density peaks clustering algorithm to obtain corresponding clustering results, i.e., group data objects into corresponding subsets.

3.1. Data Prepocessing

Let $X=\{x_1,x_2,\dots ,x_{\mathit{N}}\}$ denote a dataset consisting of $\mathit{N}$ objects and $x_i=\left({\mathit{x}}_{\mathit{i},1}^{\mathit{c}},{\mathit{x}}_{\mathit{i},2}^{\mathit{c}},\dots ,{\mathit{x}}_{\mathit{i},\mathit{p}}^{\mathit{c}},{\mathit{x}}_{\mathit{i},\mathit{p}+1}^{\mathit{r}},{\mathit{x}}_{\mathit{i},\mathit{p}+2}^{\mathit{r}},\dots ,{\mathit{x}}_{\mathit{i},\mathit{m}}^{\mathit{r}}\right)$ be the ith data object represented by $\mathit{p}$ categorical attribute values ${\mathit{x}}_{\mathit{i},1}^{\mathit{c}},{\mathit{x}}_{\mathit{i},2}^{\mathit{c}},\dots ,{\mathit{x}}_{\mathit{i},\mathit{p}}^{\mathit{c}}$ and $\mathit{m}-\mathit{p}$ numerical attribute values ${\mathit{x}}_{\mathit{i},\mathit{p}+1}^{\mathit{r}},{\mathit{x}}_{\mathit{i},\mathit{p}+2}^{\mathit{r}},\dots ,{\mathit{x}}_{\mathit{i},\mathit{m}}^{\mathit{r}}$ for $1\le \mathit{i}\le \mathit{N}.$

To facilitate extracting useful feature representations from original mixed data by stacked denoising autoencoders, we should transform categorical attribute values into binary numerical ones and normalize numerical attribute values at first.

3.1.1. Encoding of Categorical Attributes

As a simple and effective encoding technique, one-hot encoding is the most popular approach to handling numeralization of categorical attributes [43]. It can convert each value of categorical attributes to a binary vector, whose element with value 1 is only one, the others are zeros. The element with value 1 indicates the presence of corresponding possible value of a categorical attribute. For example, assuming that the categorical attribute color has only three possible values—e.g., red, green, and blue—with one-hot encoding, the red can be encoded to $\left(1,0,0\right)$, the green can be converted to $\left(0,1,0\right)$, while the blue can be transformed into $\left(0,0,1\right)$.

The jth categorical attribute ${\mathit{A}}_{\mathit{j}}\left(1\le \mathit{j}\le \mathit{p}\right)$ with ${\mathit{n}}_{\mathit{j}}$ possible values, can be encoded by a one-hot encoding mapping, as illustrated in (1).

$${\mathit{f}}_{\mathit{j}}^{\mathit{c}}:{\mathit{A}}_{\mathit{j}}\to B^{{\mathit{n}}_{\mathit{j}}}$$

(1)

where $B^{{\mathit{n}}_{\mathit{j}}}$ denotes a set containing binary vectors with ${\mathit{n}}_{\mathit{j}}$ dimensionality and each element is composed of a single one and ${\mathit{n}}_{\mathit{j}}-1$ zeros.

After one hot encoding, the jth categorical attribute value ${\mathit{x}}_{\mathit{i},\mathit{j}}^{\mathit{c}}$ of the ith data object will be transformed into a binary row vector $z_{\mathit{i},\mathit{j}}^{}$ with ${\mathit{n}}_{\mathit{j}}$ dimensionality, where $z_{\mathit{i},\mathit{j}}^{}={\mathit{f}}_{\mathit{j}}^{\mathit{c}}({\mathit{x}}_{\mathit{i},\mathit{j}}^{\mathit{c}}).$ By combining the encoding vectors $z_{1,\mathit{j}},z_{2,\mathit{j}},\dots ,z_{\mathit{N},\mathit{j}}$ of $\mathit{N}$ objects of a categorical attribute ${\mathit{A}}_{\mathit{j}}\left(1\le \mathit{j}\le \mathit{p}\right)$ from top to bottom, we can get a $\mathit{N}\times {\mathit{n}}_{\mathit{j}}$ matrix $Z_{\mathit{j}}.$ Finally, we concatenate $Z_1,Z_2,\dots ,Z_{\mathit{p}}$ to a $\mathit{N}\times {\displaystyle \sum _{\mathit{j}=1}^{\mathit{p}}{\mathit{n}}_{\mathit{j}}}$ matrix $Z=\left[Z_1,Z_2,\dots ,Z_{\mathit{p}}\right]$, which can be viewed as the encoded matrix of all categorical attribute values of original dataset.

3.1.2. Normalization of Numerical Attributes

Usually, different numerical attributes have different magnitudes or units, which may greatly affect the value of similarity or dissimilarity between two distinct data objects. The larger the magnitude of a numerical attribute is, the more the attribute contributes to the similarity or dissimilarity between two data objects. To eliminate the effect of discrepancy in magnitude between different numerical attributes, numerical attribute values should be normalized. One of popular normalization methods is min–max normalization, which can linearly transform each numerical attribute value into a value between 0 and 1.

For a numerical attribute ${\mathit{A}}_{\mathit{j}}$ $\left(\mathit{p}+1\le \mathit{j}\le \mathit{m}\right)$, the jth attribute value ${\mathit{x}}_{\mathit{i},\mathit{j}}^{\mathit{r}}$ of the ith data object $x_{\mathit{i}}$ can be normalized by min–max normalization, as shown in (2).

$${\mathit{y}}_{\mathit{i},\mathit{j}}^{\mathit{r}}=\frac{{\mathit{x}}_{\mathit{i},\mathit{j}}^{\mathit{r}}-{\mathit{x}}_{\min ,\mathit{j}}^{\mathit{r}}}{{\mathit{x}}_{\max ,\mathit{j}}^{\mathit{r}}-{\mathit{x}}_{\min ,\mathit{j}}^{\mathit{r}}}$$

(2)

where, ${\mathit{y}}_{\mathit{i},\mathit{j}}^{\mathit{r}}$ denotes the normalized numerical attribute value of ${\mathit{x}}_{\mathit{i},\mathit{j}}^{\mathit{r}},$ ${\mathit{x}}_{\max ,\mathit{j}}^{\mathit{r}}$ and ${\mathit{x}}_{\min ,\mathit{j}}^{\mathit{r}}$ represent the maximum and minimum value of numerical attribute ${\mathit{A}}_{\mathit{j}}$ among all the data objects, respectively.

After normalization of numerical attribute values of $\mathit{N}$ objects, we can arrange all the normalized values into a $\mathit{N}\times \left(\mathit{m}-\mathit{p}\right)$ matrix $Y={\left[{\mathit{y}}_{\mathit{i},\mathit{j}}^{\mathit{r}}\right]}_{\mathit{N}\times \left(\mathit{m}-\mathit{p}\right)}$, which represents the normalized matrix of all numerical attribute values of original dataset.

Let $D=\left[Z,Y\right]={\left(d_1,d_2,\dots ,d_{\mathit{N}}\right)}^{\mathit{T}}$ denotes the transformed data matrix with the size of $\mathit{N}\times \left(\mathit{m}-\mathit{p}+{\displaystyle \sum _{\mathit{j}=1}^{\mathit{p}}{\mathit{n}}_{\mathit{j}}}\right)$, which $d_{\mathit{i}}\left(1\le \mathit{i}\le \mathit{N}\right)$ refers to the transformed attribute vector of the ith data object. The transformed attribute vectors of all data objects can input to stacked denoising autoencoders to extract feature representations instead of original dataset.

3.2. Feature Extraction by Stacked Denoising Autoencoders

3.2.1. AutoEncoder (AE)

First, we introduce the principle of traditional autoencoder (AE) [44,45], which is the base of denoising autoencoder (DAE) [33]. The typical structure of autoencoder (AE) can be illustrated by Figure 2.

As shown in Figure 2, a traditional autoencoder is a simple neural network with three layers containing an input layer, representation layer, and reconstruction layer. Different from the common neural network, the output of AE is the reconstruction of input data, with the training objective to minimize the reconstruction error.

Let $a$ denote input vector of a data object, which can be encoded to a feature representation $b$ by a nonlinear activation function ${\mathit{f}}_{\theta }$, as shown in (3).

$$b={\mathit{f}}_{\theta }\left(a\right)$$

(3)

Then, another nonlinear function ${\mathit{g}}_{{\theta }^{\prime }}$ can be employed to decode the feature representation $b$ to a vector $\widehat{a}$, the reconstruction of the input vector $a$, as illustrated in (4).

$$\widehat{a}={\mathit{g}}_{{\theta }^{\prime }}\left(b\right)$$

(4)

The parameters $\theta$ and ${\theta }^{\prime }$ are optimized according to the principle of minimizing the sum of reconstruction error of all data objects.

3.2.2. Stacked Denoising Autoencoders (SDAE)

A denoising autoencoder (DAE) [33], a simple variant of AE, was designed to acquire more robust representations by learning from a corrupted input with adding some noise into initial input, such as Gaussian noise, or some values being setting to zeros. Stacked denoising autoencoders (SDAE), has succeeded in a variety of application domains, such as the user response prediction based on encoded categorical attributes by one-hot technique [43] and acoustic object recognition [46]. Let $\tilde{d}$ denote a corrupted version of an initial input vector $d.$ Similar to AE, $\tilde{d}$ can be encoded to a representation $h$ by ${\mathit{f}}_{\theta }$, then the reconstruction vector $\widehat{d}$ of the uncorrupted input vector can be obtained from $h$ by ${\mathit{g}}_{{\theta }^{\prime }}$.

Giving the initial values at random, the parameters $\theta$ and ${\theta }^{\prime }$ can be optimized with adaptive gradient algorithm (Adagrad) [47] by minimizing the sum of reconstruction error between reconstructed vector ${\widehat{d}}_{\mathit{i}}$ and corresponding uncorrupted input vector $d_{\mathit{i}}$ over entire dataset, as shown in (5).

$$\mathrm{E}={\displaystyle \sum {\Vert d_{\mathit{i}}-{\widehat{d}}_{\mathit{i}}\Vert }_2^2}={\displaystyle \sum {\Vert d_{\mathit{i}}-{\mathit{g}}_{{\theta }^{\prime }}\left({\mathit{f}}_{\theta }\left({\tilde{d}}_{\mathit{i}}\right)\right)\Vert }_2^2}$$

(5)

With optimized parameters $\theta$, we can calculate the first feature representation $h_1$ of each input vector $d$ by ${\mathit{f}}_{\theta }$.Then, by taking $h_1$ as the input of second DAE and discarding the reconstruction layer, the second feature representation $h_2$ can be acquired using optimized parameters. In this way, stacked denoising autoencoders (SDAE) can be constructed by stacking some DAEs, with taking the feature representation vector of previous DAE as the input vector of the next DAE and discarding the reconstruction layer of previous DAE after layer-wise training. In the SDAE, each feature representation layer is corresponding to a hidden layer of common neural network. A typical structure of SDAE can be illustrated as Figure 3.

By adding noise to the uncorrupted input vector $d$, the vector $d$ is transformed into corrupted input $\tilde{d}$, which can be mapped into the first feature representation $h_1$ by training the first DAE. By taking $h_1$ as the uncorrupted input vector of the second DAE, the second feature representation $h_2$ can be acquired in a similar way. Similarly, feature representations $h_3,\dots ,h_{\mathit{L}}$ can also be obtained, where $L$ is the number of feature representation layers, i.e., the number of hidden layers. As Zhang et al. suggested in [43], the optimal number of hidden layers in SDAE can be set at three, which is also validated by our experiment. The feature representations extracted by SDAE will deteriorate when the number of hidden layers is too big or too small.

In the SDAE, besides the optimization layer by layer, the parameters ${\theta }_{\mathit{i}}(1\le \mathit{i}\le \mathit{L})$ can be fine-tuned simultaneously to minimize the sum of reconstruction error between the uncorrupted preprocessed data vectors and corresponding representation vectors in the $\mathit{L}\mathrm{th}$ representation layer using the back propagation algorithm [48], where $\mathit{L}$ is the number of representation layers. In addition, dropout noise [49] is used to prevent our SDAE from overfitting. By reconstructing a repaired input from an input with noise, the feature representations extracted by SDAE become more robust to noise [33].

3.2.3. Feature Construction for Clustering

After layer-wise optimization and fine-tuning of SDAE mentioned above, the optimal values of parameter ${\theta }_{\mathit{i}}(1\le \mathit{i}\le \mathit{L})$ can be acquired. For each data object $x_i$, we can calculate its corresponding representation vectors of $\mathit{L}$ layers,$h_1^{\mathit{i}},h_2^{\mathit{i}},\dots ,h_{\mathit{L}}^{\mathit{i}}$, based on these optimal values of parameters ${\theta }_{\mathit{i}}(1\le \mathit{i}\le \mathit{L})$.

As Shelhamer et al. illustrated in [50], we can concatenate the representation vectors of $\mathit{L}$ layers and express the clustering features of data object $\mathit{i}$ as $f_{\mathit{i}}$ in (6),

$$f_{\mathit{i}}=\left(h_1^{\mathit{i}},h_2^{\mathit{i}},\dots ,h_{\mathit{L}}^{\mathit{i}}\right)$$

(6)

where $\mathit{i}=1,2,\dots ,\mathit{N}.$

Consequently, the original mixed dataset $X$ can be converted to a real feature matrix $F$, as shown in (7).

$$F={\left(f_1,f_2,\dots ,f_{\mathit{N}}\right)}^T$$

(7)

After preprocessing and feature extraction, the original mixed data space are transformed into a real Euclidean space, in which traditional numerical clustering algorithms can be employed to cluster mixed data objects. In this paper, due to its effectivity and insensitivity to data distribution, the modified density peaks clustering (DPC) method was chosen to perform our clustering tasks.

3.3. Density Peak Clustering and Its Improvement

The core of density peaks clustering (DPC) [20] is the determination of the cluster centers, which are data objects with high density and large local distance from other data objects having higher density. For each data object, we firstly define the distance between two data objects so as to calculate the density and local distances from other data objects with higher density. Let $f_{\mathit{i}}$ and $f_{\mathit{j}}$ denote clustering features of data object $x_{\mathit{i}}$ and data object $x_{\mathit{j}}$ respectively, then distance ${\mathit{d}}_{\mathit{i}\mathit{j}}\left(1\le \mathit{i}\le \mathit{N},1\le \mathit{j}\le \mathit{N}\right)$ can be defined by (8).

$${\mathit{d}}_{\mathit{i}\mathit{j}}={\Vert f_{\mathit{i}}-f_{\mathit{j}}\Vert }_2$$

(8)

Giving a neighborhood ratio $\mathit{t}$ in DPC, the cutoff distance ${\mathit{d}}_{\mathit{c}}$ can be found from sorted distances of all object pairs. Then, we calculate the local density ${\rho }_{\mathit{i}}$ of data object $x_{\mathit{i}}\left(1\le \mathit{i}\le \mathit{N}\right)$ by (9).

$${\rho }_{\mathit{i}}={\displaystyle \sum _{\mathit{j}\ne \mathit{i}}^{}\exp \left(-\frac{{\mathit{d}}_{\mathit{i}\mathit{j}}^2}{{\mathit{d}}_{\mathit{c}}^2}\right)}$$

(9)

The relative distance ${\delta }_{\mathit{i}}$ of data object $x_{\mathit{i}}$ can be defined by (10).

$${\delta }_{\mathit{i}}=\{\begin{array}{cc}\underset{\mathit{j}:{\rho }_{\mathit{j}}>{\rho }_{\mathit{i}}}{\min }\left\{{\mathit{d}}_{\mathit{i}\mathit{j}}\right\},\hfill & \hfill \mathrm{if}\left\{\mathit{j}:{\rho }_{\mathit{j}}>{\rho }_{\mathit{i}}\right\}\ne \varphi \\ \underset{\mathit{j}}{\max }\left\{{\delta }_{\mathit{j}}\right\},\hfill & \hfill \mathrm{otherwise}\end{array}$$

(10)

After calculating local densities and relative distances of all data objects, we can identify the cluster centers manually with the help of a decision graph in DPC [20], a scatter plot of local density vs. relative distance. Usually, the data objects with anomalously large local densities and relative distances are recognized as cluster centers, which always locate at the upper right hand corner of the decision graph. For some complex datasets, DPC can draw a $\gamma$ graph of ${\gamma }_{\mathit{i}}\left(={\rho }_{\mathit{i}}{\delta }_{\mathit{i}}\right)$ (in descending order) vs. $\mathit{i}$ to aid users to select cluster centers. However, this human-based selection of cluster centers is a tremendous barrier in intelligent analysis of data [51]. Thus, we present an automatic parameter-free selecting cluster centers to improve DPC.

In order to eliminate the effect of the local density and relative distance in different magnitudes, we normalize them by (11) and (12), respectively.

$${\mu }_{\mathit{i}}=\frac{{\rho }_{\mathit{i}}}{\underset{1\le \mathit{j}\le \mathit{N}}{\max }\left\{{\rho }_{\mathit{j}}\right\}}$$

(11)

$${\nu }_{\mathit{i}}=\frac{{\delta }_{\mathit{i}}}{\underset{1\le \mathit{j}\le \mathit{N}}{\max }\left\{{\delta }_{\mathit{j}}\right\}}$$

(12)

To facilitate determination of the cluster number automatically, we introduce ${\gamma }_{\mathit{i}},$ which can be calculated by (13),

$${\gamma }_{\mathit{i}}={\mu }_{\mathit{i}}\cdot {\nu }_{\mathit{i}}$$

(13)

Generally speaking, the larger the value of ${\gamma }_{\mathit{i}}$ is, the more likely the data object $x_{\mathit{i}}$ becomes a cluster center. Thus, we can sort all data objects by ${\gamma }_{\mathit{i}}$ in descending order and draw an evaluation graph of sorted ${\gamma }_{\mathit{i}}$ vs. $\mathit{i},$ then choose the first $\mathit{k}$ data objects sorted in descending order according to $\gamma$ as cluster centers, which $\mathit{k}$ is determined by L-method [52,53] automatically.

The L-method is an approach to finding the knee point from a series of points by fitting two straight lines with the points on the left side of candidate knee and on the right side of it respectively according to the principle of minimizing the total root mean squared error. The intersection point of two straight lines fitted most closely can be viewed as the knee point, which can be used to determine the number of cluster centers. For data points with a long tail, the knee point found by L-method may be inconsistent with the true knee due to the serious imbalance on two sides of candidate knee point in the number of data points. To overcome this issue, a method for iterative refinement of the knee point [39] was proposed to adjust focus region iteratively so that L-method could find the true knee point in a few iterations.

To overcome the drawback of selecting the number of cluster centers by decision graph manually in DPC-based algorithms, we employed the L-method with iterative refinement to determine the cluster number automatically based on sorted $\gamma$ values of all data objects. For example, iterative refinement of the knee point requires only 3 iterations to obtain the final cluster number on Credit dataset with 653 data points in our experiment, as shown in Figure 4, Figure 5 and Figure 6, respectively.

After determination of the cluster centers, each remaining data object can be assigned to a cluster, whose cluster center is nearest to the corresponding data object.

3.4. Algorithm Summarization and Time Complexity Analysis

The proposed DPC-SDAE algorithm can be summarized as the following Algorithm 1.

Algorithm 1. DPC-SDAE algorithm
Inputs:	mixed dataset $X$ consisting of N data objects, the initial value of encoding parameters $\theta$ and decoding parameters ${\theta }^{\prime }$, and cutoff distance ${\mathit{d}}_{\mathit{c}}.$
Outputs:	cluster label vector $c=\left[{\mathit{c}}_1,{\mathit{c}}_2,\dots ,{\mathit{c}}_{\mathit{N}}\right].$
Steps: 1.	Transform categorical attributes into a binary matrix $Z$ by one-hot encoding.
2.	Normalize numerical attributes by (2) and concatenate with the binary matrix $Z$ to obtain a matrix $\mathit{D}={\left(d_1,d_2,\dots ,d_{\mathit{N}}\right)}^{\mathit{T}}$.
3.	Input $\mathit{D}$ to SDAE with the initial value of $\theta$ and ${\theta }^{\prime }$ to extract clustering features and construct a normalized feature matrix $F={\left(f_1,f_2,\dots ,f_{\mathit{N}}\right)}^{\mathit{T}}.$
4.	Calculate distances between objects based on $F$ by (8).
5.	Calculate and normalize the local densities of data objects based on cutoff distance ${\mathit{d}}_{\mathit{c}}$ by (9) and (11).
6.	Calculate and normalize the relative distances of data objects by (10) and (12).
7.	Calculate $\gamma$ by (13) and sort data objects by $\gamma$ in descending order.
8.	Determine the number of clusters $\mathit{k}$ with L-method and select $\mathit{k}$ data objects with larger $\gamma$ as cluster centers.
9.	Assign each remaining data object a cluster label as the same as its nearest cluster center.
10.	Return the cluster label vector $c=\left[{\mathit{c}}_1,{\mathit{c}}_2,\dots ,{\mathit{c}}_{\mathit{N}}\right].$

The time complexity of our proposed DPC-SDAE algorithm is illustrated as follows. To keep consistency of notations with Algorithm 1, we let $\mathit{N}$ denote the number of data objects, $\mathit{p}$ is the number of categorical attributes, $\mathit{m}-\mathit{p}$ is the number of numerical attributes, $\mathit{s}$ is the average number of distinct categorical attribute values, $\mathit{L}$ is the number of hidden layers in SDAE, and ${\mathit{N}}_0,{\mathit{N}}_1,{\mathit{N}}_2,\dots ,{\mathit{N}}_{\mathit{L}}$ represent the number of the neural units in input layer and $\mathit{L}$ hidden layers, and ${\mathit{I}}_{\mathit{m}\mathit{a}\mathit{x}}$ denotes the maximum number of iterations for adaptive gradient algorithm in each layer. The cost of data preprocessing is $\mathrm{O}(\mathit{N}(\mathit{p}\mathit{s}+\mathit{m}-\mathit{p}))$ and extracting features by SDAE requires $\mathrm{O}(\mathit{N}{\mathit{I}}_{\mathit{m}\mathit{a}\mathit{x}}{\displaystyle \sum _{\mathit{l}=1}^{\mathit{L}}{\mathit{N}}_{\mathit{l}-1}{\mathit{N}}_l})$. There are $\mathrm{O}({\mathit{N}}^2)$ to calculate distances between data objects, local densities, and relative distances of all data objects. The quick sorting of local densities needs $\mathrm{O}(\mathit{N}\log \mathit{N})$ and the cost of using L-method to determine the cluster number and assignment of remaining data objects to their clusters is $\mathrm{O}(\mathit{N}).$ Therefore, the total time cost of our proposed DPC-SDAE algorithm in this paper is $\mathrm{O}(\mathit{N}(\mathit{p}\mathit{s}+\mathit{m}-\mathit{p}))+\mathrm{O}\left(\mathit{N}{\mathit{I}}_{\mathit{m}\mathit{a}\mathit{x}}{\displaystyle \sum _{\mathit{l}=1}^{\mathit{L}}{\mathit{N}}_{\mathit{l}-1}{\mathit{N}}_{\mathit{l}}}\right)+\mathrm{O}({\mathit{N}}^2)+\mathrm{O}(\mathit{N}\log \mathit{N})$$+\mathrm{O}(\mathit{N}).$ As it usually has $\mathit{p}\mathit{s}+\mathit{m}-\mathit{p}\ll \mathit{N}$, the overall time complexity of our DPC-SDAE algorithm can be viewed as $\mathrm{O}(\mathit{N}{\mathit{I}}_{\mathit{m}\mathit{a}\mathit{x}}{\displaystyle \sum _{\mathit{l}=1}^{\mathit{L}}{\mathit{N}}_{\mathit{l}-1}{\mathit{N}}_{\mathit{l}}})+\mathrm{O}({\mathit{N}}^2)$.

4. Results and Discussion

In this section, we show our experiments on some datasets from UCI machine learning repository [54] to validate our proposed DPC-SDAE algorithm by comparing with OCIL [13], k-prototypes [12], and DPC-M [22]. In addition, the evaluation indexes adopted and the impact of five hyperparameters on clustering quality are also introduced. All experiments were conducted on a laptop with Intel Core i5-5257U 2.7 GHz CPU and 8 GB RAM implementing MATLAB 2013a with Windows 8 operating system.

4.1. Datasets

To illustrate the extensive applications of our proposed method, the datasets in our experiments contain one pure numerical attribute dataset, one pure categorical dataset, and four mixed datasets, whose basic information are shown in

Table 1. Basic information about datasets ¹.

Dataset	N	Mc	Mr	C
Iris	150	0	4	3
Vote	435-203	16	0	2
Credit	690-37	9	6	2
Abalone	4177-9	1	7	21
Dermatology	366-8	33	1	6
Adult_10000	10000	8	6	2

¹ ‘N’ denotes the number of data objects; ‘Mc’ represents the number of categorical attributes; ‘Mr’ signifies the number of numerical attributes; ‘C’ indicates the number of classes.

.

Among six datasets, Iris is a pure numerical attribute dataset and Vote is a pure categorical attribute dataset, whereas the other four datasets are composed of mixed data with both categorical attributes and numerical attributes. Before performing experiments, the data objects containing missing values were removed from corresponding datasets, for example, eight data objects with missing values are eliminated from 366 data objects in Dermatology dataset. In Abalone dataset, nine data objects are deleted as outliers due to that they belong to the classes consisting of only one or two data objects. Adult_10,000 is a subset of Adult containing 10,000 data objects selected randomly after the removal of data objects containing missing values.

4.2. Evaluation Indexes

In order to evaluate the clustering quality, we introduced two evaluation indexes for clustering: clustering accuracy (ACC) [13,14,55] and the rand index (RI) [39,56]. These two indexes are used extensively to validate the clustering algorithms.

Let $\mathit{N}$ stand for the number of data objects in the dataset, $c=\left({\mathit{c}}_1,{\mathit{c}}_2,\dots ,{\mathit{c}}_{\mathit{N}}\right)$ denotes the cluster label vector acquired from the clustering algorithm, and $b=\left({\mathit{b}}_1,{\mathit{b}}_2,\dots ,{\mathit{b}}_{\mathit{N}}\right)$ represents the ground truth label vector of all data objects. The ACC can be calculated by (14).

$$\mathrm{ACC}=\frac{{\displaystyle \sum _{\mathit{j}=1}^{\mathit{N}}\delta \left({\mathit{b}}_{\mathit{j}},\mathrm{map}\left({\mathit{c}}_{\mathit{j}}\right)\right)}}{\mathit{N}}$$

(14)

where $\mathrm{map}(\cdot )$ is the optimal mapping function obtained by Hungarian algorithm [57], which maps each cluster label into a ground truth label, and $\delta \left(\mathit{x},\mathit{y}\right)$ is the Kronecker delta function, whose value equals 1 if $\mathit{x}=\mathit{y}$ and 0 otherwise. In general, the larger the value of ACC is, the higher the clustering quality.

The second index to measure the clustering quality is rand index (RI) based on the relations between pairwise data objects from two partitions of the same dataset. Let $\mathit{P}$ and $\mathit{Q}$ be the ground truth partition and a clustering partition of a dataset with $\mathit{N}$ data objects, respectively. Let ${\mathit{a}}_1$ denote the number of pairs of data objects that belong to the same class in $\mathit{P}$ and the same cluster in $\mathit{Q}$, ${\mathit{a}}_2$ stands for the number of pairs of data objects that are placed in the different classes in $\mathit{P}$ and the different clusters in $\mathit{Q}$, ${\mathit{a}}_3$ denotes the number of pairs of data objects that are in the same class in $\mathit{P}$ and belong to the different clusters in $\mathit{Q}$, and ${\mathit{a}}_4$ represents the number of pairs of data objects that belong to the different classes in $\mathit{P}$ and are in the same cluster in $\mathit{Q}$. RI can be calculated by (15).

$$\mathrm{RI}=\frac{{\mathit{a}}_1+{\mathit{a}}_2}{{\displaystyle \sum _{\mathit{j}=1}^{\mathit{N}}{\mathit{a}}_{\mathit{j}}}}$$

(15)

Similar to ACC, the larger the RI is, the better the clustering result.

4.3. Clustering Results and Analysis

In the experiments, the clustering performance of our proposed DPC-SDAE clustering algorithm is compared with k-prototypes [12], OCIL [13], and DPC-M [22] algorithms on six different datasets. Among them, k-prototypes is the most classical clustering algorithm for mixed data, and OCIL is an efficient partition-based clustering algorithm being free of certain parameters, while DPC-M proposed in 2017 is an algorithm based on DPC for clustering mixed data by defining a united distance for categorical and numerical attributes. With the exception of the DPC-M algorithm, the clustering performance of the remaining three algorithms was affected by the initial cluster prototypes or initial parameters selected randomly. To eliminate the impact of random initialization, we executed each algorithm thirty times on each dataset and investigated them statistically except DPC-M, i.e., took into account the mean and standard deviation of evaluation index values. In k-prototypes and OCIL, the cluster number k was set at the number of actual classes. The adjustment parameter $\gamma$ in k-prototypes was set at a half of the average standard deviation of numerical attribute values, as the same as in [13]. The k-means was conducted on pure numerical attribute Iris dataset instead of k-prototypes, whereas k-prototypes was replaced by k-modes on pure categorical attribute Vote dataset. In DPC-M algorithm, the cutoff distance ${\mathit{d}}_{\mathit{c}}$ was calculated based on the neighborhood ratio $\mathit{t}$ being 1.5% [22].

In our proposed DPC-SDAE algorithm, the structure of SDAE and hyperparameters are set as identically as possible on all datasets in our experiments. Specifically, three denoising autoencoders with the same number of hidden neural units were stacked to constitute SDAE to extract clustering features of each dataset. As each denoising autoencoder contains only one hidden layer, the number of hidden layers adopted in our DPC-SDAE algorithm is three unless otherwise stated. The sigmoid function was adopted as encoding and decoding functions, whose parameters was optimized by adaptive gradient algorithm (Adagrad) [47]. The important hyperparameters in DPC-SDAE algorithm were set as follows. The neighborhood ratio in DPC, the epsilon in Adagrad and the number of optimization epochs were set respectively at 4%, 10⁻⁸, and 100 on all datasets. The dropout ratio, the learning rate, and the number of neural units in the hidden layer were set at 0.2, 0.1, and the same as the size of input data on four small sample datasets, respectively. On Abalone and Adult_10000, the dropout ratio and the learning rate were set at 0.1 and 0.03, respectively. The number of neural units in hidden layer on these two datasets were set at 10 and 55, respectively.

Table 2. Clustering accuracy (ACC) ¹.

Dataset	DPC-SDAE	k-prototypes	OCIL	DPC-M
Iris	0.899 ± 0.078	0.793 ± 0.146	0.806 ± 0.094	0.860
Vote	0.891 ± 0.005	0.866 ± 0.004	0.887 ± 0.001	0.867
Credit	0.820 ± 0.010	0.748 ± 0.090	0.798 ± 0.101	0.809
Abalone	0.199 ± 0.009	0.165 ± 0.006	0.169 ± 0.007	0.194
Dermatology	0.806 ± 0.056	0.589 ± 0.079	0.680 ± 0.097	0.683
Adult_10000	0.716 ± 0.004	0.650 ± 0.047	0.709 ± 0.006	0.715

¹ the best performance are accentuated in bold.

and

Table 3. Clustering rand index (RI) ¹.

Dataset	DPC-SDAE	k-prototypes	OCIL	DPC-M
Iris	0.894 ± 0.049	0.828 ± 0.072	0.832 ± 0.041	0.852
Vote	0.806 ± 0.008	0.767 ± 0.005	0.804 ± 0.002	0.797
Credit	0.705 ± 0.013	0.638 ± 0.066	0.673 ± 0.071	0.690
Abalone	0.812 ± 0.012	0.754 ± 0.004	0.778 ± 0.005	0.799
Dermatology	0.933 ± 0.017	0.821 ± 0.051	0.856 ± 0.046	0.865
Adult_10000	0.593 ± 0.003	0.549 ± 0.023	0.589 ± 0.004	0.592

¹ the best performance are accentuated in bold.

provide the main results of our experiments. The values of ACC and RI corresponding to the algorithm with the best performance are accentuated in bold for each dataset.

As shown in Table 2, it is obvious that our proposed DPC-SDAE algorithm is much superior to other three clustering algorithms on five datasets in terms of ACC from the viewpoint of statistics. It can be observed that our proposed DPC-SDAE algorithm acquires a larger mean of ACC values than three benchmark algorithms on six datasets. For instance, the mean of ACC was improved by 21.7%, 12.6%, and 12.3% on the Dermatology dataset compared with k-prototypes, OCIL, and DPC-M, respectively. In addition, the standard deviation of ACC generated by the DPC-SDAE algorithm is much smaller on most of datasets, which means that the DPC-SDAE algorithm is more robust.

It can be observed from Table 3 that our proposed DPC-SDAE algorithm statistically performs better than the other three baseline clustering algorithms on six datasets in terms of IR. It can be seen that DPC-SDAE algorithm obtains much larger mean of IR values than three baseline algorithms on six datasets. In addition, the standard deviation of IR generated by DPC-SDAE algorithm is much smaller than that resulted from k-prototypes and OCIL on most of datasets, which signifies that DPC-SDAE algorithm has greater robustness. It is noteworthy that DPC-SDAE algorithm increases the IR onsix datasets. For instance, the mean of IR is improved by 11.2%, 7.7%, and 6.8% on the Dermatology dataset compared with k-prototypes, OCIL, and DPC-M, respectively.

The main reason for the higher clustering quality and more robust results achieved by our proposed DPC-SDAE algorithm is that the feature representations extracted by SDAE are more robust and discriminating than original data attributes. Moreover, the subsequent DPC algorithm adopted by DPC-SDAE can further improve clustering performance, especially on datasets with nonspherical distribution.

4.4. Discussion on Hyperparameters in DPC-SDAE Algorithm

In this part, we discuss the relations between clustering quality and five hyperparameters in DPC-SDAE algorithm, performing on the Credit dataset as an example. In our experiment, we vary only one hyperparameter by fixing other hyperparameters every time.

4.4.1. Dropout Ratio

Dropout is a technique to address overfitting of stacked denoising autoencoders by randomly dropping units (along with their connections) from the deep neural network during training [49]. As an important index for describing the degree of dropout, the dropout ratio plays a key role in the DPC-SDAE algorithm.

We varied the dropout ratio from 0 to 0.9 with an increment of 0.1 on the condition that the learning rate, the number of neural units in each hidden layer, and the neighbor ratio were set at 0.1, 46, and 4%, respectively. As shown in Figure 7, we can observed that both average ACC and RI firstly go up significantly with the increasing of dropout ratio when dropout ratio is less than 0.3, then reach the maximum value point when the dropout ratio is at 0.4. Both the average ACC and RI deteriorated slightly when the dropout ratio was greater than 0.4. Thus, 0.4 can be viewed as the optimal dropout ratio on the Credit dataset.

4.4.2. Learning Rate

The learning rate is a hyperparameter to affect the speed of weights updating in optimization of encoding and decoding functions in SDAE. To investigate the impact of learning rate, we fixed the dropout ratio, the number of neural units in each hidden layer, and the neighbor ratio at 0.2, 46, and 4%, respectively. We investigated the relationship between evaluation indexes of DPC-SDAE algorithm and the learning rate.

From Figure 8, it can be obviously seen that the average ACC and RI improve dramatically when the learning rate is less than 0.1. When the learning rate lies between 0.1 and 0.4, average ACC and RI are almost unchanged. If we continued to increase the learning rate, both the average ACC and RI would go down significantly and have larger standard deviations when the learning rate is greater than 0.4. Taking into account the larger average value and the lower deviations in terms of ACC and RI, the optimal learning rate can be set at 0.1 on the Credit dataset.

4.4.3. Number of Neural Units in Hidden Layer

The hidden layer architecture is an important factor for extracting useful clustering features. According to the suggestion from [43], the SDAE consisting of three hidden layers with equal number of neural units was employed in our experiment. Thus, we need only take into account the impact of the number of neural units in hidden layer. Fixing the dropout ratio, the learning rate, and the neighbor ratio at 0.2, 0.1, and 4%, respectively, we can observe how ACC and RI change with different numbers of neural units in the hidden layer.

As illustrated in Figure 9, both average ACC and RI rise significantly with the increasing of the number of neural units in hidden layer at first, then change little when the number of neural units in hidden layer is greater than about 50. Thus, choosing 50 as the optimal number of neural units in hidden layer is appropriate on Credit dataset by jointly considering about evaluation indexes and computational complexity.

4.4.4. Neighbor Ratio

The cutoff distance—the important parameter in DPC-SDAE determined by the neighbor ratio—significantly affects the clustering quality. As before, we fixed the dropout ratio, the learning rate, and the number of neural units in each hidden layer at 0.2, 0.1, and 46, respectively.

By varying the percent of neighbors from 0.5% to 6% with an increment 0.5%, we can observe from Figure 10 that both the average ACC and RI change little when the neighbor ratio is less than 5% whereas they go down sharply with large deviation when the neighbor ratio is greater than 5%. Taking into account the larger average value and the lower deviations in terms of ACC and RI, the optimal neighbor ratio can be set at 4% on Credit dataset.

4.4.5. Number of Hidden Layers

The number of hidden layers also plays an important role in our DPC-SDAE algorithm. To investigate the impact of the number of hidden layers, we fixed the learning rate, the dropout ratio, the number of neural units in each hidden layer, and the neighbor ratio at 0.1, 0.2, 46, and 4%, respectively. We investigated the relationship between evaluation indexes of DPC-SDAE algorithm and the number of hidden layers.

As shown in Figure 11, with the number of hidden layers being changed from 1 to 10 with an increment 1, both average ACC and RI increase at first and then deteriorate slowly when the number of hidden layers is greater than three. Thus, the optimal number of hidden layers can be set at three on Credit dataset.

4.5. Discussion on the Generalization of the Results

Our proposed DPC-SDAE framework applies to not only clustering objects with mixed attributes, but also to clustering pure categorical attribute objects or pure numerical attribute objects. The SDAE in the framework can be generalized to other common deep learning structures, such as variational autoencoder (VAE), generative adversarial network (GAN). In our experiments, the data objects with missing attribute values are deleted directly before employing DPC-SDAE. In data preprocessing stage, we can fill in missing attribute values by utilizing imputation methods of missing values. For example, we can use the mean value of each numerical attribute on entire dataset to replace the corresponding missing values or take the mode value of each categorical attribute on entire dataset as the imputation values of corresponding missing categorical attribute values. Thus, our DPC-SDAE framework can be generalized to perform clustering mixed data with missing attribute values.

4.6. Discussion on the Methodological and Practical Implications

Our proposed DPC-SDAE algorithm provides an effective method for clustering mixed data, it improves the traditional density peaks clustering, which extends its clustering objects from numerical attribute data to mixed data with both numerical attributes and categorical attributes. L-method with iterative refinement of the knee point can also be applied to determining the number of clusters automatically for other clustering methods for mixed data by replacing the sorted $\gamma$ values with corresponding clustering quality index of all data objects.

Our proposed DPC-SDAE algorithm also makes better decisions in practical problems. For example, by clustering credit card applicants in Credit dataset with DPC-SDAE algorithm, bank credit officer can obtain two clusters and corresponding cluster centers, which represent the prototypes of approval applicants and denial applicants, respectively. When the attribute values of a new credit card applicant are submitted, the distances between this credit card applicant and each cluster center can be calculated. Consequently, a bank credit officer can decide to approve or reject the credit card application by the principle of the nearest neighbor. In clinical medicine, patients with heart disease can also be recognized from healthy persons by employing DPC-SDAE algorithm.

4.7. Discussion on the Limitations of DPC-SDAE

In our proposed DPC-SDAE algorithm, the improved density peaks clustering is employed to conduct clustering on mixed data objects based on the feature representations extracted by SDAE. The cutoff distance, or corresponding neighbor ratio, plays an important role in clustering mixed data, but it is usually prespecified manually and incapable of determining automatically on each dataset. In the future research, we will investigate the methods for adaptively determining the cutoff distance or neighbor ratio on different datasets.

5. Conclusions

In this paper, we have proposed a clustering algorithm named DPC-SDAE, which integrates stacked denoising autoencoders and density peaks clustering algorithm to perform clustering for various data types. Based on the robust clustering features learned from SDAE instead of the original attributes, the DPC-SDAE algorithm can acquire more accurate and robust clustering results, which have been experimentally validated in comparison with three baseline algorithms. In addition, density peaks clustering algorithm has been improved by employing L-method to determine the number of cluster centers automatically, which overcomes the drawback of selecting the number of cluster centers by decision graph manually in original DPC algorithm. We also experimentally investigated the effect of some hyperparameters on clustering quality in DPC-SDAE. We found that each hyperparameter has a relatively large range of values, in which the evaluation indexes of DPC-SDAE keep almost unchanged. This is vitally important since it is unfeasible to optimize hyperparameters by cross validation method in most of clustering tasks due to the lack of ground truth labels of data objects. Our experiment results have also demonstrated that stacked denoising autoencoders can be applied to clustering small sample data effectively besides extensive applications in clustering large scale data.

In the future, we aim to investigate the integration of DPC algorithm and deep generative model, such as variational autoencoder (VAE), generative adversarial network (GAN), to improve the clustering quality even more. Future research will also include the methods for adaptively determining cutoff distance in DPC.