Scalable Mining of High-Utility Sequential Patterns With Three-Tier MapReduce Model

Data mining, which also can be referred as Knowledge Discovery in Databases (KDD) [1, 8], has been widely studied and utilized in many applications and domains. The fundamental knowledge in KDD can be classified as many representations, e.g., association-rule mining (ARM) [2, 17], sequential pattern mining (SPM) [3, 14, 16, 33, 35], high-utility itemset mining (HUIM) [9, 15, 20, 22, 28, 43], among others. For generic ARM, it only takes the occurrence frequency of the items into account, but the other factors, such as interestingness, weight, or importance are not considered; the discovered information from ARM may become incomplete. To address this problem, HUIM considers two factors such as unit profit of the items and the quantity of the items into account to find more meaningful patterns than that of ARM. It thus has become an important topic in the field of KDD; however, it is not suitable for time-series or sequential data in many realistic domains and applications, for example, stock market analysis or DNA sequence analysis. Besides, there is a large number of time-series and sequence characteristic data with different meanings and effects at different times in fields like consumer behaviour analysis, business intelligence, fault risk prediction, and medical diagnosis, that cannot be analyzed by the traditional HUIM nor ARM.

To solve the limitation of the traditional ARM or HUIM, SPM is used to find the interesting subsequences in a set of sequences, where the interestingness of a subsequence can be measured in terms of various criteria such as its occurrence frequency, length, and profit. SPM shows numerous real-life applications because data is naturally encoded as sequences of symbols in many fields such as bio-informatics, e-learning, market basket analysis, texts, and web-page click-stream analysis. SPM has, however, the limitation by only considering the occurrence frequency of the sequence, thus if a sequence is with low frequency but high utility, it could be ignored in SPM. For example, although the sales volume of a sequence behaviour (= buying diamond rings first then buying necklaces afterward) is lower than the sales volume of a sequence behaviour (= buying bread first then buying milk afterward), the profit of a sequence is much higher than the profit of a sequence . Clearly, sequence behaviour is more conducive to merchants. However, in general, the frequency of sequence is very low, and frequent sequence pattern mining cannot find such important information.

High-utility sequential pattern mining (HUSPM) [42, 46, 48] has broader application prospects and needs when compared with traditional SPM and HUIM. For example, HUSPM can find a high-margin product sequence by analyzing sales data of a supermarket, thereby helping the supermarket to formulate commodity promotion strategies and provide a more reasonable commodity procurement plan. In bio-informatics [51], HUSPM can simultaneously consider temporal characteristics and importance of genes, and it can analyze the relationship between the top-k efficient gene sequences and diseases (such as pneumonia) through inter-gene interactions in disease diagnosis. As HUSPM is an emerging field that has attracted the attention of an increasing number of researchers, several works [42, 46] have been initiated on HUSPM. However, the existing methods are memory-based, which means it is assumed that all the data can fit into the main memory of a single machine. Current trends show that high volumes of data are produced in real-life applications. Memory-based algorithms are not realistic for application areas with large-scale datasets. However, mining high-utility sequential patterns (HUSPs) from large-scale datasets is an emerging topic but not a simple task. The limitations of the current works are stated below, which are the motivation of this article for further improvement.

In HUIM, the distributed and parallel methods are Apriori-based [26] or using sampling model [10] to mine HUIs. The former is the same as the methods in frequent itemsets mining using iterative MapReduce that requires higher computational cost. The latter parallelizes the HUI-Miner algorithm by adopting the sampling model to obtain the approximate number of HUIs. To better solve the above limitations for efficiently and accurately revealing the number of HUSPs in the large-scale datasets, we firstly propose a distributed and parallel HUSPM framework to handle large-scale datasets. Four contributions of this article are then stated below:

Section 2 provided a detailed survey of the relevant works in this article. Section 3 stated the basic preliminary and problem statement of this article. Section 4 mentioned the proposed MapReduce models and the developed algorithms. Section 5 showed the experiments to evaluate the performance of the designed model compared to the other works. Finally, Section 6 concluded the achievements of this article and extended directions for future works.

Agrawal et al. [2] and Han et al. [17], respectively, presented the Apriori and FP-growth algorithms to solve the ARM problem. To handle the realistic situations regarding sequence ordering, Agrawal et al. [3] then first proposed the concept of SPM and designed the AprioriSome, AprioriAll, and DynamicSome algorithms for SPM. Srikant et al. [35] proposed a GSP algorithm that uses a hash tree to keep the candidate sets for improving the efficiency of the AprioriAll algorithm. The FreeSpan [16] and PrefixSpan [33] were also, respectively, presented to speed up the mining performance of SPM.

Since ARM and SPM only explore the occurrence frequency of the items in the database, it ignores many important factors, e.g., importance, interestingness, weight, unit profit of items, among others, to mine the association rules. Chan et al. [9] first introduced the concept of utility into frequent itemset mining to help decision-makers develop more favourable strategies. Yao et al. [45] proposed a formal definition of efficient itemset mining, using utility values instead of support as a measure of itemsets. Liu et al. [20] proposed the transaction-weighted utility (TWU) concept for estimating the upper bound of the itemset utility value. Tseng et al. extended the FP-tree and proposed the UP-growth+ [40] algorithm to exploit the nature of the tree for compressing the search space. Lin et al. [21] proposed HUP-tree, which is based on the TWU concept and FP-tree, and they used the tree structure to save the database, which speeds up the mining process of the proposed HUP-growth algorithm. Liu et al. [22] proposed the HUI-Miner algorithm, which converts the original database into a list structure and mines efficient itemsets from the list and thus avoids the generation of candidate sets. Zida et al. [50] designed a novel algorithm EFIM, proposed two new utility upper bounds, and more effectively reduced the search space. Presently, the research on HUIM is still in development. Kim et al. [18] then developed a utility model for handling the large-scale stream data for discovering the high-utility patterns. The designed model divides the stream data into several fixed-sized data and processes each batch of data in a window according to the added time by the designed decaying factor to differently show its importance. Vo et al. [41] suggested having the dynamic profit tables for the itemsets in real applications and presented a multi-core framework for efficiently mining the high-utility itemsets. The designed model can then greatly reduce the cost of database rescans thus the performance can be improved. Nam et al. [32] considered the influence of the recent data compared to the old one, a model focused on finding the high-utility itemsets from the time-sensitive databases was presented by applying the damped-window model. Mai et al. [31] presented a model to mine the high-utility association rules, which enables users to iteratively choose the preferred weights for the discovered rules based on the developed semi-lattice structures. To speed up mining performance, Yun et al. [47] presented a pre-large-based concept for mining high-utility itemsets in dynamic databases. The deletion operation is considered here to maintain and update the discovered patterns by nine cases of the pre-large concept to reduce the number of database rescans. Moreover, the pre-large concept is also adapted to the sensor network situation [38] to combine all the discovered high-utility itemsets in a fusion model, which is applicable in industrial applications. Several works [7, 15, 27, 29, 43] in the direction of HUIM have been extensively presented and discussed but most of them can only be performed on a single machine for running small datasets.

HUSPM is a field that has emerged in recent years. HUSPM was first used in the sequence mining of website logs [49]. Shie et al. proposed the UMSP [36] algorithm and the UM-span [37] algorithm for mining high-utility mobile sequences in mobile business applications. To exploit the usefulness of web page access sequence data, Ahmed et al. [4] proposed two tree structures, called UWAS-tree and IUWAS-tree, for processing static and dynamic databases, respectively. Subsequently, Ahmed et al. [5] proposed a HUSPM algorithm for processing general sequences, namely, the layer-by-layer search UL algorithm and the pattern-extended US algorithm. Yin et al. [46] officially defined HUSPM and proposed an efficient algorithm, USpan, for mining general sequence patterns with utility values. To simplify the parameter setting, Yin et al. [48] then proposed the TUS algorithm for discovering the top-k HUSPs. Lan et al. [25] first introduced the concept of fuzziness into sequence mining and then proposed a HUSPM algorithm to simplify the mining results and reduce the search space. Alkan et al. [6] proposed a high-utility sequential pattern extraction (HuspExt) algorithm. It calculates a Cumulated Rest of Match (CRoM) to obtain a smaller upper bound to reduce the complexity of the algorithm. Wang et al. [42] subsequently proposed the HUS-Span algorithm to reduce useless candidate sets by two utility upper bound PEUs and RSUs. The HUS-Span is the generic and serial algorithm that can be used to discover the set of HUSPs from the database based on the developed high sequence weighted utility (SWU) to maintain the downward closure property, which is the standard and the state-of-the-art algorithm for HUSPM. Their article also proposes a TKHUS-Span algorithm based on top-k and its performance was tested under three search strategies.

MapReduce [11], which was proposed by Dean and Ghemawat, is a programming framework designed to handle big datasets. It is a parallel and distributed algorithm on a cluster and contains two major procedures, Map and Reduce. Overall, MapReduce provides a reliable, dynamic, and parallel programming framework to deal with big data environments. Regarding the MapReduce framework in pattern mining, Lin et al. [24] proposed three algorithms, respectively, named SPC, FPC, and DPC, by implementing the Apriori in MapReduce framework. The SPC algorithm is used to find the frequent k-itemsets at each level based on the generate-and-test model. The FPC is used to improve the performance of the baseline SPC model using a mapper to calculate k, (k+1), and (k+2) itemsets altogether, and the DPC is used to collect the candidates at different lengths. Those three models are based on the Apriori-like approach thus more execution time is required. Li et al. [19] then proposed PFP algorithm, which parallelizes the FP-Growth algorithm on distributed machines without candidate generation. This developed model is based on novel data with the distribution scheme and MapReduce framework to virtually eliminate the communication among several parallel and distributed computers. Moens et al. [30] introduced Dist-Eclat and BigFIM algorithms for mining the frequent itemsets based on the MapReduce framework. The first Dist-Eclat is used to speed up mining performance and the latter BigFIM is then used to optimize the execution progress on the large databases. Duong et al. [12] presented a two-phase approach for frequent itemset mining in large-scale databases based on the MapReduce and distributed Apriori-like approach. The projection model is also used in the developed model to gradually reduce the database size during the MapReduce phase. In addition to frequent itemset mining, Ge et al. [13] considered the uncertainty in sequential databases and presented a MapReduce framework for mining the uncertain sequential patterns iteratively. A vertical data structure is then used to keep the necessary information of the uncertain sequence databases that can greatly reduce the computational complexity. For HUIM, Lin et al. [26] proposed PHUI-Growth for mining HUIs from big data, which is Apriori-based Apache Hadoop framework. However, this approach requires huge computational costs, thus it lacks the efficiency to handle very large databases. Chen et al. [10] presented a parallel algorithm of HUI-Miner implemented based on Apache Spark using sampling technologies to reduce the size of input data and approximately mine the HUIs. Based on this model, the approximate set of HUIs is then discovered but the performance can be greatly improved by the sampling model. However, this model could not provide accurate results in terms of the number of HUIs or even the utility of the itemset. It is thus a limitation for making the accurate and precise decision. Wu et al. [44] then applies the Hadoop framework for mining the fuzzy high-utility patterns, which is the first work to adapt the fuzzy-set theory into the HUIM. However, this model cannot handle large-scale databases. As the rapid growth for the research of HUSPM, it is necessary to develop an efficient model to discover the set of HUSPs from a large-scale efficiency. Sumalatha and Subramanyam [39] then presented a distributed high utility time interval sequential pattern mining (DHUTISP) algorithm based on the MapReduce framework. Two upper-bound models are then designed to reduce the computational cost. However, this model is mainly focused on distributing data into several nodes and the two designed upper-bounds mainly replied on the past works.

Let I = {, , , } be a set of m different items. A quantitative sequence database (q-sequence database) is a set of transactions (or in the running example, where id is the transaction id in the database) D = {, , , }, where each transaction is a quantitative sequence (q-sequence) and q is its unique identifier (= id). A quantitative itemset (q-itemset) denoted as X = [(, ), (, ), , (, )] is a subset of and each item in a q-itemset is a quantitative item that is a pair of the form (i, q), where and q is a positive integer representing the internal weight locally associated with an item in a transaction/sequence (internal utility). The quantity of a q-item i in a q-itemset X is denoted as q(i, X). Each item () is also associated with a weight denoted as pr() representing the external weight globally associated with an item (external utility). In addition, without a loss of generality, since the items are unordered in an itemset, it is assumed that q-items in a q-itemset are sorted in lexicographical order. A quantitative sequence (q-itemset) is composed of multiple itemsets in an ordered arrangement, which is denoted as s = , , , . The order of q-itemsets in a q-sequence, containing temporal order and spatial order, can represent the order of purchase, building order, among others.

Table 1 shows a quantitative sequential database. This database has five quantitative sequences and six items. Table 2 shows a utility table of the items that appear in Table 1. In Tables 1 and 2, (a), (b), (c), and so on, represent the items; (a:2) indicates that the purchased quantity of item a is 2 (q-item for short); [(a:2) (c:3)] indicates a set of items with a purchased quantity 2 of item a and purchased quantity 3 of item c (referred to as q-itemset); and [(a:2)(c:3)], [(e:3)] means that it is a sequence containing two q-itemsets [(a:2)(c:3)] and [(e:3)] (q-sequence for short), where [(a:2)(c:3)] and [(e:3))] have a sequential relationship in the sequence.

Take in Table 1 as an example to give the concrete explanations, apple(a) is purchased with cake(c) together, respectively, with the amounts of 2 and 3 (e.q., [(a:2)(c:3)]). After that, apple(a), bread(b) and cake(c) are purchased together, respectively, with the amounts of 3, 1, and 2 (e.q., [(a:3)(b:1)(c:2)]). In addition, apple(a), bread(b), and donuts(d) are purchased together, respectively, with the amounts of 4, 5, and 4 (e.q., [(a:4)(b:5)(d:4)]). Finally, egg(e) is then purchased with the amount of 3 (e.q., [(e:3)]). Thus, it can be seen that four sequential orders are in . First, the utility of an item in a q-itemset X can be defined as follows.

To calculate the utility of an itemset X (or q-itemset) in a q-sequence s, the following definition and an example are given below.

Based on the above definitions, we can then calculate the utility of a q-sequence s in the database by the following definition.

To calculate the utility of a quantitative sequential database D, the following definition and its example are given below.

To show all the elements (or item/sets) of an itemset, the formal definition and the relevant example are given below.

To show whether an itemset is contained in a sequence, the formal definition and the example are given below to clearly show their relationships.

To elaborate the relationships of a sequence to a sequence, the following definition with a simple example is given below.

To handle the quantitative number of the items in the sequential database, the definition is then given below to show the relationship of a sequence and its sub-sequences.

To show the number of matches regarding the sub-sequences within a sequence, the following definition and its example are then given below.

This example shows that a target sequence in HUSPM may have multiple utility values in a transaction, which is quite different from generic HUIM and ARM. Different evaluation criteria choose different utility values, and here the maximum value is used as the utility value of the target sequence in HUSPM.

To handle multiple databases in the distributed and parallel environment, let D be a quantitative sequence database and , , , are the partitions of D satisfied by D = { } and {, } D, = . For example, the database D in Table 1 can be split into two partitions, and , as Tables 3 and 4 show. Table 2 is also the profit table of these two quantitative databases.

To find the utility of a sequence t in the partitions, the definition is given as follows.

Problem Statement. Given a large-scale quantitative database D and a minimum utility threshold , the task of HUSPM using a distributed and parallel method for handling the large-scale dataset is to discover the complete set of sequences whose global utility is not less than by efficiently parallel mining the partition of D.

In this article, we first develop a three-stage MapReduce framework for discovering HUSPs from large-scale databases, which is the first work to adopt a MapReduce model in HUSPM. Furthermore, two data structures, respectively, called sidset and Utility-Linked List are utilized here to keep the necessary information for the mining progress. Figure 1 first shows an overview of the designed framework. The framework is divided into three phases which are Identification, Local Mining, and Integration. Each MapReduce is then performed for each phase in the designed framework. The three MapReduce operations used in the designed framework are, respectively, representing the three phases and are described below.

Several experiments were conducted to evaluate the performance of the presented MapReduce framework in the Spark model. The designed three-stage MapReduce framework without any structure (i.e., sidset and Utility-Linked List) is named M-HUSPM in the experiments, and the designed three-stage MapReduce framework with both the sidset and the Utility-Linked List structures for the implementation is named ML-HUSPM in the experiments. The state-of-the-art algorithm HUS-Span [42] is also used as the serial algorithm for comparisons and evaluation. Each algorithm is then performed 10 times for the evaluation. Experiments were performed with a local Spark cluster on a workstation having Intel Xenon CPU 2.10 GHz with 8 cores, 16 threads, 16 GB RAM, and 1.5 TB of disk storage. Spark-2.1.1 is installed over Ubuntu 20.04, 64 bit running on the workstation. Note that the data structure is stored using Hadoop Distributed File System (HDFS) storage system. To save the shared structure, we used the Hadoop sequence file, which is a binary file format containing all of the data in the shared structures presented by key, value pairs in a serialized form. Four real-life datasets [34] were used in the experiments. The characteristics of the four original datasets are shown in Table 6. The parameters of the datasets are indicated using the following four attributes: |D| states the total number of sequences; |I| is the number of distinct items; C is the average number of itemsets per sequence, and MaxLen states the maximum number of items per sequence. In real cases, there is no large-scale datasets for performing the designed model for efficiency evaluation, thus, the original datasets in Table 6 are then enlarged, that is, the original size is multiplied by various numbers (i.e., 1, 20, 50, 100, 200, 500, 1,000, 2,000, 5,000, 10,000). The sizes of the conducted large-scale datasets are also illustrated in Table 7.

A three-stage MapReduce framework is designed in this article to handle the HUSPM in large-scale databases. To speed up mining performance, two data structures called sidset and Utility-Linked List are applied in the designed model. Moreover, two properties are then developed to hold the correctness and completeness of the discovered patterns. From the conducted results in the experiments, the designed model showed better performance compared to the traditional HUSPM models in terms of runtime, memory usage, and scalability, particularly in large-scale databases. In future works, the designed model can be extended to the other constraint-based approaches, i.e., top-k, maximal or closed HUSPM. Moreover, the evolutionary computation models can also be discussed and utilized in the designed model to improve the effectiveness and efficiency of the mining progress.