Affiliations: [a] Department of Computer Science and Numerical Analysis, University of Cordoba, Cordoba, Spain | [b] Department of Computer Science, University of Jaén, Jaén, Spain | [c] Faculty of Computing and Information Technology, King Abdulaziz University, Saudi Arabia Kingdom | [d] Knowledge Discovery and Intelligent Systems in Biomedicine Laboratory, Maimonides Biomedical Research Institute of Cordoba, Cordoba, Spain | [e] Faculty of Computing and Information Technology, North Jeddah, Saudi Arabia Kingdom | [f] Department of Computer Science and Artificial Intelligence, University of Granada, Granada, Spain
Correspondence:
[*]
Corresponding author: S. Ventura, Department of Computer Science and Numerical Analysis, University of Cordoba, 14071 Cordoba, Spain. Tel.: +34 957212218; Fax: +34 957218630; E-mail: [email protected].
Abstract: Association rule mining is one of the most important tasks to describe raw data. Although many efficient algorithms have been developed to this aim, existing algorithms do not work well on huge volumes of data. The aim of this paper is to propose a new genetic programming algorithm for mining association rules in Big Data. The genetic operators of our proposal have been specifically designed to avoid a growing in the complexity of the solutions without an improvement in their fitness function values. Furthermore, it introduces a repairing operator to improve the convergence. Additionally, to facilitate its application on real world problems a grammar has been included, allowing it to introduce subjective knowledge into the mining process and to reduce the search space. Due to the growing interest in data gathering, a unique implementation of the proposed algorithm is not useful so different implementations (considering different architectures such as RMI, Hadoop and Spark) are required depending on the data size. All these adaptations obtain exactly the same solutions as those of the original algorithm since they only differ on the software architectures. The experimental study considers more than 75 datasets and 14 algorithms and the results reveal that the proposed algorithm obtains excellent results for more than 12 quality measures. The scalability of the proposal is also analyzed by considering the three parallel implementations on high dimensional datasets (3,000 millions of instances) and file sizes up to 800 GB.
Keywords: Association rules, Big Data, MapReduce, Hadoop, Spark
