Abstract
While the k-Nearest Neighbors (\(k\)NN) join fetches the k closest objects from a dataset for each element of a reference collection, a \(k\)NN join with result diversification aims at retrieving the k nearest objects to each reference entry that are dissimilar among themselves. Under the Metric Space Model, the distance-based ternary relationship between each search reference, the dataset, and the result set can be explicitly used to define a coverage-based criterion, namely Influence, that ensures diversification by dismissing nearby regions during the join operation. However, adding result diversification to \(k\)NN joins by means of Influence criteria in large-scale, big data frameworks is still an open issue since existing algorithms do not consider shared-nothing environments. To fulfill this gap, we extend the nested Better Results with Influence Diversification algorithm (BRID\(_k\)) to a Map-Reduce framework. In particular, this study introduces two new algorithms: the P-BRID\(_k\) and the SP-BRID\(_k\). The P-BRID\(_k\) method relies on partitioning the objects by their proximity to a set of pivots so that the search space locality is preserved throughout the mapped distance-based comparisons. The SP-BRID\(_k\) method expands the P-BRID\(_k\) by using a data replication strategy where a window of the search space is copied across the partitions for enhancing the Influence-based pruning of the nearest objects. We performed an extensive evaluation of both methods over low and high-dimensional datasets on an Apache Hadoop cluster, and the results indicate that (i) P-BRID\(_k\) has consistently outperformed the nested BRID\(_k\) implementation, with gains up to 80\(\%\) in terms of Recall (fraction of points among true diversified neighbors), (ii) fine-tuned SP-BRID\(_k\) has enhanced the P-BRID\(_k\) performance at a small overhead cost in data replication, and (iii) the SP-BRID\(_k\) elapsed time has scaled smoothly with the number of partitions, yielding high Recalls for \(k\)NN joins with result diversification for a controlled overhead ratio.
This study was supported by CAPES, CNPq and FAPERJ (grant numbers SEI-016517/2021 and E-26/202.806/2019).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
If \(M\times k\) exceeds the worker available memory, then the reduce task can be recursively split into two workers (and one merger) with \((M\times k/2)\) space requirement.
- 2.
Source-code and datasets in https://github.com/rviniciussouza/BRIDkD.
- 3.
\(ID = \lceil \mu _\mathcal {O}^2/2 \cdot \sigma _\mathcal {O}^2\rceil \), where \(\mu _\mathcal {O}\) and \(\sigma _\mathcal {O}\) are the mean and standard deviation of the pairwise distance distribution within \(\mathcal {O}\), respectively [4].
References
Armbrust, M., Das, T., et al.: Delta lake: high-performance ACID table storage over cloud object stores. VLDB 13(12), 3411–3424 (2020)
Bohm, C., Krebs, F.: The k-nearest neighbour join: turbo charging the KDD process. Knowl. Inf. Syst. 6(6), 728–749 (2004)
Čech, P., Maroušek, J., Lokoč, J., Silva, Y.N., Starks, J.: Comparing MapReduce-based k-NN similarity joins on Hadoop for high-dimensional data. In: Cong, G., Peng, W.-C., Zhang, W.E., Li, C., Sun, A. (eds.) ADMA 2017. LNCS (LNAI), vol. 10604, pp. 63–75. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69179-4_5
Chávez, E., Navarro, G., Baeza-Yates, R., Marroquín, J.: Searching in metric spaces. Comput. Surv. 33(3), 273–321 (2001)
Chen, G., Yang, K., Chen, L., Gao, Y., Zheng, B., Chen, C.: Metric similarity joins using MapReduce. TKDE 29(3), 656–669 (2016)
Chen, L., et al.: Indexing metric spaces for exact similarity search. Comput. Surv. 55(6), 1–39 (2022)
Drosou, M., Jagadish, H., Pitoura, E., Stoyanovich, J.: Diversity in big data: a review. Big Data 5, 73–84 (2017)
Hetland, M.L.: The basic principles of metric indexing. In: Coello, C.A.C., Dehuri, S., Ghosh, S. (eds.) Swarm Intelligence for Multi-objective Problems in Data Mining. SCI, vol. 242, pp. 199–232. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03625-5_9
Jacox, E.H., Samet, H.: Metric space similarity joins. TODS 33(2), 1–38 (2008)
Jasbick, D., Santos, L., Marques, P., Traina, A., Oliveira, D., Bedo, M.: Pushing diversity into higher dimensions: the LID effect on diversified similarity searching. Inf. Syst. 114, 102–116 (2023)
Kim, C., Shim, K.: Supporting set-valued joins in NoSQL using MapReduce. Inf. Syst. 49, 52–64 (2015)
Rong, C., Cheng, X., Chen, Z., Huo, N.: Similarity joins for high-dimensional data using Spark. Concurr. Comput.: Pract. Experience 31(20), 1–17 (2019)
Santos, L.F.D., Carvalho, L.O., Oliveira, W.D., Traina, A.J.M., Traina, C.: Diversity in similarity joins. In: Amato, G., Connor, R., Falchi, F., Gennaro, C. (eds.) SISAP 2015. LNCS, vol. 9371, pp. 42–53. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25087-8_4
Santos, L., Oliveira, W., Ferreira, M., Traina, A., Traina, C., Jr.: Parameter-free and domain-independent similarity search with diversity. In: SSDBM (2013)
Silva, Y.N., Reed, J.M., Tsosie, L.M.: MapReduce-based similarity join for metric spaces. In: WCI, pp. 1–8 (2012)
Ukey, N., Yang, Z., Li, B., Zhang, G., Hu, Y., Zhang, W.: Survey on exact kNN queries over high-dimensional data space. Sensors 23(2), 629 (2023)
Wu, J., Zhang, Y., Wang, J., Lin, C., Fu, Y., Xing, C.: Scalable metric similarity join using MapReduce. In: ICDE, pp. 1662–1665 (2019)
Yianilos, P.N.: Data structures and algorithms for nearest neighbor. In: ACM-SIAM Symposium on Discrete Algorithms, vol. 66, p. 311 (1993)
Zhu, Y., Chen, L., Gao, Y., Jensen, C.S.: Pivot selection algorithms in metric spaces: a survey and experimental study. VLDB J. 31(1), 23–47 (2021). https://doi.org/10.1007/s00778-021-00691-4
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Souza, V., Carvalho, L.O., de Oliveira, D., Bedo, M., Santos, L.F.D. (2023). Adding Result Diversification to \(k\)NN-Based Joins in a Map-Reduce Framework. In: Strauss, C., Amagasa, T., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2023. Lecture Notes in Computer Science, vol 14146. Springer, Cham. https://doi.org/10.1007/978-3-031-39847-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-39847-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-39846-9
Online ISBN: 978-3-031-39847-6
eBook Packages: Computer ScienceComputer Science (R0)