Abstract
The Data Mesh architecture is gaining popularity as a new approach to data management within modern organizations. A key component of this architecture is the metadata catalog, which provides a centralized repository for documenting and discovering data products across different teams. However, maintaining an accurate and secure catalog in the context of this architecture is challenging. In this paper, we explore the benefits of using blockchain technology to power the metadata catalog in a Data Mesh architecture. Indeed, blockchain provides a decentralized and immutable ledger that can help to ensure the accuracy and consistency of metadata across different teams. It provides a secure and transparent way of tracking changes and access to the metadata. We discuss the potential advantages of using blockchain for metadata catalog, including data security, data provenance, and data ownership. We also examine the potential challenges and limitations of using blockchain technology in the context of Data Mesh architecture. Overall, we argue that blockchain-powered metadata catalogs have the potential to enhance the efficiency and reliability of data management in a Data Mesh.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Casino, F., Dasaklis, T.K., Patsakis, C.: A systematic literature review of blockchain-based applications: current status, classification and open issues. Telemat. Inform. (2019)
Chessell, M., Jones, N.L., Limburn, J., Radley, D., Shan, K.: Designing and operating a data reservoir. In: IBM Redbooks (2015)
Dehghani, Z.: Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly (2022)
Demichev, A., Kryukov, A., Prikhodko, N.: The approach to managing provenance metadata and data access rights in distributed storage using the hyperledger blockchain platform. In: Ivannikov Ispras Open Conference. IEEE (2018)
García-Barriocanal, E., Sánchez-Alonso, S., Sicilia, M.-A.: Deploying metadata on blockchain technologies. In: Garoufallou, E., Virkus, S., Siatri, R., Koutsomiha, D. (eds.) MTSR 2017. CCIS, vol. 755, pp. 38–49. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70863-8_4
Hai, R., Geisler, S., Quix, C.: Constance: an intelligent data lake system. In: Proceedings of the 2016 International Conference on Management of Data (2016)
Halevy, A.Y., et al.: Managing Google’s data lake: an overview of the goods system. IEEE Data Eng. Bull. 39(3), 5–14 (2016)
Inmon, W., Strauss, D., Neushloss, G.: DW 2.0: The Architecture for the Next Generation of Data Warehousing. Elsevier (2010)
Koscina, M., Manset, D., Negri-Ribalta, C., Perez, O.: Enabling trust in healthcare data exchange with a federated blockchain-based architecture. In: International Conference on Web Intelligence-Companion Volume (2019)
Kumar, D.S., Rahman, M.A.: Simplified HDFS architecture with blockchain distribution of metadata. Int. J. Appl. Eng. Res. (2017)
Laurent, A., Laurent, D., Madera, C.: Data Lakes, Chapter 1, Introduction to Data Lakes: Definitions and Discussions, vol. 2. Wiley (2020)
Liu, L., Li, X., Au, M.H., Fan, Z., Meng, X.: Metadata privacy preservation for blockchain-based healthcare systems. In: Bhattacharya, A., et al. (eds.) DASFAA 2022. LNCS, vol. 13245, pp. 404–412. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-00123-9_33
Majchrzak, J., Balnojan, S., Siwiak, M., Sieraczkiewicz, M.: Data Mesh in Action. Manning Publishing (2022)
Sawadogo, P., Darmont, J.: On data lake architectures and metadata management. J. Intell. Inf. Syst. 56(1), 97–120 (2021)
Sawadogo, P.N., Darmont, J., Noûs, C.: Joint management and analysis of textual documents and tabular data within the AUDAL data lake. In: Bellatreche, L., Dumas, M., Karras, P., Matulevičius, R. (eds.) ADBIS 2021. LNCS, vol. 12843, pp. 88–101. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-82472-3_8
Zhao, Y.: Metadata management for data lake governance. Ph.D. thesis, Univ. Toulouse 1 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dolhopolov, A., Castelltort, A., Laurent, A. (2024). Exploring the Benefits of Blockchain-Powered Metadata Catalogs in Data Mesh Architecture. In: Chbeir, R., Benslimane, D., Zervakis, M., Manolopoulos, Y., Ngyuen, N.T., Tekli, J. (eds) Management of Digital EcoSystems. MEDES 2023. Communications in Computer and Information Science, vol 2022. Springer, Cham. https://doi.org/10.1007/978-3-031-51643-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-51643-6_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-51642-9
Online ISBN: 978-3-031-51643-6
eBook Packages: Computer ScienceComputer Science (R0)