-
KIF: A Wikidata-Based Framework for Integrating Heterogeneous Knowledge Sources
Authors:
Guilherme Lima,
João M. B. Rodrigues,
Marcelo Machado,
Elton Soares,
Sandro R. Fiorini,
Raphael Thiago,
Leonardo G. Azevedo,
Viviane T. da Silva,
Renato Cerqueira
Abstract:
We present a Wikidata-based framework, called KIF, for virtually integrating heterogeneous knowledge sources. KIF is written in Python and is released as open-source. It leverages Wikidata's data model and vocabulary plus user-defined mappings to construct a unified view of the underlying sources while keeping track of the context and provenance of their statements. The underlying sources can be t…
▽ More
We present a Wikidata-based framework, called KIF, for virtually integrating heterogeneous knowledge sources. KIF is written in Python and is released as open-source. It leverages Wikidata's data model and vocabulary plus user-defined mappings to construct a unified view of the underlying sources while keeping track of the context and provenance of their statements. The underlying sources can be triplestores, relational databases, CSV files, etc., which may or may not use the vocabulary and RDF encoding of Wikidata. The end result is a virtual knowledge base which behaves like an "extended Wikidata" and which can be queried using a simple but expressive pattern language, defined in terms of Wikidata's data model. In this paper, we present the design and implementation of KIF, discuss how we have used it to solve a real integration problem in the domain of chemistry (involving Wikidata, PubChem, and IBM CIRCA), and present experimental results on the performance and overhead of KIF
△ Less
Submitted 24 July, 2024; v1 submitted 15 March, 2024;
originally announced March 2024.
-
A Polystore Architecture Using Knowledge Graphs to Support Queries on Heterogeneous Data Stores
Authors:
Leonardo Guerreiro Azevedo,
Renan Francisco Santos Souza,
Elton F. de S. Soares,
Raphael M. Thiago,
Julio Cesar Cardoso Tesolin,
Ann C. Oliveira,
Marcio Ferreira Moreno
Abstract:
Modern applications commonly need to manage dataset types composed of heterogeneous data and schemas, making it difficult to access them in an integrated way. A single data store to manage heterogeneous data using a common data model is not effective in such a scenario, which results in the domain data being fragmented in the data stores that best fit their storage and access requirements (e.g., N…
▽ More
Modern applications commonly need to manage dataset types composed of heterogeneous data and schemas, making it difficult to access them in an integrated way. A single data store to manage heterogeneous data using a common data model is not effective in such a scenario, which results in the domain data being fragmented in the data stores that best fit their storage and access requirements (e.g., NoSQL, relational DBMS, or HDFS). Besides, organization workflows independently consume these fragments, and usually, there is no explicit link among the fragments that would be useful to support an integrated view. The research challenge tackled by this work is to provide the means to query heterogeneous data residing on distinct data repositories that are not explicitly connected. We propose a federated database architecture by providing a single abstract global conceptual schema to users, allowing them to write their queries, encapsulating data heterogeneity, location, and linkage by employing: (i) meta-models to represent the global conceptual schema, the remote data local conceptual schemas, and mappings among them; (ii) provenance to create explicit links among the consumed and generated data residing in separate datasets. We evaluated the architecture through its implementation as a polystore service, following a microservice architecture approach, in a scenario that simulates a real case in Oil \& Gas industry. Also, we compared the proposed architecture to a relational multidatabase system based on foreign data wrappers, measuring the user's cognitive load to write a query (or query complexity) and the query processing time. The results demonstrated that the proposed architecture allows query writing two times less complex than the one written for the relational multidatabase system, adding an excess of no more than 30% in query processing time.
△ Less
Submitted 15 March, 2024; v1 submitted 7 August, 2023;
originally announced August 2023.
-
DevOps and Microservices in Scientific System development
Authors:
Maximillien de Bayser,
Vinicius Segura,
Leonardo Guerreiro Azevedo,
Leonardo P. Tizzei,
Raphael Melo Thiago,
Elton Soares,
Renato Cerqueira
Abstract:
There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware plat…
▽ More
There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware platforms, combined with the increasing multidisciplinarity of science itself have exacerbated the problem because self-taught "end user developers" are not familiar with the disciplines needed to tackle this increased complexity. On a more positive note, agile programming methods have approached software development practices to the way scientific software is produced. In this work, we present the experience of a multi-year industry research project where agile methods, microservices and DevOps were applied. Our goal is to validate the hypothesis that the use of microservices would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration. Hence, scientific multidisciplinary systems would gain in a twofold way: (i) Subject Matter Experts(SME) use their preferable tools to develop the specific scientific part of the system; (ii) software engineers provide the high quality software code for the system delivery.
△ Less
Submitted 22 December, 2021;
originally announced December 2021.
-
Workflow Provenance in the Lifecycle of Scientific Machine Learning
Authors:
Renan Souza,
Leonardo G. Azevedo,
Vítor Lourenço,
Elton Soares,
Raphael Thiago,
Rafael Brandão,
Daniel Civitarese,
Emilio Vital Brazil,
Marcio Moreno,
Patrick Valduriez,
Marta Mattoso,
Renato Cerqueira,
Marco A. S. Netto
Abstract:
Machine Learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibil…
▽ More
Machine Learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design principles to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the principles enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.
△ Less
Submitted 25 August, 2021; v1 submitted 30 September, 2020;
originally announced October 2020.
-
Managing Data Lineage of O&G Machine Learning Models: The Sweet Spot for Shale Use Case
Authors:
Raphael Thiago,
Renan Souza,
L. Azevedo,
E. Soares,
Rodrigo Santos,
Wallas Santos,
Max De Bayser,
M. Cardoso,
M. Moreno,
Renato Cerqueira
Abstract:
Machine Learning (ML) has increased its role, becoming essential in several industries. However, questions around training data lineage, such as "where has the dataset used to train this model come from?"; the introduction of several new data protection legislation; and, the need for data governance requirements, have hindered the adoption of ML models in the real world. In this paper, we discuss…
▽ More
Machine Learning (ML) has increased its role, becoming essential in several industries. However, questions around training data lineage, such as "where has the dataset used to train this model come from?"; the introduction of several new data protection legislation; and, the need for data governance requirements, have hindered the adoption of ML models in the real world. In this paper, we discuss how data lineage can be leveraged to benefit the ML lifecycle to build ML models to discover sweet-spots for shale oil and gas production, a major application in the Oil and Gas O&G Industry.
△ Less
Submitted 10 March, 2020;
originally announced March 2020.
-
Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering
Authors:
Renan Souza,
Leonardo Azevedo,
Vítor Lourenço,
Elton Soares,
Raphael Thiago,
Rafael Brandão,
Daniel Civitarese,
Emilio Vital Brazil,
Marcio Moreno,
Patrick Valduriez,
Marta Mattoso,
Renato Cerqueira,
Marco A. S. Netto
Abstract:
Machine Learning (ML) has become essential in several industries. In Computational Science and Engineering (CSE), the complexity of the ML lifecycle comes from the large variety of data, scientists' expertise, tools, and workflows. If data are not tracked properly during the lifecycle, it becomes unfeasible to recreate a ML model from scratch or to explain to stakeholders how it was created. The m…
▽ More
Machine Learning (ML) has become essential in several industries. In Computational Science and Engineering (CSE), the complexity of the ML lifecycle comes from the large variety of data, scientists' expertise, tools, and workflows. If data are not tracked properly during the lifecycle, it becomes unfeasible to recreate a ML model from scratch or to explain to stakeholders how it was created. The main limitation of provenance tracking solutions is that they cannot cope with provenance capture and integration of domain and ML data processed in the multiple workflows in the lifecycle while keeping the provenance capture overhead low. To handle this problem, in this paper we contribute with a detailed characterization of provenance data in the ML lifecycle in CSE; a new provenance data representation, called PROV-ML, built on top of W3C PROV and ML Schema; and extensions to a system that tracks provenance from multiple workflows to address the characteristics of ML and CSE, and to allow for provenance queries with a standard vocabulary. We show a practical use in a real case in the Oil and Gas industry, along with its evaluation using 48 GPUs in parallel.
△ Less
Submitted 21 October, 2019; v1 submitted 9 October, 2019;
originally announced October 2019.
