Crossref’s Post

Are you having issues discovering the results of your DQ tests when using Great Expectations? We've got you covered! 1. Easily integrate GX results into OpenMetadata & Collate 2. Visualize the results in a single place 3. Create observability alerts directly from our UI 4. Collaborate with your team using our Incident Manager DQ has never been easier :)

At Zeenea, we develop advanced connectors to automatically synchronize your metadata from all your sources. Today's connector highlight: Snowflake 🚀 With Zeenea, discover your Snowflake data in seconds. Zeenea collects your Snowflake Datasets, associated Fields, and Data Processes as well as any technical and operational metadata, and other documentation you may have provided at the field table level. In addition, Zeenea's advanced features for Snowflake data include: ✔️ Data Lineage ✔️ Data Profiling ✔️ Data Sampling More info 👉 https://hubs.ly/Q02G8--y0 #zeenea #snowflake #connector #data #metadata #datalineage #dataprofiling #datasampling

Classical enterprise data metadata can be presented as an Enterprise Knowledge Graph (EKG). From a pragmatic view of EKG, we can highlight these points, which allow gradual growth of knowledge inside the company. 1. Unified Data Integration: EKG integrate and unify data from diverse sources, eliminating fragmentation. They provide a single access point for viewing, exploring, and analysing data. 2. Holistic View: Acting as a central hub, knowledge graphs combine actual data and metadata. This holistic view helps understand relationships between different information sources. 3. Semantic Precision: EKGs define precise meanings and relationships by using semantic modelling techniques (such as ontologies and controlled vocabularies). This resolves ambiguity issues common in traditional data management systems. 4. Global Context: EKG can enhance proprietary information by leveraging global knowledge. External sources enrich the understanding of data, going beyond what’s contained within the graph. 5. Real-Time Processing: The enterprise needs to process vast amounts of data effectively. Traditional approaches fall short. Knowledge graphs offer a viable solution suitable for any enterprise size and complexity.

Understanding Relational Entity Hierarchy in Databricks🧩 Metastore: The central repository for metadata, serving as the highest level of the hierarchy. #Metastore 🗄️ Catalog: Used to group schemas (databases) within the metastore, providing organization at an intermediate level. #Catalog 📁 Schema (Database): Contains tables and provides logical organization within catalogs, representing a level below catalogs. #Schema 📂 Table: The most granular level of data organization, residing within schemas and containing actual data. #Table 📊

Metadata identification and documentation can be a tremendously expensive exercise for large, complex organisations - especially those that didn’t have the hygiene and discipline in this area in the early stages (when the data landscape was simple). But how can we approach this when it’s already perceived as too complex, too difficult, or too costly? In my opinion, it boils down to: 1) common understanding of why the exercise is important, and what value each stakeholder can derive from it. 2) prioritising metadata efforts on use cases that benefits the broadest range of stakeholders, and/or having the highest value impact. Thoughts? #datathoughtoftheday #metadata #datamanagement #datagovernance

Quick Video Tutorial: 🎥 Visualize your Great Expectations data quality tests in OpenMetadata. Centralize your data quality metadata alongside your data discovery, governance, lineage, insight and more: https://buff.ly/3zK6Hkv PS: Stay tuned for upcoming Anomaly Detection in OpenMetadata!

#DATAENGINEERING_WITH_SNOWFLAKE HI all, #connections Here with the complete hands on document on TIME TRAVEL in snowflake #snowflake #timetravel #dataengineering #datascience #snowflake_31 covered the following 1. About "Time Travel" 2. Retrieve data using 'before' 3. Data restoration using direct and indirect methods 4. UNDROP command 5. Retention time Thanks, #all

👋 Hello LinkedIn community! Let's dive into a fascinating conversation about data lakes and delta lakes, two integral elements of our data ecosystem. 🧩   ✔️ A traditional Data Lake provides raw, unfiltered data storage, but it may lack ACID compliance, versioning capabilities and schema evaluation. 🔄 On the other hand, Delta Lake offers a robust transactional layer with ACID compliance ensuring reliable data integrity during concurrent reads and writes. ✅ What sets Delta Lake apart is its ability to handle Schema Evolution seamlessly. It allows us to add or modify fields without breaking existing pipelines or queries - a game changer in dynamic environments. 💫 The icing on the cake? Delta Live Tables! They offer an end-to-end structured ETL platform that automatically manages metadata for you. This translates into less time spent on maintenance and more time for insights! In conclusion: - Data Lakes 🌊 : Raw Storage - Delta Lakes ⛰️ : ACID Compliant + Versioning + Schema Evaluation - Delta Live Tables 📊 : Automated Metadata Management Let's continue harnessing these tools to create more efficient, scalable solutions 💡 #DataLake #DeltaLake #DeltaLiveTable #DataEngineering #CloudComputing

Efficient management and structured cataloguing of #metadata enable a clear understanding of data provenance and its trustworthiness. Find out more about our Deep Dive for companies with heterogeneous data landscapes: https://lnkd.in/dNYi2MeC #DataManagement #Metadata #DataDrivenCompany

Embedded vs. discrete #metadata. From the perspective of where metadata is stored, there are two kinds. 1.     Embedded Metadata is part of the data object itself, like a Last Updated Column on a record. 2.     Discrete Metadata is metadata about a data object that is stored somewhere other than in the data object itself, like a definition of a column stored in a #datacatalog. A couple of worries: (a) For the same data object, some metadata may be Discrete and some Embedded, so we don’t get a single view of its metadata; (b) The relationship between Discrete Metadata and the data object it is describing needs more management effort to make sure the metadata remains accurate. Overall conclusion is that Metadata Architecture is a thing, and we need to pay attention to it. Agree, or am I mistaken? #datagovernance #database #datamanagment #datascience #analytics #metadata #datamesh