Prague data management meetup 2018-03-27
•
1 like•215 views
This document discusses different data types and data models. It begins by describing unstructured, semi-structured, and structured data. It then discusses relational and non-relational data models. The document notes that big data can include any of these data types and models. It provides an overview of Microsoft's data management and analytics platform and tools for working with structured, semi-structured, and unstructured data at varying scales. These include offerings like SQL Server, Azure SQL Database, Azure Data Lake Store, Azure Data Lake Analytics, HDInsight and Azure Data Warehouse.
Report
Share
Prague data management meetup 2018-03-27
- 3. Considering Data Types Audio, video, images. Meaningless without adding some structure Unstructured JSON, XML, sensor data, social media, device data, web logs. Flexible data model structure Semi-Structured Structured CSV, Columnar Storage (Parquet, ORC). Strict data model structure Relational databases (RDBMS) work with structured data. Non-relational databases (NoSQL) work with semi-structured data Relational data and non-relational data are data models, describing how data is organized. Structured, semi-structured, and unstructured data are data types
- 4. Big Data = All Data! What is Big Data? • Variety: It can be structured, semi-structured, or unstructured • Velocity: It can be streaming, near real-time or batch • Volume: It can be 1GB or 1PB • Big data is the new currency
- 6. Any BI tool Advanced Analytics Any languageBig Data processing Data warehousing Relational data Dashboards | Reporting Mobile BI | Cubes Machine Learning Stream analytics | Cognitive | AI .NET | Java | R | Python Ruby | PHP | Scala Non-relational data Datavirtualization OLTP ERP CRM LOB The Data Management Platform for Analytics Social media DevicesWeb Media On-premises Cloud
- 8. SMP vs MPP • Uses many separate CPUs running in parallel to execute a single program • Shared Nothing: Each CPU has its own memory and disk (scale-out) • Segments communicate using high-speed network between nodes MPP - Massively Parallel Processing • Multiple CPUs used to complete individual processes simultaneously • All CPUs share the same memory, disks, and network controllers (scale-up) • All SQL Server implementations up until now have been SMP • Mostly, the solution is housed on a shared SAN SMP - Symmetric Multiprocessing
- 9. Big Data Solutions Decision Tree Thanks to Ivan Kosyakov: https://biz-excellence.com/2016/08/30/big-data-dt/
- 11. Velocity Volume Per Day Real-world Transactions Per Day Real-world Transactions Per Second Relational DB Document Store Key Value or Wide Column 8 GB 8.64B 100,000 As Is 86 GB 86.4B 1M Tuned* As Is 432 GB 432B 5M Appliance Tuned* As Is 864 GB 864B 10M Clustered Appliance Clustered Servers Tuned* 8,640 GB 8.64T 100M Many Clustered Servers Clustered Servers 43,200 GB 43.2T 500M Many Clustered Servers * Tuned means tuning the model, queries, and/or hardware (more CPU, RAM, and Flash)
- 12. Microsoft data platform solutions Product Category Description More Info SQL Server 2016 RDBMS Earned top spot in Gartner’s Operational Database magic quadrant. JSON support. Linux TBD https://www.microsoft.com/en-us/server- cloud/products/sql-server-2016/ SQL Database RDBMS/DBaaS Cloud-based service that is provisioned and scaled quickly. Has built-in high availability and disaster recovery. JSON support https://azure.microsoft.com/en- us/services/sql-database/ SQL Data Warehouse MPP RDBMS/DBaaS Cloud-based service that handles relational big data. Provision and scale quickly. Can pause service to reduce cost https://azure.microsoft.com/en- us/services/sql-data-warehouse/ Analytics Platform System (APS) MPP RDBMS Big data analytics appliance for high performance and seamless integration of all your data https://www.microsoft.com/en-us/server- cloud/products/analytics-platform- system/ Azure Data Lake Store Hadoop storage Removes the complexities of ingesting and storing all of your data while making it faster to get up and running with batch, streaming, and interactive analytics https://azure.microsoft.com/en- us/services/data-lake-store/ Azure Data Lake Analytics On-demand analytics job service/Big Data-as-a- service Cloud-based service that dynamically provisions resources so you can run queries on exabytes of data. Includes U- SQL, a new big data query language https://azure.microsoft.com/en- us/services/data-lake-analytics/ HDInsight PaaS Hadoop compute/Hadoop clusters-as-a-service A managed Apache Hadoop, Spark, R, HBase, Kafka, and Storm cloud service made easy https://azure.microsoft.com/en- us/services/hdinsight/ Azure Cosmos DB PaaS NoSQL: Document Store Get your apps up and running in hours with a fully managed NoSQL database service that indexes, stores, and queries data using familiar SQL syntax https://azure.microsoft.com/en- us/services/documentdb/ Azure Table Storage PaaS NoSQL: Key-value Store Store large amount of semi-structured data in the cloud https://azure.microsoft.com/en- us/services/storage/tables/
- 13. Microsoft Big Data Portfolio SQL Server Stretch Business intelligence Machine learning analytics Insights Azure SQL Database SQL Server 2016 SQL Server 2016 Fast Track Azure SQL DW ADLS & ADLA Cosmos DB HDInsight Hadoop Analytics Platform System Sequential Scale Out + AcrossScale Up Key Relational Non-relational On-premisesCloud Microsoft has solutions covering and connecting all four quadrants – that’s why SQL Server is one of the most utilized databases in the world
- 14. Azure SQL Data Warehouse A relational data warehouse-as-a-service, fully managed by Microsoft. Industries first elastic cloud data warehouse with enterprise-grade capabilities. Support your smallest to your largest data storage needs while handling queries up to 100x faster.
- 15. Azure Data Lake Store A hyper-scale repository for Big Data analytics workloads Hadoop File System (HDFS) for the cloud No limits to scale Store any data in its native format Enterprise-grade access control, encryption at rest Optimized for analytic workload performance
- 16. Data lake is the center of a big data solution A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until it is needed. • Inexpensively store unlimited data • Collect all data “just in case” • Store data with no modeling – “Schema on read” • Complements EDW • Frees up expensive EDW resources • Quick user access to data • ETL Hadoop tools • Easily scalable • With Hadoop, high availability built in
- 17. Data Analysis Paradigm Shift
- 18. Data Lake Transformation (ELT not ETL) New Approaches All data sources are considered Leverages the power of on-prem technologies and the cloud for storage and capture Native formats, streaming data, big data Extract and load, no/minimal transform Storage of data in near-native format Orchestration becomes possible Streaming data accommodation becomes possible Refineries transform data on read Produce curated data sets to integrate with traditional warehouses Users discover published data sets/services using familiar tools CRMERPOLTP LOB DATA SOURCES FUTURE DATA SOURCESNON-RELATIONAL DATA EXTRACT AND LOAD DATA LAKE DATA REFINERY PROCESS (TRANSFORM ON READ) Transform relevant data into data sets BI AND ANALYTCIS Discover and consume predictive analytics, data sets and other reports DATA WAREHOUSE Star schemas, views other read- optimized structures
- 19. Data Analysis Paradigm Shift OLD WAY: Structure -> Ingest -> Analyze NEW WAY: Ingest -> Analyze -> Structure This solves the two biggest reasons why may EDW projects fail: • Too much time spent modeling when you don’t know all of the questions your data needs to answer • Wasted time spent on ETL where the net effect is a star schema that doesn’t actually show value
- 20. Data Lake layers • Raw data layer– Raw events are stored for historical reference. Also called staging layer or landing area • Cleansed data layer – Raw events are transformed (cleaned and mastered) into directly consumable data sets. Aim is to uniform the way files are stored in terms of encoding, format, data types and content (i.e. strings). Also called conformed layer • Application data layer – Business logic is applied to the cleansed data to produce data ready to be consumed by applications (i.e. DW application, advanced analysis process, etc). This is also called by a lot of other names: workspace, trusted, gold, secure, production ready, governed • Sandbox data layer – Optional layer to be used to “play” in. Also called exploration layer or data science workspace Still need data governance so your data lake does not turn into a data swamp!
- 21. Azure HDInsight Hadoop and Spark as a Service on Azure Fully-managed Hadoop and Spark for the cloud 100% Open Source Hortonworks data platform Clusters up and running in minutes Managed, monitored and supported by Microsoft with the industry’s best SLA Familiar BI tools for analysis, or open source notebooks for interactive data science 63% lower TCO than deploy your own Hadoop on-premises* *IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”
- 22. Azure Data Lake Analytics A new distributed analytics service Distributed analytics service built on Apache YARN Elastic scale per query lets users focus on business goals—not configuring hardware Includes U-SQL—a language that unifies the benefits of SQL with the expressive power of C# Integrates with Visual Studio to develop, debug, and tune code faster Federated query across Azure data sources Enterprise-grade role based access control
- 23. Query data where it lives Easily query data in multiple Azure data stores without moving it to a single store Benefits • Avoid moving large amounts of data across the network between stores (federated query/logical data warehouse) • Single view of data irrespective of physical location • Minimize data proliferation issues caused by maintaining multiple copies • Single query language for all data • Each data store maintains its own sovereignty • Design choices based on the need • Push SQL expressions to remote SQL sources • Filters, Joins • SELECT * FROM EXTERNAL MyDataSource EXECUTE @”Select CustName from Customers WHERE ID=1”; (remote queries) • SELECT CustName FROM EXTERNAL MyDataSource LOCATION “dbo.Customers” WHERE ID=1 (federated queries) U-SQL Query Query Azure Storage Blobs Azure SQL in VMs Azure SQL DB Azure Data Lake Analytics Azure SQL Data Warehouse Azure Data Lake Storage
- 24. PolyBase Query relational and non-relational data with T-SQL PolyBase extents T-SQL onto data via views
- 27. Azure Data Lake Store Azure Blob Storage Purpose Optimized for big data analytics General purpose bulk storage Use Cases Batch, Interactive, Streaming App backend, backup data, media storage for streaming Units of Storage Accounts / Folders / Files Accounts / Containers / Blobs Structure Hierarchical File System Flat namespace WebHDFS Implements WebHDFS No (WASB) Security AD SAS keys Storage Auto Shared/Files chunked Manually manage expansion/Files intact Size Limits No limits on account size, file size, # files 500TB account, 4.75TB file Service State Generally Available Generally Available Billing Pay for data stored and for I/O Pay for data stored and for I/O Region Availability Two US regions (East, Central) & North Europe (At the moment) All Azure Regions ADL Store vs Blob Store
- 28. Want Hadoop? Need exact same on- prem Need interactive / streaming? Mandatory No strong opinion Azure Marketplace (IaaS) • Need all workloads exactly like on- premises • Need 100% Hortonworks/Cloudera/MapR Azure HDInsight • Most Hadoop workloads • Fully managed by Microsoft • Sell HDI + ADLS • Stickier to Microsoft than VMs • Can do interactive (Spark) and streaming (Storm/Spark) Azure Data Lake Analytics • Easiest experience for admin: no sense of clusters, instant scale per job • Easiest experience for developers: Visual Studio/U-SQL (C#+SQL) • Sell ADLA + ADLS • Batch workloads only Need everything exactly like on-prem Need core projects Yes Batch is OK Always present ADLA if .NET or Visual Studio Shop If .NET or VS shop?
- 29. Azure SQL DW HDInsight Hive HDInsight Spark ADLS/ADLA SQL Server (IaaS) Volume Petabytes Petabytes Petabytes Petabytes Terabytes Security Encryption, TD, Audit ADLS / Apache Ranger ADLS AAD Security Groups (data) Encryption, TD Audit Languages T-SQL (subset) HiveQL SparkSQL, HiveQL, Scala, Java, Python, R U-SQL T-SQL Extensibility No Yes, .NET/SerDe Yes, Packages Yes, .NET Yes, .NET CLR External File Types ORC, TXT, Parquet, RCFile ORC, CSV, Parquet + others Parquet, JSON, Hive + others Many ORC, TXT, Parquet, RCFile Admin Low-Medium Medium-High Medium-High Low High Cost Model DWU Nodes & VM Nodes & VM Units/Jobs VM Schema Definition Schema on Write / Polybase Schema on Read Schema on Read Schema on Read Schema on Write / Polybase Max DB Size 240TB Comp (5X = 1PB) Unlimited 256B (64 4TB drives)
- 30. Data Lake Data Warehouse Complementary to DW Can be sourced from Data Lake Schema-on-read Schema-on-write Physical collection of uncurated data Data of common meaning System of Insight: Unknown data to do experimentation / data discovery System of Record: Well-understood data to do operational reporting Any type of data Limited set of data types (ie. relational) Skills are limited Skills mostly available All workloads – batch, interactive, streaming, machine learning Optimized for interactive querying
- 31. Roles when using both Data Lake and DW Data Lake/Hadoop (staging and processing environment) • Batch reporting • Data refinement/cleaning • ETL workloads • Store historical data • Sandbox for data exploration • One-time reports • Data scientist workloads • Quick results Data Warehouse/RDBMS (serving and compliance environment) • Low latency • High number of users • Additional security • Large support for tools • Easily create reports (Self-service BI) • A data lake is just a glorified file folder with data files in it – how many end-users can accurately create reports from it?
- 33. Lambda Architecture : Interactive Analytics Pipeline
- 34. Lambda Architecture : Interactive Analytics Pipeline
- 35. Lambda Architecture : Interactive Analytics Pipeline Layer Description Azure Capabilities Batch Layer Stores master dataset , high latency , horizontal scalable Data will get appended and stored (Batch View) Azure HDInsight , Azure Blob storage Speed Layer Stream processing of data , stored limited data, dynamic computation Processed in real-time and stored for both read & write operations (real-time view) Azure Stream Analytics , Azure HDInsight Spark Serving Layer Queries batch & real-time views , merge results Indexes batch views / out of date results Power BI
- 36. Microsoft Products vs Hadoop/OSS Products Microsoft Product Hadoop/Open Source Software Product Office365/Excel OpenOffice/Calc DocumentDB MongoDB, HBase, Cassandra SQL Database SQLite, MySQL, PostgreSQL, MariaDB Azure Data Lake Analytics/YARN None Azure VM/IaaS OpenStack Blob Storage HDFS, Ceph (Note: These are distributed file systems and Blob storage is not distributed) Azure HBase Apache HBase (Azure HBase is a service wrapped around Apache HBase), Apache Trafodion Event Hub Apache Kafka Azure Stream Analytics Apache Storm, Apache Spark, Twitter Heron Power BI Apache Zeppelin, Apache Jupyter, Airbnb Caravel, Kibana HDInsight Hortonworks (pay), Cloudera (pay), MapR (pay) Azure ML Apache Mahout, Apache Spark MLib Microsoft R Open R SQL Data Warehouse Apache Hive, Apache Drill, Presto IoT Hub Apache NiFi Azure Data Factory Apache Falcon, Apache Oozie, Airbnb Airflow Azure Data Lake Storage/WebHDFS HDFS Ozone Azure Analysis Services/SSAS Apache Kylin, Apache Lens, AtScale (pay) SQL Server Reporting Services None Hadoop Indexes Jethro Data (pay) Azure Data Catalog Apache Atlas PolyBase Apache Drill Azure Search Apache Solr, Apache ElasticSearch (Azure Search build on ES) Others Apache Flink, Apache Ambari, Apache Ranger, Apache Knox Note: Many of the Hadoop/OSS products are available in Azure
- 37. Business apps Custom apps Sensors and devices Events Events Spark Streaming Stream Processing Azure Stream Analytics Event Processing Azure Event Hubs Kafka Events Events
- 38. Choosing a Ingestion Technology Kafka Azure Event Hubs Managed No Yes Ordering Yes Yes Delivery At-least-once At-least-once Lifetime Configurable 1-30 Days Replication Configurable within Region Yes Throughput *nodes 20 throughput units Parallel Clients Yes No MapReduce Yes No Record Size Configurable 256K Cost Low + Admin Low
- 39. Choosing a Stream Processing Technology Azure Stream Analytics Storm Spark Streaming Managed Yes Yes Yes Temporal Operators Windowed aggregates, and temporal joins are supported out of the box. Temporal operators must to be implemented Temporal operators must to be implemented Development Experience Interactive authoring and debugging experience through Azure Portal on sample data. Visual Studio, etc Visual Studio, etc Data Encoding formats Stream Analytics requires UTF-8 data format to be utilized. Any data encoding format may be implemented via custom code. Any data encoding format may be implemented via custom code. Scalability Number of Streaming Units for each job. Each Streaming Unit processes up to 1MB/s. Max of 50 units by default. Call to increase limit. Number of nodes in the HDI Storm cluster. No limit on number of nodes (Top limit defined by your Azure quota). Call to increase limit. Number of nodes in the HDI Spark cluster. No limit on number of nodes (Top limit defined by your Azure quota). Call to increase limit. Data processing limits Users can scale up or down number of Streaming Units to increase data processing or optimize costs. Scale up to 1 GB/s User can scale up or down cluster size to meet needs. User can scale up or down cluster size to meet needs. Late arrival and out of order event handling Built-in configurable policies to reorder, drop events or adjust event time. User must implement logic to handle this scenario. User must implement logic to handle this scenario.