Microsoft Azure Databricks
•
6 likes•1,339 views
Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure. Designed in collaboration with the founders of Apache Spark, Azure Databricks combines the best of Databricks and Azure to help customers accelerate innovation with one-click set up, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts. As an Azure service, customers automatically benefit from the native integration with other Azure services such as Power BI, SQL Data Warehouse, and Cosmos DB, as well as from enterprise-grade Azure security, including Active Directory integration, compliance, and enterprise-grade SLAs.
Microsoft Azure Databricks
- 3. Ease of Use Generality Runs Everywhere Logistic Regression 140 120 100 80 40 20 0 60 Hadoop Spark 0.9
- 4. Speed Generality Runs Everywhere text_file = spark.textFile("hdfs://...") text_file.flatMap(lambda line: line.split()) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a+b) Word count in Spark's Python API
- 6. Speed Ease of Use Runs Everywhere Spark Core Engine Spark SQL Interactive Queries Spark MLlib Machine Learning Spark Streaming Stream Processing GraphX Graph Computation
- 7. Speed Ease of Use Generality Runs Everywhere Spark Core Engine Spark SQL Interactive Queries Spark MLlib Machine Learning Spark Streaming Stream Processing GraphX Graph Computation Yarn Mesos Standalone Scheduler
- 8. Read from HDFS Write to HDFS Read from HDFS Write to HDFS Read from HDFS
- 10. Data Sources (HDFS, SQL, NoSQL, …) Cluster Manager Worker Nodes Driver Program SparkContext
- 13. Microsoft Azure
- 14. Optimized Databricks Runtime Engine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Enhance Productivity Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST Build on secure & trusted cloud Scale without limits
- 15. CONTROL EASE OF USE Azure Data Lake Analytics Azure Data Lake Store Azure Storage Any Hadoop technology, any distribution Workload optimized, managed clusters Data Engineering in a Job-as-a-service model Azure Marketplace HDP | CDH | MapR Azure Data Lake Analytics IaaS Clusters Managed Clusters Big Data as-a-service Azure HDInsight Frictionless & Optimized Spark clusters Azure Databricks BIGDATA STORAGE BIGDATA ANALYTICS ReducedAdministration
Editor's Notes
- Follow the instructions in insidesqlr\readme.txt