This presentation provides a survey of the advanced analytics strengths of Microsoft Azure from an enterprise perspective (with these organizations being the bulk of big data users) based on the Team Data Science Process. The talk also covers the range of analytics and advanced analytics solutions available for developers using data science and artificial intelligence from Microsoft Azure.
Azure Databricks—Apache Spark as a Service with Sascha DittmannDatabricks
The driving force behind Apache Spark (Databricks Inc.) and Microsoft have designed a joint service to quickly and easily create Big Data and Advanced Analytics solutions. The combination of the comprehensive Databricks Unified Analytics platform and the powerful capabilities of Microsoft Azure make it easy to analyse data streams or large amounts of data, as well asthe training of AI models. Sascha Dittmann shows in this session how the new Azure service can be set up and used in various real-world scenarios. He also shows, how to connect the various Azure Services to the Azure Databricks service.
This document discusses designing a modern data warehouse in Azure. It provides an overview of traditional vs. self-service data warehouses and their limitations. It also outlines challenges with current data warehouses around timeliness, flexibility, quality and findability. The document then discusses why organizations need a modern data warehouse based on criteria like customer experience, quality assurance and operational efficiency. It covers various approaches to ingesting, storing, preparing, modeling and serving data on Azure. Finally, it discusses architectures like the lambda architecture and common data models.
Databricks is a Software-as-a-Service-like experience (or Spark-as-a-service) that is a tool for curating and processing massive amounts of data and developing, training and deploying models on that data, and managing the whole workflow process throughout the project. It is for those who are comfortable with Apache Spark as it is 100% based on Spark and is extensible with support for Scala, Java, R, and Python alongside Spark SQL, GraphX, Streaming and Machine Learning Library (Mllib). It has built-in integration with many data sources, has a workflow scheduler, allows for real-time workspace collaboration, and has performance improvements over traditional Apache Spark.
Spark as a Service with Azure DatabricksLace Lofranco
Presented at: Global Azure Bootcamp (Melbourne)
Participants will get a deep dive into one of Azure’s newest offering: Azure Databricks, a fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure. In this session, we will go through Azure Databricks key collaboration features, cluster management, and tight data integration with Azure data sources. We’ll also walk through an end-to-end Recommendation System Data Pipeline built using Spark on Azure Databricks.
Building data pipelines for modern data warehouse with Apache® Spark™ and .NE...Michael Rys
This presentation shows how you can build solutions that follow the modern data warehouse architecture and introduces the .NET for Apache Spark support (https://dot.net/spark, https://github.com/dotnet/spark)
The document discusses how companies can use big data analytics and Azure Databricks to improve their customer experiences and grow their business. It provides an overview of how Wide World Importers seeks to expand its customers through an omni-channel strategy using analytics from data across its retail stores, website, and mobile apps. The document also outlines logical architectures for ingesting, storing, preparing, training models on, and serving data using Azure Databricks and other Azure services.
Ai & Data Analytics 2018 - Azure Databricks for data scientistAlberto Diaz Martin
This document summarizes a presentation given by Alberto Diaz Martin on Azure Databricks for data scientists. The presentation covered how Databricks can be used for infrastructure management, data exploration and visualization at scale, reducing time to value through model iterations and integrating various ML tools. It also discussed challenges for data scientists and how Databricks addresses them through features like notebooks, frameworks, and optimized infrastructure for deep learning. Demo sections showed EDA, ML pipelines, model export, and deep learning modeling capabilities in Databricks.
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...Databricks
At Lennox International, we have thousands of IoT connected devices streaming data into the Azure platform with a minute level polling interval. The challenge was to use these data sets, combine with external data sources such as weather, and predict equipment failure with high levels of accuracy along with their influencing patterns and parameters. Previously the team was using a combination of on-premise and desktop tools to run algorithms on a sample set of devices. The result was low accuracy levels (around 65%) on a process that took more than 6 hours.
The team had to work through several data orchestration challenges and identify a machine learning platform which enabled them to collaborate between our engineering SME’s, Data Engineers and Data Scientists. The team decided to use Azure Databricks to build the data engineering pipelines, appropriate machine learning models and extract predictions using PySpark. To enhance the sophistication of the learning, the team worked on a variety of Spark ML models such as Gradient Boosted Trees and Random Forest. The team also implemented stacking, ensemble methods using H2O driverless AI and sparkling water on Azure Databricks clusters, which can scale up to 1000 cores.
Join us in this session and see how this resulted in models that run in 40 minutes with minimal tuning and predict failures with accuracy of about 90%.
Data Con LA 2020
Description
Data warehouses are not enough. Data lakes are the backbone of a modern data environment. Data Lakes are best built leveraging unique services of the cloud provider to reduce operations complexity. This session will explain why everyone's talking about data lakes, break down the best services in Azure to build a Data Lake, and walk through code for querying and loading with Azure Databricks and Event Hubs for Kafka. Attendees will leave the session with a firm grasp of why we build data lakes and how Azure Databricks fits in for ETL and querying.
Speaker
Dustin Vannoy, Dustin Vannoy Consulting, Principal Data Engineer
Einstieg in Machine Learning für DatenbankentwicklerSascha Dittmann
Hast Du Dich als Datenbankentwickler schon einmal gefragt, wie Du Deine Datenbank-Projekte mit Machine Learning Technologien erweitern kannst?
Wie kannst Du Dein vorhandenes Wissen wiederverwenden und was muss Du noch lernen?
In dieser Session stellt Sascha Dittmann verschiedene Lernpfade vor, um als Datenbankentwickler in die Welt des Data Science eintauchen zu können. Für seine Praxisbeispiele nutzt er dabei verschiedene Werkzeuge, wie beispielsweise die SQL Server ML Services, Azure Databricks und die Azure ML Services, um bekanntes Wissen mit Neuen zu vereinen.
This document discusses using Azure HDInsight for big data applications. It provides an overview of HDInsight and describes how it can be used for various big data scenarios like modern data warehousing, advanced analytics, and IoT. It also discusses the architecture and components of HDInsight, how to create and manage HDInsight clusters, and how HDInsight integrates with other Azure services for big data and analytics workloads.
Building Advanced Analytics Pipelines with Azure DatabricksLace Lofranco
Participants will get a deep dive into one of Azure’s newest offering: Azure Databricks, a fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure. In this session, we start with a technical overview of Spark and quickly jump into Azure Databricks’ key collaboration features, cluster management, and tight data integration with Azure data sources. Concepts are made concrete via a detailed walk through of an advance analytics pipeline built using Spark and Azure Databricks.
Full video of the presentation: https://www.youtube.com/watch?v=14D9VzI152o
Presentation demo: https://github.com/devlace/azure-databricks-anomaly
Cortana Analytics Workshop: Azure Data LakeMSAdvAnalytics
Rajesh Dadhia. This session introduces the newest services in the Cortana Analytics family. Azure Data Lake is a hyper-scale data repository designed for big data analytics workloads. It provides a single place to store any type of data in its native format. In this session, we will show how the HDFS compatibility of Azure Data Lake as a Hadoop File System enables all Hadoop workloads including Azure HDInsight, Hortonworks and Cloudera. Further, we will focus on the key capabilities of the Azure Data Lake that make it an ideal choice for storing, accessing and sharing data for a wide range of analytics applications. Go to https://channel9.msdn.com/ to find the recording of this session.
These are the slides for my talk "An intro to Azure Data Lake" at Azure Lowlands 2019. The session was held on Friday January 25th from 14:20 - 15:05 in room Santander.
This document provides an overview of Azure Databricks, including:
- Azure Databricks is an Apache Spark-based analytics platform optimized for Microsoft Azure cloud services. It includes Spark SQL, streaming, machine learning libraries, and integrates fully with Azure services.
- Clusters in Azure Databricks provide a unified platform for various analytics use cases. The workspace stores notebooks, libraries, dashboards, and folders. Notebooks provide a code environment with visualizations. Jobs and alerts can run and notify on notebooks.
- The Databricks File System (DBFS) stores files in Azure Blob storage in a distributed file system accessible from notebooks. Business intelligence tools can connect to Databricks clusters via JDBC
Machine learning allows us to build predictive analytics solutions of tomorrow - these solutions allow us to better diagnose and treat patients, correctly recommend interesting books or movies, and even make the self-driving car a reality. Microsoft Azure Machine Learning (Azure ML) is a fully-managed Platform-as-a-Service (PaaS) for building these predictive analytics solutions. It is very easy to build solutions with it, helping to overcome the challenges most businesses have in deploying and using machine learning. In this presentation, we will take a look at how to create ML models with Azure ML Studio and deploy those models to production in minutes.
Modern DW Architecture
- The document discusses modern data warehouse architectures using Azure cloud services like Azure Data Lake, Azure Databricks, and Azure Synapse. It covers storage options like ADLS Gen 1 and Gen 2 and data processing tools like Databricks and Synapse. It highlights how to optimize architectures for cost and performance using features like auto-scaling, shutdown, and lifecycle management policies. Finally, it provides a demo of a sample end-to-end data pipeline.
Cortana Analytics Suite is a fully managed big data and advanced analytics suite that transforms your data into intelligent action. It is comprised of data storage, information management, machine learning, and business intelligence software in a single convenient monthly subscription. This presentation will cover all the products involved, how they work together, and use cases.
The breath and depth of Azure products that fall under the AI and ML umbrella can be difficult to follow. In this presentation I’ll first define exactly what AI, ML, and deep learning is, and then go over the various Microsoft AI and ML products and their use cases.
Automated machine learning (automated ML) automates feature engineering, algorithm and hyperparameter selection to find the best model for your data. The mission: Enable automated building of machine learning with the goal of accelerating, democratizing and scaling AI. This presentation covers some recent announcements of technologies related to Automated ML, and especially for Azure. The demonstrations focus on Python with Azure ML Service and Azure Databricks.
Slides from my talk at Big Data Conference 2018 in Vilnius
Doing data science today is far more difficult than it will be in the next 5-10 years. Sharing, collaborating on data science workflows in painful, pushing models into production is challenging.
Let’s explore what Azure provides to ease Data Scientists’ pains. What tools and services can we choose based on a problem definition, skillset or infrastructure requirements?
In this talk, you will learn about Azure Machine Learning Studio, Azure Databricks, Data Science Virtual Machines and Cognitive Services, with all the perks and limitations.
This document discusses Azure Machine Learning services for data scientists. It provides an overview of Azure Machine Learning Studio for building and deploying machine learning models with over 100 modules. Numbers show hundreds of thousands of deployed models serving billions of requests. It also discusses Azure Batch AI for scalable machine learning training without managing infrastructure, and Azure Databricks for Apache Spark as a managed service on Azure. The document outlines the machine learning development lifecycle supported in Azure and tools for experimentation, model management, and operationalization of models.
This document provides an overview of a course on implementing a modern data platform architecture using Azure services. The course objectives are to understand cloud and big data concepts, the role of Azure data services in a modern data platform, and how to implement a reference architecture using Azure data services. The course will provide an ARM template for a data platform solution that can address most data challenges.
Introduction to Machine learning and Deep LearningNishan Aryal
Overview of Machine Learning and Deep Learning. Brief introduction to different types of BI Reporting tools like Power BI, SSMS, Cortana, Azure ML, TenserFlow and other tools.
Azure Machine Learning Services provides an end-to-end, scalable platform for operationalizing machine learning models. It allows users to deploy models everywhere from containers and Kubernetes to SQL Datawarehouse and Cosmos DB. It also offers tools to boost data science productivity, increase experimentation, and automate model retraining. The platform seamlessly integrates with Azure services and is built to deploy models globally at scale with high availability and low latency.
Join us for a deep dive into Windows Azure. We’ll start with a developer-focused overview of this brave new platform and the cloud computing services that can be used either together or independently to build amazing applications. As the day unfolds, we’ll explore data storage, SQL Azure™, and the basics of deployment with Windows Azure. Register today for these free, live sessions in your local area.
Big Data Advanced Analytics on Microsoft Azure 201904Mark Tabladillo
This talk summarizes key points for big data advanced analytics on Microsoft Azure. First, there is a review of the major technologies. Second, there is a series of technology demos (focusing on VMs, Databricks and Azure ML Service). Third, there is some advice on using the Team Data Science Process to help plan projects. The deck has web resources recommended. This presentation was delivered at the Global Azure Bootcamp 2019, Atlanta GA location (Alpharetta Avalon).
MLflow and Azure Machine Learning—The Power Couple for ML Lifecycle ManagementDatabricks
The ML Lifecycle management process is quickly becoming the bottleneck for a lot of ML projects. With MLflow’s newest release, and its enhanced integration with Azure Machine Learning, this process is now showing the right promise and capabilities on Azure. In this talk, we intend to take a tour of the integration details and how MLOps is now becoming a strength of the platform. We’ll talk about versioning, maintaining run history, production pipeline automation, deployment to cloud and edge, and CI/CD pipelines with MLOps as the backdrop.
Be prepared for an interactive conversation as we intend to seek a lot of feedback on the integration and capabilities being lit up.
Developing and deploying AI solutions on the cloud using Team Data Science Pr...Debraj GuhaThakurta
Presented at: Global Big AI Conference, Santa Clara, Jan 2018 Developing and deploying AI solutions on the cloud using Team Data Science Process (TDSP) and Azure Machine Learning (AML)
AI for an intelligent cloud and intelligent edge: Discover, deploy, and manag...James Serra
Discover, manage, deploy, monitor – rinse and repeat. In this session we show how Azure Machine Learning can be used to create the right AI model for your challenge and then easily customize it using your development tools while relying on Azure ML to optimize them to run in hardware accelerated environments for the cloud and the edge using FPGAs and Neural Network accelerators. We then show you how to deploy the model to highly scalable web services and nimble edge applications that Azure can manage and monitor for you. Finally, we illustrate how you can leverage the model telemetry to retrain and improve your content.
Borys Rybak “How to make your data smart with Artificial Intelligence and Mac...Lviv Startup Club
1. The document discusses Microsoft's artificial intelligence capabilities including cognitive services, bots, Azure machine learning, and tools for building AI applications and managing models.
2. It provides an overview of Microsoft's offerings for AI including cognitive services, bots, Azure machine learning studio for building experiments visually, and Azure machine learning services for experimentation and model management.
3. The document emphasizes that Microsoft's goal is to make AI accessible and useful to every person and organization by providing a broad set of tools, frameworks, and infrastructure for developing, training, deploying, and managing AI applications and models.
When it comes to Large Scale data processing and Machine Learning, Apache Spark is no doubt one of the top battle-tested frameworks out there for handling batched or streaming workloads. The ease of use, built-in Machine Learning modules, and multi-language support makes it a very attractive choice for data wonks. However bootstrapping and getting off the ground could be difficult for most teams without leveraging a Spark cluster that is already pre-provisioned and provided as a managed service in the Cloud, while this is a very attractive choice to get going, in the long run, it could be a very expensive option if it’s not well managed.
As an alternative to this approach, our team has been exploring and working a lot with running Spark and all our Machine Learning workloads and pipelines as containerized Docker packages on Kubernetes. This provides an infrastructure-agnostic abstraction layer for us, and as a result, it improves our operational efficiency and reduces our overall compute cost. Most importantly, we can easily target our Spark workload deployment to run on any major Cloud or On-prem infrastructure (with Kubernetes as the common denominator) by just modifying a few configurations.
In this talk, we will walk you through the process our team follows to make it easy for us to run a production deployment of our Machine Learning workloads and pipelines on Kubernetes which seamlessly allows us to port our implementation from a local Kubernetes set up on the laptop during development to either an On-prem or Cloud Kubernetes environment
.Net development with Azure Machine Learning (AzureML) Nov 2014Mark Tabladillo
Azure Machine Learning provides enterprise-class machine learning and data mining to the cloud. This presenter will cover 1) what AzureML is, 2) technical overview of AzureML for application development, 3) a reminder to consider SQL Server Data Mining, and 4) a recommend path for resources and next steps.
A practical guidance of the enterprise machine learning Jesus Rodriguez
This session provides an analysis of the machine learning market in the enterprise. The analysis includes vendors, platforms and best practices that should be considered by companies implementing data science solutions at an enterprise scale
2018 11 14 Artificial Intelligence and Machine Learning in AzureBruno Capuano
Slides used during my session "Artificial Intelligence and Machine Learning in Azure" for The Azure Group (Canada's Azure User Community) on November 14 2018.
Public group
Differentiate Big Data vs Data Warehouse use cases for a cloud solutionJames Serra
It can be quite challenging keeping up with the frequent updates to the Microsoft products and understanding all their use cases and how all the products fit together. In this session we will differentiate the use cases for each of the Microsoft services, explaining and demonstrating what is good and what isn't, in order for you to position, design and deliver the proper adoption use cases for each with your customers. We will cover a wide range of products such as Databricks, SQL Data Warehouse, HDInsight, Azure Data Lake Analytics, Azure Data Lake Store, Blob storage, and AAS as well as high-level concepts such as when to use a data lake. We will also review the most common reference architectures (“patterns”) witnessed in customer adoption.
Similar to Big Data Adavnced Analytics on Microsoft Azure (20)
How to find low-cost or free data science resources 202006Mark Tabladillo
This document provides guidance on finding low-cost or free data science resources. It categorizes resources for beginners, intermediates, and advanced learners. Beginner resources like KD Nuggets and YouTube aim for likes and clicks. Intermediate resources like Meetup and MOOCs want identity and participation. Advanced resources want trading relationships where value is exchanged. The document recommends specific free resources from Microsoft like Azure documentation and AI Business School. It emphasizes understanding the motivations behind "free" to build respectful relationships within the free economy.
Microsoft has released Automated ML technologies for developers through ML.NET, Azure ML Service, and Azure Databricks. This presenter is a data scientist and Microsoft architect, and will give a comprehensive overview of the utility and use case of this automated technology for production solutions. The presentation includes code you can try now.
ML.NET 1.0 release is the first major milestone of a great journey that started in May 2018 when we released ML.NET 0.1 as open source. ML.NET is an open-source and cross-platform machine learning framework for .NET developers. Using ML.NET, developers can leverage their existing tools and skillsets to develop and infuse custom AI into their applications by creating custom machine learning models for common scenarios like Sentiment Analysis, Recommendation, Image Classification and more.
This presentation provides an overview of the technology with demos run in a Deep Learning Virtual Machine running Windows Server 2016. Code examples are in C# and F# and run in Visual Studio Community 2019. This technology is ready for production implementation and runs on .NET Core.
This presentation is the first of four related to ML.NET and Automated ML. The presentation will be recorded with video posted to this YouTube Channel: http://bit.ly/2ZybKwI
Automated machine learning (automated ML) automates feature engineering, algorithm and hyperparameter selection to find the best model for your data. The mission: Enable automated building of machine learning with the goal of accelerating, democratizing and scaling AI.
This presentation covers some recent announcements of technologies related to Automated ML, and especially for Azure. The demonstrations focus on Python with Azure ML Service and Azure Databricks.
This presentation is the fourth of four related to ML.NET and Automated ML. The presentation will be recorded with video posted to this YouTube Channel: http://bit.ly/2ZybKwI
NimbusML enables data scientists to use ML.NET to train models in Azure Machine Learning or anywhere else they use Python. NimbusML provides state-of-the-art ML algorithms, transforms and components, aiming to make them useful for all developers, data scientists, and information workers and helpful in all products, services and devices. The components are authored by the team members, as well as numerous contributors from MSR, CISL, Bing and other teams at Microsoft. NimbusML is interoperable with scikit-learn estimators and transforms, while adding a suite of highly optimized algorithms written in C++ and C# for speed and performance.
The trained machine learning model can be used in a .NET application with ML.NET. This presentation will outline the features of NimbusML and provide a notebook-based demonstration using Azure Notebooks.
This presentation is the third of four related to ML.NET and Automated ML. The presentation will be recorded with video posted to this YouTube Channel: http://bit.ly/2ZybKwI
201906 02 Introduction to AutoML with ML.NET 1.0Mark Tabladillo
ML.NET 1.0 release is the first major milestone of a great journey that started in May 2018 when we released ML.NET 0.1 as open source. ML.NET is an open-source and cross-platform machine learning framework for .NET developers. Using ML.NET, developers can leverage their existing tools and skillsets to develop and infuse custom AI into their applications by creating custom machine learning models for common scenarios like Sentiment Analysis, Recommendation, Image Classification and more.
“Automated ML” is a collection of new technologies from Microsoft to enhance the data science development process. Still in preview, Auto ML for ML.NET 1.0 will be demonstrated in a Deep Learning Virtual Machine running Windows Server 2016. Code examples are in C# and run in Visual Studio Community 2019.
This presentation is the second of four related to ML.NET and Automated ML. The presentation will be recorded with video posted to this YouTube Channel: http://bit.ly/2ZybKwI
201905 Azure Certification DP-100: Designing and Implementing a Data Science ...Mark Tabladillo
This document provides an overview and learning resources for the DP-100: Designing and Implementing a Data Science Solution on Azure certification. It includes links to learn about the benefits of Microsoft certification, an overview of Microsoft certification, a learning path for DP-100, and descriptions of key Azure technologies for data science like Azure Machine Learning Studio, the Data Science VM, and the Azure Machine Learning service. The document also recommends additional certifications to pursue and provides ways to connect with the author.
This presentation anchors best practices for Enterprise Data Science based on Microsoft's "Team Data Science Process". The talk includes introducing the concepts, describing some real-world advice for project planning, and discusses typical titles of professionals who make enterprise data science successful. These techniques also apply for AI (artificial intelligence), deep learning, machine learning, and advanced analytics.
Power BI has become an increasingly important data analytics tool. This presentation focuses on the advanced analytics options currently available in Power BI. Attendees to this talk will see:
· Microsoft’s perspective on advanced analytics development: the Team Data Science Process
· What the general options are for advanced analytics on Azure
· What the specific native advanced analytics capabilities are in Power BI
· Some ideas on pairing Power BI with other technologies in advanced analytics architectures
Microsoft Cognitive Toolkit (Atlanta Code Camp 2017)Mark Tabladillo
The Microsoft Cognitive Toolkit (CNTK) is a unified deep-learning toolkit that describes neural networks as a series of computational steps via a directed graph. In this directed graph, leaf nodes represent input values or network parameters, while other nodes represent matrix operations upon their inputs.
The objectives of this presentation is to 1) describe what CNTK is, 2) present a comparative evaluation with similar technologies, 3) outline potential applications, and 4) demonstrate the technology with Jupyter Python examples.
Machine learning services with SQL Server 2017Mark Tabladillo
SQL Server 2017 introduces Machine Learning Services with two independent technologies: R and Python. The purpose of this presentation is 1) to describe major features of this technology for technology managers; 2) to outline use cases for architects; and 3) to provide demos for developers and data scientists.
Microsoft Technologies for Data Science 201612Mark Tabladillo
The document discusses Microsoft technologies that can be used for data science, including SQL Server, Azure ML, Cortana Intelligence Suite, and R Server. It provides definitions of key terms like data science, machine learning, and data mining. It also shares links to resources for learning about Microsoft's data science tools and platforms.
How Big Companies plan to use Our Big Data 201610Mark Tabladillo
Underneath the shiny popular apps on tablets, smartphones, and entertainment channels are typically large cloud-based data centers. App developers leverage the cloud to provide advertisers with targeted sales opportunities, which has been accounting for an ongoing shift from paper to online media. This presentation will provide updated trends and statistics for 2016 on big data usage (based on consumer use), statistical concerns with big data, and the Microsoft big data story.
Georgia Tech Data Science Hackathon September 2016Mark Tabladillo
This document summarizes a hackathon event at Georgia Tech in September 2016 about storytelling with data. It provides examples from the book "Storytelling with Data" by Cole Nussbaumer Knaffic and discusses Microsoft's data science story. It also demonstrates Power BI and the components of Cortana Intelligence including Power BI, Machine Learning, HDInsight and other analytics tools. Contact information is given to connect on LinkedIn and Twitter.
Delivered to SQL Saturday Columbus, GA
Microsoft provides several technologies which can be used for casual to serious data science. This presentation provides an authoritative overview of two major categories: products and services. The products include: SQL Server Analysis Services, Excel Add-in for SSAS, Semantic Search, SQL Server R Services, Microsoft R Technologies, and F#. The services include Cortana Intelligence and Bing Predicts. These technologies have been used by the presenter in various companies and industries, and he will be speaking toward how Microsoft uses these technologies today for its largest Azure customers.
Insider's guide to azure machine learning 201606Mark Tabladillo
This document contains information about SQL Server 2016 including that it provides a consistent experience from on-premises to cloud, has in-memory capabilities built-in across all workloads, and can handle real-time and massive scale workloads. It also contains graphics comparing SQL Server's performance to other database management systems and analytics platforms on the TPC-H benchmark and shows SQL Server ranking first or in the top three.
Window functions are powerful analytic functions built into SQL Server. SQL Server 2005 introduced the core window ranking functions, and SQL Server 2012 added time and statistical percentage window functions. These functions allow for advanced variable creation, and are of direct benefit to people creating features for data science. This talk will also recommend further reading on this topic. The slide deck contains a link to the code on GitHub.
Microsoft Technologies for Data Science 201601Mark Tabladillo
Microsoft provides several technologies in and around SQL Server which can be used for casual to serious data science. This presentation provides an authoritative overview of five major options: SQL Server Analysis Services, Excel Add-in for SSAS, Semantic Search, Microsoft Azure Machine Learning, and F#. Also included are tips on working with Python and R. These technologies have been used by the presenter in various companies and industries.
Microsoft Data Science Technologies: Back Office EditionMark Tabladillo
Microsoft provides several technologies in and around SQL Server which can be used for casual to serious data science. This presentation provides an authoritative overview of five major options: SQL Server Analysis Services, Excel Add-in for SSAS, Semantic Search, Microsoft Azure Machine Learning, and F#. Also included are tips on working with Python and R. These technologies have been used by the presenter in various companies and industries. This presentation will emphasize the back office story for supporting big data processing.
This document summarizes Microsoft technologies that can be used for data science, including SQL Server, Excel, Azure Machine Learning, and F#. It provides an overview of these tools and how they can be used for tasks like data mining, semantic search, and machine learning. Examples of using these technologies for text analysis and benchmarking document processing times are also presented. Links to documentation, forums, and conferences related to applying Microsoft technologies to data science are included.
❻❸❼⓿❽❻❷⓿⓿❼ SATTA MATKA DPBOSS KALYAN MATKA RESULTS KALYAN CHART KALYAN MATKA MATKA RESULT KALYAN MATKA TIPS SATTA MATKA MATKA COM MATKA PANA JODI TODAY
8. The opportunity and challenge of data science in
enterprises
Opportunity: 17% had a well-developed Predictive/Prescriptive Analytics
program in place, while 80% planned on implementing such a program within
five years – Dataversity 2015 Survey
Challenge: Only 27% of the big data projects are regarded as successful –
CapGenimi 2014
Tools & data platforms have matured -
Still a major gap in executing on the potential
9. One reason: Process challenge in Data Science
Organization
Collaboration
Quality
Knowledge Accumulation
Agility
Global Teams
• Geographic Locations
Team Growth
• Onboard New
Members Rapidly
Varied Use Cases
• Industries and Use
Cases
Diverse DS
Backgrounds
• DS have diverse
backgrounds,
experiences with
tools, languages
20. Microsoft | Amplifying Human Ingenuity
Enhance traditional line-of-
business analytics solutions with
Machine Learning
Solve complex business
problems with Deep Learning
Engage customers with
intelligent automated solutions
35. A P A C H E S P A R K
An unified, open source, parallel, data processing framework for Big Data Analytics
Spark Core Engine
Spark SQL
Interactive
Queries
Spark Structured
Streaming
Stream processing
Spark MLlib
Machine
Learning
Yarn Mesos
Standalone
Scheduler
Spark MLlib
Machine
Learning
Spark
Streaming
Stream processing
GraphX
Graph
Computation
37. CONTROL EASE OF USE
Azure Data Lake
Analytics
Azure Data Lake Store
Azure Storage
Any Hadoop technology,
any distribution
Workload optimized,
managed clusters
Data Engineering in a
Job-as-a-service model
Azure Marketplace
HDP | CDH | MapR
Azure Data Lake
Analytics
IaaS Clusters Managed Clusters Big Data as-a-service
Azure HDInsight
Frictionless & Optimized
Spark clusters
Azure Databricks
BIGDATA
STORAGE
BIGDATA
ANALYTICS
ReducedAdministration
K N O W I N G T H E V A R I O U S B I G D A T A S O L U T I O N S
38. Azure HDInsight
What It Is
• Hortonworks distribution as a first party service on Azure
• Big Data engines support – Hadoop Projects, Hive on Tez, Hive
LLAP, Spark, HBase, Storm, Kafka, R Server
• Best-in-class developer tooling and Monitoring capabilities
• Enterprise Features
• VNET support (join existing VNETs)
• Ranger support (Kerberos based Security)
• Log Analytics via OMS
• Orchestration via Azure Data Factory
• Available in most Azure Regions (27) including Gov Cloud
and Federal Clouds
Guidance
• Customer needs Hadoop technologies other than, or in addition
to Spark
• Customer prefers Hortonworks Spark distribution to stay closer
to OSS codebase and/or ‘Lift and Shift’ from on-premises
deployments
• Customer has specific project requirements that are only
available on HDInsight
Azure Databricks
What It Is
• Databricks’ Spark service as a first party service on Azure
• Single engine for Batch, Streaming, ML and Graph
• Best-in-class notebooks experience for optimal productivity and
collaboration
• Enterprise Features
• Native Integration with Azure for Security via AAD (OAuth)
• Optimized engine for better performance and scalability
• RBAC for Notebooks and APIs
• Auto-scaling and cluster termination capabilities
• Native integration with SQL DW and other Azure services
• Serverless pools for easier management of resources
Guidance
• Customer needs the best option for Spark on Azure
• Customer teams are comfortable with notebooks and Spark
• Customers need Auto-scaling and
• Customer needs to build integrated and performant data
pipelines
• Customer is comfortable with limited regional availability (3 in
preview, 8 by GA)
Azure ML
What It Is
• Azure first party service for Machine Learning
• Leverage existing ML libraries or extend with Python and R
• Targets emerging data scientists with drag & drop offering
• Targets professional data scientists with
– Experimentation service
– Model management service
– Works with customers IDE of choice
Guidance
• Azure Machine Learning Studio is a GUI based ML tool for
emerging Data Scientists to experiment and operationalize with
least friction
• Azure Machine Learning Workbench is not a compute engine &
uses external engines for Compute, including SQL Server and
Spark
• AML deploys models to HDI Spark currently
• AML should be able to deploy Azure Databricks in the near future
L O O K I N G A C R O S S T H E O F F E R I N G S
40. P R O V I S I O N I N G A Z U R E D A T A B R I C K S W O R K S P A C E
41. G E N E R A L S P A R K C L U S T E R A R C H I T E C T U R E
Data Sources (HDFS, SQL, NoSQL, …)
Cluster Manager
Worker Node Worker Node Worker Node
Driver Program
SparkContext
42. A Z U R E D A T A B R I C K S C L U S T E R A R C H I T E C T U R E
Azure DB
for
PostgreSQL
Webapp
Azure Compute
Cluster
Manager
Databricks’ Azure Account User’s Azure Account
Azure Compute
Spark
Driver
Azure Compute
Spark
Worker
Azure Compute
Spark
Worker
Jobs
FileSystem
Service
Spark
History
Server
Log
Daemon
Log
Daemon
43. C L U S T E R M A N A G E R A R C H I T E C T U R E
JobsWebapp
Cluster Manager
Cluster
Azure Compute
Spark
Driver
Azure Compute
Spark
Worker
Azure Compute
Spark
Worker
Database
Instances, Clusters, Libraries, Hive Metastore, …
Cluster
Azure Compute
Spark
Driver
Azure Compute
Spark
Worker
Azure Compute
Spark
Worker
Instance Manager
Container
Management
Library Manager
45. D A T A B R I C K S A C C E S S C O N T R O L
Access control can be defined at the user level via the Admin Console
Workspace Access
Control
Defines who can who can view, edit, and run notebooks in
their workspace
Cluster Access Control
Allows users to who can attach to, restart, and manage
(resize/delete) clusters.
Allows Admins to specify which users have permissions to
create clusters
Jobs Access Control
Allows owners of a job to control who can view job results or
manage runs of a job (run now/cancel)
REST API Tokens
Allows users to use personal access tokens instead of
passwords to access the Databricks REST API
Databricks
Access
Control
47. C L U S T E R S
▪ Azure Databricks clusters are the set of Azure Linux VMs that
host the Spark Worker and Driver Nodes
▪ Your Spark application code (i.e. Jobs) runs on the provisioned
clusters.
▪ Azure Databricks clusters are launched in your subscription—but
are managed through the Azure Databricks portal.
▪ Azure Databricks provides a comprehensive set of graphical
wizards to manage the complete lifecycle of clusters—from
creation to termination.
51. A Z U R E D A T A B R I C K S N O T E B O O K S O V E R V I E W
Notebooks are a popular way to develop, and run, Spark Applications
▪ Notebooks are not only for authoring Spark applications but
can be run/executed directly on clusters
• Shift+Enter
•
•
▪ Notebooks support fine grained permissions—so they can be
securely shared with colleagues for collaboration (see
following slide for details on permissions and abilities)
▪ Notebooks are well-suited for prototyping, rapid
development, exploration, discovery and iterative
development Notebooks typically consist of code, data, visualization, comments and notes
52. M I X I N G L A N G U A G E S I N N O T E B O O K S
You can mix multiple languages in the same notebook
Normally a notebook is associated with a specific language. However, with Azure Databricks notebooks, you can
mix multiple languages in the same notebook. This is done using the language magic command:
• %python Allows you to execute python code in a notebook (even if that notebook is not python)
• %sql Allows you to execute sql code in a notebook (even if that notebook is not sql).
• %r Allows you to execute r code in a notebook (even if that notebook is not r).
• %scala Allows you to execute scala code in a notebook (even if that notebook is not scala).
• %sh Allows you to execute shell code in your notebook.
• %fs Allows you to use Databricks Utilities - dbutils filesystem commands.
• %md To include rendered markdown
53. L I B R A R I E S O V E R V I E W
Enables external code to be imported and stored into a Workspace
55. D A T A B R I C K S S P A R K I S F A S T
Benchmarks have shown Databricks to often have better performance than alternatives
SOURCE: Benchmarking Big Data SQL Platforms in the Cloud
57. S P A R K S Q L O V E R V I E W
Spark SQL is a distributed SQL query engine for processing structured data
58. L O C A L A N D G L O B A L T A B L E S
Databricks registers global
tables to the Hive metastore and
makes them available across all
clusters.
Only global tables are visible in
the Tables pane
Azure Databricks Tables
Databricks does not registers local
tables in the Hive metastore and
are only available within one
cluster.
Also known as temporary tables
59. A Z U R E S Q L D W I N T E G R A T I O N
Integration enables structured data from SQL DW to be included in Spark Analytics
Azure SQL Data Warehouse is a SQL-based fully managed, petabyte-scale cloud solution for data warehousing
Azure Databricks Azure SQL DW
▪ You can bring in data from Azure SQL
DW to perform advanced analytics that
require both structured and unstructured
data.
▪ Currently you can access data in Azure
SQL DW via the JDBC driver. From within
your spark code you can access just like
any other JDBC data source.
▪ If Azure SQL DW is authenticated via
AAD then Azure Databricks user can
seamlessly access Azure SQL DW.
60. P O W E R B I I N T E G R A T I O N
Enables powerful visualization of data in Spark with Power BI
Power BI Desktop can connect to Azure Databricks
clusters to query data using JDBC/ODBC server that
runs on the driver node.
• This server listens on port 10000 and it is not accessible
outside the subnet where the cluster is running.
• Azure Databricks uses a public HTTPS gateway
• The JDBC/ODBC connection information can be obtained
from the Cluster UI directly as shown in the figure.
• When establishing the connection, you can use a Personal
Access Token to authenticate to the cluster gateway. Only
users who have attach permissions can access the cluster
via the JDBC/ ODBC endpoint.
• In Power BI desktop you can setup the connection by
choosing the ODBC data source in the “Get Data” option.
61. C O S M O S D B I N T E G R A T I O N
The Spark connector enables real-time analytics over globally distributed data in Azure Cosmos DB
▪ With Spark connector for Azure Cosmos DB, Apache Spark
can now interact with all Azure Cosmos DB data models:
Documents, Tables, and Graphs.
• efficiently exploits the native Azure Cosmos DB managed indexes
and enables updateable columns when performing analytics.
• utilizes push-down predicate filtering against fast-changing
globally-distributed data
▪ Some use-cases for Azure Cosmos DB + Spark include:
• Streaming Extract, Transformation, and Loading of data (ETL)
• Data enrichment
• Trigger event detection
• Complex session analysis and personalization
• Visual data exploration and interactive analysis
• Notebook experience for data exploration, information sharing,
and collaboration
Azure Cosmos DB is Microsoft's globally distributed, multi-model database service for mission-critical applications
The connector uses the Azure DocumentDB Java SDK
and moves data directly between Spark worker nodes
and Cosmos DB data nodes
62. A Z U R E B L O B S T O R A G E I N T E G R A T I O N
Data can be read from Azure Blob Storage using the Hadoop FileSystem interface. Data can be read from public storage accounts
without any additional settings. To read data from a private storage account, you need to set an account key or a Shared Access
Signature (SAS) in your notebook
spark.conf.set ( "fs.azure.account.key.{Your Storage Account Name}.blob.core.windows.net", "{Your Storage Account Access Key}")
Setting up an account key
Setting up a SAS for a given container:
spark.conf.set( "fs.azure.sas.{Your Container Name}.{Your Storage Account Name}.blob.core.windows.net", "{Your SAS For The Given Container}")
Once an account key or a SAS is setup, you can use standard Spark and Databricks APIs to read from the storage account:
val df = spark.read.parquet("wasbs://{Your Container Name}@m{Your Storage Account name}.blob.core.windows.net/{Your Directory Name}")
dbutils.fs.ls("wasbs://{Your ntainer Name}@{Your Storage Account Name}.blob.core.windows.net/{Your Directory Name}")
63. A Z U R E D A T A L A K E I N T E G R A T I O N
To read from your Data Lake Store account, you can configure Spark to use service credentials with the following snippet in
your notebook
spark.conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential")
spark.conf.set("dfs.adls.oauth2.client.id", "{YOUR SERVICE CLIENT ID}")
spark.conf.set("dfs.adls.oauth2.credential", "{YOUR SERVICE CREDENTIALS}")
spark.conf.set("dfs.adls.oauth2.refresh.url", "https://login.windows.net/{YOUR DIRECTORY ID}/oauth2/token")
After providing credentials, you can read from Data Lake Store using standard APIs:
val df = spark.read.parquet("adl://{YOUR DATA LAKE STORE ACCOUNT NAME}.azuredatalakestore.net/{YOUR DIRECTORY NAME}")
dbutils.fs.list("adl://{YOUR DATA LAKE STORE ACCOUNT NAME}.azuredatalakestore.net/{YOUR DIRECTORY NAME}")
65. S P A R K M L A L G O R I T H M S
Spark ML
Algorithms
67. S P A R K S T R U C T U R E D S T R E A M I N G O V E R V I E W
▪ Unifies streaming, interactive and batch queries—a single API for both
static bounded data and streaming unbounded data.
▪ Runs on Spark SQL. Uses the Spark SQL Dataset/DataFrame API used
for batch processing of static data.
▪ Runs incrementally and continuously and updates the results as data
streams in.
▪ Supports app development in Scala, Java, Python and R.
▪ Supports streaming aggregations, event-time windows, windowed
grouped aggregation, stream-to-batch joins.
▪ Features streaming deduplication, multiple output modes and APIs for
managing/monitoring streaming queries.
▪ Built-in sources: Kafka, File source (json, csv, text, parquet)
A unified system for end-to-end fault-tolerant, exactly-once stateful stream processing
69. D A T A B R I C K S R E S T A P I
Cluster API Create/edit/delete clusters
DBFS API Interact with the Databricks File System
Groups API Manage groups of users
Instance Profile API
Allows admins to add, list, and remove instances
profiles that users can launch clusters with
Job API Create/edit/delete jobs
Library API Create/edit/delete libraries
Workspace API List/import/export/delete notebooks/folders
Databricks
REST API
70. D A T A B R I C K S A P I - A U T H E N T I C A T I O N
Personal access tokens or passwords can be used to authenticate and access Databricks REST APIs
-H "Authorization: Bearer TOKEN_VALUE"