How Azure Databricks helped make IoT Analytics a Reality with Janath Manohararaj and Prasad Chandravihar
At Lennox International, we have thousands of IoT connected devices streaming data into the Azure platform with a minute level polling interval. The challenge was to use these data sets, combine with external data sources such as weather, and predict equipment failure with high levels of accuracy along with their influencing patterns and parameters. Previously the team was using a combination of on-premise and desktop tools to run algorithms on a sample set of devices. The result was low accuracy levels (around 65%) on a process that took more than 6 hours. The team had to work through several data orchestration challenges and identify a machine learning platform which enabled them to collaborate between our engineering SME’s, Data Engineers and Data Scientists. The team decided to use Azure Databricks to build the data engineering pipelines, appropriate machine learning models and extract predictions using PySpark. To enhance the sophistication of the learning, the team worked on a variety of Spark ML models such as Gradient Boosted Trees and Random Forest. The team also implemented stacking, ensemble methods using H2O driverless AI and sparkling water on Azure Databricks clusters, which can scale up to 1000 cores. Join us in this session and see how this resulted in models that run in 40 minutes with minimal tuning and predict failures with accuracy of about 90%.
More Related Content
What's hot
What's hot (20)
Similar to How Azure Databricks helped make IoT Analytics a Reality with Janath Manohararaj and Prasad Chandravihar
Similar to How Azure Databricks helped make IoT Analytics a Reality with Janath Manohararaj and Prasad Chandravihar (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohararaj and Prasad Chandravihar
- 1. Prasad Chandravihar, Lennox International Janath Manohararaj, Lennox International How Azure Databricks helped make IoT Analytics a reality #Ent7SAIS
- 2. Agenda • About Lennox • Problem Statement • How did we start • Challenges • Machine Learning using Databricks • Wrap – up • Q&A 2#Ent7SAIS
- 3. About Lennox #Ent7SAIS#Ent7SAIS • Lennox International Inc. is an intercontinental provider of climate control products for the heating, ventilation, air conditioning, and refrigeration markets. • Founded in 1895, in Marshalltown, Iowa, by Dave Lennox. • Three core businesses: Residential Heating and Cooling, Commercial Heating and Cooling, and Refrigeration. • The company headquarters are in Richardson, Texas, near Dallas.
- 4. Problem Statement Particular type of mechanical switch trips pretty often and causes discomfort to the users Use data sets captured from the IoT devices to analyze Build ML models to predict the trips and also detect patterns/influencing variables #Ent7SAIS#Ent7SAIS
- 5. How did we start • Initial Model was developed in desktop tools with about 15 devices • Accuracy levels – Recall : 65%, Specificity : 80%
- 6. Challenges • Consuming the entire data set • Data orchestration – 10 Billion rows • Assembling different skill sets • Team structure, working model • Right Computing platform – Cloud ? • collaboration among Data Scientists & Engineers
- 7. Small file problem • Traverse through 7 million directory paths to gather data Solution: Developed a batch program which would gather data from each file in each destination and append to a table
- 8. Machine Learning Core team Data Engineering • Helps on identifying the right data sets • Helps in data orchestration Big Data Cloud Architecture - helps the team on the best use of the tools and resources • Constantly works with Data science and Engineering members during data orchestration Data Science • Helps on getting the data ready for the model (parts of ETL) • Explore different data science models Platform Eng. • Analyzes the feasibility to productize Eng. SME • Feedback on the outputs • Helps in making sure the analysis is progressing in the right direction
- 9. Why Azure Databricks ? • Unified platform for Machine Learning & Data engineering • Provides collaboration between team members • Spark with minimal tuning • Average time to start a cluster – 4 mins • Automatic scaling – very important as different jobs are different sizes • Ability to integrate sparkling water, H20 driverless AI and Tensorflow through GUI • Ability to integrate PowerBI
- 10. Machine Learning using Azure Databricks • Equipment information • Weather information • Demographic information • ELT pipeline • Training the ML model • Validation and scoring • Sends data every minute 70% of the time – structuring and preparing the data, creating labels etc..
- 11. Machine Learning using Azure Databricks Class Imbalance: We had way more 0’s than 1’s as labels , for ex: for every 590M 0’s we had 5K 1’s. We then tried 3 different approaches: • Over sample the 1’s (SMOTE, duplicating rows etc..) • Under sample the 0’s (relatively small set of 0’s which represent the population) • Multiple samples of 0’s and the same set of 1’s – each set of 0’s & 1’s will have its own model – Created the ensemble of models
- 12. Machine Learning using Azure Databricks First sample of 0’s & 1’s with the assembler of features #Bins, #trees, #depth and threshold tuned using an exhaustive grid search Model was tuned for the best Area under Precision Recall Sample output from Grid search
- 13. Machine Learning using Azure Databricks ……… Test Data Ensemble Train 1 Train 2 Train n
- 14. Machine Learning using Azure Databricks • Right from data engineering to ML models was done in one single notebook • Coding language - PySpark • Each ML model was tuned to get the best hyper parameters using an automated grid search • The ensemble model helped us reduce a lot of false positives
- 15. Wrap – up Journey to ML 15 devices 500 devices 25K devices Accuracy TP - 65% TP - 80% TP – 84.6% TN - 80% TN - 85% TN – 99.5% Data Volume 2 Million 100 Million 10 Billion Tools Desktop tools Spark ML,H2O Azure Databricks Cloud Services Provider n/a Microsoft Microsoft ML models used Decision Tree Multiple (Gradient Boosted Trees, Random Forest, etc) Multiple (Gradient Boosted Trees, Random Forest, etc) Time to run ML model 6 hours 30 mins 50 mins
- 16. Wrap – up • Build a cross functional team to execute machine learning projects • In most of projects 70% of the time is spent on cleansing and transforming the data set • Give a lot of focus into engineering features • Explore sparkling water (H20 on databricks) gives a lot of auto ML options • Platform which lets team members collaborate and develop the project end to end
- 17. 17#Ent7SAIS