Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

From Data Science to MLOps

•Download as PPTX, PDF•
•
C
Carl W. Handlin

The document discusses moving from data science to MLOps. It defines MLOps as extending DevOps methodology to include machine learning, data science, and data engineering assets. Key concepts of MLOps include iterative development, automation, continuous integration and delivery, versioning, testing, reproducibility, monitoring, source control, and model/feature stores. MLOps helps address challenges of moving models to production like the deployment gap by establishing best practices and tools for testing, deploying, managing, and monitoring models.

1 of 29
From DSc to MLOps
¿Who am I? Hi my name is Carl! ● MSc Computer Science (ITESM) ● Head of Data Science RappiPay ● DataPub @CDMX carlwhandlin.com linkedin.com/in/carlhandlin/
About 7/10 companies report little or no impact with the use of AI.* 40% of organizations with significant investments in AI report no benefits.* Reality is: ● AI is a source of opportunities and advantages ● Implementing AI is a risk ● Implementing AI correctly is difficult * According to the MITSloan and BCG 2019 survey
● Gap between development and deployment into production Only 22% of companies using ML have successfully deployed an ML model into production* 87% of data science projects never make it into production.* The main challenges people face when developing ML capabilities are scale, version control, model reproducibility, and aligning stakeholders. Deployment Gap *According to the 2019 Algorithmia’s “State of Enterprise ML” survey
Ideal Collection and Transformation Monitoring and Feedback Process and Training Evaluation and Validation Enablement and Deployment* ML Cycle
IRL (In Real Life…)
Why? ML Code Data Collection Testing and Debugging Model Analysis Resource Management Process Management Data Verification Automation Configuration Feature Engineering Infrastructure Monitoring
BUT WAIT… I’m Data Scientist why should worry about this? HINT: you want people to use it and your model to work!
MLOps Machine Learning DevOps Data Engineering MLOps “The extension of the DevOps methodology to include Machine Learning, Data Science and Data Engineering assets as first-class citizens within the DevOps ecology”
As ML & AI propagate in software products, we need to establish best practices and tools to test, deploy, manage, and monitor ML models in real-world production. Key Pillars DESIGN a.k.a Think DEVELOPMENT a.k.a Build OPERATION a.k.a Run
Key Concepts & Components ● Iterative-Incremental Dev ● Automation ● CT/CI/CD ● Versioning ● Testing ● Reproducibility ● Monitoring ● Source Control ● Test & Build Services ● Deployment Services ● Model Registry ● Feature Store ● ML Metadata Store ● ML Pipeline Orchestrator
Maturity Level 1 Data Data Extraction & Analysis Data Preparation Model Training Model Evaluation & Validation Trained Model Registry Serving Prediction Service ML Ops
Maturity Level 2 / Automation Data Data Extraction & Analysis Data Preparation Model Training Model Evaluation & Validation Source Code Repository Prediction Service Feature Store Automated Pipeline Trained Model Registry Monitoring Service
The MLOps Tech Stack A Tech Stack should able (at least in some way) to do this: ● Data engineering ● Version control of data ● ML models and code ● Continuous integration and continuous delivery pipelines ● Automating deployments and experiments ● Model performance assessment ● Model monitoring in production. Think in terms of concepts instead of components
The MLOps Tech Stack MLOps Setup Components Tools Data Analysis Python, Pandas Source Control Git Test & Build Services PyTest & Make Deployment Services Git, DVC Model & Dataset Registry DVC[aws s3] Feature Store Feast ML Metadata Store DVC ML Pipeline Orchestrator Airflow
Traceability / Reproducibility ● What went wrong? ● DVC Data Version Control $ dvc init $ git commit -m "Initialize DVC" $ dvc remote add -d myremote/tmp/storage $ dvc add my-dataset.csv $ dvc push
Automating the ML Pipeline ● Apache Airflow ● Kubeflow ● Luigi ● Argo ● MLFlow ● … Can I use CI/CD tools?
● Airflow is a platform to create, monitor and schedule flows. ● Each flow in airflow is presented as a DAG (Directed Acyclic Graph) of Tasks They run independently. ● Flows are created from Python code. Apache Airflow
Apache Airflow
Flask ● Python API Framework ● Works with all Python ML frameworks Serving https://github.com/cwallaceh/sklearnflask_docker
Deploying Level 1 Simplest architecture Model Server with API ML Model Client Client Serverless ML Model
Docker container Deploying Level 2 Simplest architecture containerized Model Server with API ML Model Client
Containers ● Containers have everything the app needs to run including libraries, system tools, code, and runtime ● Containers emulate the operating system ● Lightweight and fast! ● This allow for microservices ● Docker!
Containers ● Reproducibility ● Isolation ● Security ● Environment Management ● Continuous Integration ● Scalability ○ ...with Kubernetes or Swarm
Kubernetes Deploying Level 3 Simplest architecture containerized and scalable! Client Docker container Docker container Docker container Docker container
Other alternatives ● Tensorflow Serving ● MLflow ● Cloud Options: ○ AWS ○ GCP ○ Azure ● Tensorflow.js directly into the browser!
● Mostly problems can be: ○ Data Monitoring (Inputs): ■ Data Drift ■ Input Distribution ■ Data Checks ■ ... ○ Prediction Monitoring (Outputs) ■ Prediction Distribution ■ Model Performance ■ ... ○ Operations issues ■ System Performance ■ Uptime ■ Response time Monitoring!
**Monitor 1:** Dependency changes result in notification **Monitor 2:** Data invariants hold in training and serving inputs **Monitor 3:** Training and serving features compute the same values **Monitor 4:** Models are not too stale **Monitor 5:** The model is numerically stable **Monitor 6:** The model has not experienced a dramatic or slow-leak regressions in training speed, serving latency, throughput, or RAM usage **Monitor 7:** The model has not experienced a regression in prediction quality on served data Key Monitoring Principles *The ML test score: A rubric for ML production readiness and technical debt reduction.
Thank you!

More Related Content

From Data Science to MLOps

  • 1. From DSc to MLOps
  • 2. ¿Who am I? Hi my name is Carl! ● MSc Computer Science (ITESM) ● Head of Data Science RappiPay ● DataPub @CDMX carlwhandlin.com linkedin.com/in/carlhandlin/
  • 3. About 7/10 companies report little or no impact with the use of AI.* 40% of organizations with significant investments in AI report no benefits.* Reality is: ● AI is a source of opportunities and advantages ● Implementing AI is a risk ● Implementing AI correctly is difficult * According to the MITSloan and BCG 2019 survey
  • 4. ● Gap between development and deployment into production Only 22% of companies using ML have successfully deployed an ML model into production* 87% of data science projects never make it into production.* The main challenges people face when developing ML capabilities are scale, version control, model reproducibility, and aligning stakeholders. Deployment Gap *According to the 2019 Algorithmia’s “State of Enterprise ML” survey
  • 5. Ideal Collection and Transformation Monitoring and Feedback Process and Training Evaluation and Validation Enablement and Deployment* ML Cycle
  • 6. IRL (In Real Life…)
  • 7. Why? ML Code Data Collection Testing and Debugging Model Analysis Resource Management Process Management Data Verification Automation Configuration Feature Engineering Infrastructure Monitoring
  • 8. BUT WAIT… I’m Data Scientist why should worry about this? HINT: you want people to use it and your model to work!
  • 9. MLOps Machine Learning DevOps Data Engineering MLOps “The extension of the DevOps methodology to include Machine Learning, Data Science and Data Engineering assets as first-class citizens within the DevOps ecology”
  • 10. As ML & AI propagate in software products, we need to establish best practices and tools to test, deploy, manage, and monitor ML models in real-world production. Key Pillars DESIGN a.k.a Think DEVELOPMENT a.k.a Build OPERATION a.k.a Run
  • 11. Key Concepts & Components ● Iterative-Incremental Dev ● Automation ● CT/CI/CD ● Versioning ● Testing ● Reproducibility ● Monitoring ● Source Control ● Test & Build Services ● Deployment Services ● Model Registry ● Feature Store ● ML Metadata Store ● ML Pipeline Orchestrator
  • 12. Maturity Level 1 Data Data Extraction & Analysis Data Preparation Model Training Model Evaluation & Validation Trained Model Registry Serving Prediction Service ML Ops
  • 13. Maturity Level 2 / Automation Data Data Extraction & Analysis Data Preparation Model Training Model Evaluation & Validation Source Code Repository Prediction Service Feature Store Automated Pipeline Trained Model Registry Monitoring Service
  • 14. The MLOps Tech Stack A Tech Stack should able (at least in some way) to do this: ● Data engineering ● Version control of data ● ML models and code ● Continuous integration and continuous delivery pipelines ● Automating deployments and experiments ● Model performance assessment ● Model monitoring in production. Think in terms of concepts instead of components
  • 15. The MLOps Tech Stack MLOps Setup Components Tools Data Analysis Python, Pandas Source Control Git Test & Build Services PyTest & Make Deployment Services Git, DVC Model & Dataset Registry DVC[aws s3] Feature Store Feast ML Metadata Store DVC ML Pipeline Orchestrator Airflow
  • 16. Traceability / Reproducibility ● What went wrong? ● DVC Data Version Control $ dvc init $ git commit -m "Initialize DVC" $ dvc remote add -d myremote/tmp/storage $ dvc add my-dataset.csv $ dvc push
  • 17. Automating the ML Pipeline ● Apache Airflow ● Kubeflow ● Luigi ● Argo ● MLFlow ● … Can I use CI/CD tools?
  • 18. ● Airflow is a platform to create, monitor and schedule flows. ● Each flow in airflow is presented as a DAG (Directed Acyclic Graph) of Tasks They run independently. ● Flows are created from Python code. Apache Airflow
  • 20. Flask ● Python API Framework ● Works with all Python ML frameworks Serving https://github.com/cwallaceh/sklearnflask_docker
  • 21. Deploying Level 1 Simplest architecture Model Server with API ML Model Client Client Serverless ML Model
  • 22. Docker container Deploying Level 2 Simplest architecture containerized Model Server with API ML Model Client
  • 23. Containers ● Containers have everything the app needs to run including libraries, system tools, code, and runtime ● Containers emulate the operating system ● Lightweight and fast! ● This allow for microservices ● Docker!
  • 24. Containers ● Reproducibility ● Isolation ● Security ● Environment Management ● Continuous Integration ● Scalability ○ ...with Kubernetes or Swarm
  • 25. Kubernetes Deploying Level 3 Simplest architecture containerized and scalable! Client Docker container Docker container Docker container Docker container
  • 26. Other alternatives ● Tensorflow Serving ● MLflow ● Cloud Options: ○ AWS ○ GCP ○ Azure ● Tensorflow.js directly into the browser!
  • 27. ● Mostly problems can be: ○ Data Monitoring (Inputs): ■ Data Drift ■ Input Distribution ■ Data Checks ■ ... ○ Prediction Monitoring (Outputs) ■ Prediction Distribution ■ Model Performance ■ ... ○ Operations issues ■ System Performance ■ Uptime ■ Response time Monitoring!
  • 28. **Monitor 1:** Dependency changes result in notification **Monitor 2:** Data invariants hold in training and serving inputs **Monitor 3:** Training and serving features compute the same values **Monitor 4:** Models are not too stale **Monitor 5:** The model is numerically stable **Monitor 6:** The model has not experienced a dramatic or slow-leak regressions in training speed, serving latency, throughput, or RAM usage **Monitor 7:** The model has not experienced a regression in prediction quality on served data Key Monitoring Principles *The ML test score: A rubric for ML production readiness and technical debt reduction.