Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Global ai conf_final

•
•
Nisha Talagala
Nisha Talagala

Building trustworthy and effective AI solutions. - Many cloud vendor AI services (AWS, GCP, Azure) - Demo of a workflow with AWS Sagemaker - What is AI Trust - What is explainability - How to add this to a workflow with S3, Sagemaker, Lambda (server less) and Postman

1 of 15
Building Trustworthy and Effective Production AI Applications Nisha Talagala CEO, Pyxeda AI
Sophisticated AI technologies, more available every day Each logo is a (separate) service offered by GCP, AWS or Azure for part of an AI workflow
Problem: Minimal adoption https://www.oreilly.com/library/view/the-new-artificial/9781492048978/ https://emerj.com/ai-sector-overviews/valuing-the-artificial-intelligence-market-graphs-and-predictions/ Despite the advanced services available, AI usage still minimal
Why? • Flow is complex and multi- faceted • Tools are at different levels and tackle subsets of workflow • Multiple roles collaborate • Other considerations must be factored in across tools • Trust • Explainability Data Train Model(s) Develop Model(s) Test Model(s) Deploy Model(s) Connect to Business app App developers Data Scientists ML Engineers Operations Business Need Monitor and Optimize Many levels of complexity
In this talk • Typical AI Workflow • Tools available for each stage from major cloud vendors • How to think about Trust and Explainability • Tips for creating trustworthy cloud ML workflows • Demo
A typical flow • Use case definition • Data prep • Modeling • Training • Deploy • Integrate • Monitor/Optimize • Iterate Data Train Model(s) Develop Model(s) Test Model(s) Deploy Model(s) Connect to Business app App developers Data Scientists ML Engineers Operations Business Need Monitor and Optimize
Multiple levels of Services Substantial tooling available for each stage from multiple cloud vendors Labeling Data Prep and Visualization Modeling and Deployment Marketplaces Service APIs Manipulate raw data Build, tune or deploy your own models Buy a model or algorithm Use a pre-built AI (example voice to text, etc.) Infrastructure: Compute, Authentication, Data source, Logs etc. Where your AI runs and what monitors it
• Publicized “mistakes” that damage corporate brands and generate business risk • Example Racism in Microsoft Tay bot and Bias in Amazon HR hiring tool • Intersection of AI decisions and human social values • Core tenets of Trust • Integrity (which includes Explainability) • Bias AI Trust
ML Integrity – Pillars for Trust • Together ensure that the ML is operating correctly and free from intrusion • Details about how and why predictions and made • Reproduce cases if needed • We focus on explainability in this talk https://www.forbes.com/sites/cognitiveworld/2019/01/29/ml-integrity-four-production-pillars-for-trustworthy-ai/#432a488b5e6f
• How did an algorithm make its decision? • Increasingly important to provide this to the consumer who is impacted by the decision • Simple for basic ML (like linear regression), still open research problem (with some advances) for advanced techniques like deep learning • Regulations: Example GDPR are pushing in this direction Explainability
How to develop “Trustworthy” Workflows? • Depends on level of service • If it is an end to end service – research the service to see what claims are made in this area • Some end to end services tested by third parties for trust and bias • Explainability • Select explainable algorithms • Get explanations with every prediction • See if tools offer generic solutions • Track activities across flow for reproducibility and security • Maintain a database of artifacts (models, etc.) Many tools help with this • Use security and authentication models to control access
Demo • Dataset: A UCI classification dataset (Avila) • 10 features • A target variable with 12 categories/classes. • Tools: AWS Sagemaker, Lambda, API Gateway and Postman • Flow: • Training (Sagemaker) • Deployment as REST (Lambda, API Gateway) • Testing (Postman)
Demo Labeling Data Prep and Visualization Modeling and Deployment Marketplaces Service APIs Dataprep/Labeling offline Dataset (Avila) Usecase: Identifying author/copyist from handwriting images. http://archive.ics.uci.edu/ml/datasets/ Avila AWS Sagemaker used to train and deploy Free Scikit Learn Decision Tree Classifier Custom built service API AWS Lambda (for explainability) Infrastructure: Compute, Authentication, Data source, Logs etc. AWS
S3 Adding Explainability Dataset prep and transform AWS Sagemaker Sagemaker endpoint + Marketplace inference code External endpoint (basic Lambda) External endpoint (custom inference code in Lambda) Dataset Model Artifact Postman Postman Request Request Prediction Prediction + Explanation AWS Marketplace Modified Dataset API Gateway API Gateway Explainable path Default path
Thank you • Tell us about your ML challenges - see ML Therapy at booth! • Take our survey • Contact: nisha@pyxeda.ai https://forms.gle/ejEnwdhQL3371Jvw7

More Related Content

Global ai conf_final

  • 1. Building Trustworthy and Effective Production AI Applications Nisha Talagala CEO, Pyxeda AI
  • 2. Sophisticated AI technologies, more available every day Each logo is a (separate) service offered by GCP, AWS or Azure for part of an AI workflow
  • 4. Why? • Flow is complex and multi- faceted • Tools are at different levels and tackle subsets of workflow • Multiple roles collaborate • Other considerations must be factored in across tools • Trust • Explainability Data Train Model(s) Develop Model(s) Test Model(s) Deploy Model(s) Connect to Business app App developers Data Scientists ML Engineers Operations Business Need Monitor and Optimize Many levels of complexity
  • 5. In this talk • Typical AI Workflow • Tools available for each stage from major cloud vendors • How to think about Trust and Explainability • Tips for creating trustworthy cloud ML workflows • Demo
  • 6. A typical flow • Use case definition • Data prep • Modeling • Training • Deploy • Integrate • Monitor/Optimize • Iterate Data Train Model(s) Develop Model(s) Test Model(s) Deploy Model(s) Connect to Business app App developers Data Scientists ML Engineers Operations Business Need Monitor and Optimize
  • 7. Multiple levels of Services Substantial tooling available for each stage from multiple cloud vendors Labeling Data Prep and Visualization Modeling and Deployment Marketplaces Service APIs Manipulate raw data Build, tune or deploy your own models Buy a model or algorithm Use a pre-built AI (example voice to text, etc.) Infrastructure: Compute, Authentication, Data source, Logs etc. Where your AI runs and what monitors it
  • 8. • Publicized “mistakes” that damage corporate brands and generate business risk • Example Racism in Microsoft Tay bot and Bias in Amazon HR hiring tool • Intersection of AI decisions and human social values • Core tenets of Trust • Integrity (which includes Explainability) • Bias AI Trust
  • 9. ML Integrity – Pillars for Trust • Together ensure that the ML is operating correctly and free from intrusion • Details about how and why predictions and made • Reproduce cases if needed • We focus on explainability in this talk https://www.forbes.com/sites/cognitiveworld/2019/01/29/ml-integrity-four-production-pillars-for-trustworthy-ai/#432a488b5e6f
  • 10. • How did an algorithm make its decision? • Increasingly important to provide this to the consumer who is impacted by the decision • Simple for basic ML (like linear regression), still open research problem (with some advances) for advanced techniques like deep learning • Regulations: Example GDPR are pushing in this direction Explainability
  • 11. How to develop “Trustworthy” Workflows? • Depends on level of service • If it is an end to end service – research the service to see what claims are made in this area • Some end to end services tested by third parties for trust and bias • Explainability • Select explainable algorithms • Get explanations with every prediction • See if tools offer generic solutions • Track activities across flow for reproducibility and security • Maintain a database of artifacts (models, etc.) Many tools help with this • Use security and authentication models to control access
  • 12. Demo • Dataset: A UCI classification dataset (Avila) • 10 features • A target variable with 12 categories/classes. • Tools: AWS Sagemaker, Lambda, API Gateway and Postman • Flow: • Training (Sagemaker) • Deployment as REST (Lambda, API Gateway) • Testing (Postman)
  • 13. Demo Labeling Data Prep and Visualization Modeling and Deployment Marketplaces Service APIs Dataprep/Labeling offline Dataset (Avila) Usecase: Identifying author/copyist from handwriting images. http://archive.ics.uci.edu/ml/datasets/ Avila AWS Sagemaker used to train and deploy Free Scikit Learn Decision Tree Classifier Custom built service API AWS Lambda (for explainability) Infrastructure: Compute, Authentication, Data source, Logs etc. AWS
  • 14. S3 Adding Explainability Dataset prep and transform AWS Sagemaker Sagemaker endpoint + Marketplace inference code External endpoint (basic Lambda) External endpoint (custom inference code in Lambda) Dataset Model Artifact Postman Postman Request Request Prediction Prediction + Explanation AWS Marketplace Modified Dataset API Gateway API Gateway Explainable path Default path
  • 15. Thank you • Tell us about your ML challenges - see ML Therapy at booth! • Take our survey • Contact: nisha@pyxeda.ai https://forms.gle/ejEnwdhQL3371Jvw7