DataOps introduction : DataOps is not only DevOps applied to data!
•Télécharger en tant que PPTX, PDF•
1 j'aime•487 vues
This presentation introduces the audience to the DataOps practice, which inheritates from Agile and DevOps, but also handle additional aspects.
Signaler
Partager
Contenu connexe
Tendances
Tendances (20)
Similaire à DataOps introduction : DataOps is not only DevOps applied to data!
Similaire à DataOps introduction : DataOps is not only DevOps applied to data! (20)
Plus de Adrien Blind
Plus de Adrien Blind (20)
Dernier
Dernier (11)
DataOps introduction : DataOps is not only DevOps applied to data!
- 1. SNOWCAMP 2020 DATAOPS n’est pas que DEVOPS appliqué aux projets DATA !
- 2. Frédéric PETIT Head Of Architecture Head Of Data Department MNT-VYV Adrien BLIND DataOps Evangelist Docker Captain SAAGIE @madgicweb @AdrienBlind @mutuelleMNT @Groupe_VYV @saagie_io
- 3. Kevin KELLY “ The business plans of the next 10.000 startups are easy to forecast : Take X and add AI” Photo : @USIEvents Source : ici Rédacteur en chef et fondateur du magazine “Wired”
- 4. Approche de programmation “Produit Traditionnel” Compute Source Code Data Output FeedBack Approche “Traditionnelle”
- 5. Approche de programmation d’un “Produit intelligent” Compute Source Code Data Output FeedBack Compute Training Code Labeled Data Model(s) FeedBack Approche “Machine Learning” Approche “Traditionnelle”
- 6. La “DATA” est la matière première du système ! Compute Source Code Output FeedBack Compute Training Code Labeled Data Model(s) FeedBack Data Factory Data Sources Back-End Data Extraction APIs FeedBack Data VIZRequest Analytics Consumer Data Code DashboardsDashBoard Code
- 7. SI Data vs SI Opérationnel Compute Source Code Output FeedBack Compute Training Code Labeled Data Model(s) FeedBack Data Factory Data Sources Back-End Data Extraction APIs FeedBack Data VIZRequest Analytics Consumer Data Code DashboardsDashBoard Code SI Data SI Opérationnel
- 8. DATA is the new oil - Clive Humby
- 9. DATA is the new oil - Clive Humby
- 10. Les “cultures” associées Source Code Output FeedBack Training Code Labeled Data Model(s) FeedBack Data Sources Back-End Data Extraction APIs FeedBack Data VIZRequest Analytics Consumer Data Code DashboardsDashBoard Code MLOps DevOps DataOps
- 11. Les “cultures” associées Source Code Output FeedBack Training Code Labeled Data Model(s) FeedBack Data Sources Back-End Data Extraction APIs FeedBack Data VIZRequest Analytics Consumer Data Code DashboardsDashBoard Code MLOps DevOps DataOps #1 Data Engineers needs pipelines to deliver DATA #2 Data Scientists needs pipelines to deliver MODELS #3 Developers needs pipelines to deliver APPLICATION
- 12. Les pipelines associés Develop Build Test ReleaseNeeds Deploy APPLICATIO N Operated Develop Training Test Evaluate Extract Prepare Analyse Storage Release Publish DATA Exposed MODELS Optimized Driven by : Intelligence Data Capital
- 13. Le sujet, ce n’est pas le datalake, c’est le data PROCESSING Datamarts, Shared Dataset(s) Data processing Consumers Si la data est le nouvel or noir, alors : ● Vos datalakes sont vos nappes de pétrole, votre capital (grandes masses de données brutes) ● Hive/Impala & co. sont vos puits de pétrole, permettant de requêter la ressource ● Mais ce sont en fait les orchestrateurs, vos raffineries permettant de transformer le capital en valeur d’usage DATALAKE Data storing: datalakes, object storage, ... Extract Prepare Analyse Publish
- 14. Du batch processing au streaming et à l’event processing
- 15. Les pipelines associés Develop Build Test ReleaseNeeds Deploy APPLICATIO N Operated Develop Training Test Evaluate Extract Prepare Analyse Storage Release Publish DATA Exposed MODELS Optimized Driven by : Intelligence Data Capital MLOps DevOps DataOps Innovation Pipeline Value Pipeline Analysis Pipeline
- 16. Quelle culture holistique regroupe toutes ces initiatives ? Develop Build Test ReleaseNeeds Deploy Applicatio n Operated Develop Training Test Evaluate Extract Prepare Analyse Storage Analys is ReleaseValue Publish Dataset Exposed Models Optimized Driven by : MLOps DevOps DataOps Innovation Pipeline Value Pipeline Analytics Pipeline Faites vos propositions ! http://bit.ly/36eqHqL
- 17. On s’égare , revenons au DataOps ! Abordons le sujet de DataOps en partant d’un postulat : Vous avez d’ores et déjà mis en place la culture DevOps dans votre entreprise :) Source : Giphy @ Snuls
- 18. Domaines de compétences techniques nécessaires Develop Build Test ReleaseNeeds Deploy APPLICATIO N Operated Develop Training Test Evaluate Extract Prepare Analyse Storage Release Publish DATA Exposed MODELS Optimized Driven by : Intelligence Data Capital DataOpsMLOpsDevOps
- 19. Domaines de compétences opérationnelles nécessaires DevOps DEV OPS BIZ DataOps DEV OPSBIZ Data Scientist Data Engineer
- 20. Organisation de la culture DevOps Tribe Squad Squad Squad Chapter Dev Chapter Ops
- 21. Organisation de la culture DataOps qui semblerait naturelle Tribe Squad Squad Squad Chapter Dev Chapter ... Chapter DATA
- 22. Organisation observée de la culture DataOps DATA LABS Tribe Squad Squad Squad Chapter Dev Chapter ... Squad Data Oriented Squad Data Oriented
- 23. Organisation de la culture DataOps “post-maturation” Tribe Squad Squad Squad Chapter Dev Chapter ... Chapter DATA DATA LABS Squad Data Oriented
- 24. Data Factory CentralizedDistributed Data Dictionnary Data Extraction / Lineage Data Catalog Data Exposition Data Processing Data WareHouse / Data Lake Data Collection Data Exploration & Analysis tools Data Viz ML Code ML Trainning (Model) Monitoring Data Viz Data Verification Data Quality Gouvernance / Security Modelization Service Presentation
- 25. Shadow Data
- 26. Human after all
- 27. TAKE AWAY POWER (OF IA) IS NOTHING WITHOUT (DATA) CONTROL Sans maîtrise (de la donnée) la puissance (de l’IA) n’est rien
- 28. Rencontrez-vous ! THE BEST TAKE AWAY @madgicweb @AdrienBlind Frédéric PETIT Adrien BLIND