ODSC May 2019 - The DataOps Manifesto

@ODSC
THE
DATAOPS
MANIFESTO
Boston | April 30 - May 4, 2019

Topics
Why DataOps Is Essential
Seven Steps to DataOps
DataOps

Strategic Trend: DataOps
• Increased rate of market adoption of
DataOps principles by leaders of data and
analytic teams
• Gartner Hype Cycle in late 2018
• Increased Analysts Coverage

DevOps has resulted in a transformative
improvement in Software Development
• High-performing IT
organizations deploy 200
times more frequently
• They have 24 times faster
recovery times and three
times lower change
failure rates
• And they spend 22
percent less time on
unplanned work and
rework
Source: State of DevOps Report

Lean has resulted in a transformative
improvement in manufacturing
• Lean manufacturing improves
efficiency, reduces waste, and
increases productivity.
• The benefits are manifold:
• Increased product quality
• Reduces rework
• Employee satisfaction
• Higher profits

DataOps – Transformative to Data Analytics
DataOps – Continuous
Delivery of Analytics
• Delivery insights faster
• Ensure high quality
• Add features at the speed of
business
• Automate, orchestrate complex
environment of people and
technology
Source: Gartner
“Organizations that adopt a DevOps- and DataOps-based approach are
more successful in implementing end-to-end, reliable, robust, scalable and
repeatable solutions.”
Sumit Pal, Gartner, November 2018
People,
Process,
Organization
Technical
Environment

From To
Change Fear Change Velocity
Manual Operations Automated Operations
Hope For Quality Integrated Quality
Hero Mentality Repeatable Processes
Tool Centric Code Centric
Vendor Lock-In Diverse Tools
How To Succeed?
A Mindset Change to DataOps…
…to power your highly agile data culture.

Currently, Teams Have High Errors
DataKitchen/Eckerson Survey (May 2019)
Errors In Production:
• 2% Acceptable Number Of Errors
(None)
• 67% High Number Of Errors
• 31% Dangerous Number Of Errors
(Greater Than 11 Per Month)
None
3%
1 to 2
18%
3 to 5
29%
6 to 10
20%
11+
30%
ON AVERAGE, HOW MANY ERRORS (E.G.,
INCORRECT DATA, BROKEN REPORTS, LATE
DELIVERY, CUSTOMER COMPLAINTS) DO YOU
HAVE EACH MONTH?
Forthcoming DataKitchen / Eckerson Research Survey of Medium – Large Companies US And Abroad

Currently, Teams Deploy Too Slowly
Minutes
9%
Hours
15%
Days
36%
Weeks
27%
Months
13%
ON AVERAGE, HOW LONG DOES IT TAKE TO MOVE A
NEW OR MODIFIED DATA ANALYTIC PIPELINE FROM
DEVELOPMENT TO PRODUCTION?
Pipeline (Model, etc.) Deployment:
• 9% Acceptable Deployment
Speed (minutes or less)
• 78% Slow Deployment Speed
• 13% Dangerously Slow
Deployment Speed (months or
longer)

Currently, Teams Struggle to Develop
New Development Environment:
• 12% Acceptable Env. Creation
Speed (minutes or less)
• 50% Slow Env. Creation Speed
• 38% Dangerously Slow Env.
Creation Speed (weeks or
longer)
Days
Hours
Minutes
Months
Weeks
(blank)
NEW DEVELOPMENT ENVIRONMENT WITH THE
APPROPRIATE TEST DATA, SERVERS, AND TOOLS

Figure 1: Only a small fraction of real-world ML systems is
composed of the ML code, as shown by the small black
box in the middle. The required surrounding infrastructure
is vast and complex.
Google
Advances in Neural Information Processing Systems 28 (NIPS 2015)

Business Need
Prep Data
Feature Extraction
Build Model
Evaluate Model
Deploy Model
Monitor Model
Iterate, Test and Improve
Model building

Topics
Why DataOps Is Essential
Next Steps With DataOps

1. Orchestrate Two Journeys
2. Add Tests And Monitoring
3. Use a Version Control
System
4. Branch and Merge
5. Use Multiple Environments
6. Reuse & Containerize
7. Parameterize Your
Processing
People,
Process,
Organization
Technical
Environment
= 7 steps

Journey 1: Orchestrate data to customer value
Analytic process are like manufacturing: materials (data) and
production outputs (refined data, charts, graphs, model)
Access:
Python Code
Transform:
SQL Code, ETL
Model:
R Code
Visualize:
Tableau
Workbook
Report:
Tableau Online
❶

Journey 2: Speed deployment to production
Analytic processes are like software development: deliverables
continually move from development to production
Data
Engineers
Data
Scientists
Data
Analysts
Diverse Team
Diverse Tools
Diverse Customers
Business
Customer
Products &
Systems
❶

Innovation and Value Pipeline Together
Focus on both orchestration and deployment while automating &
monitoring quality
Don’t want break production
when I deploy my changes
Don’t want to learn about data quality issues from my customers
❶

Add Automated Monitoring And Tests
Monitoring: To ensure that
during in the Value Pipeline, the
data quality remains high.
Tests: Before promoting work,
running new and old tests gives
high confidence that the change did
not break anything in the
Innovation Pipeline
❷

Automate Monitoring & Tests In Production
Test Every Step And Every Tool in Your Value Pipeline
Are your outputs
consistent?
And Save Test Results!
Are data inputs
free from
issues?
Is your business logic
still correct?
Access:
Python Code
Transform:
SQL Code, ETL
Model:
R Code
Visualize:
Tableau
Workbook
Report:
Tableau Online
❷

Support Multiple Types Of Tests
Testing Data Is Not Just Pass/Fail in Your Value Pipeline
Support Test Types
• Error – stop the line
• Warning – investigate later
• Info – list of changes
Keep Test History
• Statistical Process Control
❷

Types of Tests
❷

Example Tests
Simple
❷

Example Test
More Complex
Make sure all
table counts are
the same in the
production and
development
environment
❷

Example Test
Location Balance❷
Access:
Python Code
Transform:
SQL Code, ETL
Model:
R Code
Visualize:
Tableau
Workbook
Report:
Tableau Online
source 1 million
rows
database
1 million
rows
1 million
facts
300K
dimension
report
1 million
facts
300K
dimensions

Example Test Historical Balance
❷
SKU Product Product Group Volume
SKU1 P1
G1
100
SKU2 P1 50
SKU3 P2 75
SKU4 P3
G2
125
SKU5 P4 200
SKU6 P5 25
575
Production
Data, Pipeline
& Environment
Pre-Production
Data, Pipeline
& Environment
SKU Product Product Group Volume
SKU1 P1
G1
101
SKU2 P1 55
SKU3 P2 76
SKU4 P3 126
SKU5 P4
G2
200
SKU6 P5 29
587
Access:
Python
Code
Transfor
m: SQL
Code, ETL
Model:
R Code
Visualiz
e:
Tableau
Workbook
Report
: Tableau
Online
Access:
Python
Code
Transfor
m: SQL
Code, ETL
Model:
R Code
Visualiz
e:
Tableau
Workbook
Report
: Tableau
Online
Histbal
G1 225
G2 350
Histbal
G1 358
G2 229

Automated ‘Tests’ Serve
a Dual Purpose:
1. Data Tests and
Monitoring in
Production
2. Regression, Functional
and Performance Tests
in Development
Data Fixed Data Variable
Code Fixed Value Pipeline
Code Variable
Innovation
Pipeline
Quality Your
Customer
Receives
= f (data, code)
https://medium.com/data-ops/disband-your-impact-review-board-automate-analytics-testing-42093d09fe11
Duality of Tests❷

For the Innovation Pipeline
Tests Are For Also Code: Keep Data Fixed
Deploy Feature
Run all tests here before
promoting
❷

Use a Version Control System
At The End Of The Day, Analytic Work Is All Just Code
Access:
Python Code
Transform:
SQL Code,
ETL Code
Model:
R Code
Visualize:
Tableau
Workbook XML
Report:
Tableau Online
Source Code
Control
❸

Branch & Merge
Source Code
Control
Branching & Merging enables people to safely work on their own tasks
❹

Example Branch And Merge Pattern
Sprint 1 Sprint 2
f1 f2
f3
main / master / trunk
f5
❹

Access:
Python Code
Transform:
SQL Code,
ETL Code
Model:
R Code
Visualize:
Tableau
Workbook XML
Report:
Tableau Online
Use Multiple Environments
Analytic Environment
Your Analytic Work Requires Coordinating Tools And Hardware
❺

Use Multiple Environments
Provide an Analytic Environment for each branch
• Analysts need a controlled environment for their experiments
• Engineers need a place to develop outside of production
• Update Production only after all tests are run!
❺

Environments Are Complex
Analytic Environment
❺
Data Engineers,
Scientists or Analytics Team’s Analytic Tools
R
(model)
Alteryx
(business ETL)
Redshift
(data)
SQL
(ETL)
Hardware & Network
Configurations
Right Hardware and
Software Versions
Tableau
(workbook)
Python
Test Data Sets
Code Branch
Test Result
History
Analytic Environment/
Development Sandbox
Creation is Complex:
Hard to create the
right set of data,
tools, people,
history and
configuration for a
fast build test debug
cycle

Reuse & Containerize
Containerize
1. Manage the environment for each
component (e.g. Docker, AMI)
2. Practice Environment Version Control
Reuse
1. Do not create one ‘monolith’ of code
2. Reuse the code and results
❻

Parameterize Your Processing
Think Of Your Value Pipeline Like A Big Function
• Named sets of parameters will
increase your velocity
• With parameters, you can vary
• Inputs
• Outputs
• Steps in the workflow
• You can make a time machine
• Secure storage for credentials
❼

The Seven Steps In Action
1. Select story
2. Create branch
3. Create environment
4. Implement feature
5. Write new tests
6. Run new and existing tests
7. Check in to branch
8. Merge to parent
9. Delete environment
When sprint ends
• Deliver all completed features to
customer
• Merge sprint branch to master
• Roll un-merged features into the
next sprint

The 7 Steps and Data Science
Journeys Tests Version Control Branch and Merge Environments Reuse / Containerize Parameterize
Business Need Agile
Prep Data x x x x x x x
Feature Extraction x x x x x x x
Build Model x x x x x x x
Evaluate Model x
Deploy Model x x x x x x x
Monitor Model x

Where to Start With DataOps?
Look to manufacturing/DevOps
‘Theory of Constraints’
• Where are ‘bottlenecks’ (or
constraints in your data science
or analytic process?
• What impedes from creating new
insight for you customers?
• Iterate & improve

Example Bottlenecks to Start
• “I don’t want to learn about
data quality issues from my
customers”
• “I don’t want break production
when I deploy my changes”
• “I don’t like the Hatfields vs
Mccoys war between data
science and analytic teams”
None
3%
1 to 2
18%
3 to 5
29%
6 to 10
20%
11+
30%
ON AVERAGE, HOW MANY ERRORS (E.G.,
INCORRECT DATA, BROKEN REPORTS,
LATE DELIVERY, CUSTOMER
COMPLAINTS) DO YOU HAVE EACH
MONTH?

Example Bottlenecks to Start
customers”
• “I don’t want break
production when I deploy my
changes”
Minutes
9%
Hours
15%
Days
36%
Weeks
27%
Months
13%
ON AVERAGE, HOW LONG DOES IT TAKE TO MOVE A
NEW OR MODIFIED DATA ANALYTIC PIPELINE FROM
DEVELOPMENT TO PRODUCTION?

Example: Which Constraint?
customers”
Errors : Constraint
Deployment : Constraint
Team Coordination : Constraint

Example: Which Constraint?
customers”
Errors, Deployment, and Team
Coordination Are Bottlenecks or
Constraints That Inhibit
GOAL: Flow of Innovation
“How do measure team progress
and show results to leadership?”

DataKitchen Software Platform
Our cloud platform orchestrates data to customer value,
speeds features to production, and automates quality.
Kitchens
Recipes & Tests
Orders
Ingredients
1. Orchestrate Two Journeys
2. Add Tests And Monitoring
3. Use a Version Control System
4. Branch and Merge
5. Use Multiple Environments
6. Reuse & Containerize
7. Parameterize Your Processing

For More Information
• For These Slides, Contact Me:
• cbergh@datakitchen.io
• DataOps Manifesto:
• http://dataopsmanifesto.org
• DataOps Blog:
• http://medium.com/data-ops
• Follow Twitter:
• #DataOps

ODSC May 2019 - The DataOps Manifesto

More Related Content

ODSC May 2019 - The DataOps Manifesto