Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Managing an Experimentation Platform by LinkedIn Product Leader

•
•
Product School
Product School

Main Takeaways: -Establishing a culture of experimentation at scale -Developing the product vision and strategy -Backlog prioritization based on Impact Score formula

1 of 32
www.productschool.com Managing an Experimentation Platform by LinkedIn Product Leader
CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™
Corporate Training Level up your team’s Product Management skills
Free Product Management Resources BOOKS EVENTS JOB PORTAL COMMUNITIES bit.ly/product_resources COURSES
Event is Sponsored by:
Makram Mansour, PhD [Product Leader @ LinkedIn] Managing an Experimentation Platform
Makram Mansour Product Manager, LinkedIn 1 LinkedIn Profile linkedin.com/in/ilmansour 2 Education MS & PhD in EE from UIUC Stanford LEAD from Stanford GSB 3 Product @ LinkedIn LinkedIn Experimentation & Data Insights 4 Product @ Texas Instruments Online Design Tools 5 Engineer @ Intel Intel Server Chipsets 6 Values Out-of-the-box thinker who is not afraid of taking risks, gets more committed when people tell me “it cannot be done”, and strongly believe in the saying: “where there’s a will, there’s a way”
1 2 3 4 Agenda Experimentation @ LinkedIn Backlog Prioritization Workflow Optimization Final note on Vision & Strategy
Experimentation @ LinkedIn
Create economic opportunity for every member of the global workforce.
Connect the world’s professionals to make them more productive and successful.
The Economic Graph 740M 57M 14M 38K 120K 280B Members Companies Jobs Skills Schools Knowledge
LinkedIn’s data in motion 100B graph edges, 2B nodes 650K graph queries/second 1,500PB total data storage 11T messages/day on Kafka 25B model parameters in AI models 208M contributions/day 30B Feed Updates viewed/month 750PB average daily storage on Hadoop 12B page views/day 180M messages sent/day 96M profile actions/day
Complex growth engine full of “network” effects One thing we learnt over the years is that even small, localized changes can have massive impact! Maintaining and accelerating growth requires a strong discipline around experimentation and data.
Cost of not doing A/B testing can be high! Profile top ads 5-pixel height decrease CTR drop on 11/11/2013
We test everything Frontend, ranking algorithms, and backend infra Experimentation activity • 100/day new tests • 400/day tests ramps • 200/week AI/ML models Experimentation adoption • 5000+ experiment owners • 2000+ WAU • 41k tests running simultaneously Infrastructure activity • 2M QPS • 35T/day evaluations
20k Metrics 8k A/B testable Metrics Single source-of-truth Self-served, custom metrics 90k QPS 2.5PB/day 20T/day records processed on Hadoop Tracking Events Kafka Offline Infrastructure Unified Metrics Platform Visualization Experimentation Anomaly Detection & Root Cause Analysis Massive experimentation and metrics data
1. Targeting Helps run experiments on different audience attributes: - Location - Job title - Company - Industry - Education - Language - Device - Connections - etc. 2. Ramping Helps you easily and safely ramp/de-ramp a feature over time. Provides standard framework to assist users to ramp with the right balance of Speed, Quality, and Risk 3. EXperimentation Advanced experimentation infrastructure built for large-scale use. Some of the features: - Multivariate testing - Advanced randomization - Metrics Reporting & Alerting - Insights for ease of interpreting results - Variance Reduction Methods - Most Impactful Experiments TREX is LinkedIn’s Experimentation Platform It’s a unified platform for Targeting, Ramping, and EXperimentation
Backlog Prioritization
1 2 3 4 Prioritization Framework Quantifies business value Allow “apples-to-apples” stack-rank and prioritization Makes decision-making data-driven and transparent Sets clear guidelines on what data is needed for every ask
Prioritization Framework Four Pillars Value Measured by impact to Site Up, Productivity, Revenue, Key Metrics, or User Satisfaction Leverage Measured by # of users impacted in 12 months or less Urgency Measured by urgency: blocked, impaired but have short-term workaround, or inconvenient Cost Measured in engineer-quarters (engineers needed * no. of quarters to complete)
Computing Value (V) Rubric Select up to two categories and multiply their corresponding numbers E.g., V1 = G = 4, V2 = P = 2 ⇒ V = 4 * 2 = 8 GCNs (G) Productivity (P) Revenue (R) Metrics (M) User Satisfaction (S) GCN Savings: 1. 1 minor GCN/qrtr 2. 2-4 minor GCNs 3. 1-2 medium GCNs 4. 1+ major GCNs Savings Criteria: 1. < 1hr /usr/mnth 2. 2 - 5 hrs saved 3. 6 - 10 hrs saved 4. > 10 hrs saved Revenue Lift: 1. < $1M annually 2. $1M - $10M 3. $10M - $30M 4. > $30M Key Metrics Lift: 1. < 0.5% lift 2. 0.5% - 1.0% 3. 1.0% - 2.0% 4. > 2.0% NSAT Score Lift: 1. < +0.2 lift 2. +0.3 - +0.5 3. +0.6 - +0.9 4. > +1.0
Computing other Rubrics T-REX Users Impacted 1. < 100 users 2. 100 - 499 3. 500 - 1999 4. > 2000 Leverage (L) User’s work impact Costing Formula Urgency (U) Cost (C) 1. Inconvenient 2. Impaired; workaround in place 3. Blocked 1. # of Engineers 2. # of Quarters 3. C = E * Q
Prioritization - Final Scoring Guidelines Final Ranking Criteria 03 ● Rank by largest Impact Score ● Override for large ROIs (Quick Wins) Calculate ROI 02 ROI = Impact Score / Cost Calculate Impact Score (IS) 01 Impact Score = Leverage * Value * Urgency
Ideal Mix 40% 30% 20% 10% Big Bets Home Run Quick Wins Fall Back Level of Effort Ranking
Workflow Optimization
Optimize your Workflow Prioritized 50% Power Design Sign-off Plan Sign- off Feature Request Feature Release Code Ready Ideate Flow diagram, Workflows, Design, Use cases Ramp Verify solution, fix unforeseen issues, capture metrics, Derive success stories Announce Announce highlighting features and impact. Include user testimonials, training, and documentation Learn Problem statements, User stories, Impact assessment, Requirements gathering, Prioritization, PRD Design RFC, Hi-Fi Design, Acceptance test cases, WBS, Timeline Build Modular code, Bug bashes, Unit & Integration testing, Previews Design Thinking Approach with Experimentation in the Core of your Purpose Gain insights in what our customers really want and why. Build clarity in terms of Before / After state. Gradual release ramp to efficiently address unforeseen issues. Measure success metrics and announce with impact!
Sample PRD Areas of focus: Problem Statement Core Use Cases Prioritization Details Objectives and Key Results Before and After Milestones Sign-offs
Note on Vision and Strategy
Be clear on your Vision to Values Vision • The Dream. The Future. What you inspire your product will impact and become. Mission • Briefly describes the goals and purpose of your product. Why the product exists? Target Audience • List down your primary user personas • Prioritize them. Also be clear on non-users. • Group them in Producers / Consumers if you have a platform product Strategy • Series of strategic objectives and roadmap milestones Priorities • Stack ranked list of critical initiatives. “If we can only do one thing this quarter, what would it be?” Objectives (Key Metrics) • True North Metrics • Sign-Post Metrics • Guardrail Metrics Values • Your guiding principles in making day-to-day decisions
Thank you
www.productschool.com Part-time Product Management Training Courses and Corporate Training

More Related Content

Managing an Experimentation Platform by LinkedIn Product Leader

  • 1. www.productschool.com Managing an Experimentation Platform by LinkedIn Product Leader
  • 2. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™
  • 3. Corporate Training Level up your team’s Product Management skills
  • 4. Free Product Management Resources BOOKS EVENTS JOB PORTAL COMMUNITIES bit.ly/product_resources COURSES
  • 6. Makram Mansour, PhD [Product Leader @ LinkedIn] Managing an Experimentation Platform
  • 7. Makram Mansour Product Manager, LinkedIn 1 LinkedIn Profile linkedin.com/in/ilmansour 2 Education MS & PhD in EE from UIUC Stanford LEAD from Stanford GSB 3 Product @ LinkedIn LinkedIn Experimentation & Data Insights 4 Product @ Texas Instruments Online Design Tools 5 Engineer @ Intel Intel Server Chipsets 6 Values Out-of-the-box thinker who is not afraid of taking risks, gets more committed when people tell me “it cannot be done”, and strongly believe in the saying: “where there’s a will, there’s a way”
  • 8. 1 2 3 4 Agenda Experimentation @ LinkedIn Backlog Prioritization Workflow Optimization Final note on Vision & Strategy
  • 10. Create economic opportunity for every member of the global workforce.
  • 11. Connect the world’s professionals to make them more productive and successful.
  • 12. The Economic Graph 740M 57M 14M 38K 120K 280B Members Companies Jobs Skills Schools Knowledge
  • 13. LinkedIn’s data in motion 100B graph edges, 2B nodes 650K graph queries/second 1,500PB total data storage 11T messages/day on Kafka 25B model parameters in AI models 208M contributions/day 30B Feed Updates viewed/month 750PB average daily storage on Hadoop 12B page views/day 180M messages sent/day 96M profile actions/day
  • 14. Complex growth engine full of “network” effects One thing we learnt over the years is that even small, localized changes can have massive impact! Maintaining and accelerating growth requires a strong discipline around experimentation and data.
  • 15. Cost of not doing A/B testing can be high! Profile top ads 5-pixel height decrease CTR drop on 11/11/2013
  • 16. We test everything Frontend, ranking algorithms, and backend infra Experimentation activity • 100/day new tests • 400/day tests ramps • 200/week AI/ML models Experimentation adoption • 5000+ experiment owners • 2000+ WAU • 41k tests running simultaneously Infrastructure activity • 2M QPS • 35T/day evaluations
  • 17. 20k Metrics 8k A/B testable Metrics Single source-of-truth Self-served, custom metrics 90k QPS 2.5PB/day 20T/day records processed on Hadoop Tracking Events Kafka Offline Infrastructure Unified Metrics Platform Visualization Experimentation Anomaly Detection & Root Cause Analysis Massive experimentation and metrics data
  • 18. 1. Targeting Helps run experiments on different audience attributes: - Location - Job title - Company - Industry - Education - Language - Device - Connections - etc. 2. Ramping Helps you easily and safely ramp/de-ramp a feature over time. Provides standard framework to assist users to ramp with the right balance of Speed, Quality, and Risk 3. EXperimentation Advanced experimentation infrastructure built for large-scale use. Some of the features: - Multivariate testing - Advanced randomization - Metrics Reporting & Alerting - Insights for ease of interpreting results - Variance Reduction Methods - Most Impactful Experiments TREX is LinkedIn’s Experimentation Platform It’s a unified platform for Targeting, Ramping, and EXperimentation
  • 20. 1 2 3 4 Prioritization Framework Quantifies business value Allow “apples-to-apples” stack-rank and prioritization Makes decision-making data-driven and transparent Sets clear guidelines on what data is needed for every ask
  • 21. Prioritization Framework Four Pillars Value Measured by impact to Site Up, Productivity, Revenue, Key Metrics, or User Satisfaction Leverage Measured by # of users impacted in 12 months or less Urgency Measured by urgency: blocked, impaired but have short-term workaround, or inconvenient Cost Measured in engineer-quarters (engineers needed * no. of quarters to complete)
  • 22. Computing Value (V) Rubric Select up to two categories and multiply their corresponding numbers E.g., V1 = G = 4, V2 = P = 2 ⇒ V = 4 * 2 = 8 GCNs (G) Productivity (P) Revenue (R) Metrics (M) User Satisfaction (S) GCN Savings: 1. 1 minor GCN/qrtr 2. 2-4 minor GCNs 3. 1-2 medium GCNs 4. 1+ major GCNs Savings Criteria: 1. < 1hr /usr/mnth 2. 2 - 5 hrs saved 3. 6 - 10 hrs saved 4. > 10 hrs saved Revenue Lift: 1. < $1M annually 2. $1M - $10M 3. $10M - $30M 4. > $30M Key Metrics Lift: 1. < 0.5% lift 2. 0.5% - 1.0% 3. 1.0% - 2.0% 4. > 2.0% NSAT Score Lift: 1. < +0.2 lift 2. +0.3 - +0.5 3. +0.6 - +0.9 4. > +1.0
  • 23. Computing other Rubrics T-REX Users Impacted 1. < 100 users 2. 100 - 499 3. 500 - 1999 4. > 2000 Leverage (L) User’s work impact Costing Formula Urgency (U) Cost (C) 1. Inconvenient 2. Impaired; workaround in place 3. Blocked 1. # of Engineers 2. # of Quarters 3. C = E * Q
  • 24. Prioritization - Final Scoring Guidelines Final Ranking Criteria 03 ● Rank by largest Impact Score ● Override for large ROIs (Quick Wins) Calculate ROI 02 ROI = Impact Score / Cost Calculate Impact Score (IS) 01 Impact Score = Leverage * Value * Urgency
  • 25. Ideal Mix 40% 30% 20% 10% Big Bets Home Run Quick Wins Fall Back Level of Effort Ranking
  • 27. Optimize your Workflow Prioritized 50% Power Design Sign-off Plan Sign- off Feature Request Feature Release Code Ready Ideate Flow diagram, Workflows, Design, Use cases Ramp Verify solution, fix unforeseen issues, capture metrics, Derive success stories Announce Announce highlighting features and impact. Include user testimonials, training, and documentation Learn Problem statements, User stories, Impact assessment, Requirements gathering, Prioritization, PRD Design RFC, Hi-Fi Design, Acceptance test cases, WBS, Timeline Build Modular code, Bug bashes, Unit & Integration testing, Previews Design Thinking Approach with Experimentation in the Core of your Purpose Gain insights in what our customers really want and why. Build clarity in terms of Before / After state. Gradual release ramp to efficiently address unforeseen issues. Measure success metrics and announce with impact!
  • 28. Sample PRD Areas of focus: Problem Statement Core Use Cases Prioritization Details Objectives and Key Results Before and After Milestones Sign-offs
  • 29. Note on Vision and Strategy
  • 30. Be clear on your Vision to Values Vision • The Dream. The Future. What you inspire your product will impact and become. Mission • Briefly describes the goals and purpose of your product. Why the product exists? Target Audience • List down your primary user personas • Prioritize them. Also be clear on non-users. • Group them in Producers / Consumers if you have a platform product Strategy • Series of strategic objectives and roadmap milestones Priorities • Stack ranked list of critical initiatives. “If we can only do one thing this quarter, what would it be?” Objectives (Key Metrics) • True North Metrics • Sign-Post Metrics • Guardrail Metrics Values • Your guiding principles in making day-to-day decisions
  • 32. www.productschool.com Part-time Product Management Training Courses and Corporate Training