Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

The Best of Both Worlds - Combining Performance and Functional Mobile App Testing

•Download as PPTX, PDF•
•
Bitbar
Bitbar

We co-hosted a webinar with Neotys to shed some lights on - How to overcome the challenges in mobile app performance and functional testing - How to gain granular and actionable insights to measure and improve your app user experience - Best practices to get the mobile readiness for 2017 Holiday Shopping Season - A brief demo of the integration between Neoload and Bitbar Testing

1 of 38
The Best of Both Worlds - Combining Performance and Functional Mobile App Testing
HouseKeeping rules
Presenters Speaker Henrik Rexed Performance Engineer Neotys Speaker Lingkai Shao Marketing Analyst Bitbar Speaker Dave Horner Director of Sales Bitbar
Neotys Continuous Performance Testing Solution System Testing Integrated Component Testing/System-wide Testing ReleaseComponent Testing Continuous Integration & Component Testing Proactive Production Monitoring End-User Experience
“Continuously ship quality apps with a high velocity delivery pipeline” Bitbar - Mobile DevOps
Dev Ops Continuous Release & Deployment Collaborative Development Continuous Testing Continuous Monitoring Continuous Integration Continuous Feedback DevOps is no longer a unicorn’s discipline. TestDev Ops Shift Left Shift Right DevOps
According to Adobe 2016 Online Shopping Data, mobile drove 55% of visits to retail websites on Black Friday (45% coming from smartphones, 10% from tablets) and accounted for 36% of sales (25% from smartphones, 11% from tablets) * Why Functional and Performance Testing for Holiday Season *=http://news.adobe.com/press-release/marketing-cloud/media-alert-adobe-data-shows-black-friday-breaks-online-sales-record-3
Performance matters 53% of visits to mobile sites are abandoned after 3 seconds according to research from Google According to Google doubleClick faster websites has 70% longer average session lengths, 35% lower bounce rates and 25% higher ad viewability than their slower counterparts. Etam reduces its average page load time from 1.2s to 500ms and increased conversions by 20% Walmart saw up to a 2% increase in conversions for every 1 second of improvement in load time. Every 100ms improvementalso resulted in up to a 1% increase in revenue.
Top Failures During Peak Days
Concrete study
When Revenue is Generated
● Preparation: typically 10 weeks ahead of peak days ● Functional test suite including Stage One: Functional & Compatibility
Do you know the behavior of your app/web users? Defining the right user flows
Stage One: Functional & Compatibility
Stage One: Functional & Compatibility
Stage Two: Performance & Load
Standard performance testing workflow Build the performance strategy Create the testing scenarios Prepare Execute Analyse Tuning
Performance design is the key
Mobile Application Architecture Landscape Native app Hybrid app Web app Web app Web Model Mobile Model Rest Interface Business logic Data Access logic XML Legacy Relational Presentation Business Data Third party
Mobile Performance Testing Touchpoints BACK-END EXECUTION PLATFORM RIA/FRAMEWORKS DATABAS E NETWORK 3G, 4G / LTE, Wi-Fi VIDEO HTTP Adaptive StreamingThroughput Latency Packet loss SERVER INFRASTRUCTU RE OS CLIENT APP DEVICE iPhone 4, 5, 6 Samsung Blackberry … (7,000 different device types are used to access Facebook) Screen Size CPU Memory Native Browser-Based Hybrid
Impact of Mobile Network
The most important network constraints are: • Latency • Bandwidth At a certain point, increasing the bandwidth will only have a minor effect. Latency & Bandwidth . . .
Packet Loss Consequences • Packet Retransmission • End user experience degraded because packet loss increases jitter • Errors could occur if packet loss is in pure data transmission Packet Loss is Inevitable with Mobile • Mobility: Quickly switching to the next cell tower may result in temporary signal loss due to congestion on the current tower Mobile Protocols Try to Reduce Packet Loss Rate Packet Loss . . .
Lots of users • Mobile applications often have large user bases • High growth in popularity can create the need for highly scalable load generation infrastructure Geographical distributed load • Users are accessing apps from locations around the globe • Distance and location can greatly affect performance Mix of Web & Mobile Users • Many mobile apps have a corresponding web app with a shared backend • High load on the web app can affect performance on mobile High volume, Geographically Distributed
Don’t forget the User Experience….. What are is really being measured?
… Applying the Bitbar Testing Integration… 00:00:00 00:00:30 00:01:00 00:01:30 Load NeoLoad Application Server App Error Rate Virtual Users Load User Experience
RUM (Passive) vs Synthetic (Active) Stage Three: Proactive Production Monitoring
Why Synthetic Monitoring? ● Alarm and Alert ● Better MTTR! (Mean Time To Resolution) ● Find and fix issues before impact to users ● Baselining and Benchmarking (Pre-production Staging & Post Production) ● Prepare for peak traffic and new markets ● Monitor Complex Transactions ● Measure and Adhere to SLAs ● Hold 3rd Party Vendors Accountable ● Test from end user perspective ● Competitive Benchmarking Stage Three: Proactive Production Monitoring
Understand Performance of Complex User Transactions and Hold 3rd Party Services Accountable (SLAs) Benefits of Synthetic Monitoring
Faster Mean Time to Repair… Identify Issues Before Your Users Do Benefits of Synthetic Monitoring
Understand Performance In New Markets, Optimize Content Delivery, Benchmark Pre & Post Change Benefits of Synthetic Monitoring
Competitive Benchmarking: Benefits of Synthetic Monitoring
Purpose Built Synthetic Monitoring Solution
Global Points of Presence
Use Your Existing Functional Test Scripts For Monitoring
Solution: Stage Three: Proactive Production Monitoring
DEMO Demonstration
Questions? A Q

More Related Content

The Best of Both Worlds - Combining Performance and Functional Mobile App Testing

  • 1. The Best of Both Worlds - Combining Performance and Functional Mobile App Testing
  • 3. Presenters Speaker Henrik Rexed Performance Engineer Neotys Speaker Lingkai Shao Marketing Analyst Bitbar Speaker Dave Horner Director of Sales Bitbar
  • 4. Neotys Continuous Performance Testing Solution System Testing Integrated Component Testing/System-wide Testing ReleaseComponent Testing Continuous Integration & Component Testing Proactive Production Monitoring End-User Experience
  • 5. “Continuously ship quality apps with a high velocity delivery pipeline” Bitbar - Mobile DevOps
  • 7. According to Adobe 2016 Online Shopping Data, mobile drove 55% of visits to retail websites on Black Friday (45% coming from smartphones, 10% from tablets) and accounted for 36% of sales (25% from smartphones, 11% from tablets) * Why Functional and Performance Testing for Holiday Season *=http://news.adobe.com/press-release/marketing-cloud/media-alert-adobe-data-shows-black-friday-breaks-online-sales-record-3
  • 8. Performance matters 53% of visits to mobile sites are abandoned after 3 seconds according to research from Google According to Google doubleClick faster websites has 70% longer average session lengths, 35% lower bounce rates and 25% higher ad viewability than their slower counterparts. Etam reduces its average page load time from 1.2s to 500ms and increased conversions by 20% Walmart saw up to a 2% increase in conversions for every 1 second of improvement in load time. Every 100ms improvementalso resulted in up to a 1% increase in revenue.
  • 11. When Revenue is Generated
  • 12. ● Preparation: typically 10 weeks ahead of peak days ● Functional test suite including Stage One: Functional & Compatibility
  • 13. Do you know the behavior of your app/web users? Defining the right user flows
  • 14. Stage One: Functional & Compatibility
  • 15. Stage One: Functional & Compatibility
  • 17. Standard performance testing workflow Build the performance strategy Create the testing scenarios Prepare Execute Analyse Tuning
  • 19. Mobile Application Architecture Landscape Native app Hybrid app Web app Web app Web Model Mobile Model Rest Interface Business logic Data Access logic XML Legacy Relational Presentation Business Data Third party
  • 20. Mobile Performance Testing Touchpoints BACK-END EXECUTION PLATFORM RIA/FRAMEWORKS DATABAS E NETWORK 3G, 4G / LTE, Wi-Fi VIDEO HTTP Adaptive StreamingThroughput Latency Packet loss SERVER INFRASTRUCTU RE OS CLIENT APP DEVICE iPhone 4, 5, 6 Samsung Blackberry … (7,000 different device types are used to access Facebook) Screen Size CPU Memory Native Browser-Based Hybrid
  • 21. Impact of Mobile Network
  • 22. The most important network constraints are: • Latency • Bandwidth At a certain point, increasing the bandwidth will only have a minor effect. Latency & Bandwidth . . .
  • 23. Packet Loss Consequences • Packet Retransmission • End user experience degraded because packet loss increases jitter • Errors could occur if packet loss is in pure data transmission Packet Loss is Inevitable with Mobile • Mobility: Quickly switching to the next cell tower may result in temporary signal loss due to congestion on the current tower Mobile Protocols Try to Reduce Packet Loss Rate Packet Loss . . .
  • 24. Lots of users • Mobile applications often have large user bases • High growth in popularity can create the need for highly scalable load generation infrastructure Geographical distributed load • Users are accessing apps from locations around the globe • Distance and location can greatly affect performance Mix of Web & Mobile Users • Many mobile apps have a corresponding web app with a shared backend • High load on the web app can affect performance on mobile High volume, Geographically Distributed
  • 25. Don’t forget the User Experience….. What are is really being measured?
  • 26. … Applying the Bitbar Testing Integration… 00:00:00 00:00:30 00:01:00 00:01:30 Load NeoLoad Application Server App Error Rate Virtual Users Load User Experience
  • 27. RUM (Passive) vs Synthetic (Active) Stage Three: Proactive Production Monitoring
  • 28. Why Synthetic Monitoring? ● Alarm and Alert ● Better MTTR! (Mean Time To Resolution) ● Find and fix issues before impact to users ● Baselining and Benchmarking (Pre-production Staging & Post Production) ● Prepare for peak traffic and new markets ● Monitor Complex Transactions ● Measure and Adhere to SLAs ● Hold 3rd Party Vendors Accountable ● Test from end user perspective ● Competitive Benchmarking Stage Three: Proactive Production Monitoring
  • 29. Understand Performance of Complex User Transactions and Hold 3rd Party Services Accountable (SLAs) Benefits of Synthetic Monitoring
  • 30. Faster Mean Time to Repair… Identify Issues Before Your Users Do Benefits of Synthetic Monitoring
  • 31. Understand Performance In New Markets, Optimize Content Delivery, Benchmark Pre & Post Change Benefits of Synthetic Monitoring
  • 33. Purpose Built Synthetic Monitoring Solution
  • 34. Global Points of Presence
  • 35. Use Your Existing Functional Test Scripts For Monitoring
  • 36. Solution: Stage Three: Proactive Production Monitoring