Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

QASymphony Webinar - "How to Start, Grow & Perfect Exploratory Testing on your Team"

•Download as PPTX, PDF•
•
QASymphony
QASymphony

This webinar defines a clear path moving forward to gain success with exploratory testing, no matter what stage of the testing process you are currently in. Learn how to make the internal “sell” to get exploratory testing off the ground and then how to standardize and scale exploratory testing for the enterprise.Whether your organization is waterfall, agile, or somewhere in between, a properly implemented exploratory testing process is sure to increase the value of your testing team.

1 of 31
Start, Grow, and Perfect Exploratory Testing on Your Team Kevin Dunne Director of Product Strategy QASymphony Kimberly Stockett QA Manager KMS Technology
Agenda 1. What is exploratory testing (ET)? 1. What is NOT ET? 1. How can I start an ET practice? 1. How can I expand the footprint of my ET practice? 1. How can I measure & improve the effectiveness of my ET practice?
What is exploratory testing? 1. Parallel test planning, test design, and test execution 1. Specific yet flexible 1. Aligned towards investigation of potential opportunities 1. Values depth and attention to detail during testing 1. Fosters knowledge sharing and awareness
Parallel Planning, Design & Execution Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach Plan Design Execute Report Plan DesignExecute Report
Specific yet Flexible Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc. However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it Manual Scripted Testing I tested the application as the script prescribed Exploratory Testing I tested the application as the end user would
Investigating Opportunities Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters Manual Scripted Testing Exploratory Testing
Knowledge Sharing Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity Plan DesignExecute Report Transfer Learning Example Questions to Ask • Have you seen this before? • What am I not considering? • Why would someone do this? • How would you have tested this?
What is Not Exploratory Testing 1. Exploratory testing is NOT unstructured testing 1. Exploratory testing is NOT the only form of testing 1. Exploratory testing is NOT throwaway work 1. Exploratory testing is NOT impossible in a regulated environment
ET is NOT Unstructured Testing While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as: 1. A goal of the exploration 1. A log of the activity performed 1. A lens through which the testing is performed (i.e. a user persona) Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing
ET is NOT the Only Form of Testing Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage. Potential New Feature Testing Cycle Code Developer Unit Test Exploratory Testing Manual Scripted Test Automation Regression Test Exploratory Testing Feature “Delivered” “Let’s make sure this is worth writing scripts against yet” “Let’s make sure were still testing all aspects of this”
ET is NOT Throwaway Work Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it: 1. Charter 2. Session Sheet 3. Oral report 4. Debrief 5. Data Files 6. Logs "Any testing approach is manageable when you choose to manage it.” – Michael Bolton http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/
ET is NOT Impossible in a Regulated Environment Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
Starting an Exploratory Testing Practice 1. Who - choosing the pilot group 1. What - choosing the application 1. Where - choosing the team/location 1. When - choosing the release cycle 1. How - choosing the Exploratory Testing framework/approach
Who will be the first to explore? Finding the right group of initial members to pilot exploratory testing is one of the most critical aspects of successful implementation. Pilot members should be: • Comfortable with uncertainty and flexible • Well connected in the organization • A group that covers the ranges of skill and experience on the team • Solid communicators and willing to speak up about their concerns • Attuned towards new experiences as well as personal and team improvement
What will be the first application we explore? The pilot application should be one that minimizes the risk to the business of serious impact if exploratory testing is unsuccessful. Some factors to look out for in a pilot application are: Size Small to medium sized application – easier to track coverage Time on Market Mature application – easier to see net impact of explorations Business Criticality Not the flagship product – executives will allow a longer period to pilot
Where will the first exploratory team be located? To ensure the highest likelihood of success, it is important to choose a team that is located centrally with easy access to developers and visibility to executives overseeing the implementation Centrally Located Allow for collaboration and knowledge sharing early on Access to Developers When developers are near by, testers can provide feedback quickly Executive Visibility Locating the team near execs means better visibility of impact
When will we first introduce exploratory testing? It is critical to introduce exploratory testing during a time where the pilot group can contribute their full attention to the new initiative and they are not being pulled in other directions: • Avoid “busy season” – typically around big end of year releases • Avoid releases/sprints that overlap with long holidays • Avoid times of the year when application usage is high (i.e. year end closing for an accounting application) Be cautious of implementing exploratory testing while other change initiatives are going on. It can be helpful as there is a change “tailwind”, but be aware exploratory testing might be falsely attributed to other initiatives’ failure
How will we structure our exploratory testing? Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives. Session Based Test Management is a popular framework for this, as it tracks all the important data on testing: More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf • Session charter (includes a mission statement, and areas to be tested) • Testers involved Date and time executed • Task breakdown • Data files • Test notes • Issues • Bugs
Expanding the Footprint 1. Build champions within the organization 1. Share best practices and tips 1. Understand the limits and opportunities
Team #3 Team #3 Team #3 Build Champions within the Organization There are two main ways we can build champions to spread the word about exploratory testing in our organization: Initial Team New Team New Team “Bowling Pin” Model “Infiltration” Model Initial Team Tester #1 Tester #2 Tester #3
Share Best Practices and Tips A common threat in Agile organizations is the “information hoarding.” As loyalty to the Agile team grows, teams can lose site of the importance of improving the organization as a whole, and strive instead to improve there position as the best team. Ways to mitigate against this include: • Greater alignment with organizational goals • Transfer team members periodically • Reward teams for sharing knowledge • Provide greater visibility across teams
Understand the Limits and Opportunities There is a good chance that you will find early success with exploratory testing. However, it is important to temper your excitement to roll out exploratory testing across your organization. Executives often have an incredible memory when it comes to failed initiatives. Often, it is better to follow a measured approach than risk having exploratory testing barred from your organization entirely.
Measure and Improve ET Practices 1. What are we trying to accomplish by implementing ET? 1. How will we measure ET’s ability to accomplish these goals? 1. How will accomplishing these goals improve our ability to expand ET in the organization?
Benefits of Exploratory Testing 1. Increases focus on unique scenarios with greater potential to find new bugs 1. Drives engagement from testers and fosters critical thinking during the testing process 1. Boosts team morale and provides a change of pace from manual scripted testing 1. Encourages testers to think as end users and the "voice of the customer" and drive improvement in the product
SMART Goals • Specific • Measurable • Attainable • Relevant • Time-bound
Making the Case for Exploratory Testing • Claim: ET will create more educated testers • Metric - increase in average # of testers capable of test a given scenario • Claim: ET will provide faster feedback on new features • Metric - reduction in user stories revised in subsequent sprints • Claim: ET will increase the alignment with customer needs • Metric - reduction in support tickets/feature requests • Metric - increase in net promoter score/referenceable customers • Claim: ET will garner communication product, dev, and testing • Metric - reduction in open defect time • Metric - increase in acceptance criteria met
Continuous Improvement The path to success with exploratory testing does NOT end when implementation is complete. You must continually improve to make sure that your program continues to add value in the organization Plan Implement Evaluate Reassess Continuous Improvement
KMS has successfully implemented Exploratory testing techniques for all clients seeing benefit in quality and efficiency to the act of testing with an added bonus of boosted morale for the QA Analysts. • Benefits we have seen • Time boxed testing creates structure and limits • Risk is apparent from application reviews • Faster and deeper product understanding with hands on exploration • Proactive over reactive approach in discovering bugs or workflow issues • Items to keep in mind • Worm holes in testing – knowing when to end a testing path is based on expereince of the QA Analyst • Experiences drives when to annotate in the tool (QASymphony eXplorer) to make defect referencing easier • Session definition – too much or too little KMS is a fast growing global service provider with over 500 professionals trained on QA Symphony products. We are a Platinum Partner with QASymphony with over 3 years experience in Exploratory Testing Techniques. About KMS Application of ET
Efficiency In a month long project consisting of 4 1 week sprints the team uncovered 146 bugs in 143 hours averaging 1.02 bug/hour Quality The team completed Mind Maps outlining 154 Charters completed 144 charters in 143 true testing session hours identified high priority bugs Dramatic & Measurable Results from ET Value (Bugs Found) 20 High priority at 14% 57 Medium Priority at 39% 66 Low Priority at 45% 3 Closed as N/A at 2%
Resources/Thought Leaders • James Bach - http://www.satisfice.com/ • Jonathan Bach - https://jonbox.wordpress.com/ • Michael Bolton - http://www.developsense.com/ • Paul Holland - http://testingthoughts.com/blog/author/testthought • Keith Klain - http://qualityremarks.com/ • Brian Osman - https://bjosman.wordpress.com/
Questions? Kevin Dunne kevindunne@qasymphony.com Twitter: @kevindunneQA Linkedin: www.linkedin.com/in/kevindunneQA Blog: http://www.qasymphony.com/blog/ Kimberly Stockett kimberlystockett@kms-technology.com

More Related Content

QASymphony Webinar - "How to Start, Grow & Perfect Exploratory Testing on your Team"

  • 1. Start, Grow, and Perfect Exploratory Testing on Your Team Kevin Dunne Director of Product Strategy QASymphony Kimberly Stockett QA Manager KMS Technology
  • 2. Agenda 1. What is exploratory testing (ET)? 1. What is NOT ET? 1. How can I start an ET practice? 1. How can I expand the footprint of my ET practice? 1. How can I measure & improve the effectiveness of my ET practice?
  • 3. What is exploratory testing? 1. Parallel test planning, test design, and test execution 1. Specific yet flexible 1. Aligned towards investigation of potential opportunities 1. Values depth and attention to detail during testing 1. Fosters knowledge sharing and awareness
  • 4. Parallel Planning, Design & Execution Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach Plan Design Execute Report Plan DesignExecute Report
  • 5. Specific yet Flexible Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc. However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it Manual Scripted Testing I tested the application as the script prescribed Exploratory Testing I tested the application as the end user would
  • 6. Investigating Opportunities Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters Manual Scripted Testing Exploratory Testing
  • 7. Knowledge Sharing Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity Plan DesignExecute Report Transfer Learning Example Questions to Ask • Have you seen this before? • What am I not considering? • Why would someone do this? • How would you have tested this?
  • 8. What is Not Exploratory Testing 1. Exploratory testing is NOT unstructured testing 1. Exploratory testing is NOT the only form of testing 1. Exploratory testing is NOT throwaway work 1. Exploratory testing is NOT impossible in a regulated environment
  • 9. ET is NOT Unstructured Testing While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as: 1. A goal of the exploration 1. A log of the activity performed 1. A lens through which the testing is performed (i.e. a user persona) Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing
  • 10. ET is NOT the Only Form of Testing Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage. Potential New Feature Testing Cycle Code Developer Unit Test Exploratory Testing Manual Scripted Test Automation Regression Test Exploratory Testing Feature “Delivered” “Let’s make sure this is worth writing scripts against yet” “Let’s make sure were still testing all aspects of this”
  • 11. ET is NOT Throwaway Work Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it: 1. Charter 2. Session Sheet 3. Oral report 4. Debrief 5. Data Files 6. Logs "Any testing approach is manageable when you choose to manage it.” – Michael Bolton http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/
  • 12. ET is NOT Impossible in a Regulated Environment Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
  • 13. Starting an Exploratory Testing Practice 1. Who - choosing the pilot group 1. What - choosing the application 1. Where - choosing the team/location 1. When - choosing the release cycle 1. How - choosing the Exploratory Testing framework/approach
  • 14. Who will be the first to explore? Finding the right group of initial members to pilot exploratory testing is one of the most critical aspects of successful implementation. Pilot members should be: • Comfortable with uncertainty and flexible • Well connected in the organization • A group that covers the ranges of skill and experience on the team • Solid communicators and willing to speak up about their concerns • Attuned towards new experiences as well as personal and team improvement
  • 15. What will be the first application we explore? The pilot application should be one that minimizes the risk to the business of serious impact if exploratory testing is unsuccessful. Some factors to look out for in a pilot application are: Size Small to medium sized application – easier to track coverage Time on Market Mature application – easier to see net impact of explorations Business Criticality Not the flagship product – executives will allow a longer period to pilot
  • 16. Where will the first exploratory team be located? To ensure the highest likelihood of success, it is important to choose a team that is located centrally with easy access to developers and visibility to executives overseeing the implementation Centrally Located Allow for collaboration and knowledge sharing early on Access to Developers When developers are near by, testers can provide feedback quickly Executive Visibility Locating the team near execs means better visibility of impact
  • 17. When will we first introduce exploratory testing? It is critical to introduce exploratory testing during a time where the pilot group can contribute their full attention to the new initiative and they are not being pulled in other directions: • Avoid “busy season” – typically around big end of year releases • Avoid releases/sprints that overlap with long holidays • Avoid times of the year when application usage is high (i.e. year end closing for an accounting application) Be cautious of implementing exploratory testing while other change initiatives are going on. It can be helpful as there is a change “tailwind”, but be aware exploratory testing might be falsely attributed to other initiatives’ failure
  • 18. How will we structure our exploratory testing? Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives. Session Based Test Management is a popular framework for this, as it tracks all the important data on testing: More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf • Session charter (includes a mission statement, and areas to be tested) • Testers involved Date and time executed • Task breakdown • Data files • Test notes • Issues • Bugs
  • 19. Expanding the Footprint 1. Build champions within the organization 1. Share best practices and tips 1. Understand the limits and opportunities
  • 20. Team #3 Team #3 Team #3 Build Champions within the Organization There are two main ways we can build champions to spread the word about exploratory testing in our organization: Initial Team New Team New Team “Bowling Pin” Model “Infiltration” Model Initial Team Tester #1 Tester #2 Tester #3
  • 21. Share Best Practices and Tips A common threat in Agile organizations is the “information hoarding.” As loyalty to the Agile team grows, teams can lose site of the importance of improving the organization as a whole, and strive instead to improve there position as the best team. Ways to mitigate against this include: • Greater alignment with organizational goals • Transfer team members periodically • Reward teams for sharing knowledge • Provide greater visibility across teams
  • 22. Understand the Limits and Opportunities There is a good chance that you will find early success with exploratory testing. However, it is important to temper your excitement to roll out exploratory testing across your organization. Executives often have an incredible memory when it comes to failed initiatives. Often, it is better to follow a measured approach than risk having exploratory testing barred from your organization entirely.
  • 23. Measure and Improve ET Practices 1. What are we trying to accomplish by implementing ET? 1. How will we measure ET’s ability to accomplish these goals? 1. How will accomplishing these goals improve our ability to expand ET in the organization?
  • 24. Benefits of Exploratory Testing 1. Increases focus on unique scenarios with greater potential to find new bugs 1. Drives engagement from testers and fosters critical thinking during the testing process 1. Boosts team morale and provides a change of pace from manual scripted testing 1. Encourages testers to think as end users and the "voice of the customer" and drive improvement in the product
  • 25. SMART Goals • Specific • Measurable • Attainable • Relevant • Time-bound
  • 26. Making the Case for Exploratory Testing • Claim: ET will create more educated testers • Metric - increase in average # of testers capable of test a given scenario • Claim: ET will provide faster feedback on new features • Metric - reduction in user stories revised in subsequent sprints • Claim: ET will increase the alignment with customer needs • Metric - reduction in support tickets/feature requests • Metric - increase in net promoter score/referenceable customers • Claim: ET will garner communication product, dev, and testing • Metric - reduction in open defect time • Metric - increase in acceptance criteria met
  • 27. Continuous Improvement The path to success with exploratory testing does NOT end when implementation is complete. You must continually improve to make sure that your program continues to add value in the organization Plan Implement Evaluate Reassess Continuous Improvement
  • 28. KMS has successfully implemented Exploratory testing techniques for all clients seeing benefit in quality and efficiency to the act of testing with an added bonus of boosted morale for the QA Analysts. • Benefits we have seen • Time boxed testing creates structure and limits • Risk is apparent from application reviews • Faster and deeper product understanding with hands on exploration • Proactive over reactive approach in discovering bugs or workflow issues • Items to keep in mind • Worm holes in testing – knowing when to end a testing path is based on expereince of the QA Analyst • Experiences drives when to annotate in the tool (QASymphony eXplorer) to make defect referencing easier • Session definition – too much or too little KMS is a fast growing global service provider with over 500 professionals trained on QA Symphony products. We are a Platinum Partner with QASymphony with over 3 years experience in Exploratory Testing Techniques. About KMS Application of ET
  • 29. Efficiency In a month long project consisting of 4 1 week sprints the team uncovered 146 bugs in 143 hours averaging 1.02 bug/hour Quality The team completed Mind Maps outlining 154 Charters completed 144 charters in 143 true testing session hours identified high priority bugs Dramatic & Measurable Results from ET Value (Bugs Found) 20 High priority at 14% 57 Medium Priority at 39% 66 Low Priority at 45% 3 Closed as N/A at 2%
  • 30. Resources/Thought Leaders • James Bach - http://www.satisfice.com/ • Jonathan Bach - https://jonbox.wordpress.com/ • Michael Bolton - http://www.developsense.com/ • Paul Holland - http://testingthoughts.com/blog/author/testthought • Keith Klain - http://qualityremarks.com/ • Brian Osman - https://bjosman.wordpress.com/
  • 31. Questions? Kevin Dunne kevindunne@qasymphony.com Twitter: @kevindunneQA Linkedin: www.linkedin.com/in/kevindunneQA Blog: http://www.qasymphony.com/blog/ Kimberly Stockett kimberlystockett@kms-technology.com