Isabel Evans - Quality In Use - EuroSTAR 2011

Editor's Notes

1. Title slide: Quality in use: the beating heart of the customer experience Isabel Evans Quality Manager Dolphin Computer Access Graphics: Dolphin logo; illustration of heart and heartbeat; An accessible Word document describing this presentation is available from isabel.evans@dolphinuk.co.uk   At Dolphin Computer Access, the company where I am quality manager, we build assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic.   In my EuroSTAR presentation "Quality in Use - the beating heart of the customer experience" I am going to discuss the importance of usability, accessibility, and flexibility of use for our customers and for yours.... How well are people supported to carry out tasks?   I will illustrate the need to focus first on the customer to understand what technical product attributes are important. I will show you how we implemented user stories with acceptance criteria based on quality in use attributes. These help everyone focus on what is important for our customers.   I will talk about what has gone well, and what we had to improve as well as where we have had to adapt standards and methods to fit our particular needs. You will hear the lessons we have learned.
2. Introducing Dolphin Computer Access – Dolphin’s Purpose Dolphin provides assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic. Our managing director, Noel Duffy has set us a purpose as a company:   “It is our purpose to deliver independence to people with vision and print impairments all over the World. With over 160 million visually impaired people and 600 million dyslexic or learning disabled people, that's no small challenge. But we're determined to make a difference and have been working hard on that goal since Dolphin started in 1986.” We have a particular challenge.   Graphics - photographs of various Dolphin customers
3. As Dave Williams (Dolphin Blindness Product Owner and Q&D Tester) says “In an ideal world no-one would need our products – no-one chooses to be blind.” The Dolphin Challenge is that even if the products are perfect no-one chooses to be print-impaired... so we must work hard to make our products right for our customers. Graphic: photo of Dave Williams
4. Agenda diagram – radial circle, “Quality in Use” in the centre; items around the radial circle are: Our journey What? Why? How? Graphic – illustrative/decorative
5. Our Journey Dolphin recognised an opportunity for improvement with the help of Julian Harty (Google and then E-Bay) and as a result decided to take on a quality manager with the idea that the immediate need was to improve testing. Improvements were needed because there were gaps in the test capability: No (few) specialist testers Poor reporting on testing Poor application of test techniques No test automation. Graphics: illustrative/decorative
6. Solution 1: more testing, better testing We took these first steps: Improving test skills and motivation – identified testers for the different products, and started providing in-house workshops on test approaches and test design; We had a shortage of resources as well as a shortage of skills and so “Night-siding” to a test services company in New Zealand has given us extra capacity, fast turnaround, 24 hour work, and skilled testers providing role models for less experienced in-house testers; As a result of these the testers were able to build better regression test packs, with overview test conditions based on personas and user stories; We fed out information on test techniques to testers and developers across the company in weekly short bursts – a 2-4 side document “Technique of the Week” with lunchtime briefings in the following week to discuss and consolidate; We have started to automate some testing: CTE and Python has been very useful for unit tests on one product range For the access software is harder to do because of the way the software intercepts what is happening as in the operating system and any applications that are running so we have started to build automation in-house – we have three tools: Rhubarb, Beetroot and Sledgehammer in progress. Graphics: photos of Colin Shales (Q&D usability tester) and James Bowden (Developer and test automation)
7. Solution 2 - Improve test reportingTest coverage pie charts The graphic illustrates how we used pie charts to show test progress – the one chosen is from one product midway through testing in August – and shows about a third of the planned tests had been run at that stage, the majority having passed. The product has now passed all planned tests and is being released.
8. Solution 3: improve metrics The graphic show two graphs comparing the number of issues raised in testing with the number raised after release of a product plus a target improvement.   Product A release X: three quarters of the issues were raised post release, before any improvements to process or technique for development or test.   We then improved the testing. Product A release Y: some improvements to testing – now two thirds of the issues were found in testing and only a third post release.   Our aim for December is shown in pie chart 3 – by release Z in December, we want the DDP to be at 85%, and this will be achieved by development improvements as well as test improvements. This include white box testing and code review, design and architecture reviews and more automation.   Use of these pie charts had two benefits – the testers could see that they had improved and management could be persuaded what actions were necessary to take next.
9. Side effects of the cure Testing found so many defects and such important defects that we had to: Halt a project and restart; morale dropped especially among the developers; the developer workload increased. The side effects of the cure (improving testing) included meeting the change management problems of resistance and fear, trying to implement solutions beyond the maturity of the organisation, causing setbacks, and loss of balance between pressures. Automation was harder for the access products than I had anticipated. The lesson and key point: TEST IMPROVEMENT IS NOT ENOUGH!   Other problems: Accessibility – in order to keep all team members up to date we had to ensure that all the artefacts around the project were accessible. We had to balance that with using non-accessible artefacts where that did not disadvantage colleagues with visual impairments and benefited sighted colleagues. E.g. flowcharts can be used and then expressed as text. This document is intended to take the place of the slide set. Level of maturity – it is important to make things as simple as possible, in order that everyone can buy in, and then increase the complexity gradually. Change gradually over time – it is easy to get impatient – change takes time. WAI compliance – our tools have to deal a wide range of software applications and websites – for example, every time a website is non-compliant to WAI guidelines and our software cannot read it, it is us that the customer rings...
10. As well as test improvement support the company vision and purpose: Improving testing had made some things worse and had exposed problems that could not be resolved by testing. Instead of just improving testing we needed to Support the vision and purpose: Deliver appropriate solutions; be loved by customers; and Deliver commercial success.   Message to testers: just improving testing is not enough to help the business - Our skill set and comfort zone is often in the detail; we find excitingly obscure bugs and report these problems in detail. Our zeal sometimes causes us to miss the big picture of what the customer and the business really needs. We focus on software quality at the expense of human and commercial factors. We need to consider company vision and customers, we need to consider commercials. Graphic: decorative
11. Solution 4: use ISO25000 - Quality in Use Graphic shoes there are three Quality in Use attributes: Usability Flexibility Safety   Usability has the sub-attributes Effectiveness; Efficiency; Satisfaction Flexibility has the sub-attributes Context conformity; Context extendibility; Accessibility Safety has the sub-attributes Commercial damage; Operator health and safety; Public health and safety; Environmental harm ISO125000 is a standard that describes software attributes, how to specify them and how to measure them: Software Quality and Requirements Evaluation
12. Quality in Use (QiU) and Dolphin The graphic shows the Quality in Use attributes of Usability, Flexibility and Safety, which reflect a user and value view of quality –“my tasks” and “my money”.   Buyers and Dealers are looking for commercial safety and flexibility People with VI & TI are looking for usability and flexibility Dolphin sales and marketing are looking for all 3 attributes, but particularly flexibility – new things to sell.
13. People are different... The photographs show two Dolphin customers, both have visual impairments but their requirements from an access product are quite different. The gentleman on the left is using Guide. This give magnification voice output and – if wished – voice input. It also provides a simplified interface which replaces Windows and provides a simple internet browser, email, simple word processing. It is suitable for people who are less used to computers and/or have recently lost vision. The lady on the right is tech-savvy and uses applications such as Office, as well as all the work applications that her colleagues use. She is using SuperNova with multiple monitors to magnify the display. SuperNova is suitable for people who are tech-savvy and want to use all the applications you use at work or at home. These people have different personas and contexts of use. They have different quality in use scores. Even if they share user stories (I want to do on-line banking) these need to be implemented in different ways.
14. QiU scores for two Dolphin product users Quality in use – products compared: The graphic is a radar graph, showing each of the Quality in Use sub-attributes, and the scores for two Dolphin products – SuperNova and Guide. Each has users who have visual impairments but the two groups have quite different quality in use attribute scores. SuperNova users are typically working or at college. They are typically tech-savvy. They value being able to work efficiently and flexibly, using SuperNova to access a wide range of applications. They are conscious of whether SuperNova saves or costs them money. The typical Guide user has recent sight loss, and low IT experience. Typically a Guide users value the interface being stable and predictable (conformity) and knowing they are able to be effective (they are protected from making mistakes). The attributes are ordered:   SuperNova typical user: Efficiency, Extendibility, Effectiveness, Commercial, Satisfaction, Conformity, Health & Safety, Environment. Guide typical user: Effectiveness, Conformity, Satisfaction, Efficiency, Extendibility, Health & Safety, Commercial, Environment.
15. Solution 5 – don’t forget the engineering details We also need the engineering details – that is Internal Quality – How was it built? Is it faulty? Were good techniques used? Have all the “ilities” been considered?   The internal qualities are functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability   The customers who are systems administrators care about these. Dolphin Support and Development care about these. Technical testers care about these. Others only notice them when they go wrong...
16. Internal quality (IQ) – engineering detail Internal Quality – How was it built? Is it faulty? Were good techniques used? The graphic shows the internal qualities: Functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability The customers who are systems administrators care about these. Support and development care about these. Others only notices them when they go wrong...
17. QiU and IQ – Dolphin Project Team The project teams need to have representatives for quality in use and internal quality. Dolphin Sales and Marketing are interested in quality in use Dolphin Support and Development are interested in internal quality Both groups include colleagues who have visual impairments.   We make sure we have considered: Usability – Effectiveness; Efficiency; Satisfaction Flexibility - Context conformity; Context extendibility; Accessibility Safety - Commercial damage; Operator health and safety; Public health and safety; Environmental harm   Internal quality is made up of: Functional suitability, operability, portability, compatibility, maintainability, security, performance, reliability And we make sure that we prioritise these based on the Quality in Use priorities.
18. Testers for QiU and IQ Where do the testers sit? We need test skills in quality in use and internal quality. We have a group called Q&D (Quality and Design) which includes the quality manager, some specialist testers and project management. Testing is carried out by: Static testing (reviews) by people across the project team Dynamic testing by developers, Q&D, New Zealand nightsiders, representative in-house product users, sales and marketing, support, dealers, customer beta testers. At the start of projects, Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development.   Development does research, design, development, code review and unit test. Q&D (specialist) Testers carry out a Product test, helped by representative in-house product users, and by the New Zealand nightsiding team. When sufficiently stable it is passed for beta testing to representative dealers and customers. When the product test passes acceptance criteria, it is passed to Sales, Marketing, and Support to carry out market readiness testing.
19. Testers for QiU and IQ The graphic shows project stages above and below the quality in use line Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development. Development does research, design, development, unit test (specialist) Testers do Product test Sales, Marketing, Support carry out market readiness testing Externals carry out Beta testing
20. Solution 6 – customer-centric SDLC This graphic shows the steps through our project model. The Market Requirements phase is run by the product owner – typically from marketing - and project mentor – management team representative to identify the project vision for the customer, context of use, personas, user stories, the commercial aspects, benefits for the customer and for Dolphin, acceptance criteria. This is outlined in a Market Requirements Document. It could lead to a research project, prototyping or a delivery project, depending on our knowledge. Typically that would be in iterations. We run a brief post implementation review at the end of each iteration, or group of iterations as well as at project ends.
21. Solution 7 - project workshops include Sales and Marketing It is vital that Sales and Marketing are including in all the decisions. They take part in the workshops and other activities to build user stories. Marketing, Sales, Support: Focus on emotional Focus on practical Focus on tasks Focus on Vision for Customer and Dolphin User stories Acceptance criteria. Development and technical testers focus on measureable acceptance criteria and architecture. Q&D facilitate and encourage. The emotional level in the user stories provided by Sales, Marketing, and Support and in particular by product users in the company helps us focus on what is important. Example:   As a person who is blind I wanted to manage my own bank account online Because it is so humiliating when someone has to read you your bank statement   As an elderly person with declining vision I want to contact my grandchildren online Because I miss them and want to share news easily
22. Solution 8 – Sensible Iterations to focus the detail The graphic shows a series of overlapping iterations, each of 6 weeks plus week 7 for release. During the 6 weeks research, development, test for this and subsequent iterations are carried out. We found that we have work which needs to carry across several iterations. In the last 2 weeks market readiness testing is done by Sales, Marketing and Support, and they agree the contents for the next iteration.  
23. Market readiness email In the last 2 weeks of the iteration we move to Sales, Marketing and Support carrying out market readiness testing and choosing items for the next iteration. A Market readiness email is sent from Q&D to everyone in Sales, Marketing, Support, and to Training, Documentation, and in-house product users. The email says:   What is the verdict on this build? Action: verdict by (date – usually a week is allowed): Yes – there may be issues but they can be lived with for this iteration No – there are showstoppers Don't know – I have not finished my MR tests – estimated date to complete No opinion – I have not had time to test Proposed deadline for (iteration): Date for all market readiness feedback and planned date for release.  
24. Iteration content email A rough iteration plan for several iterations ahead, plus a wish list, is kept running. Items for iterations are planned in and discussed with all the stakeholders, as required and confirmed in a weekly meeting as we monitor support calls. Just after the MR email, an iteration content email is sent out. This is a series of paragraphs headed:   What is going to be in (next iteration)? List of items What about the issue with (....)? Items requested that cannot be covered in the iteration and why, plus a planned iteration How did we come up with this content list? What is the proposed schedule? What is the contingency plan? What is your verdict on this plan? Response by (date).
25. What have we learnt? Marketing, Sales and Support involvement is vital, but they have be shown the value and that progress has been made because of their contributions Improving project & quality management is very important and needs to done in small doses Improving test skills and motivation in very important and needs to be managed against other improvements in order that problems are uncovered and resolved at source Iterations forced out process weaknesses – the pain of moving to an iterative approach was large, but the problems were process, skill, control, tool or techniques problems that needed to be sorted out – the sort time spans exposed these more clearly Night-siding to New Zealand was really useful to provide a breathing space and also caused people in-house to want to improve what they were delivering it exposed time testing preventable issues as real costs Technique of the Week & lunchtime briefings helped people pick and choose topics to suit their immediate needs and meant iterative / incremental approach to improvement It’s good when other people have “your” ideas – it means they have adopted them Automation is hard sometimes – but worth persisting with Accessibility and white-boarding... don’t be dogmatic about the right way to do things – use the methods that suit the team Lots more to do...we have made huge improvements in the last year, but there is a lot more to do!   What has gone well? Sales and marketing involvement has worked well in providing context of use, personas, user stories, setting acceptance criteria, and market readiness testing. Increasing awareness in the organisation of the importance of strong project managers to control budget, timescale, risk and cost benefit analysis. Increasing confidence and awareness in the testers (from all areas) of the importance of strong quality management and testing. They hold the line, with courage, information, appreciation of risk, they offer choices based on cost benefits. Realisation among the testers and elsewhere that test skills and motivation is not enough – we also need to understand the commercial, sales, marketing and customer impact. Quality in use as the focus. Night-siding works and helped internal testers improve. Iterative skills improvement with Technique of the Week, workshops, lunchtime briefings – push out the improvements a few at a time. We tried to do too much too quickly at first – pick off the targets, and persist!
26. The journey continues We need to remember everyday our purpose as an organisation, our customers and what they need from use. Quality in use (usability, flexibility and safety) keeps us focused on what is important. Remember “In an ideal world no-one would need our products – no-one chooses to be blind”. Quality in use is for us the beating heart of the customer experience. Graphics - photographs of various Dolphin customers and staff
27. Quality in use: the beating heart of the customer experience Usability, Flexibility, Safety The journey continues – thanks and goodbye Graphics: Dolphin logo; illustration of heart and heartbeat; text boxes with usability, flexibility and safety.