EuroSTAR Software Testing Conference 2011 presentation on Quality In Use by Isabel Evans. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
1 of 27
More Related Content
Isabel Evans - Quality In Use - EuroSTAR 2011
1. Quality in use:
the beating heart of the
customer experience
Isabel Evans - Quality Manager
Dolphin Computer Access
Accessible document describing this presentation available from isabel.evans@dolphinuk.co.uk
2. Dolphin’s Purpose
“It is our purpose
to deliver independence
to people with vision and print
impairments all
over the World.”
Noel Duffy
Managing Director
Dolphin Computer Access
3. “In an ideal world no-one would need our
products – no-one chooses to be blind”
• The Dolphin Challenge:
• Even if the products are
perfect no-one chooses to
be print-impaired...
Dave Williams
Dolphin Blindness Product Owner
and Q&D Tester
5. Our journey
Dolphin recognises an
opportunity for improvement
• Takes on a quality manager
– To improve testing
• What needs improving?
– No specialist testers
– No reporting on testing
– No techniques
– No automation
6. Solution 1: more testing, better testing
• Improving test skills and motivation
• Night-siding to New Zealand
• Regression test packs, personas, user stories
• Technique of the Week & lunchtime briefings
• Automated some testing
– CTE and Python
– Rhubarb
– Beetroot
– Sledgehammer
• Lots more to do...
7. Solution 2 - Improve test reporting
Example snapshot for one product – simple reporting:
tests passed
tests failed
tests not yet run
8. Solution 3: improve metrics
Example snapshot for one product – simple reporting:
Defects found in test Defects found after release
Release X
More defects
found after release
than in test
More defects found
in testing than after
release
Release Z
Improvement
target we have set
for ourselves
Release Y
By July Aim for
December
9. Side effects of the cure
• Disadvantages of improving testing
and acceptance criteria
• Being wedded to your own solution
• The “new cleaner” syndrome
• Fear of change
• Level of maturity
• Loss of balance between pressures
• Cannot automate yet...
• TEST IMPROVEMENT IS NOT ENOUGH!
Project
stopped
and
restarted
Morale
drop
Workload
increase
10. As well as test improvement support
the company vision and purpose:
Deliver appropriate solutions
Be loved by customers
Deliver commercial success
11. Solution 4: use ISO25000 - Quality in Use
Usability
Context conformity;
Context extendibility;
Accessibility
Safety
Effectiveness;
Efficiency;
Satisfaction
Flexibility
Commercial damage;
Health & safety (operator & public);
Environment
12. Quality in Use (QiU) and Dolphin
U
F
People with Visual
Impairments (VI)
Marketing Sales
S
Buyers
Dealers
Quality in Use – “My tasks” and “My money”
14. QiU scores for two Dolphin product users
Effectiveness
10
8
6
4
2
0
Efficiency
Satisfaction
Conformity
Extendibility
Environment
Health &
Safety
Commercial
VI - SuperNova
VI - Guide
15. Solution 5: don’t forget engineering detail
Security
Functional suitability
Operability
Portability
Compatibility
Maintainability
Performance
Reliability
Internal Quality (IQ)
How was it built? Is it faulty?
Were good techniques used?
16. Internal quality (IQ) – engineering detail
Support
F_O Sys-Ads
Development
S_P_R
Po_Co_Ma
Engineers and technologists care about these...
Other people only notice them if they go wrong...
17. QiU and IQ – Dolphin Project Team
Quality in Use
Internal Quality
U
F
S
F_O
S_P_R
Po_Co_Ma
Includes People
with VI
Sales
Marketing
Support
Includes People
with VI
Development
18. Quality in Use
Internal Quality
U
F
S
F_O
S_P_R
Po_Co_Ma
Where
are
the
testers
?
Testers for QiU and IQ
19. U
F
S
Testers for QiU and IQ
F_O
S_P_R
Po_Co_Ma
Market Readiness testing
Beta testing
Research, Design,
Development, Code
Review, Unit test,
Product test
20. Solution 6 – customer-centric SDLC
Market Requirements
Iterations
U F
S IQ
Post Implementation Review
21. Solution 7 - Project workshops with
sales and marketing
• Marketing, Sales,
Support
– Focus on emotional
– Focus on practical
– Focus on tasks
• Vision
• User stories
• Acceptance criteria
• Development, Technical
testers
– Focus on measureable
acceptance criteria
– Focus on architecture
U F S
F_O
S_P_R
Po_Co_Ma
22. w7
Solution 8 – Sensible Iterations
w1 w2 w3 w4 w5 w6 w7
w1 Choose
Contents
Release
Market
Ready test
R&D&T for this iteration
R&D&T for multi iteration items
Release
to focus the detail
23. Market readiness email
• What is the verdict on this build?
• Action: verdict by (date):
– Yes
– No
– Don't know
– No opinion
• Proposed deadline for …
– Date for all market readiness feedback
– Date for release Market
Ready test
24. Iteration content email
• What is going to be in the next iteration?
• What about the issue with (....)?
• How did we come up with this content list?
• What is the proposed schedule?
• What is the contingency plan?
• What is your verdict on this plan?
• Response by (date)
Choose
Contents
25. What have we learnt?
• Marketing, Sales and Support involvement
• Improving project & quality management
• Improving test skills and motivation
• Iterations forced out process weaknesses
• Night-siding to New Zealand
• Technique of the Week & lunchtime briefings
• It’s good when other people have “your” ideas
• Automation is hard sometimes
• Accessibility and white-boarding...
• Lots more to do...
26. The journey
continues
“In an ideal world
no-one would need our products
no-one chooses to be blind”
27. Quality in Use:
The beating heart of the customer experience
Usability Flexibility Safety
The journey continues...
Thanks and goodbye
Isabel Evans – Quality Manager – Dolphin Computer Access
Editor's Notes
Title slide:
Quality in use: the beating heart of the customer experience
Isabel Evans
Quality Manager
Dolphin Computer Access
Graphics: Dolphin logo; illustration of heart and heartbeat;
An accessible Word document describing this presentation is available from isabel.evans@dolphinuk.co.uk
At Dolphin Computer Access, the company where I am quality manager, we build assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic.
In my EuroSTAR presentation "Quality in Use - the beating heart of the customer experience" I am going to discuss the importance of usability, accessibility, and flexibility of use for our customers and for yours.... How well are people supported to carry out tasks?
I will illustrate the need to focus first on the customer to understand what technical product attributes are important. I will show you how we implemented user stories with acceptance criteria based on quality in use attributes. These help everyone focus on what is important for our customers.
I will talk about what has gone well, and what we had to improve as well as where we have had to adapt standards and methods to fit our particular needs. You will hear the lessons we have learned.
Introducing Dolphin Computer Access – Dolphin’s Purpose
Dolphin provides assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic. Our managing director, Noel Duffy has set us a purpose as a company:
“It is our purpose to deliver independence to people with vision and print impairments all over the World. With over 160 million visually impaired people and 600 million dyslexic or learning disabled people, that's no small challenge. But we're determined to make a difference and have been working hard on that goal since Dolphin started in 1986.” We have a particular challenge.
Graphics - photographs of various Dolphin customers
As Dave Williams (Dolphin Blindness Product Owner and Q&D Tester) says “In an ideal world no-one would need our products – no-one chooses to be blind.” The Dolphin Challenge is that even if the products are perfect no-one chooses to be print-impaired... so we must work hard to make our products right for our customers.
Graphic: photo of Dave Williams
Agenda diagram – radial circle, “Quality in Use” in the centre; items around the radial circle are:
Our journey
What?
Why?
How?
Graphic – illustrative/decorative
Our Journey
Dolphin recognised an opportunity for improvement with the help of Julian Harty (Google and then E-Bay) and as a result decided to take on a quality manager with the idea that the immediate need was to improve testing. Improvements were needed because there were gaps in the test capability:
No (few) specialist testers
Poor reporting on testing
Poor application of test techniques
No test automation.
Graphics: illustrative/decorative
Solution 1: more testing, better testing
We took these first steps:
Improving test skills and motivation – identified testers for the different products, and started providing in-house workshops on test approaches and test design;
We had a shortage of resources as well as a shortage of skills and so “Night-siding” to a test services company in New Zealand has given us extra capacity, fast turnaround, 24 hour work, and skilled testers providing role models for less experienced in-house testers;
As a result of these the testers were able to build better regression test packs, with overview test conditions based on personas and user stories;
We fed out information on test techniques to testers and developers across the company in weekly short bursts – a 2-4 side document “Technique of the Week” with lunchtime briefings in the following week to discuss and consolidate;
We have started to automate some testing:
CTE and Python has been very useful for unit tests on one product range
For the access software is harder to do because of the way the software intercepts what is happening as in the operating system and any applications that are running so we have started to build automation in-house – we have three tools: Rhubarb, Beetroot and Sledgehammer in progress.
Graphics: photos of Colin Shales (Q&D usability tester) and James Bowden (Developer and test automation)
Solution 2 - Improve test reportingTest coverage pie charts
The graphic illustrates how we used pie charts to show test progress – the one chosen is from one product midway through testing in August – and shows about a third of the planned tests had been run at that stage, the majority having passed. The product has now passed all planned tests and is being released.
Solution 3: improve metrics
The graphic show two graphs comparing the number of issues raised in testing with the number raised after release of a product plus a target improvement.
Product A release X: three quarters of the issues were raised post release, before any improvements to process or technique for development or test.
We then improved the testing. Product A release Y: some improvements to testing – now two thirds of the issues were found in testing and only a third post release.
Our aim for December is shown in pie chart 3 – by release Z in December, we want the DDP to be at 85%, and this will be achieved by development improvements as well as test improvements. This include white box testing and code review, design and architecture reviews and more automation.
Use of these pie charts had two benefits – the testers could see that they had improved and management could be persuaded what actions were necessary to take next.
Side effects of the cure
Testing found so many defects and such important defects that we had to: Halt a project and restart; morale dropped especially among the developers; the developer workload increased. The side effects of the cure (improving testing) included meeting the change management problems of resistance and fear, trying to implement solutions beyond the maturity of the organisation, causing setbacks, and loss of balance between pressures. Automation was harder for the access products than I had anticipated. The lesson and key point: TEST IMPROVEMENT IS NOT ENOUGH!
Other problems:
Accessibility – in order to keep all team members up to date we had to ensure that all the artefacts around the project were accessible. We had to balance that with using non-accessible artefacts where that did not disadvantage colleagues with visual impairments and benefited sighted colleagues. E.g. flowcharts can be used and then expressed as text. This document is intended to take the place of the slide set.
Level of maturity – it is important to make things as simple as possible, in order that everyone can buy in, and then increase the complexity gradually.
Change gradually over time – it is easy to get impatient – change takes time.
WAI compliance – our tools have to deal a wide range of software applications and websites – for example, every time a website is non-compliant to WAI guidelines and our software cannot read it, it is us that the customer rings...
As well as test improvement support the company vision and purpose:
Improving testing had made some things worse and had exposed problems that could not be resolved by testing. Instead of just improving testing we needed to Support the vision and purpose: Deliver appropriate solutions; be loved by customers; and Deliver commercial success.
Message to testers: just improving testing is not enough to help the business - Our skill set and comfort zone is often in the detail; we find excitingly obscure bugs and report these problems in detail. Our zeal sometimes causes us to miss the big picture of what the customer and the business really needs. We focus on software quality at the expense of human and commercial factors. We need to consider company vision and customers, we need to consider commercials.
Graphic: decorative
Solution 4: use ISO25000 - Quality in Use
Graphic shoes there are three Quality in Use attributes:
Usability
Flexibility
Safety
Usability has the sub-attributes Effectiveness; Efficiency; Satisfaction
Flexibility has the sub-attributes Context conformity; Context extendibility; Accessibility
Safety has the sub-attributes Commercial damage; Operator health and safety; Public health and safety; Environmental harm
ISO125000 is a standard that describes software attributes, how to specify them and how to measure them: Software Quality and Requirements Evaluation
Quality in Use (QiU) and Dolphin
The graphic shows the Quality in Use attributes of Usability, Flexibility and Safety, which reflect a user and value view of quality –“my tasks” and “my money”.
Buyers and Dealers are looking for commercial safety and flexibility
People with VI & TI are looking for usability and flexibility
Dolphin sales and marketing are looking for all 3 attributes, but particularly flexibility – new things to sell.
People are different...
The photographs show two Dolphin customers, both have visual impairments but their requirements from an access product are quite different. The gentleman on the left is using Guide. This give magnification voice output and – if wished – voice input. It also provides a simplified interface which replaces Windows and provides a simple internet browser, email, simple word processing. It is suitable for people who are less used to computers and/or have recently lost vision. The lady on the right is tech-savvy and uses applications such as Office, as well as all the work applications that her colleagues use. She is using SuperNova with multiple monitors to magnify the display. SuperNova is suitable for people who are tech-savvy and want to use all the applications you use at work or at home. These people have different personas and contexts of use. They have different quality in use scores. Even if they share user stories (I want to do on-line banking) these need to be implemented in different ways.
QiU scores for two Dolphin product users
Quality in use – products compared: The graphic is a radar graph, showing each of the Quality in Use sub-attributes, and the scores for two Dolphin products – SuperNova and Guide. Each has users who have visual impairments but the two groups have quite different quality in use attribute scores. SuperNova users are typically working or at college. They are typically tech-savvy. They value being able to work efficiently and flexibly, using SuperNova to access a wide range of applications. They are conscious of whether SuperNova saves or costs them money. The typical Guide user has recent sight loss, and low IT experience. Typically a Guide users value the interface being stable and predictable (conformity) and knowing they are able to be effective (they are protected from making mistakes). The attributes are ordered:
SuperNova typical user: Efficiency, Extendibility, Effectiveness, Commercial, Satisfaction, Conformity, Health & Safety, Environment.
Guide typical user: Effectiveness, Conformity, Satisfaction, Efficiency, Extendibility, Health & Safety, Commercial, Environment.
Solution 5 – don’t forget the engineering details
We also need the engineering details – that is Internal Quality – How was it built? Is it faulty? Were good techniques used? Have all the “ilities” been considered?
The internal qualities are functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability
The customers who are systems administrators care about these. Dolphin Support and Development care about these. Technical testers care about these. Others only notice them when they go wrong...
Internal quality (IQ) – engineering detail
Internal Quality –
How was it built? Is it faulty?
Were good techniques used?
The graphic shows the internal qualities:
Functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability
The customers who are systems administrators care about these.
Support and development care about these.
Others only notices them when they go wrong...
QiU and IQ – Dolphin Project Team
The project teams need to have representatives for quality in use and internal quality.
Dolphin Sales and Marketing are interested in quality in use
Dolphin Support and Development are interested in internal quality
Both groups include colleagues who have visual impairments.
We make sure we have considered:
Usability – Effectiveness; Efficiency; Satisfaction
Flexibility - Context conformity; Context extendibility; Accessibility
Safety - Commercial damage; Operator health and safety; Public health and safety; Environmental harm
Internal quality is made up of:
Functional suitability, operability, portability, compatibility, maintainability, security, performance, reliability
And we make sure that we prioritise these based on the Quality in Use priorities.
Testers for QiU and IQ
Where do the testers sit? We need test skills in quality in use and internal quality. We have a group called Q&D (Quality and Design) which includes the quality manager, some specialist testers and project management. Testing is carried out by:
Static testing (reviews) by people across the project team
Dynamic testing by developers, Q&D, New Zealand nightsiders, representative in-house product users, sales and marketing, support, dealers, customer beta testers.
At the start of projects, Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development.
Development does research, design, development, code review and unit test.
Q&D (specialist) Testers carry out a Product test, helped by representative in-house product users, and by the New Zealand nightsiding team. When sufficiently stable it is passed for beta testing to representative dealers and customers. When the product test passes acceptance criteria, it is passed to Sales, Marketing, and Support to carry out market readiness testing.
Testers for QiU and IQ
The graphic shows project stages above and below the quality in use line
Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development.
Development does research, design, development, unit test
(specialist) Testers do Product test
Sales, Marketing, Support carry out market readiness testing
Externals carry out Beta testing
Solution 6 – customer-centric SDLC
This graphic shows the steps through our project model. The Market Requirements phase is run by the product owner – typically from marketing - and project mentor – management team representative to identify the project vision for the customer, context of use, personas, user stories, the commercial aspects, benefits for the customer and for Dolphin, acceptance criteria. This is outlined in a Market Requirements Document. It could lead to a research project, prototyping or a delivery project, depending on our knowledge. Typically that would be in iterations. We run a brief post implementation review at the end of each iteration, or group of iterations as well as at project ends.
Solution 7 - project workshops include Sales and Marketing
It is vital that Sales and Marketing are including in all the decisions. They take part in the workshops and other activities to build user stories. Marketing, Sales, Support:
Focus on emotional
Focus on practical
Focus on tasks
Focus on Vision for Customer and Dolphin
User stories
Acceptance criteria.
Development and technical testers focus on measureable acceptance criteria and architecture. Q&D facilitate and encourage.
The emotional level in the user stories provided by Sales, Marketing, and Support and in particular by product users in the company helps us focus on what is important.
Example:
As a person who is blind
I wanted to manage my own bank account online
Because it is so humiliating when someone has to read you your bank statement
As an elderly person with declining vision
I want to contact my grandchildren online
Because I miss them and want to share news easily
Solution 8 – Sensible Iterations to focus the detail
The graphic shows a series of overlapping iterations, each of 6 weeks plus week 7 for release. During the 6 weeks research, development, test for this and subsequent iterations are carried out. We found that we have work which needs to carry across several iterations. In the last 2 weeks market readiness testing is done by Sales, Marketing and Support, and they agree the contents for the next iteration.
Market readiness email
In the last 2 weeks of the iteration we move to Sales, Marketing and Support carrying out market readiness testing and choosing items for the next iteration. A Market readiness email is sent from Q&D to everyone in Sales, Marketing, Support, and to Training, Documentation, and in-house product users. The email says:
What is the verdict on this build?
Action: verdict by (date – usually a week is allowed):
Yes – there may be issues but they can be lived with for this iteration
No – there are showstoppers
Don't know – I have not finished my MR tests – estimated date to complete
No opinion – I have not had time to test
Proposed deadline for (iteration): Date for all market readiness feedback and planned date for release.
Iteration content email
A rough iteration plan for several iterations ahead, plus a wish list, is kept running. Items for iterations are planned in and discussed with all the stakeholders, as required and confirmed in a weekly meeting as we monitor support calls. Just after the MR email, an iteration content email is sent out. This is a series of paragraphs headed:
What is going to be in (next iteration)?
List of items
What about the issue with (....)?
Items requested that cannot be covered in the iteration and why, plus a planned iteration
How did we come up with this content list?
What is the proposed schedule?
What is the contingency plan?
What is your verdict on this plan?
Response by (date).
What have we learnt?
Marketing, Sales and Support involvement is vital, but they have be shown the value and that progress has been made because of their contributions
Improving project & quality management is very important and needs to done in small doses
Improving test skills and motivation in very important and needs to be managed against other improvements in order that problems are uncovered and resolved at source
Iterations forced out process weaknesses – the pain of moving to an iterative approach was large, but the problems were process, skill, control, tool or techniques problems that needed to be sorted out – the sort time spans exposed these more clearly
Night-siding to New Zealand was really useful to provide a breathing space and also caused people in-house to want to improve what they were delivering it exposed time testing preventable issues as real costs
Technique of the Week & lunchtime briefings helped people pick and choose topics to suit their immediate needs and meant iterative / incremental approach to improvement
It’s good when other people have “your” ideas – it means they have adopted them
Automation is hard sometimes – but worth persisting with
Accessibility and white-boarding... don’t be dogmatic about the right way to do things – use the methods that suit the team
Lots more to do...we have made huge improvements in the last year, but there is a lot more to do!
What has gone well?
Sales and marketing involvement has worked well in providing context of use, personas, user stories, setting acceptance criteria, and market readiness testing.
Increasing awareness in the organisation of the importance of strong project managers to control budget, timescale, risk and cost benefit analysis.
Increasing confidence and awareness in the testers (from all areas) of the importance of strong quality management and testing. They hold the line, with courage, information, appreciation of risk, they offer choices based on cost benefits.
Realisation among the testers and elsewhere that test skills and motivation is not enough – we also need to understand the commercial, sales, marketing and customer impact. Quality in use as the focus.
Night-siding works and helped internal testers improve.
Iterative skills improvement with Technique of the Week, workshops, lunchtime briefings – push out the improvements a few at a time.
We tried to do too much too quickly at first – pick off the targets, and persist!
The journey continues
We need to remember everyday our purpose as an organisation, our customers and what they need from use. Quality in use (usability, flexibility and safety) keeps us focused on what is important. Remember “In an ideal world no-one would need our products – no-one chooses to be blind”. Quality in use is for us the beating heart of the customer experience.
Graphics - photographs of various Dolphin customers and staff
Quality in use: the beating heart of the customer experience
Usability, Flexibility, Safety
The journey continues – thanks and goodbye
Graphics: Dolphin logo; illustration of heart and heartbeat; text boxes with usability, flexibility and safety.