How I Test (Failure) - Ben Simo

Now that I have your attention, prepare to read fast
v1.2
FAILURE.QualityFrog.com
Youre testing
And an error message appears
v1.2
v1.2
Did you expect it?

Or are you surprised?
v1.2
How does it make you feel?
v1.2
What do you do?
v1.2
Not ready reading drive A Abort, Retry, Ignore, Fail?
v1.2
Continue to the next test step
v1.2
v1.2
Report a bug
v1.2
v1.2
Resolve unknown conflict
v1.2
v1.2
Find a file
v1.2
v1.2
Eliminate something
v1.2
v1.2
Request permission
v1.2
v1.2
Try again
v1.2
v1.2
Try something new
v1.2
v1.2
Seek help
v1.2
v1.2
Set a reference to an instance
v1.2
v1.2
C lick OK
v1.2
lick OK
v1.2
v1.2
Save
v1.2
v1.2
Enter a multiple of |ECL0
v1.2
v1.2
Exit
v1.2
v1.2
Reconnect
v1.2
v1.2
Declare success
v1.2
v1.2
Assign blame
v1.2
v1.2
Laugh
v1.2
v1.2
Restart
v1.2
v1.2
Guess
v1.2
v1.2
Gasp
v1.2
v1.2
Request a raise
v1.2
v1.2
Swear
v1.2
v1.2
So
v1.2
What do you do when you encounter an error?
v1.2
While you think about that, let me tell you what I do.
v1.2
Oh, who am I?
v1.2
I am Ben Simo
Ben@QualityFrog.com
v1.2
I use software
Ben@QualityFrog.com
v1.2
I test software
Ben@QualityFrog.com
v1.2
I question software
Ben@QualityFrog.com www.QuestioningSoftware.com
v1.2
I code software
v1.2
I code software that helps test software

v1.2
I talk about software

v1.2
I write about software

v1.2
I think about software

v1.2
I like software
v1.2
I like software that serves people
v1.2
I like software that helps people
v1.2
I dislike bad software
v1.2
I dislike bad software that frustrates people
v1.2
I dislike bad software that confuses people
v1.2
I dislike bad software that angers people
v1.2
I dislike bad software that discombobulates people
v1.2
I sometimes like big words that discombobulate people
v1.2
But only for a moment
v1.2
Discombobulate means to emotionally confuse
v1.2
Discombobulate means to frustrate
v1.2
Discombobulate means to bewilder
v1.2
Discombobulate means to confound
v1.2
Discombobulate means to befuddle
v1.2
Discombobulate means to perplex
v1.2
Too many error messages discombobulate people
v1.2
Error messages should help people, not confuse them
v1.2
You see, Its not about the software
v1.2
Its about the people!
v1.2
Testers often fail to test the error messages
v1.2
They fail to think about how they affect people
v1.2
So what do I do when I encounter error messages?
v1.2
I test them
v1.2
I test them using the FAILURE mnemonic
v1.2
A mnemonic is a device that aids memory recall
v1.2
The FAILURE mnemonic is an acronym that helps me recall guideword heuristics
v1.2
Guideword heuristics are words that guide thinking through a problem
v1.2
The FAILURE mnemonics guideword heuristics help me think about testing error messages
v1.2
The F reminds me to think about Functional aspects of the error FAILURE
v1.2
I ask Functional questions like

Does the error detection function as expected?
v1.2

Does the error handling function as expected?
v1.2

Does the error reporting function as expected?
v1.2

Does the error dialog, window, or page function as expected?
v1.2

Does the software recover from the error?
v1.2

Was there really an error?
v1.2

Are other errors like this reported?
v1.2

Do dialog buttons work?
v1.2

Is error message information accurate?
v1.2

Is it functional?
v1.2
Only I dont stop at Functional
v1.2
A Functional error message is not necessarily an Appropriate error message
v1.2
The A reminds me to ask if the error reporting is Appropriate

FAILURE
v1.2
I ask Appropriate questions like

Is the error reporting timely?
v1.2

Is the error reporting accurate?
v1.2

Are UI messages free of inappropriate technical detail?
v1.2

Is there a better way to handle or report the error?
v1.2
I ask Appropriate questions like Is the error user friendly?
v1.2

Are errors detected as early in the user workflow as possible?
v1.2

Are system resource problems detected while there is still time to remedy the problem?
v1.2

Is the situation communicated appropriately for the intended audience?
v1.2

Is a user allowed to waste time and effort only to be told that their work cannot be applied?
v1.2
The I reminds me to ask about the Impact of the error

FAILURE
v1.2
I ask Impact questions like

Is the impact communicated?
v1.2

Is the impact communicated to the user?
v1.2

Is the impact communicated to others?
v1.2

Will the user understand the impact?
v1.2

Is any impact to the users data communicated?
v1.2

Does the message contain too much information?
v1.2

Was a transaction only partially completed?
v1.2
The L reminds me to ask about the error Log

FAILURE
v1.2
I ask Log questions like

Should the error be logged for support?
v1.2

Should the error be logged for system administrators?
v1.2

Should the error be logged for developers?
v1.2

Should the error be logged for technical investigation?
v1.2

Will log information be available if the user waits to report a problem?
v1.2

Are log messages standardized to support automated information mining?
v1.2

Can I grep this error?
v1.2

Do logs contain enough information?
v1.2

Do logs contain useless information?
v1.2

Is this an error that should not be logged?
v1.2

Is there too much logging?
v1.2

Does excessive logging impact performance?
v1.2

Does excessive logging impact disk space?
v1.2

Does the logging help or complicate error investigation?
v1.2
The U reminds me to think about the User Interface

FAILURE
v1.2
I ask UI questions like

Is the error reported to the user?
v1.2

Does this error need to be reported to the user?
v1.2

What user activity caused the failure?
v1.2

What user activity caused the failure? Is this communicated to the user?
v1.2

Who is the user?
v1.2

Are UI messages worded for the intended audience?
v1.2

Is this message consistent with the desired look and feel?
v1.2

Is the UI reporting consistent with reporting for other errors?
v1.2

Is the UI reporting efficient?
v1.2

Does the user need to click away excessive dialogs?
v1.2

Is this the best way to communicate the error?
v1.2

Could this error be handled without involving the user?
v1.2
The R helps me think about error Recovery

FAILURE
v1.2
I ask Recovery questions like

Is the user told how to recover from the impact of the error?
v1.2

Does the user need to contact support?
v1.2

Is support contact information provided?
v1.2

Does support know how to fix the problem?
v1.2

Does the software guide the user through the recovery process?
v1.2

Or is the user left to figure it out on their own?
v1.2
The E leads me to think about the Emotions of users

FAILURE
v1.2
I ask Emotions questions like

What emotions are likely to be raised by the error message?
v1.2

Does the error reporting evoke the desired user emotions?
v1.2

Does this message enhance user frustration?
v1.2

Does this message quiet user frustration?
v1.2

Does this message encourage users to use some other product?
v1.2

Does this message cause confusion?
v1.2
Asking these types of questions can help turn FAILURE into success
v1.2
The next time you encounter an error message
v1.2
The next time you encounter an error message Test it!
v1.2
The next time you encounter an error message Question it!
v1.2
Functional Appropriate Impact Log UI Recovery Emotions

v1.2
Try the FAILURE mnemonic

v1.2
And let me know how it works for you


v1.2
Ben Simo Ben@QualityFrog.com

http://FAILURE.QualityFrog.com
Copyright 2007 Quality Frog

How I Test (Failure) - Ben Simo

Uploaded by

Copyright:

Available Formats

How I Test (Failure) - Ben Simo

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How I Test (Failure) - Ben Simo

Uploaded by

Copyright:

Available Formats

Now that I have your attention, prepare to read fast

Did you expect it?

How does it make you feel?

What do you do?

Not ready reading drive A Abort, Retry, Ignore, Fail?

Continue to the next test step

Resolve unknown conflict

Try something new

Set a reference to an instance

Enter a multiple of |ECL0

What do you do when you encounter an error?

I code software that helps test software

I talk about software

I write about software

I think about software

I like software that serves people

I like software that helps people

I dislike bad software

I dislike bad software that frustrates people

I dislike bad software that confuses people

I dislike bad software that angers people

I dislike bad software that discombobulates people

I sometimes like big words that discombobulate people

But only for a moment

Discombobulate means to emotionally confuse

Discombobulate means to frustrate

Discombobulate means to bewilder

Discombobulate means to confound

Discombobulate means to befuddle

Discombobulate means to perplex

Too many error messages discombobulate people

Error messages should help people, not confuse them

You see, Its not about the software

Its about the people!

Testers often fail to test the error messages

They fail to think about how they affect people

So what do I do when I encounter error messages?

I test them using the FAILURE mnemonic

A mnemonic is a device that aids memory recall

The FAILURE mnemonic is an acronym that helps me recall guideword heuristics

Guideword heuristics are words that guide thinking through a problem

The F reminds me to think about Functional aspects of the error FAILURE

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

I ask Functional questions like

Only I dont stop at Functional

A Functional error message is not necessarily an Appropriate error message

The A reminds me to ask if the error reporting is Appropriate

I ask Appropriate questions like

I ask Appropriate questions like

I ask Appropriate questions like

I ask Appropriate questions like