All-pairs testing

Possible solutions:

Test all combinations [once], but delay the project so it misses its market window so that everyone
quits from stress, or the company goes out of business.
Choose one or two tests and hope for the best.

Choose the tests that you have already run, perhaps as part of programmer-led testing. Incorporate
them into a formal test plan and run them again.

Choose the tests that are easy to create and run. Ignore whether they provide useful information
about the quality of the product.

Make a list of all the combinations and choose the first few.

Make a list of all the combinations and choose a random subset.

By magic, choose a specially selected, fairly small subset that finds a great many defectsmore
than you would expect from such a subset

The answer is not to attempt to test all the combinations for all the values for all the variables but to test
all pairs of variables. This significantly reduces the number of tests that must be created and run.

Why this works? There are some documented studies:

In a case study published by Brownlie of AT&T regarding the testing of a local-area network-
based electronic mail system, pairwise testing detected 28 percent more defects than their original
plan of developing and executing 1,500 test cases (later reduced to 1,000 because of time
constraints) and took 50 percent less effort.
A study by the National Institute of Standards and Technology published by Wallace and Kuhn on
software defects in recalled medical devices reviewed fifteen years of defect data. They concluded
that 98 percent of the reported software flaws could have been detected by testing all pairs of
parameter settings.

Kuhn and Reilly analyzed defects recorded in the Mozilla Web browser database. They
determined that pairwise testing would have detected 76 percent of the reported errors.

Table generation techniques

1. Orthogonal arrays

An orthogonal array is a two-dimensional array of numbers that has this interesting

propertychoose any two columns in the array. All the pairwise combinations of its
values will occur in every column pair. Not only will all the pair combinations occur in
the array, but if any pair occurs multiple times, all pairs will occur that same number of
times.

Notation:

1
Tools: The rdExpert tool from Phadke Associates implements the orthogonal array
approach. See http://www.phadkeassociates.com

Reference: Neil J.A. Sloane maintains a very comprehensive catalog of orthogonal arrays
at http://www.research.att.com/~njas/oadir/index.html

Books: Quality Engineering Using Robust Design by Madhav S. Phadke

Using Orthogonal Arrays

The process of using orthogonal arrays to select pairwise subsets for testing is:

1. Identify the variables.

The variables are Browser, Plug-in, Client operating system, Server, and Server operating
system.

2. Determine the number of choices for each variable.

Browser - Internet Explorer 5.0, 5.5, and 6.0, Netscape 6.0, 6.1, and 7.0, Mozilla 1.1, and
Opera 7 (8 choices).
Plug-in - None, RealPlayer, and MediaPlayer (3 choices).
Client operating system - Windows 95, 98, ME, NT, 2000, and XP (6 choices).
Server - IIS, Apache, and WebLogic (3 choices).
Server operating system - Windows NT, 2000, and Linux (3 choices).
Multiplying 8 x 3 x 6 x 3 x 3 we find there are 1,296 combinations. For "complete" test
coverage, each of these combinations should be tested.

3. Locate an orthogonal array that has a column for each variable and values within the
columns that correspond to the choices of each variable.

What size array is needed? First, it must have five columns, one for each variable in this
example. The first column must support eight different levels (1 through 8). The second
column must support three levels (1 through 3). The third requires six levels. The fourth and
the fifth each require three levels. The perfect size orthogonal array would be 816133 (one
column of 1 through 8, one column of 1 through 6, and three columns of 1 through 3).
Unfortunately, one of this exact size does not exist. When this occurs, we simply pick the next
larger array.

The L64(8243) orthogonal array meets our requirements. The requirement of 8161 (one column
of 1 through 8 and 1 column of 1 through 6) is met by 82 (two columns of 1 through 8). The
requirement of 33 (three columns of 1 through 3) is met by 43 (three columns of 1 through 4).

2
The number of combinations of all the values of all the variables is 1,296 and thus 1,296 test cases
should be created and run for complete coverage. Using this orthogonal array, all pairs of all the
values of all the variables can be covered in only 64 tests, a 95% reduction in the number of test
cases.

L64(8243) Orthogonal Array L64(8243) Orthogonal Array L64(8243) Orthogonal Array

?/th> ?/th> ?/th>
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

1 1 1 1 1 1 21 2 1 5 4 3 41 7 3 1 2 4

2 1 4 3 4 4 22 2 4 7 1 2 42 7 2 3 3 1

3 1 4 2 4 4 23 2 4 6 1 2 43 7 2 2 3 1

4 1 1 4 1 1 24 2 1 8 4 3 44 7 3 4 2 4

5 1 3 5 3 3 25 4 2 1 4 3 45 7 1 5 4 2

6 1 2 7 2 2 26 4 3 3 1 2 46 7 4 7 1 3

7 1 2 6 2 2 27 4 3 2 1 2 47 7 4 6 1 3

8 1 3 8 3 3 28 4 2 4 4 3 48 7 1 8 4 2

9 3 4 1 3 3 29 4 4 5 2 1 49 6 4 1 3 2

10 3 1 3 2 2 30 4 1 7 3 4 50 6 1 3 2 3

11 3 1 2 2 2 31 4 1 6 3 4 51 6 1 2 2 3

12 3 4 4 3 3 32 4 4 8 2 1 52 6 4 4 3 2

13 3 2 5 1 1 33 5 2 1 4 2 53 6 2 5 1 4

14 3 3 7 4 4 34 5 3 3 1 3 54 6 3 7 4 1

15 3 3 6 4 4 35 5 3 2 1 3 55 6 3 6 4 1

16 3 2 8 1 1 36 5 2 4 4 2 56 6 2 8 1 4

17 2 3 1 2 1 37 5 4 5 2 4 57 8 1 1 1 4

18 2 2 3 3 4 38 5 1 7 3 1 58 8 4 3 4 1

19 2 2 2 3 4 39 5 1 6 3 1 59 8 4 2 4 1

20 2 3 4 2 1 40 5 4 8 2 4 60 8 1 4 1 4

3
 L64(8243) Orthogonal Array L64(8243) Orthogonal Array L64(8243) Orthogonal Array
?/th> ?/th> ?/th>
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

61 8 3 5 3 2 62 8 2 7 2 3 64 8 3 8 3 2

63 8 2 6 2 3

4. Map the test problem onto the orthogonal array.

For example, the Browser choices will be mapped onto column 1. Cells containing a 1 will
represent IE 5.0; cells with a 2 will represent IE5.5; cells with a 3 will represent IE 6.0; etc.

o 1 IE 5.0
o 2 IE 5.5

o 3 IE 6.0

o 4 Netscape 6.0

o 5 Netscape 6.1

o 6 Netscape 7.0

o 7 Mozilla 1.1

o 8 Opera 7

If the perfect size array does not exist, choose one that is slightly bigger and apply these two rules
to deal with the "excess."

The first rule deals with extra columns. If the orthogonal array chosen has more columns
than needed for a particular test scenario, simply delete them. The array will remain
orthogonal.
The second rule deals with extra values for a variable. In the current example, column 3
runs from 1 to 8 but only 1 through 6 is needed. It is tempting to delete the rows that
contain these cells but DON'T. The "orthogonalness" may be lost. Each row in the array
exists to provide at least one pair combination that appears nowhere else in the array. If
you delete a row, you lose that test case. Instead of deleting them, simply convert the extra
cells to valid values. Some automated tools randomly choose from the set of valid values
for each cell while others choose one valid value and use it in every cell within a column.
Either approach is acceptable. Using this second approach, we'll complete the orthogonal
array. Note that it may be difficult to maintain the "balanced" aspect of the array when
assigning values to these extra cells.

4
 L64 (8243) with a full mapping of all its columns including the L64 (8243) with a full mapping of all its columns including the
"extra" cells. "extra" cells.
?/th> ?/th>
Browser Plug-in Client Server Server Browser Plug-in Client Server Server
OS OS OS OS

1 IE 5.0 None Win 95 IIS Win NT 2000

2 IE 5.0 None Win ME IIS Win NT 22 IE 5.5 None Win 95 IIS Win
2000

3 IE 5.0 None Win 98 IIS Win NT

23 IE 5.5 None Win XP IIS Win
2000
4 IE 5.0 None Win NT IIS Win NT

24 IE 5.5 None Win 98 IIS Linux

5 IE 5.0 MediaPlayer Win WebLogic Linux
2000
25 Net 6.0 RealPlayer Win 95 IIS Linux

6 IE 5.0 RealPlayer Win 95 Apache Win

2000 26 Net 6.0 MediaPlayer Win ME IIS Win
2000

7 IE 5.0 RealPlayer Win XP Apache Win

2000 27 Net 6.0 MediaPlayer Win 98 IIS Win
2000

8 IE 5.0 MediaPlayer Win 98 WebLogic Linux

28 Net 6.0 RealPlayer Win NT IIS Linux

9 IE 6.0 None Win 95 WebLogic Linux

29 Net 6.0 None Win Apache Win NT
2000
10 IE 6.0 None Win ME Apache Win
2000
30 Net 6.0 None Win 95 WebLogic Win NT

11 IE 6.0 None Win 98 Apache Win

2000 31 Net 6.0 None Win XP WebLogic Win NT

12 IE 6.0 None Win NT WebLogic Linux 32 Net 6.0 None Win 98 Apache Win NT

13 IE 6.0 RealPlayer Win IIS Win NT 33 Net 6.1 RealPlayer Win 95 IIS Win
2000 2000

14 IE 6.0 MediaPlayer Win 95 IIS Win NT 34 Net 6.1 MediaPlayer Win ME IIS Linux

15 IE 6.0 MediaPlayer Win XP IIS Win NT 35 Net 6.1 MediaPlayer Win 98 IIS Linux

16 IE 6.0 RealPlayer Win 98 IIS Win NT 36 Net 6.1 RealPlayer Win NT IIS Win
2000

17 IE 5.5 MediaPlayer Win 95 Apache Win NT

37 Net 6.1 None Win Apache Win NT
2000
18 IE 5.5 RealPlayer Win ME WebLogic Win NT

38 Net 6.1 None Win 95 WebLogic Win NT

19 IE 5.5 RealPlayer Win 98 WebLogic Win NT

39 Net 6.1 None Win XP WebLogic Win NT

20 IE 5.5 MediaPlayer Win NT Apache Win NT

40 Net 6.1 None Win 98 Apache Win NT

21 IE 5.5 None Win IIS Linux

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
 L64 (8243) with a full mapping of all its columns including the L64 (8243) with a full mapping of all its columns including the
"extra" cells. "extra" cells.
?/th> ?/th>
Browser Plug-in Client Server Server Browser Plug-in Client Server Server
OS OS OS OS

41 Moz 1.1 MediaPlayer Win 95 Apache Win NT 53 Net 7.0 RealPlayer Win IIS Win NT
2000

42 Moz 1.1 RealPlayer Win ME WebLogic Win NT

54 Net 7.0 MediaPlayer Win 95 IIS Win NT

43 Moz 1.1 RealPlayer Win 98 WebLogic Win NT

55 Net 7.0 MediaPlayer Win XP IIS Win NT

44 Moz 1.1 MediaPlayer Win NT Apache Win NT

56 Net 7.0 RealPlayer Win 98 IIS Win NT

45 Moz 1.1 None Win IIS Win

2000 2000 57 Opera 7 None Win 95 IIS Win NT

46 Moz 1.1 None Win 95 IIS Linux 58 Opera 7 None Win ME IIS Win NT

47 Moz 1.1 None Win XP IIS Linux 59 Opera 7 None Win 98 IIS Win NT

48 Moz 1.1 None Win 98 IIS Win 60 Opera 7 None Win NT IIS Win NT
2000

61 Opera 7 MediaPlayer Win WebLogic Win

49 Net 7.0 None Win 95 WebLogic Win 2000 2000
2000

62 Opera 7 RealPlayer Win 95 Apache Linux

50 Net 7.0 None Win ME Apache Linux

63 Opera 7 RealPlayer Win XP Apache Linux

51 Net 7.0 None Win 98 Apache Linux

64 Opera 7 MediaPlayer Win 98 WebLogic Win

52 Net 7.0 None Win NT WebLogic Win 2000
2000

2. Allpairs algorithm

A second way is to use an algorithm that generates the pairs directly without resorting to an "external"
device like an orthogonal array.
James Bach provides a tool to generate all pairs combinations at http://www.satisfice.com.
Click on Test Methodology and look for Allpairs.
Ward Cunningham provides further discussion and the source code for a Java program to
generate all pairs combinations at http://fit.c2.com/wiki.cgi?AllPairs.

When a particular value in the test case doesn't matter, because all of its pairings have already
been selected, it is marked with a ~.

Output from the Allpairs program. Output from the Allpairs program.
?/th> ?/th>
Browser Client OS Plug-in Server Server OS Browser Client OS Plug-in Server Server OS

1 IE 5.0 Win 95 None IIS Win NT 2 IE 5.0 Win 98 Real Player Apache Win 2000

35
 Output from the Allpairs program.
?/th>
Browser Client OS Plug-in Server Server OS

3 IE 5.0 Win ME Media Player WebLogic Linux

4 IE 5.5 Win 95 Real Player WebLogic Win NT

5 IE 5.5 Win 98 None IIS Linux

6 IE 5.5 Win ME None Apache Win 2000

7 IE 6.0 Win 95 Media Player Apache Linux

8 IE 6.0 Win 98 Real Player IIS Win NT

9 IE 6.0 Win ME None WebLogic Win 2000

10 Netscape 6.0 Win ME Real Player IIS Linux

11 Netscape 6.0 Win NT Media Player IIS Win 2000

12 Netscape 6.0 Win 2000 None Apache Win NT

13 Netscape 6.1 Win NT None WebLogic Linux

14 Netscape 6.1 Win 2000 Media Player IIS Win 2000

15 Netscape 6.1 Win XP Real Player Apache Win NT

16 Netscape 7.0 Win NT Real Player Apache Win NT

17 Netscape 7.0 Win 2000 Media Player WebLogic Linux

18 Netscape 7.0 Win XP Media Player IIS Win 2000

19 Mozilla 1.1 Win XP Media Player WebLogic Win NT

20 Mozilla 1.1 Win 98 Media Player Apache Linux

21 Mozilla1.1 Win 95 Real Player IIS Win 2000

22 Opera 7 Win XP None WebLogic Linux

23 Opera 7 Win 98 Real Player WebLogic Win 2000

24 Opera 7 Win ME Media Player Apache Win NT

25 IE 5.5 Win 2000 Real Player ~WebLogic ~Linux

26 IE 5.5 Win NT Media Player ~IIS ~Win NT

27 Netscape 6.0 Win 95 ~None WebLogic ~Win 2000

28 Netscape 7.0 Win 95 None ~Apache ~Linux

36
 Output from the Allpairs program.
?/th>
Browser Client OS Plug-in Server Server OS

29 Mozilla 1.1 Win ME None ~IIS ~Win NT

30 Opera 7 Win NT ~Real Player IIS ~Linux

31 IE 5.0 Win NT ~None ~Apache ~Win 2000

32 IE 5.0 Win 2000 ~Real Player ~IIS ~Win NT

33 IE 5.0 Win XP ~None ~WebLogic ~Linux

34 IE 5.5 Win XP ~Real Player ~Apache ~Win 2000

35 IE 6.0 Win 2000 ~None ~Apache ~Win 2000

36 IE 6.0 Win NT ~Real Player ~WebLogic ~Win NT

37 IE 6,0 Win XP ~Media Player ~IIS ~Linux

38 Netscape 6.0 Win 98 ~Media Player ~WebLogic ~Win NT

39 Netscape 6.0 Win XP ~Real Player ~Apache ~Linux

40 Netscape 6.1 Win 95 ~Media Player ~Apache ~Win 2000

41 Netscape 6.1 Win 98 ~None ~IIS ~Win NT

42 Netscape 6.1 Win ME ~Real Player ~WebLogic ~Linux

43 Netscape 7.0 Win 98 ~None ~WebLogic ~Win 2000

44 Netscape 7.0 Win ME ~Real Player -US ~Win NT

45 Mozilla 1.1 Win NT ~None ~Apache ~Linux

46 Mozilla 1.1 Win 2000 ~Real Player ~WebLogic ~Win 2000

47 Opera 7 Win 95 ~Media Player ~IIS ~Win NT

48 Opera 7 Win 2000 ~None ~Apache ~Win 2000

37