Black box testing

Outline
￭ Testing Types
￭ Testing Techniques
￭ White-box Testing
￭ Black-box Testing
 Black-box Testing 1

• An approach to testing where the program is

considered as a ‘black-box’
• The program test cases are based on the
system specification / functional requirements.
• is testing that ignores the internal mechanism of
a system or component and focuses solely on
the outputs generated in response to selected
inputs and execution conditions.
• Test planning can begin early in the software
process
Black-box Testing 2

Inputs causing
anomalous
Input test data I behaviour
e

System

Outputs which reveal

the presence of
Output test results Oe defects
 Black-box Testing 3

• It is not an alternative to White-box Testing.

• It uncovers errors of the following categories:
1. Incorrect or missing functions
2. Interface errors
3. Data structures or external data base access
4. Performance errors
5. Initialization and termination errors
Advantages of Black Box Testing
• more effective on larger units of code than glass
box testing
• tester needs no knowledge of implementation,
including specific programming languages
• tester and programmer are independent of each
other
• tests are done from a user's point of view
• will help to expose any ambiguities or
inconsistencies in the specifications
• test cases can be designed as soon as the
specifications are complete
 Disadvantages of Black Box
Testing
• only a small number of possible inputs can actually be
tested, to test every possible input stream would take
nearly forever
• without clear and concise specifications, test cases are
hard to design
• there may be unnecessary repetition of test inputs if
the tester is not informed of test cases the programmer
has already tried
• may leave many program paths untested
• cannot be directed toward specific segments of code
which may be very complex (and therefore more error
prone)
• most testing related research has been directed toward
glass box testing
 Black-box Testing Techniques
2

• Equivalence Partitioning
• Boundary Value Analysis
• Decision Tables
• Requirements Based Testing
• Positive and Negative Testing
• Graph-Based Testing or State Based Testing
• Compatibility Testing
• User Documentation Testing
• Domain Testing
 Equivalence Partitioning 1

• The input domain is usually too large for

exhaustive testing.
• It is therefore partitioned into a finite number of
sub-domains for the selection of test inputs.
• Each sub-domain is known as an equivalence
class and serves as a source of at least one test
input.
 Equivalence Partitioning 2

• Test case design is based on an evaluation of

equivalence classes for an input condition.
• Equivalence class represents a set of valid or
invalid states for input conditions.
• For Specific value, one valid and two invalid
equivalence classes are defined.
• If the system works for any one test case in the
range (1..16,383), then it will probably work for any
other test case in the range.
– Range (1..16,383) constitutes an equivalence class.
Equivalence Partitioning 3

Invalid inputs Valid inputs

System

Outputs
 Example 1
• Range (1..16,383) defines three different
equivalence classes:
– Equivalence Class 1: Fewer than 1 record
– Equivalence Class 2: Between 1 and 16,383
records
– Equivalence Class 3: More than 16,383
records
 Guidelines for derivation of
Equivalence Classes
• If a range v1 ... v2,
then <v1, v1 ... v2, >v2
• If a value v, then <v, v, >v
• If a set S={ a, b, ..., x }, then S, any value
not in S
• If Boolean b, then b=true, b=false
 WHY LEARN EQUIVALENCE
PARTITIONING?
• Equivalence partitioning drastically cuts
down the number of test cases required to
test a system reasonably.
• It is an attempt to get a good 'hit rate', to
find the most errors with the smallest
number of test cases.
DESIGNING TEST CASES USING
EQUIVALENCE PARTITIONING
• Identify the equivalence classes
• Design test cases
• STEP 1: IDENTIFY EQUIVALENCE CLASSES Take
each input condition described in the specification and
derive at least two equivalence classes for it. One class
represents the set of cases which satisfy the condition
(the valid class) and one represents cases which do not
(the invalid class )
• Following are some general guidelines for identifying
equivalence classes:
– a) If the requirements state that a numeric value is input to the
system and must be within a range of values, identify one valid
class inputs which are within the valid range and two invalid
equivalence classes inputs which are too low and inputs which
are too high.
• For example, if an item in inventory can have a
quantity of - 9999 to + 9999, identify the
following classes:
• 1. one valid class: (QTY is greater than or equal
to -9999 and is less than or equal to 9999). This
is written as (- 9999 < = QTY < = 9999)
• 2. the invalid class (QTY is less than -9999),
also written as (QTY < -9999)
• 3. the invalid class (QTY is greater than 9999) ,
also written as (QTY >9999)
• If the requirements state that the number of items input
by the system at some point must lie within a certain
range, specify one valid class where the number of
inputs is within the valid range, one invalid class where
there are too few inputs and one invalid class where
there are, too many inputs.
• For example, specifications state that a maximum of 4
purchase orders can be registered against anyone
product. The equivalence classes are : the valid
equivalence class: (number of purchase orders is greater
than or equal to 1 and less than or equal to 4 , also
written as (1 < = no. of purchase orders < = 4) the invalid
class (no. of purchase orders> 4) the invalid class (no. of
purchase orders < 1)
• If the requirements state that a particular input
item match one of a set of values and each case
will be dealt with the same way, identify a valid
class for values in the set and one invalid class
representing values outside of the set. For
example, if the requirements state that a valid
province code is ON, QU, and NB, then identify :
the valid class code is one of ON, QU, NB the
invalid class code is not one of ON, QU, NB
 Grocery Store Example
• Consider a software module that is intended to accept
the name of a grocery item and a list of the different
sizes the item comes in, specified in ounces. The
specifications state that the item name is to be
alphabetic characters 2 to 15 characters in length. Each
size may be a value in the range of 1 to 48, whole
numbers only. The sizes are to be entered in ascending
order (smaller sizes first). A maximum of five sizes may
be entered for each item. The item name is to be entered
first, followed by a comma, then followed by a list of
sizes. A comma will be used to separate each size.
Spaces (blanks) are to be ignored anywhere in the input.
 Derived Equivalence Classes
• Item name is alphabetic (valid) • Size value includes nonnumeric
• Item name is not alphabetic (invalid) characters (invalid)
• Item name is less than 2 characters in • Size values entered in ascending order
length (invalid) (valid)
• Item name is 2 to 15 characters in • Size values entered in nonascending
length (valid) order (invalid)
• Item name is greater than 15 • No size values entered (invalid)
characters in length (invalid) • One to five size values entered (valid)
• Size value is less than 1 (invalid) • More than five sizes entered (invalid)
• Size value is in the range 1 to 48 • Item name is first (valid)
(valid) • Item name is not first (invalid)
• Size value is greater than 48 (invalid) • A single comma separates each entry
• Size value is a whole number (valid) in list (valid)
• Size value is a decimal (invalid) • A comma does not separate two or
• Size value is numeric (valid) more entries in the list (invalid)
• The entry contains no blanks (???)
• The entry contains blanks (????)
 Black Box Test Cases for the Grocery Item Example based on the Equivalence Classes Above.

Expected
# Test Data Classes Covered
Outcome
1,4,7,9,11,13,16,1
1 xy,1 T
8,20,22
AbcDefghijklmno,1 1,4,7,9,11,13,16,1
2 T
,2,3 ,4,48 8,20,23
3 a2x,1 F 2
4 A,1 F 3
5 abcdefghijklmnop F 5
6 Xy,0 F 6
7 XY,49 F 8
8 Xy,2.5 F 10
9 xy,2,1,3,4,5 F 14
10 Xy F 15
11 XY,1,2,3,4,5,6 F 17
12 1,Xy,2,3,4,5 F 19
13 XY2,3,4,5,6 F 21
14 AB,2#7 F 12
 Boundary Value Analysis
• Select test cases on or just to one side of
the boundary of equivalence classes.
– This greatly increases the probability of
detecting a fault.
• Complements equivalence class testing.
 Example 2.
(contd.)

Test case 1: 0 records Member of equivalence class 1 and

adjacent to boundary value
Test case 2: 1 record Boundary value
Test case 3: 2 records Adjacent to boundary value
Test case 4: 723 records Member of equivalence class 2
Test case 5: 16,382 records Adjacent to boundary value
Test case 6: 16,383 records Boundary value
Test case 7: 16,384 records Member of equivalence class 3
and adjacent to boundary value
 HOW DO YOU PERFORM
BOUNDARY VALUE ANALYSIS?
• Identify the equivalence classes.
• Design test cases.
• STEP 1: IDENTIFY EQUIVALENCE CLASSES Follow
the same rules you used in equivalence partitioning.
However, consider the output specifications as well.
• For example, if the output specifications for the inventory
system stated that a report on inventory should indicate
a total quantity for all products no greater than 999,999,
then you d add the following classes to the ones you
found previously: 6. The valid class ( 0 < = total quantity
on hand < = 999,999 ) 7. The invalid class (total quantity
on hand <0) 8. The invalid class (total quantity on hand>
999,999 )
• STEP 2: DESIGN TEST CASES In this
step, you derive test cases from the
equivalence classes. The process is
similar to that of equivalence partitioning
but the rules for designing test cases
differ. With equivalence partitioning, you
may select any test case within a range
and any on either side of it with boundary
analysis, you focus your attention on
cases close to the edges of the range.
• The detailed rules for generating test cases follow:
• RULES FOR TEST CASES
• 1. If the condition is a range of values, create valid test cases for
each end of the range and invalid test cases just beyond each end
of the range. For example, if a valid range of quantity on hand is -
9,999 through 9,999, write test cases that include: 1. the valid test
case quantity on hand is -9,999, 2. the valid test case quantity on
hand is 9,999, 3. the invalid test case quantity on hand is -10,000
and 4. the invalid test case quantity on hand is 10,000 You may
combine valid classes wherever possible, just as you did with
equivalence partitioning, and, once again, you may not combine
invalid classes. Don�t forget to consider output conditions as well.
In our inventory example the output conditions generate the
following test cases: 1. the valid test case total quantity on hand is 0,
2. the valid test case total quantity on hand is 999,999 3. the invalid
test case total quantity on hand is -1 and 4. the invalid test case total
quantity on hand is 1,000,000
• 2. A similar rule applies where the, condition
states that the number of values must lie within a
certain range select two valid test cases, one for
each boundary of the range, and two invalid test
cases, one just below and one just above the
acceptable range . 3. Design tests that highlight
the first and last records in an input or output file.
4.Look for any other extreme input or output
conditions, and generate a test for each of them.
 EXAMPLE OF BOUNDARY
VALUE ANALYSIS
• Function to be Tested
•
• For a function called billing, the following specifications are defined:
•
•  Generate a bill for accounts with a balance owed > 0.
•
•  Generate a statement for accounts with a balance owed < 0 (credit).
•
•  For accounts with a balance owed > 0:
•
•  place amounts for which the run date is < 30 days from the date of service in the current total,
•
•  place amounts for which the run date is = or > 30 days, but < or = 60 days, from the date of service, in the
30 to 60 day total,
•
•  place amounts for which the run date is > 60 days, but < or = 90 days, from the date of service, in the 61
to 90 day total,
•
•  place amounts for which the run date is > 90 days, from the date of service, in the 91 days and over total.
•
•  For accounts with a balance owed > or = $10.00, for which the run date is = or > 30 days from the date of
service, calculate a $3.00 or 1% late fee, whichever is greater.
 Input and Output Conditions
• Identify the input, (i.e., information is supplied to the function) and
output, (i.e., information is produced by the function) conditions for
the function.
•
• The input conditions are identified as:
•
•  balance owed,
•  balance owed for late fee.
•
• The output conditions are identified as:
•
•  age of amounts,
•  age of amounts for late fee,
•  calculation for late fee.
 Defining Tests
• Define tests for the boundary situations for each of the input and output conditions. For example:
•
• Balance Owed
•
• 1. > 0,
• 2. = 0,
• 3. < 0.
•
• Age of Amounts
•
• balance owed > 0 and
•
• 4. run date - date of service = 0,
• 5. run date - date of service = 29,
• 6. run date - date of service = 30,
• 7. run date - date of service = 31,
• 8. run date - date of service = 59,
• 9. run date - date of service = 60,
• 10. run date - date of service = 61,
• 11. run date - date of service = 89,
• 12. run date - date of service = 90,
• 13. run date - date of service = 91.
•
• Balance Owed for Late Fee
•
• run date - date of service > 30 and
•
• 14. balance owed = $9.99,
• 15. balance owed = $10.00,
• 16. balance owed = $10.01.
•
• Age of Amount for Late Fee
•
• balance owed > $10.00 and
•
• 17. run date - date of service = 29,
• 18. run date - date of service = 30,
• 19. run date - date of service = 31,
•
• Calculation for Late Fee
•
• balance owed > $10.00, run date - date of service > 30 and
•
• 20. 1% late fee < $3.00,
• 21. 1% late fee = $3.00,
• 22. 1% late fee > $3.00.
 Best Strategy!!!
• Equivalence classes together with
boundary value analysis to test both input
specifications and output specifications.
• This approach generates a small set of
test data with the potential of uncovering a
large number of faults
 Decision Table Testing

• Decision tables are used to record complex business

rules that must be implemented in the program, and
therefore tested.
• In the table, the conditions represent possible input
conditions.
• The actions are the events that should trigger,
depending upon the makeup of the input conditions.
• Each column in the table is a unique combination of
input conditions (and is called a rule) that result in
triggering the action(s) associated with the rule.
• Each rule (or column) should become a test case.
 Definition

Combinations • Components
Causes Values 1 2 3 4 5 6 7 8 • A decision table lists
Cause 1 Y, N Y Y Y Y NNNN
Cause 2 Y, N Y Y NNY Y NN causes and effects in
Cause 3 Y, N Y NY NY NY N a matrix. Each
Effects
Effect 1 X X X
column represents a
Effect 2 X X X unique combination.
• Purpose is to structure
logic
Cause = condition
Effect = action = expected results
Step 1: List all causes
Hints:
• Write down the values
Combinations
Causes Values 1 2 3 4 5 the
6 7 8cause/condition
Cause 1 Y, N Y Y Y Y NNNN
Cause 2 Y, N Y Y N N Y can
Y N N assume
Cause 3 Y, N Y NY NY NY N
Effects • Cluster related
Effect 1 X X X
Effect 2 X
causes
X X
• Put the most
dominating cause first
• Put multi valued
causes last
Step 2: Calculate combinations

Number of Values to • If all causes are simply

the power of the Y/N values:
number of causes 2number of causes
with these values
• If 1 cause with 3 values
and 3 with 2:
31 * 23 = 24
• Or, use the Values column
and multiply each value
down the column, eg.
3*2*2*2=24
 Step 3: Fill columns
• Algorithm:
Combinations
Causes Values 1 2 3 4 5 6 7 1.
8 Determine Repeating Factor
Cause 1 Y, N Y Y Y Y NNNN
Cause 2 Y, N Y Y NNY Y NN (RF): divide remaining
Cause 3 Y, N Y NY NY NY N
Effects combinations by the number of
Effect 1 X X X
Effect 2 X X X possible values for that cause
2. Write RF times the first value,
then RF times the next etc.
until row is full
3. Next row, go to 1.
Step 4: Reduce combinations
• Find indifferent
Combinationscombinations – place
Causes Values 1 2 3 4 5 6 7 8
Cause 1 Y, N Y Y Y Y N N NaN ‘-’
Cause 2 Y, N Y Y NNY Y NN
Cause 3 Y, N Y N - - Y NY N
Effects
Effect 1 X X X
Effect 2 X X X

Combinations
Causes Values 1 2 3 4 5 6 7 • Join columns where
Cause 1 Y, N Y Y Y NNNN
Cause 2 Y, N Y Y NY Y NN columns are
Cause 3 Y, N Y N - Y NY N identical
Effects
Effect 1 X X • Tip: ensure the
Effect 2 X X X
effects are the
same
 Step 5: Check covered
combinations
• Checksum
• For each column calculate the
Combinations
Causes Values 1 2 3 4
combinations it represents
Cause 1 Y, N Y Y Y N • A ‘-’ represents as many
Cause 2 Y, N Y N N -
Cause 3 Y, N - Y N - combinations as the cause has
Effects
Effect 1 X X
• Multiply for each ‘-’ down the
Effect 2 column
Checksum 2 1 1 4 8
• Add up total and compare with
step 2
Step 6: Add effects to table
• Read column by
Combinations
Causes Values 1 2 3 4
column and
Cause 1 Y, N Y Y Y N determine the effects
Cause 2 Y, N Y N N -
Cause 3 Y, N - Y N •- One effect can occur
Effects in multiple test
Effect 1 X X
Effect 2 X X combinations
Checksum 2 1 1 4 8
 Example
• A technical support company writes a
decision table to diagnose printer
problems based upon symptoms
described to them over the phone from
their clients.
 Printer troubleshooter
Rules

Conditions Printer does not print Y Y Y Y N N N N

A red light is flashing Y Y N N Y Y N N
Printer is unrecognized Y N Y N Y N Y N
Actions Check the power cable X
Check the printer-computer cable X X
Ensure printer software is installed X X X X
Check/replace ink X X X X
Check for paper jam X X
 Requirements Based Testing
• Deals with validating the requirements given in
Software Requirements Specification.
• Implicit requirements
• Explicit requirements
• Requirements Traceability Matrix
– Req. ID
– Description
– Priority
– Test conditions
– Test case ids
– Phase of testing
 Steps
• Gather complete requirements, including
details about quality expectations
• Trace and prioritize testing as part of a
requirements management process
• Create and manage testing requirements,
resulting in the most effective test plans
and the best use of test automation
 Positive and Negative Testing
• Positive testing
– Prove that a given product does what it is
supposed to do.
– When a test case verifies the requirements of
the product with a set of expected output
– Prove that the product works as per
specifications and expectations
• Negative testing
– Show that the product does not fail when an
unexpected input is given
– Purpose is to try and break the system
– Covers scenarios for which the product is not
designed and coded
 State based or Graph Based
Testing
• Useful in situation where
– The product under test is a language processor (for
example, a compiler), wherein the syntax of the
language automatically lends itself to a state machine
or a context free grammar
– Workflow modeling where, depending on the current
state and appropriate combinations of input variables,
specific workflows are carried out, resulting in new
output and new state
– Dataflow modeling, where the system is modeled as a
set of dataflow, leading from one state to another
 An application required to validate
a number
• A number can start with an optional sign
• The optional sign can be followed by any
number of digits
• The digits can be optionally followed by a
decimal point, represented by a period
• If there is a decimal point, then there should be
two digits after the decimal
• Any number- whether or not it has a decimal
point, should be terminated by a blank
 Example

Digit

+ or - Decimal
1 point Digit
2 3 Digit
4 5

Digit
Blank
Blank

6
 State transition table
Current state Input Next state
1 Digit 2
1 + 2
1 - 2
2 Digit 2
2 Blank 6
2 Decimal point 3
3 Digit 4
4 Digit 5
5 Blank 6
 General outline for using state
based testing methods w.r.t.
language processors
• Identify the grammar for the scenario
• Design test cases corresponding to each
valid state-input combination
• Design test cases corresponding to the
most common invalid combinations of
state-input
State graph to represent workflow

HR
Employe Eligible Manager
Verify
e desires ensure
eligibility Feasible
leave feasibility

Ineligible
Approve
Leave
application form Not
feasible
Reject
 Compatibility Testing
• Test case result are compared with expected
results to conclude whether the test is
successful or not
• Test case results depend on
– the product for proper functioning
– Equally on the infrastructure for delivering
functionality
• When infrastructure parameters change product
is expected to still behave correctly and produce
the desired or expected results
• This testing ensures the working of the product with different infrastructure
components
• The parameters that generally affect the compatibility of the product are
– Processor and their number
– Architecture and characteristics of machine
– Resource availability on the machine
– Equipment e.g. printers, modems, routers etc.
– Operating system (windows, linux) and operating system services (DNS, FTP)
– Middle-tier infrastructure components such as web server
– Backend components such as database servers
– Services that require special hardware-cum-software solutions (cluster
machines, load balancing)
– Any software used to generate product binaries (compiler, linker)
– Various technological components used to generate components (SDK, JDK)
• Inorder to arrive at practical combinations
or the parameters to be tested, a
compatibility matrix is created
• Matrix has as its columns various
parameters the combinations of which
have to be tested
• Each row represents a unique combination
of a specific set of values of the
parameters
 User Documentation Testing
• User documentation covers all the manuals,
user guides, installation guides, setup guides,
read me file, software release notes, and online
help that are provided along with the software to
help the end user to understand the software
system
• Objectives
– To check if what is stated in the document is available
in the product
– To check if what is there in the product is explained
correctly in the document
 Advantages
• Aids in highlighting problems overlooked during reviews
• High quality ensures consistency of documentation and
product, thus minimizing possible defects reported by
customers
• Reduce the time of the support call
• Results in less difficult support calls
• New programmers and testers who join a project group
can use the documentation to learn the external
functionality of the product
• Customers need less training and can proceed more
quickly to advanced training and product usage if the
documentation is of high quality and is consistent with
the product.
• Defects found in user documentation need
to be tracked to closure like any regular
software defect.
 Domain Testing
• White box testing required looking at the program code
• Black box testing performed testing without looking at
the program code but looking at the specifications
• Domain testing can be considered as the next level of
testing in which we don not look even at the
specifications of a software product but are testing the
product, purely based on domain knowledge and
expertise in the domain of application
• Requires business domain knowledge rather than the
knowledge of what the software specification contains or
how the software is written
• Domain testing is the ability to design and execute test
cases that relate to the people who will buy and use the
software
• It helps in understanding the problems they are trying to
solve and the ways in which they are using the software
to solve them
• It is also characterized by how well an individual test
engineer understands the operation of the system and
the business processes that system is supposed to
support
• Domain testing involves testing the product, not by going
through the logic built into the product. The business flow
determines the steps, not the software under test. This is
called “business vertical testing”
 Example
• Step 1: go to the ATM
• Step 2: Put ATM card inside
• Step 3: Enter correct PIN
• Step 4: Choose cash withdrawal
• Step 5: Enter amount
• Step 6: Take the cash
• Step 7: Exit and retrieve the card
 Scenarios for black box testing
techniques
When you want to test scenarios that The most effective black box testing
have… technique is likely to be…

Output values dictated by certain conditions Decision tables

depending upon values of input variables
Input values in ranges, with each range Boundary value analysis
exhibiting a particular functionality
Input values divided into classes (like Equivalence partitioning
ranges, list of values, and so on), with each
class exhibiting a particular functionality
Checking for expected and unexpected input Positive and negative testing
values
Workflows, process flows, or language Graph based testing
processors
To ensure that requirements are tested and Requirements based testing
met properly
To test the domain expertise rather than Domain testing
product specification
To ensure that the documentation is Documentation testing
consistent with the product
 References
• Sommerville, Ian “Software Engineering 6th
edition.”
• Pressman, Roger “Software Engineering, A
Practitioner’s Approach6th edition.”
• Mayer “The Art of Software Testing 2nd edition.”
• …