Chapter 7

Testing
“What do we mean by a Smoke test”
Simple: turn on a system to see if it works
Are tests this simple??

One of the desirable properties of engineering

requirements is that it be verifiable
 Motivation
 Development is accompanied by “bugs.” (mistakes, errors,
problems, faults)
 In engineering, a bug is defect in design, manufacture,
operation of machinery, circuitry, hardware, or software that
produces undesired results or operation.
 Catching bugs early saves money

◦ The further a bug progresses the more impact it has on system

 A bug fix requires all related modules to be retested.
 A bug fix may require redesigning related modules.
◦ For example: PCB design flaw or Coding flaw
 Testing doesn’t remove bugs, it just makes it less likely

2
Learning Objectives
By the end of this section, you should:
 Understand the concepts of black box tests, white
box tests, observability, and controllability.
 Understand the principles of debugging.
 Different tests used in practice:
◦ Understand what a unit test is
◦ Understand what is an integration test is used
◦ Understand what is an acceptance test

3
Testing
An engineering validation test (EVT) is performed on first engineering prototypes, to ensure that the basic unit performs to
design goals and specifications

 After finishing the design, how to test?

 Why testing in the first place?
 Test should be chosen to increase likelihood
of finding an error.
 How should we perform testing?
◦ There is a Tradeoff:
 Apply every possible input, or
 Apply selected inputs which might yield errors

4
 7.1 Testing Principles
 How to test that the resulting system meets
the design requirements
 The Test-Vee is a common testing model
Development Testing

Each test performed to the right

Requirements Acceptance
must be carefully engineered Specification Test
during the development of the
system in the left
System Integration
Keep in mind right when u do the Design Test
left !

Unit
Construct
Test

Debugging

5
 We need clear tests, Why?
 Test cases define what each module does
 Testing Prevents feature obsession (excessive expansion)
 Test cases provide early feedback.
 Testing prevents early problems, since the development of a
module is complete when its test is passed.
 Test cases are a form of documentation (very important).
 Test cases force designer to consider design before building

6
7.1.1 Types of testing
 Black box tests
◦ No knowledge of internal organization
◦ Only access input and outputs
◦ Change inputs and compare the system outputs to
their expected values
 White box tests
◦ Knowledge of internal organization
◦ Might have expectation of fault model
◦ Create test instance which reveal physical or logical
errors
7
Example CE amplifier
 Vi, Vo Vcc
Black box testing:
(Rectangular box)
Vi, Vcc and Vo
Check whether there is
a liner amplification

8
Example CE amplifier
 Vcc Vi Va Vc Ve Vo
White box testing:
(detailed voltages)
Check whether all satisfy
expected design values.
(DC bias and AC
amplification behavior)

9
Example in signal analysis

 The black box could be a filter design using a

set of components

10
 Black vs White
 The major difference between two types of testing is
knowledge of the internal structure of the circuit.
 A black box test is performed without knowledge of
the system, while white box testing is performed with
knowledge of the systems construction.
 Consequently, black box tests are generally written to
check extreme inputs and outputs, and transitional
behaviors.
 White box tests are generally written to force internal
nodes of the system to specific values.

11
Testability, Controllability, Observability

An important goal in designing systems is to increase testability

 Testability

Failure of a component can be Quickly detected and located

One way to increase testability is to increase controllability and

obervability

 Controllability
Any node of the system can be set to a desired value
 Observability
Any node of the system can be measured.

Black box has low observability and no controllability

12
Testability, Controllability, Observability
 Testability – failure of a component can
◦ Quickly detected
◦ Quickly located
 Controllability
◦ When any node of the system can be set to a desired
value
◦ Black box has no controllability
 Observability
◦ When any node of the system can be measured.
◦ Black box has low observability

13
 Example: Transistor amplifier design
 Amplify input signal vi to
produce a linearly proportional
output signal vo=A.vi
 Rectangle boundary represents

black box view

 Black box testing consists of

varying the input signal and

observing the output signal
(Low controllability , low to
medium observability)

White box testing would require node

voltages VB, VC, VE to be measured
14
Stub
Stub is a device that is used to simulate a
subcomponent of a system
◦ Used to Simulates input or monitors outputs
◦ For a unit under test (UUT)
◦ Insure good behavior
 before wreaking havoc on the rest of the system
 For example
◦ A function generator for an audio input

(A method stub in software development is a piece of

code used to stand in for some other programming
functionality. )
15
Stub example for transistor amplifier

stub simulating stub simulating

expected input expected output
R C

function
BJT Amplifier RL
generator

16
Test case properties
 Accurate - The test should check what it is supposed to and
exercise an area of intent.
 Economical - The test should be performed in a minimal number
of steps.
 Limited in complexity - Tests should consist of a moderate number
(10-15) of steps.
 Repeatable - The test should be able to be performed and repeated
by another person.
 Appropriate - The complexity of the test should be such that it is
able to be performed by other individuals who are assigned the
testing task.
 Traceable - The test should verify a specific requirement.
 Self cleaning - The system should return to the pre-test state after
the test is complete.
17
7.2 Constructing Tests
 The 4 different types of tests

18
Debugging
(the process of identifying and removing errors from hardware or software.)

 Iterative process: run tests and depending on

the results go back and run new tests
 Strategy similar to if-then-else structure: “if

this test is negative, then I’ll pursue this line

of attack, otherwise the error could be in
another system.

19
Steps for the debugging process:
 Observe: Observe the problem under different
operating conditions
 Hypothesize: Form a hypothesis as to what the

potential problem is
 Experiment: Conduct experiments to confirm or

eliminate the hypothesized source of the

problem
 Repeat: Repeat until the problem is eliminated

20
7.2.2 Unit Testing
 A unit test is a test of the functionality of a
system module in isolation
 Should be traceable to the detailed design.
 Consists of a set of test cases
 Each test case establishes that a subsystem
performs a single unit of functionality to
some specification.
 Test cases should be written with the express
intent of finding undiscovered defects.

21
 Code example
 Consider a hardware module that converts an
input Celsius temperature into Fahrenheit.
if (16 < input < 32)
output = ROM[input – 16];
Else
output = (2 * input) + 43;

Need to have at least two test cases, one for the “if” clause
and one for the “else”
Also need to check the boundary conditions separating the if
and else clauses
Should consider the extreme values of the input.
Ex. if input is a signed 8-bit number, then should check -128
and +127. 0 should be checked as well. 22
Matrix Tests
 A matrix test is a test that is best suited to
cases where the inputs submitted are
structurally the same and differ only in their
values
 In this case what is needed is a list of inputs

and their expected outputs

Consider a test of ADC that is used in Ch 5.

Assume clock frequency of 10 kHz and the
input ranges from 0 to 5 v. The unit test will
consist of submitting different inputs to the
ADC and verifying the outputs. Since each test
varies only the input with no change in
procedure, test matrix is used as shown next 23
24
 Step-by-Step Tests
 A step-by-step test case is a prescription for
generating the test and checking the results.
 Good when the test consists of a complex

sequence of steps

Ex. Recall the state diagram for the vending

machine in Ch. 6 that accepts nickels and dimes and
dispense candy when total $0.25 (or more)

25
 Step-by-Step unit Test for a vending machine

Test Writer: Sue L. Engineer

  Test Case Name: Finite State Machine Path Test #1 Test ID #: FSM-Path-01

  Description: Simulate insertion of money with a mix of nickels and Type:  white box
dimes. Verifies FSM, outputs candy in response to a  black box
total deposit of $0.30.

Tester Information

  Name of Tester: Date:

  Hardware Ver: 1.0 Time:

Setup: Make sure that the system was reset sometime prior and is in state $0.00.

St Action Expected Result P F N Comments

e a a /
p s i A
s l
1 Strobe Nickel State should go to $0.05

2 Strobe Dime State should go to $0.15

3 Wait State should remain $0.15

4 Strobe Nickel State should go to $0.20

5 Strobe Dime State should go to $0.25

6 Nothing State should go to $0.00

  Overall test result:         26

7.2.3 Integration Testing
 After finishing Unit testing
 Write integration test of all units working
together:
◦ Help ensure requirements are being met.
◦ Requires the designer think about the
expected behavior of the subsystems.
◦ Requires designer to think about extreme
behaviors of subsystems.

27
Write the Integration Test
 What are the different paths of execution
through the system?
 Have all the interface signals been tested?
 Does the system process information at the
required rate and met timing requirements?

28
 E. Acceptance Testing
 Might be formal legal document
 Written along with requirements
 Traceable to engineering requirements
 Identifies
◦ Scope – how much of the system is tested?
◦ Level – how deep will testing be performed?
◦ Evaluates the system delivered by developers
◦ Carried out by the client. May involve executing
typical transactions on site on a trial basis
◦ Goal: Demonstrate that the system meets the
requirements and is ready to use.
29
7.3 Application: Autonomous Robot
 Autonomous navigating robot
 Engineering requirements
◦ The robot’s center must stay within 12 to 18
centimeters of the wall over 90% of the course,
while traveling parallel to a wall over a 3 meter
course.
◦ The robot’s heading should never deviate no
more than 10 degrees from the wall’s axis, while
traveling parallel to a straight wall over a 3 meter
course.

30
 Robot Acceptance Test
Test Writer: Sue L. Engineer
  Test Case Name: Robot acceptance test #1 Test ID #: Robot-AT-01

  Description: Checks the engineering requirement: The robot’s center must Type:  white box
stay within 12 to 18 centimeters of the wall over 90% of the  black box
course, while traveling parallel to a wall over a 3 meter course.

Tester Information
  Name of Tester: Date:
  Hardware Ver: Robot 1.0 Time:
Setup: Completed robot should be fully charged and placed on 3 meter test track.
Action Expected Result Comments
Step

Pass
Fail

N/A
1 Write a program to Program should be statically tested to verify
monitor the robots position accuracy. Should sample wall at a sufficient
from the wall. rate depending on speed.

2 Put robot on test track, run The robot should travel down the entire
test, and download data. length of the test track and then stop.

3 Plot test data in a Plot of position vs. time should be within 12

spreadsheet program. – 18 cm 90% of the time.
  Overall test result:

31
Example: Robot architecture

Switches

Digital
H-bridge DC motor
Compass

MCU

Range
H-bridge DC motor
Finder

LCD

32
Some Integration Test Possibilities
 MCU + motors + bridge + switches

 Chassis + digital compass + MCU + motors

+ bridge +LCD

 Chassis + range finder + MCU + motors +

bridge

33
Unit testing possibilities
 MCU (hardware)
 LCD
 Switches
 Compass
 Range finder
 H-bridge
 Motors
 Chassis
 MCU (software)

34
Summary: Reasons to develop and
conduct tests
 Testing reduces the number of bugs in existing and
new features.
 Tests are good documentation.
 Tests improve design.
 Tests defend against other designers.
 Testing forces you to slow down and think.
 Testing is fun.

35
What did we learn?
 Testing helps to ensure proper operation of
system.
 Broad categories of testing:
 Block box
 White box

 Types of Testing
◦ Unit Testing
◦ Matrix tests
◦ Step-by-step test
◦ Integration Testing
◦ Acceptance Testing
36