ITECH7409: Software Testing

Lecture 1. Introduction. Bird’s-eye view.

This lecture
• What is software testing?
• Halting Problem and Software Testing.
• Simple example. Traditional approach to testing.
• Basic notions.
• Verification and validation.
• Major testing types.
• Testing methods.
• Testing levels.
• The international software testing standards.
• Basic terminology
• Five approaches to testing.
• Quality issues.
Software testing. What is it?
Software testing:
• Is a process of checking software to discover if it is suitable to the
task for which it is being developed. I.e., are the customer
requirements met correctly?
• Is the software free of defects/ bugs? The tester should find as many
defects as practical and have them corrected.
• Is an essential part of the software development lifecycle.
Software Testing and Halting Problem.
• Halting Problem: can a program be written that predicts whether or
not any other program halts (terminates) after a finite number of
steps?
• If a program that was developed to solve some problem never halts, it
certainly has a bug.
• So, if we had an algorithm that finds all bugs in an arbitrary program, we
could use it to solve Halting Problem.
• It is well known that Halting Problem is algorithmically unsolvable
(the proof might be discussed in the lab).
• Therefore, it is impossible to develop a formal procedure (algorithm)
that detects all bugs in an arbitrary program.
Implications of Halting Problem for Software Testing.
The Halting Problem has an extensive scope and plays a crucial role in
understanding the limitations of computation. It sets the boundary for what
computers can and cannot solve and has substantial implications for multiple
areas of study, from Artificial Intelligence to Cybersecurity. Let’s outline some
implications of the algorithmic unsolvability of Halting Problem for software
testing:
• It is impossible to develop a rigorous formal testing procedure that is
suitable to test any program.
• It is impossible to develop a perfect software (including AI products) that
will find all bugs in an arbitrary program. Although there are lots of
programmatic tools for testing, they can only help testers, not replace
them.
• In general, software testing is a creative process that may require
development of special new techniques and approaches for each new case.
Software Testing: Practical Approach.
• Although, as we have seen, there is inherent obstacle for existence of
“perfect” testing procedure, testing still must be done.
• To partially achieve this goal, various formal testing techniques and
approaches were developed that will be discussed throughout the
course.
• Some of these techniques require familiarity with basic mathematical
notions and results that also will be discussed in the lectures.
• A good tester must be inventive and have various testing techniques
in their “toolbox”.
Example: Student Program
• Task: Write a Java program that asks a user for their Federation
University student ID number and then outputs the ID number in a
dialogue screen. Student ID numbers should have 8 digits starting
with the digit 3.

• It seems the program works correctly.

Example: Student Program
Let’s try another input:

This is not the expected behaviour of the program as the task says:
Student ID numbers should have 8 digits starting with the digit 3.
Example: Student Program
Is this program
• suitable to the task for which it is being developed?
• free of defects?

Software testing should

• check to see that customer requirements are met correctly.
• find as many defects as practical and have them corrected.

In this example, student or tester should undertake

• exhaustive testing to find as many problems as possible with the program’s
behaviour.
• correct all identified problems.
Example: Student Program
We would expect the following behaviour:
Example: Student Program
We would expect the following behaviour:
Example: Student Program
We would expect the following behaviour:
In the example:
Traditional testing approach:
• Positive testing
• Is trying to show that the program or system works correctly, i.e., does
what it was written for.
• This is often what students do when they illustrate to others that their
program is working.
• Negative testing
• tries to find faults in the program or system to discover and correct
problems.
• This is what lecturers do to student programs – they try to break them.
• Exhaustive testing
• is testing all possible inputs in the search for problems.
• In most instances, exhaustive testing is an impossible task to achieve.
Testing goals. Verification and Validation.
Another definition: Software testing is a process of checking software to
discover if the software is verified and valid.
 Verification means checking whether the system is being built right. I.e., is
concerned with detecting bugs.
 Validation means checking whether the right system is being built. I.e., checking
that the software (client) requirements are met.

Validation is performed after the verification and inspects the whole system.
Testing goals. Verification and Validation.
According to International Standard ISO/IEC/ IEEE 29119-1:
• Verification is confirmation, through the provision of objective
evidence, that specified requirements have been fulfilled in a given
work item.
• Validation demonstrates that the work item can be used by the
users for their specific tasks.
Major testing types
• Functional Testing is a type of software testing that validates the
software system against the functional requirements/specifications.
• The purpose of Functional tests is to test each function of the software
application, by providing appropriate input and verifying the output against
the Functional requirements.
• Functional testing mainly involves black box testing, and it is not concerned
about the source code of the application.
• Non-functional testing checks the performance, reliability, scalability
and other non-functional aspects of the software system. It should be
performed after functional testing.
Testing Methods
• Black Box Testing – the tester doesn’t have access to the source code.
• Usually is performed during integration, system and acceptance testing levels.
• White Box Testing – the tester has full access to the source code.
• Usually is performed during unit, integration and system testing levels.
• Grey Box Testing – is a mixture of both.
Testing Levels
 Unit testing – basic units of software are tested. The goal of this is to
verify every unit of the product after the test.
 Integration testing – units need to combine and test as a single
cluster. The main idea is to expose the faults while integrating units
of the project.
 System testing – the whole project is tested. The idea is to assess the
system's conformity with its intended requirements.
 Acceptance testing – a system is tested for adequacy. This test is
purposefully done for evaluating the compliance of the system with
business requirements.
Finally
• Alpha testing is final testing at the end of the development process
before the software is released and it is done at the developers site. It
includes testing in an environment that is similar to the intended use.
• Beta testing is done at the customer's site. It is done just before the
launch of the product, and it is designed to see if users can install the
software and use it under real-world working conditions.
• Beta testing can be considered “pre-release” testing.
The international software testing standards
ISO/IEC/IEEE 29119 is intended for software testing acts as an
internationally approved collection of standards in software testing
suitable for any Software Development Life Cycle model. Currently, it
has five standards:
• ISO/IEC 29119-1: Deals with concepts and definitions of software.
• ISO/IEC 29119-2: Deals with test processes in a product.
• ISO/IEC 29119-3: Deals with test documentation of the product.
• ISO/IEC 29119-4: Deals with testing techniques and strategies.
• ISO/IEC 29119-5: Deals with keyword-based software testing.
Basic Terminology
According to the glossary of International Software Testing Qualification Board
(ISTQB) (http://www.istqb.org):

Error – a human action that produces an incorrect result. Synonym mistake.

Defect – An imperfection or deficiency in a work product where it does not meet its
requirements or specifications. Synonyms bug, fault.
Failure – an event in which a component or system does not perform a required
function within specified limits.
Test case – A set of preconditions, inputs, actions (where applicable), expected
results and postconditions, developed based on test conditions.
Test – A set of one or more test cases.
Examples of Defects and Errors
Defects in the Interface specification
• Mismatch between what the client needs and what the server offers.
• Mismatch between requirements and implementation.
Algorithmic Defects
• Missing initialization
• Branching errors (too soon, too late)
• Missing test for nil
Mechanical Defects (very hard to find)
• Documentation does not match actual conditions or operating procedures
Errors
• Stress or overload errors
• Capacity or boundary errors
• Timing errors
• Throughput or performance errors
Testing Approaches
Level 0: There’s no difference between testing and debugging.

Problem with this approach:

• It does not distinguish between actual mistakes in a program or a
poor design causing incorrect behaviour.
• It does not help develop software that is reliable.
Testing Approaches
Level 1: The purpose of testing is to show correctness (student
approach).

Problem with this approach:

• Correctness is impossible to achieve unless there is exhaustive
testing.
• Good results could be the result of bad tests.
• When is testing enough?
Testing Approaches
Level 2: The purpose of testing is to show that the software doesn’t
work (lecturer approach).

Problem with this approach:

• The goal is to show failures.
• It puts testers and developers into an adversarial relationship.
Testing Approaches
Level 3: The purpose of testing is not to prove anything specific, but to
reduce the risk of using the software.

Problem with this approach:

• Testing can only show presence of failures.
• Testers and developers work together to reduce risk of problems.
Testing Approaches
Level 4: Testing is a mental discipline that helps all IT professionals
develop higher quality software.

Testing is the only way to increase software quality.

• Test engineers have a primary responsibility to measure and improve
software quality.
• Their expertise makes them technical leaders on a project and their
role is to help the developers deliver a quality product.
Software quality: The 3 Cs
Correctness – the software is designed to meet the requirement
specifications and the needs of its users (validation and verification).

Completeness – the software implements full functionality needed to

solve the problem.

Consistency – all functions of the software behave in the same manner

with each other and over various runs.
Software quality
Quality cannot be measured directly:
• Is it possible to know that all bugs are eliminated in testing?
• Or If the software will perform well on a new operating system, or
after the addition of new modules?

Quality software is correct, complete and consistent with all explicitly

documented functional and non-functional requirements, as well as
requirements for reliability, usability, safety and security.
In addition, it should adhere to standards imposed on the software,
such as ISO/IEC/ IEEE 29119 standard.
Measures of Quality
Measures of Quality