Introduction to Testing

Software testing is the process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features ofthe software item . Software testing is an activity that should be done throughout the whole development process. Software testing is one of the verification and validation, or V&V, software practices. Verification (the first V) is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Verification activities include testing and reviews. Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements. At the end of development validation (the second V) activities are used to evaluate whether the features that have been built into the software satisfy the customer requirements and are traceable to customer requirements.

The Economics of Software Testing In software development, there are costs associated with testing our programs. We need to write out test plan and our test cases, we need to set up the proper equipment, we need to systematically execute the test cases, we need to follow up on problems that are identified, and we need to remove most of the faults we find. Actually, sometimes we can find low-priority faults in our code and decide that it is too expensive to fix the fault

The Basics of Software Testing There are two basic classes of software testing, black box testing and white box testing. For now, you just need to understand the very basic difference between the two classes, clarified by the definitions below: Black box testing (also called functional testing) is testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions. White box testing (also called structural testing and glass box testing) is testing that takes into account the internal mechanism of a system or component.

The classes of testing are denoted by colors to depict the opacity of the testers of the code. With black box testing, the software tester does not (or should not) have access to the source code itself. The code is considered to be a big black box to the tester who cant see inside the box. The tester knows only that information can be input into to the black box, and the black box will send something back out. Based on the requirements knowledge, the tester knows what to expect the black box to send out and tests to make sure the black box sends out what its supposed to send out. Alternatively, white box testing focuses on the internal structure of the software code. The white box tester (most often the developer of the code) knows what the code looks like and writes test cases by executing methods with certain parameters. In the language of V&V, black box testing is often used for validation (are we building the right software?) and white box testing is often used for verification (are we building the software right?).

What is Black Box Testing?

Black box testing is a software testing techniques in which functionality of the software under test (SUT) is tested without looking at the internal code structure, implementation details and knowledge of internal paths of the software. This type of testing is based entirely on the software requirements and specifications. Here are the generic steps followed to carry out any type of Black Box Testing. *Initially requirements and specifications of the system are examined. *Tester chooses valid inputs (positive test scenario) to check whether SUT processes them correctly . Also some invalid inputs (negative test scenario) are chosen to verify that the SUT is able to detect them. *Tester determines expected outputs for all those inputs. *Software tester constructs test cases with the selected inputs. *The test cases are executed. *Software tester compares the actual outputs with the expected outputs. *Defects if any are fixed and re-tested.

Types of Black Box Testing

There are many types of Black Box Testing but following are the prominent ones Functional testing This black box testing type is related to functional requirements of a system; it is done by software testers. Non-functional testing This type of black box testing is not related to testing of a specific functionality , but nonfunctional requirements such as performance, scalability, usability. Regression testing Regression testing is done after code fixes , upgrades or any other system maintenance to check the new code has not affected the existing code. Six Types of Testing There are several types of testing that should be done on a large software system. Each type of test has a specification that defines the correct behavior the test is examining so that incorrect behavior (an observed failure) can be identified. 1. Unit Testing Opacity: White box testing Specification: Low-level design and/or code structure Unit testing is the testing of individual hardware or software units or groups of related units. Using white box testing techniques, testers (usually the developers creating the code implementation) verify that the code does what it is intended to do at a very low structural level. Unit testing is generally done within a class or a component.

2. Integration testing Opacity: Black- and white-box testing Specification: Low- and high-level design Integration test is testing in which software components, hardware components, or both are combined and tested to evaluate the interaction between them. Using both black and white box testing techniques, the tester (still usually the software developer) verifies that units work together when they are integrated into a larger code base. Just because the components work individually, that doesnt mean that they all work together when assembled or integrated. For example, data might get lost across an interface, messages might not get passed properly, or interfaces might not be implemented as specified. To plan these integration test cases, testers look at high- and low-level design documents.

3. Functional and system testing Opacity: Black-box testing Specification: high-level design, requirements specification Using black box testing techniques, testers examine the high-level design and the customer requirements specification to plan the test cases to ensure the code does what it is intended to do. Functional testing involves ensuring that the functionality specified in the requirement specification works. System testing involves putting the new program in many different environments to ensure the program works in typical customer environments with various versions and types of operating systems and/or applications. System testing is testing conducted on a complete, integrated system to evaluate the system compliance with its specified requirements. Because system test is done with a full system implementation and environment, several classes of testing can be done that can examine non-functional properties of the system. It is best when function and system Stress testing testing conducted to evaluate a system or component at or beyond the limits of its specification or requirement. For example, if the team is developing software to run cash registers, a non-functional requirement might state that the server can handle up to 30 cash registers looking up prices simultaneously. Stress testing might occur in a room of 30 actual cash registers running automated test transactions repeatedly for 12 hours. There also might be a few more cash registers in the test lab to see if the system can exceed its stated requirements. Performance testing testing conducted to evaluate the compliance ( fulfillment )of a system or component with specified performance requirements. To continue the above example, a performance requirement might state that the price lookup must complete in less than 1 second. Performance testing evaluates whether the system can look up prices in less than 1 second (even if there are 30 cash registers running simultaneously). Usability testing testing conducted to evaluate the extent to which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component. While stress and usability testing can be and is often automated, usability testing is done by human-computer interaction specialists that observe humans interacting with the system. 4. Acceptance testing Opacity: Black-box testing Specification: requirements specification After functional and system testing, the product is delivered to a customer and the customer runs black box acceptance tests based on their expectations of the functionality. Acceptance testing is formal testing conducted to determine whether or not a system satisfies its acceptance criteria (the criteria the system must satisfy to be accepted by a customer) and to enable the customer to determine whether or not to accept the system These tests are often pre-specified by the customer and given to the test team to run before attempting to deliver the product. The customer reserves the right to refuse delivery of the software if the acceptance test cases do not pass. However, customers are not trained software testers. Customers generally do not specify a complete set of acceptance test cases.

5. Regression testing Opacity: Black- and white-box testing Specification: Any changed documentation, high-level design Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements. Regression tests are a subset of the original set of test cases. Thesetest cases are re-run often, after any significant changes (bug fixes or enhancements) are made to the code. The purpose of running the regression test case is to make a spot check to examine whether the new code works properly and has not damaged any previously-working functionality by propagating unintended side effects. Most often, it is impractical to re-run all the test cases when changes are made. Since regression tests are run throughout the development cycle, there can be white box regression tests at the unit and integration levels and black box tests at the integration, function, system, and acceptance test levels. The following guidelines should be used when choosing a set of regression tests (also referred to as the regression test suite): Choose a representative sample of tests that exercise all the existing software functions; Choose tests that focus on the software components/functions that have been changed; and Choose additional test cases that focus on the software functions that are most likely to be affected by the change. A subset of the regression test cases can be set aside as smoke tests. A smoke test is a group of test cases that establish that the system is stable and all major functionality is present and works under normal conditions. 6. Beta testing Opacity: Black-box testing Specification: None. When an advanced partial or full version of a software package is available, the development organization can offer it free to one or more (and sometimes thousands) potential users or beta testers. These users install the software and use it as they wish, with the understanding that they will report any errors revealed during usage back to the development organization. These users are usually chosen because they are experienced users of prior versions or competitive products. The advantages of running beta tests areas follows: Identification of unexpected errors because the beta testers use the software in unexpected ways. A wider population search for errors in a variety of environments (different operating systems with a variety of service releases and with a multitude of other applications running). Low costs because the beta testers generally get free software but are not compensated. The disadvantages of beta testing are as follows: Lack of systematic testing because each user uses the product in any manner they choose. Low quality error reports because the users may not actually report errors or may report errors without enough detail. Much effort is necessary to examine error reports particularly when there are many beta testers.

These six levels of testing are summarized in the table shown below

Performing Black Box Testing

Black box testing, also called functional testing and behavioral testing, focuses ondetermining whether or not a program does what it is supposed to do based on its functional requirements. Black box testing attempts to find errors in the external behavior of the code in the following categories: (1) incorrect or missing functionality (2) interface errors (3) errors in data structures used by interfaces (4)behavior or performance errors (5) initialization and termination errors. Throughthis testing, we can determine if the functions appear to work according to specifications. However, it is important to note that no amount of testing can unequivocally demonstratethe absence of errors and defects in your code. It is best if the person who plans and executes black box tests is not the programmer of the code and does not know anything about the structure of the code. The programmers of the code are innately biased and are likely to test that the program does what they programmed it to do. What are needed are tests to make sure that the program does what the customer wants it to do. As a result, most organizations have independent testing groups to perform black box testing. These testers are not the developers and are often referred to as third-party testers. Testers should just be able to understand and specify what the desired output should be for a given input into the program, as shown in Figure

Figure : Black Box Testing. A black-box test takes into account only the input and output of the software without regard to the internal code of the program.

The Anatomy of a Test Case

The format of your test case design is very important. We will use a particular format for our test cases, as shown in Table below . It is always recommended to use use this template in the test planning.

First, we give each test case a unique identifier. When we are tracking large projects, you might need to itemize those test cases that have not yet passed. This identifier is recorded in the first column. Next in the second column of the table, you specifically describe the set of steps and/or input for the particular condition you want to test (including what needs to be done to prepare for the test case to be run). The third column is the expected results for an input/output oracle what is expected to come out of the black box based upon the input (as described in the description). An oracle is any program, process, or body of data that specified the expected outcome of a set of tests as applied to a tested object; and input/output oracle is an oracle that specifies the expected output for a specified input. In the last column, the actual results are recorded after the tests are run. If a test passes, the actual results will indicate Pass. If a test fails, it is helpful to record Fail and a description of the failure (what came out) in the actual results column.