Basic Software Testing Skills: Logigear Certified Test Professional I
Basic Software Testing Skills: Logigear Certified Test Professional I
Basic Software Testing Skills: Logigear Certified Test Professional I
Agenda
Chapter 1: Software Development Life Cycles and Testing Chapter 2: Software Testing Overview
CHAPTER 1
CHAPTER 1
OUTLINE
SDLC and Testing Waterfall Model Spiral Model V-Model Concurrent Model Agile Model Other SDLC Models Testing Phases and Milestones
CHAPTER 1
OBJECTIVES
An introduction to Common Software Development Life Cycle (SDLC) practices and how software testing fits in those contexts. Understanding testing phases, milestones, checkpoints in a SDLC and their purposes Learn about how each SDLC choice affects software testing and its implementation
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
How testing is directly affected by the SDLC choice: Documentation availability to test against Time to test Time to automate or produce effective automation Knowledge and understanding of the application as we plan our tests Amount of regression testing
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
Waterfall Model
CHAPTER 1.2
Requirements Definition verify Functional Design verify Technical Design verify Coding verify Testing verify Deployment
The Waterfall model is a sequential software development model. Transition between phases is done by a formal review. The review is a checkpoint to see that you are on the right track.
CHAPTER 1.2
Testing is not inherent to every phase of the Waterfall model. Constant testing from the design, implementation and verification phases is required to validate the phases preceding them. Testing phase: Only after coding phase, software testing begins. Different testing methods are available to detect the bugs that were committed during the previous phases. A number of testing tools and methods are already available for testing purposes.
10
CHAPTER1.2
Requirements Definition verify Functional Design verify
11
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
12
CHAPTER 1.3
Spiral Model
Planned and structured releases Usually has documentation to test against Each spiral iteration can be thought of as a miniwaterfall; there are defined testing phases Previous releases must be regression tested
14
CHAPTER 1.4
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
15
CHAPTER 1.4
V-Model
16
Testing in V-Model
CHAPTER 1.4 Verification: Checks that a deliverable is complete (Contains all requires information, follows standards; verify a procedure). Validation: Checks that the deliverables satisfy requirements specified in the previous stage or an earlier stage, and that the business case is met (Validate a function or requirement). Testing: Ensures that the specification is properly assembled and implemented (Test to see if it works). In testing, these are important changes in software development from the V-model: Write the test cases at requirements review. Unit testing
17
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
18
Concurrent Model
CHAPTER 1.5
None
Under development
Analysis activity
Under review
Baselined
CHAPTER 1.5
Planning is concurrent to design; design is concurrent to developmenteverything is happening at the same time. The whole project is not well planned or well structured. Planning, design and development are most dynamic. Product is in constant change. Very difficult to test; impossible to effectively plan testing project. Often no documentation to test against. Testing is ad hoc. Coverage usually cannot be measured. Structured regression testing is impossible. Bugs will be missed because of so much change and so little planning. Risk analysis and reporting is crucial.
20
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
21
CHAPTER 1.6
Agile Model
Delivery cycles are short. Development is very focused. Diagrams, use cases, user stories, index cards, or discussions of functionality serve as documentation to test against. Agile projects usually have more unit testing by developers. Dynamic nature of development needs structured regression testing. Testing is often ad hoc, but focused. Dynamic style or projects include much change but the change is discussed and side-effects noted.
23
CHAPTER 1.7
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
24
CHAPTER 1.7
Rapid Application Development (RAD) Model Test First Rational Unified Process (RUP) eXtreme Programming (XP) the list goes on
25
CHAPTER 1.8
1.1 SDLC and Testing 1.2 Waterfall Model 1.3 Spiral Model 1.4 V-Model 1.5 Concurrent Model 1.6 Agile Model 1.7 Other SDLC Models 1.8 Testing Phases and Milestones
26
Development Phase: A time block within SDLC. There are a number of activities that have to be done in this time block. Examples of some Test Phases: Unit Test, Integration Test, System Test, User Acceptance Test, Release Test and Maintenance Test. Development Milestone: A significant event in the software developing process in which, the software moves from one phase to another. The milestone let us know where the application is in SDLC Criteria: Built or created from the standard of the project.
27
CHAPTER 1.8
Examples
Product Design and Specification Complete First Functionality Pre-Alpha or Alpha Candidate Alpha Pre-Beta or Beta Candidate Beta User Interface Freeze Pre-Final or Golden Master Candidate (GMC) Final Test Release or Golden Master
28
CHAPTER 1.8
Test Phase Pre Alpha Unit
Performed by Definition Developers Test one unit of code at a time
Developers/(Technical) Testers Testers Iteratively more complex system components integrated. Checking the units of code work together incrementally Bug Finding Big bang. The entire product as it is intended to be used
Outcome
Fewer Bugs
Bug Finding/QC
Goals of Testing
To find bugs, to learn about the product, to validate the integration requirements
To find bugs, to validate the system requirements, data flow, scenarios, load
29
Example
CHAPTER 1.8
Example Tasks and Activities for the Test Team: Test Plan reviewed Test Plan finalized Requirements reviewed and base-lined Test system approved and built Bug database opened and available for bug entry
30
Example
CHAPTER 1.8
The Test Plan covering all test phases: Source Level Testing, Application Testing, and Integration Testing are complete and approved. Developers testing sign-off document is complete Approved Unit Test Checklists are available for review Test Design/ Cases are complete and approved Build acceptance test is executed successfully Regression test has been defined Automation test has been defined (if applicable) The hardware and software necessary for the tests, as identified in the Test Plan, are available and ready
31
Example
CHAPTER 1.8
All test results have been documented All identified application testing has been executed Regression test has been executed Automation test has been executed (if applicable) All software fixes identified for resolution in this test phase are fixed, retested and closed Software is baselined Test report is completed All P1&P2 defects have been resolved and closed
32
CHAPTER 1
SUMMARY
Depending on the type of application to be built and or customers requirement, we choose a suitable SDLC model. There are several phases in the SDLC. In each phase, there are milestones and criteria for each milestone. The criteria, milestones and phases of the project help the product to be developed correctly, timely and effectively. Depending on the SDLC, the testing and its implementation is affected, usually in time, knowledge about the application, documentation and amount of regression test done.
33
CHAPTER 2
CHAPTER 2
OUTLINE
Testing Group Objectives of Testing Test Planning Framework Test Coverage Manual Testing vs. Automated Testing Key Lessons-Learned
35
OBJECTIVES
CHAPTER 2
An introduction to the concept of quality assurance (QA), quality control (QC), and software testing The roles of software testing groups Objectives of testing and some testing issues An introduction to Manual Testing and Automated Testing An introduction to LogiGear Software Test Planning Framework and possible test strategies, approaches, methods, types and techniques used in various software application development phases, for specific quality objectives
36
CHAPTER 2.1
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
37
Testing, QA, QC
CHAPTER 2.1 Quality Assurance: A process for providing adequate assurance that the software products and processes in the product life cycle conform to their specific requirements and adhere to their established plans. Quality Control: A set of activities designed to evaluate a developed working product. Testing: The process of executing a system with the intent of finding defects including test planning prior to the execution of the test cases. The key difference to remember is that Quality Assurance is interested in the process whereas Testing and Quality Control are interested in the product. Having a testing component in your development process demonstrates a higher degree of quality (as in QA).
38
39
CHAPTER 2.1
Find Problems
Find bugs. Find design issues. Find more efficient ways to find bugs.
Communicate Problems
Report bugs and design issues. Report on testing progress. Evaluate and report the programs stability.
40
CHAPTER 2.2
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
41
The programs specification: This program is designed to add two numbers, which you will enter Each number should be one or two digits
42
43
44
CHAPTER 2.2
There are 199 x 199 = 39,601 test cases for valid values:
definitely valid: might be valid: 0 to 99 -99 to -1
45
CHAPTER 2.2
Equivalence ClassTwo tests belong to the same equivalence class if the expected result of each is the same. Executing multiple tests of the same equivalence class is by definition, redundant testing. BoundariesMark the point or zone of transition from one equivalence class to another. The program is more susceptible to errors a the boundary conditions. Therefore, these are powerful sets of tests within the equivalent class to use. Generally, each class is partitioned by the boundary values. Nevertheless, not all equivalence classes have boundaries. For example, the browser equivalence classes consist of Netscape Navigator class and Microsoft Internet Explorer class.
Read Testing Applications on the Web, 2nd Edition, Wiley for more information
46
CHAPTER 2.2
Test Design
8*
1
Partitions Valid Inputs
9*
2
LB+1 Lower Boundary (LB) LB-1
For each equivalence class partition, well have at most, 9 tests to execute. It is essential to understand that each identified equivalence class represents a specific risk that it may pose.
2009 LogiGear Corporation All Rights Reserved
47
CHAPTER 2.2
Test Design
General Steps
Identify the classes Identify the boundaries Identify the expected output(s) for valid input(s) Identify the expected error-handling for invalid inputs Generate a table of tests (maximum, 9 tests for each partition of each class).
48
CHAPTER 2.2
Test Design
Example 1:
49
CHAPTER 2.2
A programmer implements the range 1 to 12 at an input, which stands for the month January to December in a date, has in his code a line checking for this range. This may look like: if (month > 0 && month < 13) But a common programming error may check a wrong range, starting the range at 0 by writing: if (month >= 0 && month < 13) For more complex range checks in a program this may be a problem which is not so easily spotted as in the above simple example. Applying boundary value analysis To set up boundary value analysis test cases you first have to determine which boundaries you have at the interface of a software component. This has to be done by applying the equivalence class technique. Boundary value analysis and equivalence partitioning are inevitably linked together. For the example of the month in a date you would have the following partitions: ... -2 -1 0 1 ............. 12 13 14 15 .....
--------------|-------------------|--------------------invalid partition 1 valid partition invalid partition 2 2009 LogiGear Corporation All Rights Reserved
50
Objectives of Testing
CHAPTER 2.2
Your objective should not be to verify that the program works correctly because:
If you cant test the program completely, you cant verify that it works correctly. The program doesnt work correctly, so you cant verify that it does. Then as a tester, you fail to reach your goal every time you find an error. Youll be more likely to miss problems than if you want and expect the program to fail.
51
CHAPTER 2.2
Objectives of Testing
52
CHAPTER 2.2
Objectives of Testing
53
CHAPTER 2.2
SUMMARY
We want to cover as many areas of the application as possible within the time constraint. We want to find the most serious bugs as quickly as possible. We want to be very familiar the test design technique called equivalent class and boundary condition analysis. It is an effective way to find bugs quickly.
2009 LogiGear Corporation All Rights Reserved
54
CHAPTER 2.3
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
55
CHAPTER 2.3
In glass box testing, the tester (usually the programmer) uses his/her knowledge of the source code to create test cases. - Hung Nguyen Tests design based on the knowledge of the code to exercising the code - Doug Hoffman Glass-box Test Case example:
Testing the piece of code for the Yahoo login functionality. This test type is usually executed by the developer.
56
Black-box Testing
CHAPTER 2.3
Black box tests execute the running program, without reference to the underlying code. This is testing from the customers view rather than from the programmers. - Hung Nguyen Focus on input, outputs, and an externally derived theory of operation - Cem Kaner Black-box Test Case example: Testing the Yahoo Mail login:
57
Gray-Box Testing
CHAPTER 2.3
Gray-box testing consists of methods and tools derived from the knowledge of the application internals and the environment with which it interacts, that can be applied in black-box testing to enhance testing productivity, bug finding and bug analyzing efficiency. - Hung Nguyen Testing involving inputs, outputs, but test design is educated by information about the code and the program operation of a kind that would normally be out of scope of the view of the tester. - Cem Kaner Gray-box Test Case example: Testing the Yahoo login with HTML commands
58
Testing Effectiveness
CHAPTER 2.3
Technical Knowledge
59
CHAPTER 2.3
T E S T
S T R A T E G Y
Determine
Select Select
Bug Types
T E S T
P L A N
T E S T
C A S E S
Phases
Apply Test
Select
60
Test Strategy
CHAPTER 2.3
Definition: A holistic plan that starts with a clear understanding of the core objective of testing from which, we derive a structure for testing by selecting from many testing styles and approaches available to help us meet our objectives. There are three focal points (SP3) that one can make adjustment toPeople, process, and practice (methods and tools) to get the best results. Process means the ways of doing things that people have agreed to. Practice means a successful combination of testing methods and tools that help meet the agreed objectives. People means the human resource that is capable of applying the practice. The description of key factors which are important during the preparation and execution of the test. Test Goal Why are we testing?
Minimize number of defects released to production? Validate Requirement? Find Bugs to improve quality? Find Design issues? Maximize customer satisfaction, by giving the customers the features they want? Other?
61
CHAPTER 2.3
Test Strategy
Goals for testing usually fall into 2 categories: Bug finding or validation. Once you have settled upon the goal for the test project, you can strategize how you will accomplish that goal. The test strategy describes how you will test the application.
62
CHAPTER 2.3
Quality Objectives
Definition: Quality objective is concrete requirement term that specifies the type of the quality objective and the level of acceptability for each.
Example: Usability, security, environment, communication, stress, volume, functionality, operations, recovery and performance
63
CHAPTER 2.3
Test Approaches
Definition: A testing philosophy or style that drives the selection of certain test design methods.
Examples: Requirement-based, Exploratory, Scenario-based, Modelbased, Attack-based, Risk-based, Fault Injection, DAST, etc.
64
CHAPTER 2.3
Test Types
Definition: A category of test activities with the objective of exposing specific types of bugs at certain interface, applicable for various level of product maturity. The selection of the test types must satisfy the test or quality objectives.
Examples: Requirement Review, Design Review, Code Review, Code Walkthrough, Design Walkthrough, Unit, Module, API, External Functional, Usability, Accessibility, Configuration, Compatibility, I18N, L10N, Regression , Performance, Load, Stress, Soak, Failover/Recovery, Installation, Security, Data verification testing, Compliance, etc.
(will be discussed more in later session)
65
Bug Types
CHAPTER 2.3
66
CHAPTER 2.3
Targeted Interface
Definition: A boundary across which, two independent systems meet and act on or communicate with each other. In computer technology, there are several types of interfaces. (www.webopedia.com)
Examples: Source-level (static testing such as code review and analysis), Calling Functions and Classes, API, GUI, CLI
67
CHAPTER 2.3
Definition: The state or quality of being stable of a software application. In software development, a stable application or functionality in an application is one that is reliable and dependable, and will consistently produce the same expected outputs given on the same set of inputs and conditions.
Example: Software application which has passed the alpha testing phase is more stable than when it was in the beginning of the alpha testing phase.
68
CHAPTER 2.3
Test Phases
Definition: A test phase is a time block often derived from the SDLC phases. Within each SDLC phase, there might be more than one test phase. Each test phase has different objectives related to bug finding or validation, described in each section. Testing staff participates in ongoing project management activities.
Examples: Pre-coding, Coding, Pre-integration, System, System, Integration, Final, Beta/User Acceptance, Post, Production
69
CHAPTER 2.3
Definition: A test design method is a systematic procedure in which you create tests to be executed. A test type or test approach may use one or more test design methods.
Examples:
Boundary condition and equivalent class analysis Decision-tree/table (variables in combination), Combinatorial design State-transitioning (will be discussed more in later session)
70
CHAPTER 2.3
Test Tools
Examples: TestArchitec for Action-Based Test (keyword test automation for functionality and regression testing) Webload for load and performance testing InCtrl for environment change analysis
71
CHAPTER 2.3
Test Plan
Definition: A management document outlining risks, priorities, and schedules for testing. A document prescribing the approach to be taken for intended testing activities.
72
Test Case
CHAPTER 2.3
Definition: A specific of test data and associated procedures developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement (IEEE 729-1983) Definition: A test that (ideally) executes a single well defined test objective (i.e., a specific behavior of a feature under a specific condition) (Testing Computer Software Kaner, Falk, Nguyen) will be discussed more in a later session.
73
CHAPTER 2.3
SUMMARY
There are 3 common paradigms in testing engineering: glass-box testing, black-box testing, white-box testing.
In glass box testing, the tester (usually the programmer) uses his/her knowledge of the source code to create test cases Black box tests execute the running program, without reference to the underlying code. This is testing from the customers view rather than from the programmers Gray-box testing consists of methods and tools derived from the knowledge of the application internals and the environment with which it interacts, that can be applied in black-box testing to enhance testing productivity, bug finding and bug analyzing efficiency.
LogiGear uses a Test Planning Framework to build testing strategy. In which, we determine Quality Objectives, select Test Approaches, Test Types, Bug Type, Targeted Interface, Application Stability Requirements, determine Execution Test Phases, apply Test Design Methods/Techniques, select Test Tools.to write test plan and test cases
74
CHAPTER 2.4
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
75
Test Coverage
CHAPTER 2.4
We know: Complete Testing is impossible. There are too many tests to run. We have limited time. So, what do we test? How deep? Where? With what data? On what platforms?
76
Coverage Review
CHAPTER 2.4
Coverage measures of the amount of testing done of a certain type. Since testing is done to find bugs, coverage is a measure of your effort to detect a certain class of potential errors.
So what kind(s) and level(s) of coverage are considered appropriate? There is no magic answer.
77
More on Coverage
CHAPTER 2.4
Basic or simplistic coverage is the percentage of product tested based on some aspect of the project.
78
Test Coverage
CHAPTER 2.4
Lets look at possible coverage metrics, or measurements of how much is tested and how much is not.
functionality requirements functionality screens windows use cases forms user scenarios data platforms modules lines of code number of APIs # of parameters of each API test cases data platforms
External
Quantity of testing
test cases
Internal
79
Code Coverage
CHAPTER 2.4
An analysis method that determines which parts of the software have been executed (covered) by the test case or suite and which parts have not been executed and therefore, may require additional attention. Various flavor of coverage: Function coverage - Has each function in the program been executed? Statement coverage - Has each line of the source code been executed? Condition coverage - Has each evaluation point (such as a true/false decision) been executed? Path coverage - Has every possible route through a given part of the code been executed? Entry/exit coverage - Has every possible call and return of the function been executed?
80
Problems of Coverage
CHAPTER 2.4
IEEE Unit Testing Standard is 100% Statement Coverage 100% Branch Execution (IEEE Std. 982.1-1988, 4.17, Minimal Unit Test Case Determination) Most companies do not achieve this. Several people seem to believe that complete statement and branch coverage means complete testing. Or at least, sufficient testing. It is a useful measurement but not a perfect measure of complete coverage.
81
Problems of Coverage
CHAPTER 2.4
A common metric requested from test teams today is test coverage. There are often problems with this metric. If you have requirements traceability (mapping) or dynamic code coverage measurements from test case execution, test case coverage has meaning. Other than that, test case coverage can give a false sense of security. What are the test cases tied or linked to, if not requirements or code? User Scenarios? Paths (workflows)? What? Were the test cases approved as being adequate to cover the development for this release?
82
CHAPTER 2.5
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
83
CHAPTER 2.5
Manual Testing is performed by carrying out all the actions manually against the application under test (AUT), step by step and revealing whether a particular step was completed successfully or failing. Manual testing is always a part of any testing effort. It is especially useful in the initial phase of software development, when the software and its user interface are not stable enough, thus test automation does not make sense.
84
CHAPTER 2.5
Automated Testing Using of software to control the execution of tests, and the comparison of actual outcomes with the expected (specified) outcomes. Setting up test-preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
(http://en.wikipedia.org/wiki/Test_automation)
Software testing assisted by software tools that enable test engineers to preprescribe test cases: (1) the input data and the interfaces of the UAT which will receive the data, (2) the expected/referenced output data value, (3) where to extract the output data from the UAT interfaces, and which output data will be compared against the referenced output data to determine passed/failed. At runtime, the software tools will read the pre-prescribed test cases, execute them, analyze and evaluate the results to determine passed/failed automatically. Common application of automated testing includes, but not limited to functional and/or regression testing. (LogiGear)
85
CHAPTER 2.6
An Overview of Testing
2.1 Testing Group 2.2 Objectives of Testing 2.3 Test Planning Framework 2.4 Test Coverage 2.5 Manual Testing vs. Automated Testing 2.6 Key Lessons-Learned
86
Key Lessons-Learned
CHAPTER 2.6
The Program Doesnt Work. Your task is to find errors. Be Methodical Complete Testing is Impossible Use Powerful Representatives of Tests Communication Must be Effective Change is Normal
87
All programs have bugs. Nobody would pay you to test if their program didnt have bugs. Any change to a program can cause new bugs, and any aspect of the program can be broken. You DONT verify that the program is working. You FIND bugs.
If you set your mind to show that a program works correctly, youll be more likely to miss problems than if you want and expect the program to fail.
88
Be Methodical
CHAPTER 2.6
Testing isnt just banging away at the keyboard. To have any hope of doing an efficient job, you must be methodical: Break the testing project down into tasks and subtasks so that you have a good idea of what has to be done and what has been done. Track what has been done so that people dont repeat the same tasks and you know what tasks are left. Prioritize the tasks.
89
CHAPTER 2.6
There are a nearly infinite number of paths through any nontrivial program. There are a virtually infinite set of combinations of data that you can feed the program. You cant test them all. Therefore, your task is to find bugs -- not to find all the bugs. You want to Find as many bugs as possible Find the most serious bugs Find bugs as early as possible
90
CHAPTER 2.6
Develop powerful tests by analyzing Equivalence classes Boundary conditions Input combinations Data/functionality relationships (These will be discussed in later sections)
91
CHAPTER 2.6
Your bug reports advise people of difficult situations. The problems that you report can affect The project schedule The companys cash flow
The clearer your reports are, the more likely it will be that the company will make reasonable business decisions based on them.
Persuasive and technical writing, oral argument, face-to-face negotiation, and diplomacy are core skills for your job.
92
Change is Normal
CHAPTER 2.6
Project requirements change, as do design and specifications. The market changes. Development groups come to understand the product more thoroughly as it is being built. Some companies accept late changes into their products as a matter of course. As a new tester, you might decide quickly that this is a bad, amateurish way to do business. It might be, and it might not be.
Take steps to make yourself more effective in dealing with late changes.
93
CHAPTER 2
SUMMARY
Testing Group includes QA, QC, and Testing staff who perform very important services such as: find and report bugs, communicate problems, and evaluate the product quality...
94
CHAPTER 3
Test Requirement
CHAPTER 3
OUTLINE
Products Document What is Test Requirement (TR)? Test Requirement Essentials Requirement Attributes
96
CHAPTER 3
OBJECTIVES
An introduction to requirement, specification and design An introduction to test requirements and how to use test requirement to implement test case
97
CHAPTER 3.1
Test Requirement
3.1 Products Document 3.2 What is Test Requirement (TR)? 3.3 Test Requirement Essentials 3.4 Requirement Attributes
98
CHAPTER 3.1
Products Documents
Definitions: Requirements - the customer's problem or desires Specification - what you intend to make to solve the problem or meet the desire Design - how the thing works Examples: Requirement: the room should be warm Specification: The room will be kept between 25 and 28 degrees. Design: Heat pump, augmented by electric heater
99
CHAPTER 3.2
Test Requirement
3.1 Products Document 3.2 What is Test Requirement (TR)? 3.3 Test Requirement Essentials 3.4 Requirement Attributes
100
Test Requirement
CHAPTER 3.2
A statement of what should be tested against the AUT Functional Requirement: The requirement for the functions that the application should do Non-functional requirement:The requirement for the properties that the functions should have or should look like. There are three types of non-function TR: Look n feel Boundary Negative
101
CHAPTER 3.3
Test Requirement
3.1 Products Document 3.2 What is Test Requirement (TR)? 3.3 Test Requirement Essentials 3.4 Requirement Attributes
102
CHAPTER 3.3
The formats can be very different. What they look like is unique to every company and writer. They are hopefully full of useful information. Target Platform System Requirements A description of the functionality Intended Use Expected Problem Areas
103
CHAPTER 3.4
Test Requirement
3.1 Products Document 3.2 What is Test Requirement (TR)? 3.3 Test Requirement Essentials 3.4 Requirement Attributes
104
CHAPTER 3.4
Boehm (1984) lists the attributes of good software requirements Written Specific Complete Correct Feasible Consistent Prioritized Clear/ Unambiguous Verifiable (key attribute) Concise In Understandable language
105
CHAPTER 3
EXAMPLE
Can save a file after opening a new file Can save a file with text (use your name) and text with space, such as michealhackett and micheal hackett Can save in the following format XML Web page Plain text Works 6
106
CHAPTER 3
Application Under Test: Yahoo Mail
EXERCISE
Lets practice create some Test Requirements for the Yahoo Mail. Below is an example.
Exercise: Please create 3 more Test Requirements for the Yahoo Mail.
107
CHAPTER 3
SUMMARY
Usually, there are three types of project document: Requirement, specification and design. A test requirement can be written from a user story or from other sources of the software. Tester uses test requirement to implement test cases.
(We will learn more about how to create a test requirements from user story in LCTP II)
108
CHAPTER 4
CHAPTER 4
OUTLINE
Test Case
Test case criteria Test case essentials Test case syntax
110
CHAPTER 4
OBJECTIVES
An introduction to commonly used test methods, test types and how to apply them to test an application An overview of test case and some techniques to design test case
111
112
Test Method
CHAPTER 4.1
Definitions: A definitive procedure for the identification, measurement, and evaluation of a material, product, system, or service that produces a test result. (ASTM American Society for Testing and Materials)
113
CHAPTER 4.1
Test approaches:
Requirement-based Testing (Passive Testing) Exploratory Testing (Ad Hoc Testing) . The best practice is a combination of a few methods to satisfy the project needs
Test Types
Regression Testing Smoke Testing Stress Testing Load Testing Performance Testing
114
CHAPTER 4.1
Requirement-based Testing
The primary goal of Requirement-based Testing is Requirement Validation. It is not experimental, behavioral, exploratory or a bugfinding exercise in nature. It validates.
115
CHAPTER 4.1
Requirement-based Testing
The Mechanical Approach for Test development: Analyze requirements Gather more information Make at least a test case for every requirement Execute the test cases
116
Exploratory Testing
CHAPTER 4.1
Exploratory Testing can have many names: Ad hoc, Discovery, Error Guessing
The primary goal of exploratory testing is to uncover new defects in the product on-the-fly.
The task is finding ways to creatively design experiments, exploring, experimenting, going from a known place to an unknown place.
117
Exploratory Testing
CHAPTER 4.1
Exploratory Testing is not requirement-driven but it is intense testing. The goal is to probe for weak areas of the program. Some various error conditions should be checked are:
Out of boundary Null input and overflow (storage of the sum) Non-numeric Hardware and memory error handling
Exploratory Testing is the testing approach which can produce the highest number of bugs
118
Regression Testing
CHAPTER 4.1
Most common type of work we do! Re-run Quality Control- not testing, not bug finding Expensive First to automate
119
Regression Testing
CHAPTER 4.1
Regression Testing: The testing that is performed after making a functional improvement or repair to the program. Regression Purpose:
Selectively re-run the test cases to verify that the modifications have not caused any unintended effects and that system still complies with its specified requirements. Re-run the test cases to make sure that the bugs marked as fixed by developers are really fixed.
Re-test all test cases can be done using automated testing tools.
120
Regression Testing
CHAPTER 4.1
Some of the most popular strategies for selecting regression test suites:
Retest all: Rerun all test cases. Simple but impossible in the time that we have in our everyday practice. Retest Risky Use Cases: Choose baseline tests to re-run by risk heuristics. Retest by Profile: Choose baseline tests to re-run by allocating time in proportion to operational profile. Retest Changed Segment: Choose baseline tests to re-run by comparing code changes.
121
Smoke Testing
CHAPTER 4.1 A smoke test is a collection of written tests that are performed on a system prior to being accepted for further testing, also known as build verification test. This is a "shallow and wide" approach to the application. The tester "touches" all areas of the application without getting too deep, looking for answers to basic questions like, "Can I launch the test item at all?", "Does it open to a window?", "Do the buttons on the window do things?". There is no need to get down to field validation or business flows. If you get a "No" answer to basic questions like these, then the application is so badly broken, there's effectively nothing there to allow further testing.
These written tests can either be performed manually or using an automated tool.
122
CHAPTER 4.1
Acceptance (into-testing) Test Feature-Level Test Regression Test Configuration & Compatibility Test Documentation and Online Help Test Utilities/ Tool Kits and Collateral Test Install/ Uninstall Test Integrity & Release Testing Import/ Export User Interface Test Usability test Performance Test Benchmark Test Acceptance by Customer Beta Testing Initial Stability Assessment
123
SUMMARY
CHAPTER 4.1
124
125
CHAPTER 4.2
Test Case
Definition: A test that (ideally) executes a single well defined test objective (i.e., a specific behavior of a feature under a specific condition) (Testing Computer Software Kaner, Falk & Nguyen) Definition: A specific set of test data and associated procedures developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement (IEEE 729-1983)
126
CHAPTER 4.2
Accountability Reproducibility Tracking Automation To find bugs To verify that tests are being executed correctly Use as a Training Tool for new testers For compliance To measure test coverage
127
CHAPTER 4.2
128
CHAPTER 4.2
A Good Test:
Refers and maintains tight control over specific test data Has detailed enough Test Design Steps so that any testers with basic knowledge of the system can execute the test Is aware of the Testers experience Has clear criteria for Pass or Fail
A Not-so-Good Test
Leaves it up to the user to find test data Gives very high level instructions that leave too much room for artistic interpretation Does not consider the Testers experience Leaves out follow-up verification steps which make it difficult to determine Pass or Fail criteria.
129
130
CHAPTER 4.2
The most important part of a test case is the one-line title describing the objective of the test That one-line title can be called:
Test Title Test Name Test Case Test Objective Test Goal/Purpose
131
CHAPTER 4.2
132
CHAPTER 4.2
133
CHAPTER 4.2
134
Validation Points
CHAPTER 4.2
135
136
CHAPTER 4.3
Equivalence Partitioning Boundary Value Analysis State Transition/ Model-Based Testing Cause-Effect Grapping Syntax Testing Statement Testing Branch/Decision Testing Data Flow Testing Branch Condition Testing Branch Condition Combination Testing Modified Condition Decision Testing LCSAJ Testing (Linear Code Sequence and Jump) Random (Ad hoc) Testing
137
CHAPTER 4.3
Equivalence Class: Two tests belong to the same equivalence class if the expected result of each is the same. Executing multiple test cases of the same equivalence class is by definition, redundant testing. Boundaries: Mark the point or zone of transition from one equivalence class to another. The program is more susceptible to errors a the boundary conditions. Therefore, these are powerful sets of test cases within the equivalent class to use. Generally, each class is partitioned by the boundary values. However, not all equivalence classes have boundaries. For example, the browser equivalence classes consist of Netscape Navigator class and Microsoft Internet Explorer class.
It is important to recognize that two tests are equivalent only with respect to a specific risk.
138
CHAPTER 4.3
For each equivalence class partition, well have at most, 9 test cases to execute. It is essential to understand that each identified equivalence class represents a specific risk that it may pose.
* Smallest/Largest Possible Values Allowed via UI
8*
9*
2
LB+1 Lower Boundary (LB) LB-1 UB+1
5
3
Upper Boundary (UB) UB-1
140
CHAPTER 4.3
General Steps Identify the classes Identify the boundaries Identify the expected output(s) for valid input(s) Identify the expected error-handling (EH) for invalid inputs Generate a table of test cases (maximum, 9 test cases for each partition of each class).
141
CHAPTER 4.3
Other Equivalence Classes and Special Cases Negative values Number of digit or characters 0 Non-printable characters Upper ASCII (128-254) O/S reserved characters Space Nothing etc. See Input Dialog Test Matrix
142
Character Groupings
CHAPTER 4.3
Low ASCII These are the Ctrl keys. Interesting ones include 000-null, 007-beep, 008-BS, 009-Tab, 010-LF, 011-VT/home, 012-FF, 013CR, 026-EOF (end of file, very nasty sometimes), 027-ESC Low non-alphanumeric (ASCII codes 32-47) space ! # $ % & ( ) * + , - . / Non-alphanumeric, standard, printing ASCII characters. We often lump these together even though theyre in four distinct groups. Digits Upper and lower case alpha Upper ASCII Modifier keys These keys include (depending on the keyboard) Alt, Right-Alt, Shift, Control, Command, Option, Left-Amiga, Right-Amiga, Open-Apple, etc. They generally have no effect when pressed alone, but when pressed in conjunction with some other key, they create a It often pays to test all the interesting standard values, plus a sample of others. It is often best to assign a separate chart column to each modifier key, i.e. one column for Ctrl, one for Shift, etc. Function keys Cursor keys Numeric keypad keys European keyboards Test them alone and in combination with the modifier keys. Test them alone and in combination with the modifier keys. Its common for every modifier key to have a different effect on cursor keys. These are not necessarily equivalent to number keys elsewhere on the keyboard. The left and right Alt keys often have different effects on non-English keyboards. Also, these keyboards provide deal keys -- you press a dead key to specify an accent, then press a letter key and (if its a valid character) the computer displays that let Intermediate non-alphanumeric (ASCII 58-64) : ; < = > ? @ More intermediates (ASCII 91-96) Top of standard ASCII (ASCII 123-126) (ASCII 48-57) (ASCII 65-90) (ASCII 97-122) (ASCII 128-254) 0123456789 A B C D E F G H I J K L M N O P Q R S T U V W XY Z a b c d e f g h i j k l m n o p q r s t u vw x y z These subdivide further depending on the character set, the users language, and the application. [ \ ] ^_ ` {|}~
143
144
Constraint Analysis
CHAPTER 4.3
Constraint Analysis
Equivalence class and boundary analysis look at each variable (field) in isolation. This is important, but we also have to look at the relationship between different variables. Cause-Effect Graphing is one approach to this, but it can be complicated. Here is a simpler approach that is still quite valuable. Identify the dependent variables and study their respective constraints. Identify all possible input and out put interfaces for the variables List the variables in a tabular format with their input/output interfaces and their respective constraints.
General Steps
145
CHAPTER 4.3
Data Relationships
Look at Edit Event dialog box from the Palm Desktop Software.
Note that the End Date for repeating this event is before the Start Date. That is not possible.
146
CHAPTER 4.3
Data Relationships
The program checks the End Date against the Start Date and rejects this pair as impossible because the task cant end before it starts. The value of End Date is constrained by Start Date, because End Date cant be earlier than Start Date.
The value of Start Date constrains End Date, because End Date cant be earlier than Start Date.
2009 LogiGear Corporation All Rights Reserved
147
CHAPTER 4.3
Data Relationships
This table shows how the variables are related. Here are the columns:
Field: End Date, and Start Date are examples of fields. Entry Source (Also known as input interface): What dialog boxes can you use to enter data into this field? Can you import data into this field? Can data be calculated into this field? List every way to fill the field -every screen, etc.
2009 LogiGear Corporation All Rights Reserved
148
Data Relationships
CHAPTER 4.3
Display (Also known as output interface): List every dialog box, error message window, etc., that can display the value of this field. When you re-enter a value into this field, will the new entry show up in each screen that displays the field? (Not always -- sometimes the program makes local copies of variables and fails to update them.) Print (Also known as output interface): List all the reports that print the value of this field (and any other functions that print the value). Constrained By: List every variable that constrains the value of this variable. (What if you enter a legal value into this variable, then change the value of a constraining variable to something that is incompatible with this variables value?) Constraint: List every variable that this one constrains.
149
Functionality Relationships
CHAPTER 4.3
The Let Windows manage my virtual memory settings radio button constrains the availability of the features in the Let me specify my own virtual memory settings group.
Inactive Features
Active Features
150
151
CHAPTER 4.3
State Transition
State Transitioning
Involves an analysis of the relationship among the states and the events or actions that cause the transitions from one state to another.
General Steps
Identify all supported states For each test define
The starting state The input events that cause the transitions The output results of each transition The end state
152
CHAPTER 4.3
Example 1
Navigation Command
153
CHAPTER 4.3
Example 1
Navigation Command Current View Mode
154
CHAPTER 4.3
Example 1
155
Transitioning Example
CHAPTER 4.3
156
157
Condition Combination
CHAPTER 4.3
Condition Combination
Involves an analysis of the combination relationship of the variables such as browser settings. Each combination represents a condition to be tested with the same test script or procedure.
General Steps
Identify the variables. Identify the number of possible unique values of each variable. Create a table illustrating the complete unique combination of conditions formed by the variables and their values.
2009 LogiGear Corporation All Rights Reserved
158
CHAPTER 4.3
159
CHAPTER 4.3
Test Design
160
EXAMPLE
From that Test Requirement, we can design a simple Test Case as below. TC-01: Login to Yahoo Mail with valid username/password.
161
CHAPTER 4
SUMMARY
There are many approaches to test an application: Requirement-based testing, exploratory testing, etc. The best practice is a combination of a few approaches to satisfy the project needs. Test case is written for testing an application. Some of the best techniques to designing test case are equivalent class partitioning and boundary analysis.
162
EXERCISE
Exercise: Please design a simple Test Case for one of the Test Requirements that you have created in the previous exercise.
163
CHAPTER 4
EXERCISE
164
CHAPTER 5
CHAPTER 5
OUTLINE
Test Case Tracking/Management System LogiGear TestArchitect Test Case Management System 3rd party Test Case Management System Spreadsheet-based Test Case Management System
166
CHAPTER 5
OBJECTIVES
What is a test case tracking/management system? An introduction to LogiGear TestArchitect test case management system An introduction to 3rd-party test case management systems An introduction to spreadsheet-based test case management system
167
CHAPTER 5.1
5.1 Test Case Tracking/Management System 5.2 LogiGear TestArchitect Test Case Management System 5.3 3rd-party Test Case Management Systems 5.4 Spreadsheet-based Test Case Management System
168
CHAPTER 5.1
Test case (TC) management system is the system to organize and control the process and artifacts required for the testing effort. The goal of TC management system is to allow teams to plan, develop, execute, and assess all testing activities within the overall software development effort. This includes coordinating efforts of all those involved in the testing effort, tracking dependencies and relationships among test assets and, most importantly, defining, measuring, and tracking quality goals.
169
CHAPTER 5.1
Make the test process more transparent. QA and QC teams can manage their entire testing process. Ease of use Manage execution Manage change Easier test case maintenance Requirements traceability Various execution, result and performance Measurements Coverage Metrics Maximize time to market in test planning, execution, and results analysis. Management can monitor and examine the results and status of testing. Tests and results can be shared with vendors and customers.
170
CHAPTER 5.2
5.1 Test Case Tracking/Management System 5.2 LogiGear TestArchitect Test Case Management System 5.3 3rd-party Test Case Management Systems 5.4 Spreadsheet-based Test Case Management System
171
CHAPTER 5.2
In which:
Test Module: An individual spreadsheet containing one or more related test cases. Test Requirement: An explicit statement of the purpose of a test, based on product requirements or other quality criteria. Test Case: A series of steps and verifications validating one or more test requirements
172
CHAPTER 5.2
EXAMPLE
173
CHAPTER 5.2
174
CHAPTER 5.3
5.1 Test Case Tracking/Management System 5.2 LogiGear TestArchitect Test Case Management System 5.3 3rd-party Test Case Management Systems 5.4 Spreadsheet-based Test Case Management System
175
CHAPTER 5.3
There are many 3rd party test management systems: Test Link QCIT (Quality Control Information Tool) Test Director Rational Test Manager Integrated with RequisitePro and ClearQuest Empirex e-Manager: Web Testing Manager
176
Test Link
CHAPTER 5.3
TestLink is web based Test Management system. Easily to create, manage Test cases and organize them into Test plans Test plans allow to execute Test cases and track test results dynamically, generate reports, trace software requirements, prioritize and assign tasks. The tool has web based interface with PHP and background database MySQL, Postgres or MS-SQL. It cooperates with known Bug tracking systems as is Bugzilla, Mantis, etc. TestLink is web based tool under the GPL license (free to use). The project is maintained by Open community of testers. Many developers on the team hold Quality Assurance Management positions and understand the needs of QA teams. TestLink makes Testing process easy and organized.
177
CHAPTER 5.3
179
Basic Terminology
CHAPTER 5.3
Test Case describes a testing task via steps (actions, scenario) and expected results. Test Cases are the fundamental piece of TestLink. Test Suite (Test Case Suite) organizes Test Cases to units. It structures Test Specification into logical parts. Test Plan is created when you'd like to execute Test Cases. Test Plans can be made up of the Test Cases from one or many Test Projects. Test Plan includes Builds, Milestones, user assignment and Test Results. Test Project is something that will exist forever in TestLink. Test Project will undergo many different versions throughout its lifetime. Test Project includes Test Specification with Test Cases, Requirements and Keywords. Users within the project have defined roles. User: each TestLink user has a Role that defines available TestLink features.
180
CHAPTER 5.3
General Features: Functionality helps to follow standard testing processes as are specified by IEEE 829 or BCS SIGIST. User web based GUI (Mozilla, Firefox, IE 6 compatible) help user access it without installation. Multiple databases support TestLink runs with MySQL, PostgreSQL or MS-SQL. Localization and Internacialization (English, French, German, Italian, Spanish, etc.) Authentication support (internal or external LDAP) with user self-registration support Flexible role based access control Attachments and custom fields could be added to particular objects (for example Test Cases, Test results, etc.) Test Projects: Large team can divided the products to test projects. Each test project has own user rights, test specification, software requirements and Test plans. Multiple projects support Test cases are organized in hierarchal structure (tree menu) into Test Specification. History of each test case is traceable. Keywords are supported to bring more depth to test organization. Users have defined roles in project (for example leader, tester, etc.)
181
CHAPTER 5.3
Requirements based testing: Create or import own Software Requirement document and trace requirements to test results. Each test case can be mapped to a functional requirement. User can automatically generate Test Cases Import/Export Test Cases: Allows users to easily import and export test cases and product requirements and all of their test cases to and from XML file formats. Test Plans: Use TestLink to compose rich test plans containing an choosen set of test cases. Each Test Plans can collect test results for particular builds and platforms. Testing could be prioritized, assigned to testers, defined milestones. TestLink directly cooperates with Bug Tracking systems, i.e. Bugzilla, Mantis, Jira, TrackPlus, Eventum, Trac, Seapine, Redmine Generate Reports Allows to view a variety of different test plan reports including: a bug report, a progress report, and failure rate report, and more. Export of documentation to HTML, MS Word and MS Excel is supported. User can also send reports directly via email.
182
CHAPTER 5.3
183
CHAPTER 5.3
DEMO
http://testlink.org/demo/index.php
184
CHAPTER 5.4
5.1 Test Case Tracking/Management System 5.2 LogiGear TestArchitect Test Case Management System 5.3 3rd-party Test Case Management Systems 5.4 Spreadsheet-based Test Case Management System
185
CHAPTER 5.4
A spreadsheet is a computer application simulates a paper worksheet. It displays multiple cells that together make up a grid consisting of rows and columns, each cell containing either alphanumeric text or numeric values. A spreadsheet cell may alternatively contain a formula that defines how the contents of that cell is to be calculated from the contents of any other cell each time any cell is updated. Spreadsheets are frequently used for financial information because of their ability to re-calculate the entire sheet automatically after a change to a single cell is made. LogiGear staff usually uses Excel spreadsheet as a test case management system.
186
CHAPTER 5.4
EXAMPLE
187
CHAPTER 5
SUMMARY
There are many types of test case management systems. Here are three examples:
A test case manager that also interfaces with automation tool (TestArchitect) A 3rd-party test case manager
Basing on the requirements of the projects, we choose the suitable test case management system to use
188
CHAPTER 5
EXERCISE
189
CHAPTER 6
Software Error
CHAPTER 6
OUTLINE
What is a Software Error? Common Sources of Errors 13 Common Types of Software Errors Finding, Reproducing and Analyzing a Software Error Reporting a Software Error A Common Bug Life
191
CHAPTER 6
OBJECTIVES
An introduction to software error, common sources of software error and some common types of software errors Learn to find, analyze bugs and write bug report An introduction to a common bug life
192
CHAPTER 6.1
Software Error
6.1 What is a software error? 6.2 Common Sources of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
193
A software error is present when the program does not do what its user reasonably expects it to do. It is fair and reasonable to report any deviation from high quality as a software error. The existence of software errors reflects an impediment on the quality of the product, but does not necessarily imply that the developers are incompetent.
194
CHAPTER 6.2
Software Error
6.1 What is a software error? 6.2 Common Sources of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
195
196
CHAPTER 6.3
Software Error
6.1 What is a software error? 6.2 Common Sources of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
197
CHAPTER 6.3
User Interface Error Handling Boundary-Related Calculation Initial and Later States Control Flow Handling or Interpreting Data Race Conditions Load Conditions Hardware/Environment Compatibility Source, Version, and ID Control Testing Documentation Reference: http://www.logigear.com/resources/articles_lg/common_software_errors.asp
198
CHAPTER 6.4
Software Error
6.1 What is a software error? 6.2 Root Causes of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
199
CHAPTER 6.4
1. 2. 3.
200
CHAPTER 6.4
Some bugs are always reproducible, but some are just sometimes or even rarely. Bugs dont just miraculously happen and then go away. If a bug happens intermittently, it might be under some certain conditions. When we find a bug, we are looking at a failure, which is a set of symptoms of an underlying error. We hypothesize the cause, then we try to re-create the conditions that make the error visible. If the bug is non-reproducible, you should always report it, but describe your steps and observations precisely. Programmers will often figure them out.
201
CHAPTER 6.4
Memory dependent Memory corruption Run-time errors Predicated on corrupted data Configuration dependent Software Hardware Timing related Initialization Data flow dependent Control flow dependent Error condition dependent Multi-threading dependent Special cases Algorithm Dates
2009 LogiGear Corporation All Rights Reserved
202
CHAPTER 6.4
Write down everything you remember about what you did the first time. Note which things you are sure of and which are good guesses. Note what else you did before starting on the series of steps that led to this bug. Review similar problem reports youve come across before. Use tools such as capture/replay program, debugger, debuglogger, videotape, or monitoring utilities that can help you identify things that you did before running into the bug. Talk to the programmer and/or read the code.
203
CHAPTER 6.4
Why Analyze a Reproducible Bug? Analyze bugs in order to: Make your communication effective; Make sure you are reporting what you think you are reporting. Make sure that questionable side issues are thoroughly investigated. Create accountability. Support the making of business decisions; Avoid wasting the time of the programming and management staff; Find more bugs.
204
CHAPTER 6.4
Start by making sure the error is reproducible. 1) Describe how to get the program into a known state. For example, starting the program takes it into a known (just booted) state. Describe the state clearly, so that anyone familiar with the program will know what to do to get the program into that state. 2) Specify an exact series of steps that expose the problem. 3) Test your steps to make sure that you can reproduce the problem if you do exactly (and only) what it says in the bug report.
205
CHAPTER 6.5
Software Error
6.1 What is a software error? 6.2 Types of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
206
CHAPTER 6.5
207
Written (not reported orally) Uniquely numbered (ID required) Simple (non-compound - one bug per report) Understandable Reproducible Non-judgmental (only facts, no opinions)
208
209
Report Content
CHAPTER 6.5
Summary Description Steps to Reproduce including expected behavior and observed behavior Reproducible Severity Frequency Priority Keyword (Functional Area) Resolution
210
Bug Summary
CHAPTER 6.5 This one-line description of the problem is the most important part of the report. The project manager will use it when reviewing the list of bugs that havent been fixed. Executives will read it when reviewing the list of bugs that wont be fixed. They might only spend additional time on bugs with interesting summaries. The ideal summary tells the reader what the bug is, what caused the bug, what part of the program its from and what its worst consequence is. It runs from 8 to 15 words long. You might not fit all this information in 15 words, but you might fit in the most important parts.
We use the following syntax for writing the problem summary: Symptom + Action + Operating Condition
211
Bug Summary
CHAPTER 6.5
212
Bug Summary
CHAPTER 6.5
213
CHAPTER 6.5
First, describe the problem. What is the bug? Dont rely on the summary to do this a short line sometimes cannot state all what you want to say. Next, go through the steps that you use to recreate this bug. Start from a known place (e.g. boot the program) and then describe each step until you hit the bug.
Describe the erroneous behavior and if necessary, explain what should have happened. (Why is this a bug? Be clear.) If you expect the reader to have any trouble reproducing the bug (special circumstances are required), be clear about them.
214
A clear, reproducible set of steps may be all you need to file the report. But you may be able to improve the report in four ways: 1) You might be able to simplify the report by eliminating unnecessary or irrelevant steps. 2) You might be able to simplify the report by splitting it into two reports. 3) You might be able to strengthen the report by showing that it is more serious than it first appears. 4) You might be able to strengthen the report by showing that it is more general than it first appears.
215
CHAPTER 6.5
Sometimes it is not immediately obvious what steps can be dropped from a long sequence of steps in a bug. Look for critical steps -- Sometimes the first symptoms of an error are subtle. You made a list of all the steps that you took to show the error. You are now trying to shorten the list. If youve found what looks like a critical step, try to eliminate almost everything else from the bug report. Go directly from that step to the last one (or few) that shows the bug. In general, try taking out individual steps or small groups of steps. Can you show the bug with the others?
216
When you see two related problems, you might report them together on the same report as long as you show that there are two of them. But if this lengthens the report or makes it at all confusing, write two reports instead. If you would have to run two tests to verify that a bug has been fixed, its fair to report the two symptoms that you have to test on two separate bug reports. When you report related problems, its a courtesy to cross-reference them. For example: Related bug -- see Report # xxx
217
CHAPTER 6.5
Look for follow-up errors: Keep using the program after you get this problem. Does anything else happen? Sometimes a modest-looking bug can lead to a system crash or corrupted data. Look for nastier variants: Vary the conditions under which you got the bug. For example, try to get the bug while the program is doing a background save. Does that cause data loss or corruption along with this failure?
Look for nastier configurations: Sometimes a bug will show itself as more serious if you run the program with less memory, a higher resolution output, more graphics on a page, etc.
218
CHAPTER 6.5
Look for alternative paths to the same problem: Sometimes the bug can happen in some alternative path. For example, a bug that happens when deleting a file can happen when deleting a folder also. Look for configuration dependence: Sometimes the bug happens because of some hardware configuration. For example, a graphic/UI bug might have some tie to the graphic card driver, the graphic card manufacturer or the screen resolution.
219
Reproducible
CHAPTER 6.5
You may or may not have this on your form, but you should always provide this information. Never say it is reproducible unless you have recreated the bug. (Always try to recreate the bug before writing the report.) If you have tried and tried but you can not recreate the bug, say No. Then explain what steps you tried in your attempt to recreate it. If the bug appears sporadically and you do not yet know why, say sometimes and explain.
220
Severity
CHAPTER 6.5
You will have to rate the bugs seriousness. Many companies use a three-level rating: 1 - Critical: This means fatal to the release (unacceptable to ship) 2 - Serious: Its a bad bug, but it doesnt, for example, cause data loss or a program crash. 3 - Minor: Its a bug, but its not a big deal.
Your companys definitions may be a bit different from these. And you might have a five-level rating instead of three -- check with your manager.
Many companies sort their summary reports by severity, so you want to fill in this field thoughtfully.
221
Frequency
CHAPTER 6.5
With some bug tracking systems, you also have to rate the bug frequency. Frequency is usually graded by assessing the following three characteristics:
How easy is it for the user to encounter the bug How frequent would the user encounter the bug How often the buggy feature is used
222
CHAPTER 6.5
Priority
Priority rating is either automatically generated by the bug tracking system by assessing the Severity and the Frequency ratings or assigned only by the project manager. This is the fix-priority that everyone who is responsible for working on the bug will go by.
223
CHAPTER 6.5
You may have to categorize the bug according to its functional area. The tracking system should include a list of the possible keywords. Read the list and ask questions in order to learn what each category includes. It is important to categorize these bugs consistently. Theyll often be assigned out and summary-reported by category. Burying a bug in the wrong category can lead to its never getting fixed. If youre creating the list of functional areas, keep it short, perhaps 20 areas.
224
Resolution
CHAPTER 6.5 The project manager has the privilege to assign most of the resolutions in this field. Common resolutions include: New: The newly submitted bug To Be Distributed: The bug is waiting to be distributed To Be Fixed: The bug is being fixed. QA Info Request: The bug needs more clarification from Tester. Developer Info Request: The bug needs more clarification from Developer. Not Reproducible: The bug cannot be reproduced. Fixed: The bug is fixed. Not a Problem: The application works as it is supposed to. Duplicate: The bug is just a repeat of another bug. Deferred: The bug will be fixed in a later release. Feature Limitation: There is some feature limitations that do not allow to fix the bug.
225
CHAPTER 6.5
226
CHAPTER 6.6
Software Error
6.1 What is a software error? 6.2 Types of Errors 6.3 13 Common Types of Software Errors 6.4 Finding, Reproducing and Analyzing a Software Error 6.5 Reporting a Software Error 6.6 A Common Bug Life
227
228
CHAPTER 6
SUMMARY
A software error is present when the program does not do what its user reasonably expects it to do. Software Errors usually come from 5 common sources: Coding Error, Design Issue, Requirement Issue, Documentation/Code Mismatch, Specification/Code Mismatch There is a lot bugs in a software They can normally grouped into 13 types of software error. Bug report is the main product of tester. Before writing a bug report, a tester has to find, reproduce, analyze and finally report the bug.
2009 LogiGear Corporation All Rights Reserved
229
CHAPTER 6
EXERCISE
230
CHAPTER 7
CHAPTER 7
OUTLINE
Bug Tracking/Management System LogiGear TrackGear Bug Management System Reporting Bugs Using TrackGear
232
CHAPTER 7
OBJECTIVES
An introduction to a bug tracking/management system An introduction to LogiGear TrackGear bug management system Learn to report bugs using TrackGear
233
CHAPTER 7.1
7.1 Bug Tracking/Management System 7.2 LogiGear TrackGear Bug Management System 7.3 Reporting Bugs Using TrackGear
234
A bug tracking/management system is a software application designed to help testers or QA staff, project managers and programmers keep track of reported software bugs in their work. Many bug-tracking systems used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company doing software development. Typically bug tracking systems are integrated with other software project management applications. Having a bug tracking system is extremely valuable in software development, and they are used extensively by companies developing software products
235
The major component is a database records facts about known bugs. Facts may include the time a bug was reported, its severity, the erroneous program behavior, the details on how to reproduce the bug, the identity of the person who reported it and any programmers who may be working on fixing it Typical bug tracking systems support the concept of the life cycle for a bug which is tracked through status assigned to the bug. A bug tracking system allow administrators to configure permissions based on status, move the bug to another status, or delete the bug. The system also allow administrators to configure the bug statuses and to what status a bug in a particular status can be moved to.
236
CHAPTER 7.1
A bug-tracking system may be used to measure the productivity of programmers at fixing bugs (This is a bad idea because bugs are not equal).
A local bug tracker (LBT) is usually a computer program used by a team of application support professionals to keep track of issues communicated to software developers. Using an LBT allows:
Support professionals to track bugs in their "own language" and not the "language of the developers." A team of support professionals to track specific information about users who have called to complain that may not always be needed in the actual development queue (thus, there are two tracking systems when an LBT is in place.)
2009 LogiGear Corporation All Rights Reserved
237
CHAPTER 7.2
7.1 Bug Tracking/Management System 7.2 LogiGear TrackGear Bug Management System 7.3 Reporting Bugs Using TrackGear
238
CHAPTER 7.2
A Web-based bug tracking system that is powerful, flexible, and simple to use. Offers custom workflow, e-mail notification, meaningful metrics and attachment features. Supports multiple projects while maintaining security at user, group and project levels. Create unique queries with flexible search tools. Intuitive interface - access any information with just a few clicks. Supports any client platform with Netscape Navigator 4.x or MS Explorer 4.x.
239
Powerful TrackGear
CHAPTER 7.2
Analyze project status quickly with an unlimited number of customizable report formats. Interpret bug statistics with predefined metrics and present information in meaningful charts and reports. Configure TRACKGEAR's workflow to fit bug-resolution processes. Enhance workflow with features like auto-routing and userdefined email notification. Enforce security at the user level, group level, and project level simultaneously. Scale TRACKGEAR to support multiple projects in the same or different divisions all at the same time! Attach multiple files to a single report for easy downloading later.
240
CHAPTER 7.2
Flexible TrackGear
Find bug reports quickly with TRACKGEAR's powerful search tools: Easy Find, Quick Find, and Form Find. Or, use Custom Find to create your own unique queries. Share customized report formats and queries with other users-or keep them private. Utilize TRACKGEAR's remarkable accessibility and speed to work from anywhere you use the Web: at the office, at home, or on the road. Support different divisions within a company in customizing TRACKGEAR to meet individual needs; each division can have a unique system design and custom workflow! Communicate critical information to users with admin broadcast messaging-announce scheduled maintenance shutdowns hours in advance. Set workflow rules to reflect existing or new processes. TRACKGEAR will automatically enforce them! Support any number of individual user accounts-and purchase additional concurrent user licenses as your company grows.
241
CHAPTER 7.2
Simple TrackGear
Access any TRACKGEAR feature with just a few clicks. Navigate with TRACKGEAR's intuitive user interface-no wasted time learning a new process. View reports in one of three intuitive layouts; change views instantly as your needs and tasks change. Get your product team started right away-with TRACKGEAR's easy setup and installation process you'll be up and running in no time!
242
Other Features
CHAPTER 7.2
Customize essential system attributes and optimize your company's workflow-Resolution, Priority, Keyword, Error Type and more! Trace any report's history with automatic signature and date stamping for reliable auditing and user accountability. Set preferences for user-defined start pages-each user starts their session from the TRACKGEAR screen that is most relevant to them. Construct and share multiple, complex search queries to automate repetitive search tasks. Create unique profiles for numerous hardware/software configurations and save time reentering identical information. Save data-entry time by cloning similar reports.
243
CHAPTER 7.3
7.1 Bug Tracking/Management System 7.2 LogiGear TrackGear Bug Management System 7.3 Reporting Bugs Using TrackGear
244
CHAPTER 7.3
245
CHAPTER 7.3
Project: This will have the different projects the user is assigned to. User select a project to log-in a defect. Build: If a particular project has multiple builds, user can select the same. Module: This will help us to select the modules of the application for which the bug/issue is going to logged into Summary: Give a general description of the bug/Issue. Steps: Specify the detailed steps to reproduce the bug. Assigned: Assign the defect/issue to someone may concern. Due Date: Mention the date by which you expect the solution. Attachments: To attach the screen shots of the bug or any materials which you think it can help the bug is clearer Save: Click on Save button to complete logging in the bug. Once the bug/issue is saved, a new screen for logging in new bugs/issues will be displayed. And some other fields
246
CHAPTER 7.3
EXAMPLE
247
CHAPTER 7
SUMMARY
A bug tracking/management system is a software application designed to help QA, QC, testers and programmers keep track of reported software bugs in their work. Bug tracking systems are extremely valuable in software development, and they are used by companies developing software products The major component is a database records facts about known bugs. In LogiGear, we use TrackGear for keeping track and managing bugs. It is a web-based bug tracking system that is powerful, flexible, and simple to use.
248
CHAPTER 7
EXERCISE
249
CHAPTER 8
CHAPTER 8
OUTLINE
Reporting Test Progress Goals of Status Report Sample Template of a Status Report
251
CHAPTER 8
OBJECTIVES
An introduction to testing status report and its goals Learn about how to write a status report
252
CHAPTER 8.1
8.1 Reporting Test Progress 8.2 Goals of Status Report 8.3 Sample Template of a Status Report
253
254
This is our communication tool to the rest of the team. Tests or testing that is: blocked, failed, skipped As with all writing write to the audience. This report may go out to non-technical staff. Results should be reported in terms of: schedule, budgets, features and quality. Not core-dumps or uninitialized pointers. Also keep the tone non-judgmental Be calm! Speak in numbers and trends. Reporting has to be consistent.
255
If you are a Test Lead, you have to write Weekly Status or Milestone Reports. They are very important for tracking and prompting change. Many times on a development project changes are approved and not documented. Your report may be the only place these changes are captured. Important items from status reports can be rolled into Test Plan revisions Types of Test Status Report:
Test Coverage Analysis, Bug Summery Report, Test Suite Summery, Test Case Progression
256
CHAPTER 8.2
8.1 Reporting Test Progress 8.2 Goals of Status Report 8.3 Sample Template of a Status Report
257
The Goal is communicating Test Progress This takes out wondering what the test group is doing. It gives test assurance of what is in test and what is not tested It can be just defects found, defects fixed if nothing more than that it is a start
258
CHAPTER 8.3
8.1 Reporting Test Progress 8.2 Goals of Status Report 8.3 Template of a Status Report
259
260
CHAPTER 8.3
261
CHAPTER 8.3
Status Report
With every issue, you should suggest/restate the solution Outstanding issue is the one can affect your all tasks or it may occur for a long time Issue for the day is the one that only occur on the day you report With every assignment, you should restate the percentage of completion, status of the tasks
262
CHAPTER 8
SUMMARY
Status report is used to communicate test result and testing progress. Status report can include so many things depend on the testing project is being handled. Main things usually mentioned in status report are issues, assignments, resources, test progress, test results Testers write status reports everyday and send them to their test lead. The leader will collect status reports of all team members and send a status report of the team to QA
263
Thank you!
264