CSQE Exam Preparation

Douglas Hoffman
BACS, MSEE, MBA, ASQ-CSQE, ASQ-CQMgr, ASQ Fellow Software Quality Methods, LLC. 24646 Heather Heights Place Saratoga, California 95070-9710 doug.hoffman@acm.org www.SoftwareQualityMethods.com 408-741-4830

Copyright 1997-2004, Software Quality Methods, LLC.

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 1

Copyright Notice
These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode. These notes include examples which are context specific. In class discussions I may respond to questions and provide information on approaches and solutions to students problems. I may answer questions in a way that I believe would normally be true but our answer could be completely inappropriate for your particular situation. I cannot accept any responsibility for any actions that you might take in response to my comments in this course. Courses such as this one would not be needed if the problems addressed were simple enough to analyze and resolve in a few minutes.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 2

About Doug Hoffman

I advocate and provide advice and services in software testing and quality assurance. Software quality assurance, and especially software testing, have a reputation of being where failed programmers or programmer wanta bes congregate. I dont believe its true, and its through courses like this that we can change the perception. I gravitated into quality assurance from engineering. Ive been a production engineer, developer, support engineer, tester, writer, instructor, and Ive managed manufacturing quality assurance, software quality assurance, technical support, software development , and documentation. Along the way I have learned a great deal about software testing and measurement. I enjoy sharing what Ive learned with interested people. Current employment President of Software Quality Methods, LLC. (SQM) Management consultant in strategic and tactical planning for software quality Adjunct Instructor for UCSC Extension, teaching Black Box Software Testing and Test Automation Design Education MBA MS in Electrical Engineering, (Digital Design and Information Science) B.A. in Computer Science Professional Past Chair, Silicon Valley Section, American Society for Quality (ASQ) Founding Member and Past Chair, Santa Clara Valley Software Quality Association (SSQA, 1992-1997) Certified in Software Quality Engineering (ASQ-CSQE, 1995) Certified Quality Manager (ASQ-CQMgr, 2003) Previously a Registered ISO 9000 Lead Auditor, (RAB 1993) I also participate in the Los Altos Workshop and several others on Software Testing
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 3

About This Course

The CSQE Exam Preparation Class provides a survey of the topics included in ASQs CSQE Body of Knowledge to familiarize experienced software quality practitioners with its terminology and basic concepts. The practices recommended and discussed in this course are useful as an overview of software quality engineering. There is not time to cover any of the topics in depth and therefore the class is not intended to teach specific job skills, techniques, or tools. The class content is led by the slide set, but the presentation of materials is heavily influenced by the specific questions and interests of the students in each session. I also include much material from my and the students experience which often does not correspond with the CSQE BOK or would necessarily provide the expected (correct) answers to CSQE Exam questions. In the slides I have tried to identify the non-BOK materials with a red asterisk (*) to footnote them as not specifically being part of the BOK, and I try to make it clear in the discussions when ideas are not likely to be on the exam or may possibly be contrary to expected exam responses.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 4

The Course and The Exam

In this class I explicitly present information from the BOK I believe is likely to be included on the exam, ideas about interpreting the BOK to pass the exam, and specific test taking techniques. This is all based upon my experience and understanding of the material, the history of the CSQE BOK, ASQ certification policies and procedures, and feedback from other CSQEs. I do not have any special insider information about the BOK, exam materials, or the exam itself. Students earning their CSQE do so based upon their own knowledge and capabilities. The class does help substantially improve the chances of passing the exam by reviewing the technical material, explaining vocabulary, setting expectations, and providing techniques for maximizing scores and making the best use of the examination time. As a result of combining information from the formal BOK with experience, students from prior classes have reported learning very practical, real world concepts immediately applicable to their work. Past students have also achieved a pass rate on the CSQE Exam more than double the overall National pass rate.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 5

CSQE Body of Knowledge

1. General Knowledge 2. Software Quality Management 3. Software Engineering Processes 4. Program and Project Management 5. Software Metrics, Measurement, and Analytical Methods 6. Software Verification and Validation 7. Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 6

se in Pha f Time to Percen

Defec t Sev erity B reakd own

s mpile of Co er Numb
Pat hC ove rag e

Software Quality
Lines o f Docum entatio n Numbe r of Pro cedure s

e rag ove tC Tes

Numbe r of Com ments

ts Defec ber of Num

Software Development Life Cycle

Defect Discovery Rate

es Rat rtion Inse lt Fau

Tes t Eff ort p er D efec t

s Date duled che ne S ilesto M

Def ect Den sity

Code Complexity

e od of C es Lin

CSQE Class Schedule

(1) (2) (3-4) (4-5) (6) (7) (8)
Douglas Hoffman

Introduction; General Knowledge Software Quality Management Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation Configuration Management; Exam Strategies; Review
Copyright 1997-2004, SQM, LLC. Slide 8

Introductions
Me You

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 9

Class Objectives
Survey the SQE Body of Knowledge Prepare for CSQE Examination Share quality related experiences Gain information on selected SQ topics
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 10

Expectations
Mine Yours

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 11

General Knowledge
General Knowledge, Conduct, and Ethics
Software Quality Management Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 12

CSQE Requirements
Obtaining CSQE Certificate Maintaining Certification
Blooms Levels Of Cognition CSQE Subject Areas

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 13

Certification Requirements
Education and/or Experience
8 years in quality field up to 5 years credit for degrees

Proof of Professionalism Examination

proctored, open book exam 160 questions
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 14

Recertification Requirements
Recertify every 3 years 18 points needed

Douglas Hoffman

Professional Development Employment Instructor/Student Meetings Committees Certifications Proctoring Publishing

Copyright 1997-2004, SQM, LLC. Slide 15

CSQE Body of Knowledge

General Knowledge (10% - 16 questions) Software Quality Management (19% - 30) Software Engineering Processes (16% - 26) Program and Project Management (15% - 24) Software Metrics (15% - 24) Verification and Validation (V&V) (15% - 24) Configuration Management (10% - 16)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 16

CSQE BOK Subject Areas

General Knowledge, Conduct, and Ethics Software Quality Management Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 17

Levels of Cognition1
Knowledge Comprehension Application Analysis Synthesis Evaluation
1Bloom,

B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives handbook 1: Cognitive domain. New York: McKay.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 18

Levels of Cognition
Level
Knowledge Comprehension Application Analysis Synthesis Evaluation
Douglas Hoffman

Verbs
- write, list, name, define, label, state - explain, describe, summarize, illustrate, paraphrase - use, solve, apply, construct, demonstrate, compute - analyze, compare, contrast, separate - create, design, invent, develop - judge, recommend, critique, justify
Copyright 1997-2004, SQM, LLC. Slide 19

General Knowledge, Conduct, and Ethics

Quality philosophy and principles Standards, specifications, and models Leadership tools and skills Ethical conduct and professional development

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 20

Quality Philosophies and Principles

Benefits of software quality Prevention vs. Detection philosophies Software TQM principles and applications Organization and process benchmarking

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 21

Benefits of software quality

To Customers:
Satisfaction Improved Reliability Reduced Errors in Operations Matching with Requirements

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 22

Benefits of software quality

To the Organization:

Douglas Hoffman

Meeting Customer Requirements Stable Requirements Verification that Requirements are Met Consistent Application of Processes Improvement Over Time Quality of Life *
*

Not part of CSQE 2002 BOK

Slide 23

Copyright 1997-2004, SQM, LLC.

Quality philosophies*
Philip Crosby (Conformance to Requirements) Dr. W. Edwards Deming
(Never Ending Improvement)

Dr. Joseph Juran (Fitness for Use)

*

Not part of CSQE 2002 BOK

Slide 24

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Philip Crosby*
Four Absolutes:
Quality Means Conformance to Requirements Quality Comes from Prevention Zero Defects Quality Measurement is the Price of Nonconformance

14 Steps to Improvement
*

Not part of CSQE 2002 BOK

Slide 25

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Dr. W. Edwards Deming*

Never Ending Improvement Fourteen Points Seven Deadly Diseases
*

Not part of CSQE 2002 BOK

Slide 26

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Demings Fourteen Points*

Douglas Hoffman

Constancy of Purpose for Improvement. Adopt the New Philosophy. Cease Dependence on Mass Inspection. Cease Doing Business on Price Tag Alone. Continual Improvement of Process. Institute Training on the Job. Institute Leadership. Drive Out Fear. Break Down Barriers Between Departments. Eliminate Slogans, Exhortations, and Targets. Eliminate Numerical Quotas. Allow Pride in Workmanship. Institute a Program of Self-Improvement. * Not part of CSQE 2002 BOK Do It.
Copyright 1997-2004, SQM, LLC. Slide 27

Demings Seven Deadly Diseases*

Lack of Constancy of Purpose Emphasis on Short-Term Profits Personal Evaluation Appraisal Mobility of Management (Job Hopping) Use of Visible Figure for Management Excessive Medical Costs Excessive Costs of Warranty
*

Not part of CSQE 2002 BOK

Slide 28

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Dr. Joseph Juran*

Fitness for Use Revolutionary Rate of Improvement Top Management is in Charge Specific Goals for Quality Improvement in the Business Plan Assign Responsibility for Improvements to Individuals Train People Empower the Workforce
*

Not part of CSQE 2002 BOK

Slide 29

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Prevention vs. Detection

Detection
Identify Correct

Prevention
Start Earlier Look Upstream for Improvements

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 30

Software TQM Principles*

Continuous Improvement Management by Facts Measurement of Progress Quality Teams Management of Resources Leadership
*

Not part of CSQE 2002 BOK

Slide 31

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Types of Quality Teams*

Quality Council Cross-Functional Team Quality Action Team Tiger Team
*

Not part of CSQE 2002 BOK

Slide 32

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Software TQM Applications*

Assessments Activities
Plan-Do-Check-Act SEI CMM Goal/Question/Metric Paradigm
*

Not part of CSQE 2002 BOK

Slide 33

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Types of Organizational Benchmarking

Process Benchmarking
macro (organizational) level micro (process and project) level

Performance Benchmarking Project Benchmarking Strategic Benchmarking

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 34

The Benchmarking Process

Determine Own Current Practices Identify Industry Best Practices Analyze Best Practices Model Best Practices
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 35

Standards, Processes, and Models

Software standards Software quality and process initiatives, ventures, and consortia Quality management system models Software assessment models
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 36

Domestic and International standards and specifications

ISO
ISO 2382 (Data Processing Vocabulary) ISO 9000 (Quality Management System) ISO 15504 (SPICE) IEEE/EIA 12207 (Software Life Cycle Processes)

IEEE
40 Standards in a 4 volume set (1999)

CMM and CMMI

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 37

ISO 9000
Family of Standards and Guides First approved in 1987, latest revision 2000 Process (not product) focus Covers all processes affecting quality of goods and services of organization Open ended Descriptive (not prescriptive)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 38

ISO 9000-1987
9000 Guidelines for Selection and Use 9001 Model for Design, Development, Production, Installation, and Servicing 9002 Model for Production, Installation, and Servicing 9003 Model for Final Inspection and Test 9004 Guidelines for Quality Management and Quality System Elements

O S

E L

E T

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 39

ISO 9000:2000
9000 Quality Management Systems Fundamentals and Vocabulary 9001 Quality Management Systems Requirements Adds process improvement Combines 9001, 9002, and 9003 9004 Quality Management Systems Guidelines for Performance Improvements
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 40

ISO 9000 Part 3

9000-3 Guidelines for Software Development, Supply, and Maintenance of Software Interprets ISO 9001

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 41

ISO/IEC JTC 1
ISO and International Electrotechnical Commission Develops standards for information technology Subcommittee SC7 for software engineering standards Working Groups for specific standards ISO/TC176 to harmonize activities
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 42

Domestic and International standards and specifications

IEEE
Standard Collection for Software Engineering ANSI/IEEE 610.12: Standard Glossary of Software Engineering Terminology ANSI/IEEE 730: Software QA Plans ANSI/IEEE 828: Software CM Plans ANSI/IEEE 830: Requirements Specifications ANSI/IEEE 1028: Reviews and Audits ANSI/IEEE 1012: Software V & V Plans ANSI/IEEE 1074: Life Cycle Processes
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 43

Other Domestic and International (ISO) Standards and Specifications

ISO 9126 (Software Product Evaluation) ISO 10006 (Project Management) ISO 10007 (Configuration Management) ISO 10011 (Guidelines for Auditing)

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 44

Software Process Models

Bellcore TR-179 SEIs CMM and CMMI Trillium Model BOOTSTRAP ISO SPICE
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 45

Bellcore TR-179
Based on ISO 9001 and 9000-3 Bellcore Capability Assessment Bellcore Model
Quality System Framework Quality System Life Cycle Activities Quality System Supporting Activities

Includes Additional Requirements for Telecommunications Suppliers

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 46

TL 9000 Standards
Quality Excellence for Suppliers of Telecommunications (QuEST) Forum Set of common standards including ISO 9000 (both 1994 and 2000) Quality System Requirements Quality System Metrics

Douglas Hoffman

Common Software Hardware Service

Copyright 1997-2004, SQM, LLC. Slide 47

SEIs Capability Maturity Model

Levels of Maturity
Level 1: Initial Level 2: Repeatable Level 3: Defined Level 4: Managed Level 5: Optimizing
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 48

Capability Maturity Model

Key Process Areas (KPA) Software Capability Evaluation (SCE) Software Process Assessment (SPA) Interim Profile CMM-Based Appraisal for Internal Process Improvement (CBA IPI)
Copyright 1997-2004, SQM, LLC. Slide 49

Douglas Hoffman

Capability Maturity Model Integration

Combines SE-CMM and SW-CMM Continuous or Staged process models Software Process Assessment (SPA) Appraisal Requirements for CMMI (ARC) Standard CMMI Appraisal Method for Process Improvement (SCAMPI)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 50

CMMI Common Process Areas

Category
Process Management Project Management

Process Areas
Organizational: Process Focus , Process Definition, Training, Process Performance, and Innovation and Deployment Project Planning and Monitoring and Control Supplier Agreement Management Integrated Project Management Risk Management Integrated Teaming Quantitative Project Management Requirements Development and Management Technical Solution Product Integration Verification and Validation Configuration Management Process and Product Quality Assurance Measurement and Analysis Organizational Environment for Integration Decision Analysis and Resolution Causal Analysis and Resolution

Engineering Support

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 51

CMMI Model Representations

Continuous
Allows selecting the order of improvement Enables comparisons Easy comparison with ISO/IEC 15504 Easy migration from EIA/IS 731

Staged
Proven sequence of improvements Permits comparisons of maturity levels Single summary rating Easy migration from SW-CMM to CMMI

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 52

SEIs CMMI - Continuous

Improvement across organization and by process area Generic Goals (GGs) and Practices (GPs) Specific Goals and Practices Capability Level Profile
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 53

SEIs CMMI - Continuous

Levels of Maturity
Level 0: Incomplete Level 1: Performed Level 2: Managed Level 3: Defined Level 4: Quantitatively Managed Level 5: Optimizing
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 54

SEIs CMMI - Staged

Improvement across organization by capability level Generic and Specific Goals and Practices KPAs mapped from CMM

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 55

SEIs CMMI - Staged

Levels of Maturity
Level 1: Initial Level 2: Managed Level 3: Defined Level 4: Quantitatively Managed Level 5: Optimizing
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 56

CMM - CMMI Levels

CMM Level 0 Level 1 Level 2 Level 3 Level 4 Level 5 Initial Repeatable Defined Managed Optimizing CMMI Continuous Incomplete Initial Performed Managed Managed Defined Defined Quantitatively Quantitatively Managed Managed Optimizing Optimizing CMMI Staged

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 57

ISO SPICE
Software Process Improvement for Capability Determination ISO 15504 To Harmonize CMM-Based Efforts Used For:

Douglas Hoffman

Process Assessment Process Improvement Capability Determination Qualification and Training of Assessors
Copyright 1997-2004, SQM, LLC. Slide 58

Trillium Model
Trillium Scale
Level 1: Unstructured Level 2: Repeatable and Project Oriented Level 3: Defined and Process Oriented Level 4: Managed and Integrated Level 5: Fully Integrated
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 59

Trillium Model
Capability Areas

Douglas Hoffman

Organizational Process Quality CHR Development and Management Process Management Quality System Development Practices Development Environment Customer Support
Copyright 1997-2004, SQM, LLC. Slide 60

Trillium Model
Capability Evaluation/Joint-Assessment Capability Assessment and Improvement Capability Self-Assessment Continuous Improvement (CI) Program Capability Profile, Levels, Road Maps, and Practices
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 61

BOOTSTRAP
AKA ESPRIT Bootstrap Assessment Bootstrap Evaluation Process Improvements

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 62

BOOTSTRAP Assessment
Assesses Organization and Projects Quality-Attribute Hierarchy
Clusters Elementary Attributes

Process Dimensions
Organization Methodology Technology
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 63

Leadership Tools and Skills

Organizational leadership Analyzing Current Situations SWOT Analysis Implementing and Managing Change Quality Initiatives Cross-Functional Collaboration Knowledge Management Motivation Techniques
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 64

SWOT Analysis

Strengths and Weaknesses

Internal to organization

Opportunities and Threats

External sources

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 65

Management Science
Fredrick Taylor (time and motion) Hawthorne Studies (cause and effect) Abraham Maslow (need hierarchy) Douglas McGregor (Theory X and Y) Frederick Herzberg (2 motivation factors) Organizational Empowerment Motivating the Work Force
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 66

Team Roles and Responsibilities

Champion Sponsor Facilitator Coach Leader Team member Recorder Timekeeper
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 67

Team Life Cycle

Bruce Tuckermans Stages:
Forming [Build Phase] Storming [Build Phase] Norming [Develop Phase] Performing [Optimize Phase]
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 68

Team Applications
Team Type Structure
department 8-10 from a department

Applications
Quality or productivity issues

Improvement 8-10 from a Quality

Initially quality or performance; can evolve to self directed teams ranges; may be part Focus on a specific project; Project or all managers disbands when done 8-12 from different Like project teams but for Cross areas policies and operational issues Functional Can work from objectives; Self Directed 6-15; usually a natural area team requires training
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 69

Team Dynamics
Selecting Team Members Team Size Team Diversity Dominant or Disruptive Team Members Common Team Problems Dealing with Team Problems
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 70

Team Tools
Nominal Group Technique (NGT) Multivoting Brainstorming Joint Application Development (JAD) Rapid Application Development (RAD) Facilitated Application Specification Technique (FAST)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 71

Facilitation skills
Team leader Facilitator Team participants Avoiding chaos Maintaining balance Keeping focused
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 72

Conflicts
Expect them Manage conflicts Resolve conflicts Handle conflicts Force Field Analysis Negotiation
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 73

Meetings
Meeting Management Team performance Operating guidelines Meeting structure

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 74

Verbal Communication Skills

Purpose of Presentation Type of Presentation Analyze Audience Needs Determine Objective or Outcome Structure to Support Main Idea Get the Audience Attention Meet Audience Objectives Strong Conclusion Practice Delivery Use Visual Aids
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 75

Written Communication Skills

Memo Report Letter

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 76

Interviewing Skills
Organize the Interview
Have a Plan Simple Questions Stay Focused Dont Threaten Ask Permission Avoid Ambiguity Avoid Manipulation

Closed and Open Questions Stay In-Bounds

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 77

Effective Listening Skills

Non-Listener Marginal Listener Evaluative Listener Active Listener

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 78

Ethical Conduct
Professional codes of ethics ASQ Code of Ethics Professional conduct and competence Conflicts of interests Software Licensing Software Copyrights
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 79

Professional Conduct Terminology

Conflict of interest Ethical Ethics Legal Malpractice Negligence Notification Recall Regulation
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 80

Conflicts of Interest Examples

Providing recommendations on the purchase of products or services while owning an interest in the producer Presenting independent assessment results to a client with recommendations to use your services Participating in awarding a contract to someone who manages you in another context Evaluating the corrective actions of another party which your organization has performed for them
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 81

Software Liability and Safety Issues

Legal issues involving software

product liability product safety negligence customer notification requirements

Other legal or regulatory issues

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 82

Professional Development
Software Quality Training Quality Auditor Training Software Engineering Training Professional Certification Training Needs Analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 83

Professional Training and Development

Subject areas for SQE Available training resources, materials, and providers SQE Professional
Societies Technical Associations Organizations
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 84

SQE Subject Areas

Douglas Hoffman

Design of Experiments Failure Mode Analysis Internal Auditing Reliability Engineering Statistical Process Control Software Quality Engineering Strategic Quality Planning
Copyright 1997-2004, SQM, LLC. Slide 85

SQE Training Resources

RAB Accredited ISO Courses ASQ SEI Colleges and Universities Private Organizations and Individuals

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 86

Professional Societies for SQEs

ASQ (SQE Certification) SEI (SPINs) Deming User Group ACM IEEE

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 87

Quality Management
General Knowledge, Conduct, and Ethics

Software Quality Management

Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 88

Software Quality Management

Douglas Hoffman

Goals and objectives Principals Policies Strategic quality goals Tactical quality goals Process quality Tools and methods Audits
Copyright 1997-2004, SQM, LLC. Slide 89

Goals and Objectives

Quality goals and objectives Outsourced services Planning Software quality management systems documentation Customer requirements
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 90

SQA Tasks
Generation of QA Documentation Review of Project Materials Auditing Monitoring of Project Status Inspection of Delivered Items Monitoring Corrective Actions Participation in Project Activities Guidance of Project Testing Oversight
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 91

Outsourced Services
Process work (operations) Project work Global (strategic) Tactical (partial) Subcontracting (targeted) Offshore outsourcing Subcontractor management (SW-CMM)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 92

Outsourcing Pros and Cons

Forces for:
Lower costs Risk sharing or reduction Economies of scale Greater skill pool Greater focus More control More professional Cash infusion
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 93

Forces against:
Higher costs Risk exposure Dis economies of scale Limited access to knowledge Loss of intellectual capital Conflicting agendas

Software Quality Management

Planning Tracking Organizational and Professional Software QA Training

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 94

SQ Management Planning
Product and Project SQ Goals and Objectives Customer Requirements for Quality Quality and Customer Support Activities Software Security, Safety, and Hazard Analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 95

Planning Issues
Software Development is Private Traditional Programmer Training Misses Planning and Measurement Standards not Followed Cowboy Culture QE Considered an Outsider
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 96

Planning Solutions
On the Job Training Positive Experience with Standards and Metrics Professional Attitude Discipline Planning Initiative (WWWWWH)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 97

Planning Initiative Focus

Defect Prevention Identification of Problem Areas Risk Analysis and Mitigation Correct Reviews and Inspections Documentation of Required Procedures
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 98

Planning Initiative Levers

Contractually Imposed Standards Analysis of System Requirements Schedules Development Costs User Concerns
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 99

Product and Project SQ Goals and Objectives

Predictability Consistency Objective Measurements and Evidence

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 100

Quality Planning System

ISO DOD-MIL NASA Stds. Bellcore IEEE FCC

Policies

Standard Processes

Procedures (Test, CM, QA)

Procedure Artifacts

Internal/External Guides (SEI-CMM, Coding Standards, etc.)

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 101

Customer Requirements for Quality

Reliability (MTTF) Pass/Fail Criteria and Evidence Traceability Software Baseline Control Product Consistency
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 102

Planning for Quality

Phase Based Activities Software Quality Assurance Plan (SQAP) ISO 9000-3 requirements
Planning Tracking

Documentation
SQM Plan (IEEE 730) Requirements
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 103

Requirements*
Solve a problem Must be met to satisfy the contract Requirements analysis vs. Requirements specification Problem analysis vs. Product description
* IEEE 610.12
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 104

Requirements Analysis
Object Oriented Function Oriented State Oriented Behavioral Analysis Nonbehavioral Analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 105

Nonbehavioral Quality Factors

Efficiency Reliability Security Testability Understandability Maintainability
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 106

Human engineering Portability Visibility Capacity Standards compliance

Requirements Elicitation
Problem recognition Evaluation and synthesis Modeling Specification Review
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 107

Customer/User Reviews
JAD RAD QFD FAST Context-free questions
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 108

Requirement Analysis and Specification

Domains of the problem
Information Function Behavior

Problem partitioning Representing requirements

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 109

Quality Records*
Records and Data Collection Use Changes Storage Maintenance Retention
*

Not part of CSQE 2002 BOK

Slide 110

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Issues for Quality Related Data

Data Entry Change Control Data Availability Data Integrity Data Security
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 111

Quality Management Methods

Reviews, Inspections, and Testing Change Management Cost of Quality (COQ) Quality Data Tracking Problem Reporting and Corrective Actions Quality Improvement Processes
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 112

SQA Tools
Standards Analyzer Data-flow Analyzer Interface Analyzer Test Generator Test Management System

Requirements Tracer Database Analyzer Complexity Analyzer Logic Analyzer Reliability Model Simulators

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 113

Process and Technology Change Management

Software process modeling SEPGs Software process assessment and evaluation Software process and technology change management theory and methods Barriers to the implementation or success of quality improvement efforts and quality systems
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 114

Audit and Assessment Management

Program development and administration Audit preparation and execution Audit reporting and follow up

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 115

Software Quality Factors

SQFs Definitions Metrics

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 116

Software Quality Methods

Bag of Tricks Inspections Tests Assessments
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 117

Cost of Quality
Traditional Concepts Levels Locations

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 118

Quality Cost Analysis

Quality Cost Measurement is a cost control system used to identify opportunities for reducing the controllable quality-related costs. The Cost of Quality is the total amount the company spends to achieve and cope with the quality of its product. This includes the companys investments in improving quality, and its expenses arising from inadequate quality. A key goal of the quality engineer is to help the company minimize its cost of quality.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 119

Quality-Related Costs
Prevention Cost of preventing customer dissatisfaction, including errors or weaknesses in software, design, documentation, and support. Internal Failure Appraisal Cost of inspection (testing, reviews, etc.).

External Failure

Cost of dealing with errors Cost of dealing with errors discovered during development that affect your customers, and testing. Note that the after the product is released. company loses money as a user (who cant make the product work) and as a developer (who has to investigate, and possibly fix and retest it).
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 120

Examples of Quality Costs

Prevention Staff training Requirements analysis & early prototyping Fault-tolerant design Defensive programming Usability analysis Clear specification Accurate internal documentation Pre-purchase evaluation of the reliability of development tools Internal Failure Bug fixes Regression testing Wasted in-house user time Wasted tester time Wasted writer time Wasted marketer time Wasted advertisements Direct cost of late shipment Opportunity cost of late shipment Appraisal Design review Code inspection Glass box testing Black box testing Training testers Beta testing Usability testing Pre-release out-of-box testing by customer service staff External Failure Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Supporting multiple versions in the field Refunds, recalls, warranty, liability costs Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers
Slide 121

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Customers Quality Costs

Seller: external costs Customer: failure costs (sellers externalized costs)
These illustrate costs absorbed by the These illustrate costs absorbed by the seller that releases a defective customer who buys a defective product. product.
Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Refunds, recalls, warranty, liability costs Government investigations Supporting multiple versions in the field Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers
Douglas Hoffman

Wasted time Lost data Lost business Embarrassment Frustrated employees quit Failure during one-time-only tasks, e.g. demos to prospective customers Cost of replacing product Reconfiguring the system Cost of recovery software Tech support fees Injury / death
Slide 122

Copyright 1997-2004, SQM, LLC.

The $q / $Q Debate
Topic
Products Processes Customer Industries Cost of Poor Quality
Douglas Hoffman

Little Q
Manufactured goods Process directly related to production Clients who buy the products Manufacturing

Big Q
All products, goods, and services All processes, support, business, etc. All who are affected All industries

Costs associated with All costs that would deficient manufactured disappear if everything goods were perfect
Copyright 1997-2004, SQM, LLC. Slide 123

Quality Data Tracking

Principles
Understanding Evaluation Control Prediction

Models Impact on the organization Management Support

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 124

Problem Reporting and Corrective Actions

Reporting Procedures Corrective Actions Quality Improvement Processes
Defect prevention Defect detection and removal Trend analysis Pareto Analysis Reviews Testing

Barriers to Quality Improvements

Ill-defined process Poor process management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 125

Problem Reporting and Corrective Action Procedures

Defect Tracking Systems Event Recording Systems Development Needs QA Requirements Support Issues
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 126

Supporting Activities
Defect Tracking Technical Support Change Control Board (CCB)

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 127

Software Audits

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 128

Software Audits
Program Development and Administration Audit Preparation and Execution Audit Reporting and Follow-up

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 129

Audit Program Objectives

Auditing stages
Planning and preparation Performance Reporting Corrective action and follow-up

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 130

Audit Objectives
Compliance with standards Effectiveness of controls Opportunity for improvement Regulatory requirements Permit registration
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 131

Reasons for Audits

Contractually required Verify QMS meets a standard Confirm QMS implementation Verify QMS effectiveness Evaluate compliance with standards.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 132

Audit Responsibilities
Audit team member Lead Auditor Client Auditee
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 133

Auditing Standards
ANSI/ISO/ASQC Q10011-1994 Guideline for Auditing QMS ISO 9001:2000
[ISO 9000-3 Guideline]

IEEE 1028 TickIT SEI (CMMI)

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 134

Software Audits
Terms Types Methods Steps Process Planning
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 135

Audit Terms
Audit Checklists Compliance Conformance Customers Interviews Objective Evidence
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 136

Audit Types
First party Second party Third party Internal External
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 137

Audit Types
(Continued) System Process Product Compliance Regulatory
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 138

Audit Types
(Continued) Management Quality Functional configuration Physical configuration Administrative Audit etc.
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 139

Audit Methodology
Purpose Perspectives Frequency Criteria Procedures
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 140

Audit Precepts
Function of management Qualified auditors Measures against standards Objective evidence Focus on control system
Douglas Hoffman Copyright 1997-2004, SQM, LLC.

Slide 141

Management Audits
To determine that: Adequate controls exist Controls are being implemented Controls really work
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 142

Audit Perspectives
Customers
Auditee Client Organization

Auditor

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 143

Audit Process
Objectives Scope Entry criteria Assessment criteria Exit criteria
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 144

Audit Frequency
Project milestone Quality milestone External requirements Internal request Major organizational change
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 145

Audit Tools
Checklists Authoritative Documents Flowcharts Interviewing Data Collection
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 146

Data Collection
Interviews
Data gathering Not objective evidence Conducting an interview

Sampling Trace Forward Trace Back

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 147

Audit Data Sources

Physical properties Observations Documents and records Interviews Patterns of information

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 148

Audit Phases
Preparation Performance Reporting Closure
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 149

Audit steps

Douglas Hoffman

Planning Preparation Execution Reporting Corrective action Verification Follow-up

Copyright 1997-2004, SQM, LLC. Slide 150

Engineering Processes
General Knowledge, Conduct, and Ethics Software Quality Management

Software Engineering Processes

Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 151

Software Engineering Processes

Environmental conditions Requirements management Requirements engineering Analysis, design, and development methods and tools Maintenance management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 152

Software Environmental Conditions

Life cycles

Systems architecture

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 153

Why Worry About SDLC?

"Cosmic Glue" Where Quality Goes In Basis for Planning Basis for Communication Set Milestones Reduces Ambiguity Defines Roles Surfaces Issues
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 154

Development Life Cycles

Three generic phases
Definition Development Maintenance

Many models
Balance among tradeoffs Choose one that adjusts to expected changes
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 155

Generic Life Cycle Phases

Definition
What information What functions What interfaces What design constraints What validation criteria

Maintenance
Error corrections Adaptations evolved Enhancements made

Development phase

Douglas Hoffman

Related activities
Quality assurance Configuration management Project monitoring Measurement
Slide 156

How data is structured How system is architected How procedures work How programmed How tested

Copyright 1997-2004, SQM, LLC.

Software Development Life Cycle Models

Douglas Hoffman

Waterfall Prototyping* and Mock-ups Spiral Incremental Development Decomposition/Integration Cleanroom* Fourth Generation Techniques* Object-Oriented Hybrid Models *
Copyright 1997-2004, SQM, LLC.

Not included in Primer

Slide 157

Waterfall
Feasibility Requirements Product Design Detailed Design Code/Unit Test Integration/Product Verification Implementation/System Test Operation and Maintenance/ Revalidation Time
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 158

Phases

Prototyping
Start Stop Determine Requirements Rapid Design Complete the Engineering

Build/fix Prototype Revise Requirements User Feedback

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 159

Mock-Up Approach
Revise Requirements Evaluate Prototype Detailed Design Code/Unit Test Integration/Product Verification Implementation/System Test Operation and Maintenance/ Revalidation
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 160

Revise Prototype

Copyright 1996-2002 SQM, LLC.

Spiral Model
Determine objectives, alternatives, constraints Evaluate Alternatives, Identify and Resolve Risks
Risk Analysis Risk Analysis Risk Analysis Prototype Prototype Operational Prototype

Risk Anal Proto

Re Pla quirem n ent De s Concept vel Pla op s me n ent nt rem ui Inte Req gra Te st P tion a lan nd

Initial Design

Detailed Design

Implementation

Plan Next Phases

Douglas Hoffman Copyright 1997-2004, SQM, LLC.

Develop, Verify Next Level Product

Slide 161

Incremental Development
Component 1 System Engineering Component 2 System Engineering

... ...

Component N System Engineering

Phases

Analysis Design Code

Analysis Design

...
Code

Analysis

...

Design

... ...

Code Component Test Component Integration Testing

Component Test

Component Test

Component Integration Testing

...

Time
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 162

Decomposition/Integration
Operation System Test Integration

Requirements Specification

Douglas Hoffman

n tio osi mp co De

Analysis

Design Development

Unit Test

Copyright 1997-2004, SQM, LLC.

Int egr ati on

Slide 163

Cleanroom
Development Team
Des and Dev 1st Module Baselined S/W Specification Development Planning Des and Dev 1st & 2nd Module Test & Proof 1st & 2nd Modules Test & Proof Modules 1, 2, 3

Requirements & Spec Team

Test & Proof 1st Module

Verify 1st Module

Des and Dev Modules 1, 2, 3

Verify 1st & 2nd Module Verify Modules 1, 2, 3

Verification/Certification Team
Slide 164

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Fourth Generation

Requirements Gathering Design Strategy Implementation Using 4GL Testing

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 165

Object-Oriented
OOA OOD Component Library Examine for a Reusable Component

No Go
Customer Evaluation

Not in Library

Reuse Component Put in Library Test Code Design Spec

OK

Put in Use

Construct Prototype

Develop New Component

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 166

Lifecycle Selection
Life Cycle
Waterfall Decomposition/ Integration Prototyping Spiral Cleanroom
Douglas Hoffman

Project Requirements
Known, unchanging Known, unchanging Unknown, changing Unknown, unchanging Known, provable

Copyright 1997-2004, SQM, LLC.

Slide 167

Hybrid Life Cycles

Should be planned Based on Project Requirements Focus on Milestone deliverables

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 168

Pitfalls In SDLCs
Poor Choice of SDLC Lack of Understanding Conflicts of Interest Violation of Process Poor Feedback

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 169

Systems Architecture
Collection of components Interactions between components Patterns of similar components and interactions

Douglas Hoffman

Client Server (C/S) N-tier B to B (B2B) B to C (B2C) B to E (B2E) Web based

Copyright 1997-2004, SQM, LLC. Slide 170

Web Architectures (Internet/Intranet/Extranet)

Wireless Messaging Collaboration Software Market to Market (M2M) B to Government (B2G) B to Anyone (B2A) Application to Application (A2A)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 171

Software Requirements Management

Requirements prioritization and evaluation Requirements change management

Bi-directional requirements traceability

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 172

Requirements Prioritization
Critical, Important, Desirable Required, Conditionally Required, Objective Assignment to future releases Requirements should be:
Unambiguous Correct Verifiable Consistent Complete
Copyright 1997-2004, SQM, LLC. Slide 173

Douglas Hoffman

Requirements Change Management

Submitting Evaluating Reviewing Scheduling Implementing
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 174

Bi-directional Traceability
Of: Customer Requirements Engineering Requirements Design Components Code Tests V & V Results Built one way (top-down) Bottom-up trace derived
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 175

Verifying: Requirements to Design Design to Code Requirements to Tests Validating: Requirements to Test Results

Software Requirements Engineering

Requirements types Requirements elicitation Requirements analysis and modeling System and software requirements specifications
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 176

Types of Requirements
Input Output Reliability Availability Maintainability Performance Accessibility Environmental conditions Ergonomic Safety Security Facility requirements Transportability Training Documentation External interfaces Testing Quality provisions Policy and regulatory Compatibility to existing systems Standards and technical policies Conversion Growth capacity Installation Contractual Regulatory
Slide 177

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Quality Requirements*
Product description
General requirements Identifications and indications Functionality Reliability Usability Efficiency Maintainability Portability Completeness Correctness Consistency Understandability Ease of overview

Programs and Data

Functionality Reliability Usability Efficiency Maintainability Portability

User Documentation

ISO/IEC 12119: 1994(E) (IEEE 1465-1998)

Slide 178

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Requirements Elicitation
Problem recognition Evaluation and synthesis Modeling Specification Review
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 179

Requirements Gathering Processes

Douglas Hoffman

Interviews Document Analysis Brainstorming Requirements Workshop Prototyping Use Cases Storyboards Interfaces Analysis Modeling
Copyright 1997-2004, SQM, LLC. Slide 180

Requirements Gathering Practices*

1. Project vision and scope 2. Project glossary 3. Joint requirements w/users 4. Requirement rationales 5. Train analysts and users about requirements 6. Have requirements CM 7. Prioritize requirements 8. Incremental development
Douglas Hoffman

9. Use reviews and inspections 10. Use a requirements tool 11. Use proven techniques 12. Include domain experts 13. Evolve mechanisms 14. Use CPI 15. Involve users throughout 16. V&V requirements
*

R.R.Young,Effective Requirements Practices (Addiston-Wesley, 2001)

Slide 181

Copyright 1997-2004, SQM, LLC.

Requirements Obstacles
User procedures Current capabilities Formal business rules Gold plating

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 182

User Procedure Issues

Invalid Practices Workarounds SOPs Local Policies Antiquated Business Practices
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 183

Current Capability Issues

Current system functionality Planned enhancements Offline processes Rogue applications Current system limitations
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 184

Formal Business Rules

Legislation Local policy Industry standards Partnering agreements Regulations
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 185

Gold Plating
Cutting-edge technology User desired features Management information Technical staff desire to provide better products Requirements creep
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 186

Quality Function Deployment (QFD)

Feature interaction
N P N N P P P = POSITIVE INTERACTIONS N = NEGATIVE INTERACTIONS

Product Features Importance of Needs Needs

CUSTOMER NEEDS COMPREHENSIVE LOW COST UP-TO-DATE EASILY AVAILABLE TEST QUESTIONS RATINGS 4 2 5 4 5 5 0 4 0 5

DESIGN FEATURES
COMPETITION COMPARISON

1 4 1 1 0

2 3 0 3 2

1 1 3 0 5

1 5 0 4 1

3 3 2 0 5

5 0 5 0 5 RANKINGS 5 = MOST IMPORTANT 0 = NO IMPORTANCE

Competitive Position

65 21 36 46 35 53 70

Computed Rating
TARGET VALUES

Measurable Outcomes
Douglas Hoffman

Contribution

House of Quality
Copyright 1997-2004, SQM, LLC. Slide 187

Requirements Analysis and Modeling

Represent the data domain Model
System data Functions Behaviors

Partition into layers Move from essential descriptions toward implementation details
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 188

Requirements Analysis and Modeling

Data Flow Diagram (DFD) Control Flow Diagram (CFD) Data Dictionary Entity Relationship Diagrams (E-R) State Transition Diagram Data Content Model Process Specifications Control Specifications
Copyright 1997-2004, SQM, LLC. Slide 189

Douglas Hoffman

Data Flow Diagram (DFD)

Notation
External entity

A producer or consumer of information that resides outside the bounds of the system to be modeled. A transformer of information that resides within the bounds of the system to be modeled. A data item or collection of data items; the arrowhead indicates the direction of data flow. A repository of data that is to be stored for use by one or more processes; may be as simple as a buffer or queue or as sophisticated as a relational data base.
Copyright 1997-2004, SQM, LLC. Slide 190

Process

Data item

Data store

Douglas Hoffman

Control Flow Diagram (CFD)

Illustration
Part status buffer Movement Alarm Robot Initiation Control

Bit String Monitor fixture and Operator Interface

Start/Stop Flag

Process Activate

Operator Commands Operator Settings Proecess Robot Controls Robot Movement Record Position Commands

Robot Command file

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 191

Data Dictionary
Represents data content of objects Lists all data elements Rigorously defines

Douglas Hoffman

Inputs Outputs Data Stores Intermediate calculations

Copyright 1997-2004, SQM, LLC. Slide 192

Entity Relationship Diagrams (E-R)

WRITERS

CREATE

TRAINING MANUAL

E-R Diagram Illustration

TITLE SECTIONS

QUESTIONS

EDIT

REVIEW

ETC.

Data Object Table Illustration

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 193

Requirements Analysis Common Characteristics

Methods: Support analysis Hierarchical representation Force careful consideration of interfaces Provide foundation for design and implementation
Douglas Hoffman

Characteristics: Mechanisms for analysis Represent functions and/or behaviors Define interfaces Partition the problem Support abstraction Represent essential and implementation views
Slide 194

Copyright 1997-2004, SQM, LLC.

Data Structure Oriented Analysis

Common characteristics:
Identify key information objects Assures hierarchical information structure Allows mapping the data into a program

Warnier-Orr Method (DSSD) Jackson System Development (JSD) Object Oriented Requirements Analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 195

Data Structure Systems Development (DSSD) a.k.a. Warnier-Orr Method

Shipping order fulfillment

Customer Phone Order

Shipment Order No. Sales Clerk receives records order Order # Title Author Unit $ Total

UPS

Order # Cust name Address Order Date S/W # Charge #

S/W # Quantity Order date Cust Address

Accounting System Order File

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 196

Jackson System Development (JSD)

Action Sequence Action Iteration Action Selection

Notation
Bus

Leave (1)

Midtown Route

Arrive(1)

Garage (i)

Move vehicle

0 Arrive 42nd St. (i)

0 Leave 42nd St. (i)

* Accelerate

Illustration
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 197

Approaches for Object Oriented Requirements Analysis

Categorical Behavioral Domain Use-Case Textual Structured
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 198

Requirements Specification Process

Systems and Software Engineering Iterative process Top down synthesis Functions and behaviors uncovered, analyzed, and allocated to components Defines the scope (limits of the product) Trade-offs Becomes the System Specification
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 199

Requirements Trade-off Criteria

Project considerations Business considerations Technical analysis Manufacturing evaluation Human considerations Environmental interfaces Off-the-shelf solutions
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 200

Systems Requirements Specification*

Functions and capabilities Business, organizational and user requirements Safety, security, and ergonomic requirements Operational, interface and maintenance requirements Constraints Qualification requirements
Douglas Hoffman
* IEEE

1233 Guide IEEE 1220 Process ISO/IEC 12207 Life Cycles

Slide 201

Copyright 1997-2004, SQM, LLC.

Software Engineering Processes

Analysis, design, and development methods and tools
Software design methods Types of software reuse Clean room and other formal methods Software development tools
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 202

Information Domain
Information flow
Transformations Introduced data Output data

Information content
Individual data and control items Combined into records

Information structure and relationships

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 203

Models
Aid in understanding system
Information Functions Behavior

Become the focal point for review and testing Are the foundation for the design
(Pressman, 1993)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 204

Object Oriented Design

Find the Classes and Objects Identify Structures Identify Subjects Define Attributes Define Services
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 205

Structured Analysis and Design

Input External Entity Output Information

External Entity

Computer Based System

External Entity

External Entity

External Entity

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 206

Unified Modeling Language (UML)

Four Layer Architecture
Layer Description Example
MetaClass, MetaAttribute, MetaOperation The infrastructure for a metametamodel metamodeling architecture.

Defines the language for specifying metamodels. metamodel An instance of a metametamodel. Defines the language for specifying a model. An instance of a metamodel. model Defines a language to describe an information domain. An instance of a model. user Defines a specific information objects (user data) domain.
Douglas Hoffman

Class, Attribute, Operation, Component

StockShare, askPrice, sellLimitOrder, StockQuoteServer <Acme_Software_Share_98789> , 654.56, sell_limit_order, <Stock_Quote_Svr_32123>

Slide 207

Copyright 1997-2004, SQM, LLC.

UML Representations
Static Diagrams; classes and objects Use-Case Diagram; systems functions and actors Sequence Diagrams; sequential interactions and messages Collaboration Diagram; objects interactions that together perform a function non-sequentially Statechart Diagram; the state machine Activity Diagram; a state machine where states are activities, and operations and transitions are triggered on activity completion (a.k.a swimlane diagram) Implementation Diagram; component source code structure and runtime processing descriptions
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 208

UML Perspectives
Logical View; the structure of the model (for analysts and designers) Process View; the performance, scalability, and throughput (for system integrators) Implementation View; the software components (for programmers) Use-Case View; the system functionality (for end-users) Deployment View; the system topology and installation (for system engineers)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 209

Software Reuse
Development activity Components designed for more than one use Benefits
Cost savings Reliability Efficiency
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 210

Software Reengineering
Maintaining Functionality Inventory Analysis Documentation Analysis Code Restructuring Data Restructuring

Reverse Engineering
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 211

Reverse Engineering
Design Recovery
Code and Data structures Code and Data flows E-R Models

Re-implementation

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 212

Defect Prevention
Use Standards
Life Cycle Models CM Documentation Testing

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 213

Classical Cleanroom
Development Team
Des and Dev 1st Module Baselined S/W Specification Development Planning Des and Dev 1st & 2nd Module Test & Proof 1st & 2nd Modules Test & Proof Modules 1, 2, 3

Requirements & Spec Team

Test & Proof 1st Module

Verify 1st Module

Des and Dev Modules 1, 2, 3

Verify 1st & 2nd Module Verify Modules 1, 2, 3

Verification/Certification Team
Slide 214

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Modified Cleanroom
Explicit component and functional requirements Explicit detailed design Explicit correctness arguments for design Formal inspections of code
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 215

Defect Removal
Detection plus Repair Testing Reviews
Peer or independent Formal or informal Document or code
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 216

Technical Reviews

Three to five people Advance preparation Maximum of two hour sessions Peers only

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 217

Development Tools
Management Modeling Design Code analysis and Testing Documentation Databases
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 218

Maintenance Processes
Steps
Defect report Impact analysis Implement the fix Test Issue revisions

Types
Corrective Adaptive Perfective Preventative

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 219

Maintainability
Factors that control maintainability
Qualified staff System structure System handling (CM) Standards use Structure of documentation Test cases
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 220

Side effects
Code Data Documentation

Operational Maintenance
Framework
User Environment Maintenance processes Software product Personnel
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 221

Maintenance Documents
Requirements System architecture Program architectures Specifications and designs Source code Validation documents Maintenance guide
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 222

Maintenance Factors
System documentation Lehmans Laws
Continuing change Increasing complexity Large program evolution Organizational stability Conservation of familiarity

Program understandability Maintenance personnel Tools

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 223

Maintenance Tools
Reverse Engineering
Program slicer Static analyzer Dynamic analyzer Data flow analyzer Cross-referencer Dependency analyzer Transformation tool
Douglas Hoffman

Testing
Simulator Test case generator Test path generators

Other
CM Documentation Complexity assessment
Slide 224

Copyright 1997-2004, SQM, LLC.

Project Management
General Knowledge, Conduct, and Ethics Software Quality Management Software Engineering Processes

Program and Project Management

Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 225

Program and Project Management

Planning Tracking and Controlling Risk Management

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 226

Program and Project Management

Planning
Project planning elements Goal-setting and deployment Project planning tools Cost and value data

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 227

Planning
Project Plan document Project planning factors Project planning methods and tools Goal setting and deployment factors Maintenance types Software maintenance and adaptability program planning Supplier management methodologies
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 228

Contract Review

Douglas Hoffman

Scope Standards Deliverables Risks and Contingencies Proprietary information Resources identified Subcontractor requirements
Copyright 1997-2004, SQM, LLC. Slide 229

Contract Review (cont)

Customer interfaces
Design Test Acceptance

Douglas Hoffman

Security Acceptance criteria Change process Training

Copyright 1997-2004, SQM, LLC. Slide 230

Project Plan (SMP)

Objectives and scope Contracts and standards Technical constraints Issues and alternatives WBS Estimates
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 231

Project Planning Elements

Project Definition Scope Management Estimation - Forecasting Definition Phase Customer Goals Contract Review Software Management Plan
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 232

Project Initiation
Contract Request for quote/proposal Feasibility study R&D Customer request Business need Strategic need
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 233

Project Factors

Douglas Hoffman

Work to be done Risks Required resources Tasks Milestones Effort required Schedule
Copyright 1997-2004, SQM, LLC. Slide 234

Goal-setting and Deployment

Elements with goals
Size, cost, schedule, task duration Critical project resources Product quality Software technical activities Measures and metrics

Goal Setting Goal Deployment

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 235

Estimation Elements
Effort Resources Scheduled deliveries

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 236

Estimation Means
Analogy Engineering assessment Subject Matter Experts (SME) Parametric Modeling

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 237

Parametric Modeling Tools

Douglas Hoffman

COCOMO Before You Leap (BYL) Wang Institute Cost Model (WICOMO) DEC Plan SLIM Checkpoint
Copyright 1997-2004, SQM, LLC. Slide 238

Scheduling Projects
Critical path scheduling Resource availability Task cost estimates Task crashing times Task crashing costs
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 239

Software Project Management

Work Breakdown Structure Phase Based Software Activities Planned Roles and Activities

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 240

Milestones
Events Entry and exit criteria Correspond with phase transitions Key for process control
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 241

PERT Example
Task3 (5) Task6 (4) Task7 (4) Task2 (3) B Task5 (3) Task4 (4) D F

Task1 (2) Start

Task10 (3) End

Task9 (3) G

Task11 (6)

Task8 (4)

Program Evaluation and Review Technique

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 242

PERT / CPM Example

Task Start 1 2 3 4 5 6 7 8 9 10 11 End T im e 2 3 5 4 3 4 4 4 3 3 6 Early 0 0 2 3 3 7 7 6 11 11 14 20 Late 0 4 2 7 7 13 7 10 11 17 14 20

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 243

PERT/CPM
2 Task1 3 Task2 5 Task3 4 4 Task4 3 Task5 Task7 3 Task9 4 Task8 6 Task11 4 Task6 3 Task10

Critical Path Method

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 244

Gantt Chart Example

Task Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6 Wk 7 Wk 8 Wk 9 Wk 10 Wk 11 Wk 12Wk 13 Wk 14 Wk 15Wk 16 Wk 17 Wk 18 Wk 19 Wk 20 1 2 3 4 5 6 7 8 9 10 11

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 245

Work Breakdown Structure

Estimate size Decompose tasks Identify dependencies Build vs. Buy

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 246

Project Sizing Using Function Points

Counting Externals
Inputs (X4) Outputs (X5) Inquires (X4) Interfaces (X7)

Add count of internal [logical] files (X10) Summed values adjusted by experience All values weighted by complexity
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 247

Cost and Value Data

Resource planning Cost estimation Cost budgeting Cost Control

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 248

Resource Planning
Input data:
Task duration from WBS Project scope statement Activity duration estimates

Output data:
Resource requirements
People Platforms Equipment Materials
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 249

Cost Estimation
Input data:
WBS Resource requirements Historical data and information Resource rates Risks considered for cost estimates

Output data:
Cost estimates (as cost and schedule)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 250

Cost Budgeting
Input data:
Cost estimates WBS Project schedule Risk management plan

Output data:
Cost baseline

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 251

Cost Control Process

Manage to the budget Procedures for changing cost baseline
Cost tracking system Approvals for authorizing changes

Performance measurement techniques

Assess the magnitude of cost variations Earned value measures
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 252

Earned Value Management

Planned value
Physical work schedule to be performed Estimated value of the work

Earned value
Physical work accomplished Estimated value of the work

Actual costs
Amount of resources consumed
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 253

Project Progress
Phase transitioning control techniques Collecting Cost of Quality data Cost of Quality categories Cost, progress, and deliverable tracking

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 254

Tracking and Controlling

Phase transition control techniques Reporting and interpreting Cost of Quality (COQ) data Tracking elements and methods Project reviews

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 255

Tracking and Monitoring

Project Plan Tracing and control of Size, cost, schedule Management approval for Computer resource external commitments Technical activities Internal and external Software technical, cost, change management resource, schedule, and Product quality quality risks Reviews of status, plans, Measures, metrics and replanning data performance and issues Post-release lessons Formal reviews at learned review milestones
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 256

Phase Transition Controls

PERT CPM WBS Schedule

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 257

Scheduling Factors
Development environment Programmer skill levels Language and compiler stability Adjunct resources (e.g., test beds) Schedule commitments SDLC Corporate culture
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 258

Budgeting
BCWP ACWP BCWS Earned Value Analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 259

Earned Value Analysis Computations

Scheduled Completion Task 1 Task 2 Task 3 Task 4 Task n <count to date> <count to date> <cost to date> <cost to date> Actual Completion Budgeted Cost Actual Cost

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 260

Earned Values
Cost Variance = BCWP ACWP Schedule Variance = BCWP BCWS Cost Performance Index = BCWP/ACWP Earned Value = BCWP
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 261

Earned Value Analysis

ACWP
Cost Overrun Target Cost

BCWS

BCWP

Cost Variance Schedule Variance

Slippage

Time
Douglas Hoffman

Target Completion

Overrun Date

Copyright 1997-2004, SQM, LLC.

Slide 262

Other Cost Metrics

Planned Value (PV) Earned Value (EV) Actual Cost (AC) Budgeted [cost] at Completion (BC) Latest Revised Estimate (LRE) Schedule Variance = EV -PV Cost Variance = EV - AC Schedule Performance Indicator = EV/PV Cost Performance Indicator = EV/AC
Copyright 1997-2004, SQM, LLC. Slide 263

Douglas Hoffman

Quality Cost Responsibilities

QA Accounting Management General Manager

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 264

Quality Cost Improvement

Identify categories Collect costs Oversee accuracy Analyze data

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 265

Quality Cost Program

Budgeting Cost Categories Advantages/Limitations Optimum Costs

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 266

Optimum Quality Costs

Cost Per Good Unit
Total Costs Failure Costs

Optimum Point Cost of Appraisal Plus Prevention

100% Bad
Douglas Hoffman Copyright 1997-2004, SQM, LLC.

100% Good
Slide 267

Project Reviews
Senior Management Review Project Status Review Closed Loop Methodologies
Project Plan updates Corrective action plans Lessons learned
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 268

Implementation

Project management tools Reporting Cost of Quality data Trade-offs in release decisions

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 269

Risk Management
Risk management planning methods Risk probability Product release decisions Software security, safety, and hazard analysis issues
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 270

Risk Management Planning

Project risks
Product performance Actual costs higher than budgeted Late product delivery

Risk areas

Douglas Hoffman

Technical (performance) Supportability (performance) Programmatic (performance) Cost Schedule

Copyright 1997-2004, SQM, LLC. Slide 271

Risk Management
Features - Costs - Schedule Uncertainties
Requirements People Metrics Data Subjective values Model correctness

Risk management process

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 272

Continuous Risk Management

Identify Analyze Plan Track Control Mitigate Communicate
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 273

Analyze

Plan

Identify

Communicate

Track

Mitigate

Control

Uncertainty in Quantification

Product requirements Variability in personal performance Inaccuracies in measurements Judgement variations

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 274

Risk Probabilities
Components of risk
An [unwanted] event A likelihood of occurrence The impact of the event Ability to notice the event

Risk identification Qualitative risk analysis

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 275

Risk Response - Contingency Plans

Avoidance: action to eliminate the risk Transference: shift the consequence elsewhere Mitigation: reduce the likelihood or consequence Acceptance: decision not to change
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 276

Product Release Decisions

Decision attributes
Content (functionality) Schedule Quality

Release Criteria
Identified deliverables Documents reviewed Testing complete with acceptable pass rate Regression testing complete Prioritized defects resolved Planned release criteria all met Risks identified and mitigated Exceptions to plans are resolved Customer notified Product Support plan in place Quality criteria have been met
Slide 277

Management team decision

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Project Corrective Actions

Factors requiring trade-off
Schedule Cost Quality Customer requirements Functionality

Typical decisions
Outsourcing software development (contractors) Phased deliveries of increasing functionality Phased delivery of customer requirements Delivery with known non-critical defects Re-negotiating cost or schedule

Schedule compression most common issue

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 278

Security, Safety, and Hazards

Faults Authentication Safety-Critical software (HACCP) Real-time logic Petri net models Fault tree analysis
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 279

Common Security Faults

Programmer errors Input Overflows Fault prevention Syntax checking flaws Making a function too general Documentation
Lapses Wrong Missing Confusing Faulty

Easter Eggs Miss-configured security functions Default passwords

Lack of security awareness

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 280

Metrics and Measurement

General Knowledge, Conduct, and Ethics Software Quality Management Software Engineering Processes Program and Project Management

Software Metrics, Measurement, and Analytical Methods

Software Verification and Validation (V&V) Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 281

Background for Software Metrics

Definitions Process and product measurement Analytical techniques

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 282

Software Metrics Questions

How much do we need to know before measuring How do we know what weve measured What do the measures really mean What arithmetic makes sense

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 283

Measurement Theory Definitions

Entity (object or event) Attribute (property of an entity) Measurement (assign a value) Primitive (measurable entity) Measure (value assigned) Reliability (repeatability) Validity (freedom from bias) Measurement Error (systematic and random) Representational Condition (model)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 284

Definitions of Metrics and Measures

Douglas Hoffman

Software Metrics Defect Error Failure Fault KCSI KLOC KSLOC Dependent/Independent Variables
Copyright 1997-2004, SQM, LLC. Slide 285

Measurement Theory
Conditions for measurement
Sets Relations Axioms Functions

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 286

Types of Measures
Scale Nominal Ordinal Interval Description Example Names or Colors categories Defect Ordered classifications Differences Temperature can be (F) measured Inherent zero Money start value Dispersion Tests Information Mode Chi-square only Sign or run Median Percentages test Standard or Arithmetic t-test average Mean F test deviation correlation Geometric or Percent analysis harmonic variation mean Center

Ratio

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 287

Central Limit Theorem

Population with mean variance x2 large number of samples (n of them) distribution of sample means is normal with population mean and variance (, x2)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 288

Measures of Central Tendency

Mean (X-bar)
arithmetic average

Mode
most frequent

Median
middle value (when ordered)
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 289

Measures of Dispersion
Range
largest - smallest

Variance (2, S2)

2 = (X - )2/N S2 = (X - X-bar)2/(n-1) = SQRT [ (X - )2/N ] S = SQRT [ (X - X-bar)2/(n-1) ]

Standard Deviation Coefficient of Variation

COV =( S/X-bar) * 100
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 290

Psychology of Metrics
Potential for problems Results based Clear objectives Process and product focus Avoid misuse Hawthorne effect
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 291

GQM Metric Selection Paradigm *

Goal
What you want to learn

Question
What answer you need to know

Metric
How to measure/ compute it
*

Basili and Rombach (1984)

Slide 292

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Process and product measurement

Process, product, and resource metrics Commonly used metrics Software quality attributes Defect detection effectiveness measures Program performance and process effectiveness
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 293

Designing Measures
Types of software measures
Product Process Resources

Purpose of metrics
Assessment Prediction

Model is required
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 294

Examples of Software Metrics

Product Spec Design Code Test Data

Process Develop spec Detailed design Testing

Resource Personnel Software Hardware Work area

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 295

Designing Measures
IEEE Standard 1061 Software Metrics Methodology IEEE Standard 982.1 Standard Dictionary of Measures IEEE Standard 982.2 Guide to using IEEE 982.1
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 296

IEEE 1061 Methodology

Establish Software Quality Requirements Identify Software Quality Metrics Implement the Metrics Analyze the Metrics Results Validate the Metrics
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 297

Metrics Examples
Halstead - Software Science Boehm - COCOMO Albrecht - Function Points McCabe - Cyclomatic Complexity De Marco - Bang Metrics LOC Measures
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 298

Software Quality Attributes

Attribute Metrics
Attribute
Verifiability Usability Reusability Testability Expandability or Extendability Performance Robustness Traceability

Metrics
Number of implementable requirements Average number of errors made by users in a given time Value of resources saved Number of requirements with test criteria; number of requirements planned Amount of spare capacity available Number of transactions per second Time to restart after system failure Number of requirements included in, assigned to, or tested

Faults/LOC Correctness Faults/Requirements Faults/Standards Actual or Allocated Efficiency Utilization Average labor days to Flexibility change Faults/Lines Integrity Faults relative to security Effort to couple or Inter-operability develop Maintainability Average labor days to fix Effort to transport or to Portability develop Reliability Faults/LOC

Tables 7.7 and 7.8 (pp. VII-26 & 27)

Douglas Hoffman Copyright 1997-2004, SQM, LLC.

(IEEE 610.12)
Slide 299

General Software Factors*

Correctness Reliability Efficiency Integrity Security Usability Maintainability Testability Flexibility Portability Reusability Inter-operability Understandability
*

Capers Jones, 1997

Slide 300

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Rayleigh Model

Defect Rate

A = high level design inspection B = low level design inspection C = code inspection D = component test E = system test F = customer usage

D Development Stage

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 301

Exponential Distribution

Defects per KCSI or KLOC

Time --> Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 302

Software Models
Time Between Failures (TBF) Fault Counts (FC) Examples

Douglas Hoffman

Jelinski-Moranda (J-M) (TBF) Littlewood (LW) (TBF) Goel-Okumoto (G-O) (FC) Goel-Okumoto (NHPP) (FC) Musa-Okumoto (FC) Delayed S and Inflection S (FC)
Copyright 1997-2004, SQM, LLC. Slide 303

Defect Removal Effectiveness

100
100
100

Errors _ found _ by _ inspection Error detection effectiveness = Total _ errors _ before _ inspection

(Michael Fagan, 1976)

Defects _ found _ by _ operation Defect detection effectiveness = Defects _ present _ at _ operation

Defect removal effectiveness =

Defects _ found Defects _ found _ + _ Defects _ not _ found

(Capers Jones, 1991)

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 304

Phase Containment
Development Phase
Requirements High level design Low level design Coding Unit test Integration System test

Defect Cause

Defect Detection

Requirements gathering Requirements analysis Requirements specification Requirements review Design work Design work Coding Bad defect fixes Bad defect fixes Bad defect fixes Inspection Inspection Inspection Unit testing Integration testing System testing

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 305

Process Measures
Steps *
Actual vs. Plan Extrapolate Assess impact of problems ID and evaluate alternatives Make decisions Monitor for changes

Trends Thresholds & Targets

*
Douglas Hoffman Copyright 1997-2004, SQM, LLC.

McGarry, 2002
Slide 306

Process Effectiveness
ISO 9000-2000
Procedures followed Measured results Process ownership Management support Incentive alignment Proper training Feedback encouraged Improvement results Uses technology

Measurements
Performance Reliability Usability Cost of process Cost of Quality Responsiveness Adaptability Dependability Applicability Accuracy Quality of products
Slide 307

CMM/CMMI Level
Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Metrics Data Integrity

Data correctness
Collection Analysis Comparisons

Data quality
Bias Rounding Sequence Timeliness Population statistics Entry errors Hunches Environment
Slide 308

Data errors Data management

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Software Data Error Types*

Calculation error Blank field Transfer data (prior project) Entry error Transfer data (this project) Impossible values Process sequence not followed
*
Douglas Hoffman Copyright 1997-2004, SQM, LLC.

Disney, 1998
Slide 309

Seven Quality Analysis Tools*

Douglas Hoffman

Check Sheet Pareto Diagram Histogram Scatter Diagram Graph Control Chart Cause and Effect Diagram
*
Copyright 1997-2004, SQM, LLC.

Ishikawa, 1991
Slide 310

Check Sheet (Checklist)

TEST CASE DOCUMENTATION
Run instructions complete, concise, and conforms to standards Evaluation instructions complete, concise, and conforms to standards All instructions provided in an on-line file Installation instructions complete, concise, and conforms to standards Special instructions complete, concise, and conforms to standards Configuration Management instructions complete, concise, and conforms to standards Tests are traceable to a test design specification.
11/1/00 Keith Stobie - Testware Evaluation

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 311

Pareto Diagram
93 76 59 86 80 60 40 20 0 C1 C2 C3 C4 Causes C5 C6 C7 %
Copyright 1997-2004, SQM, LLC. Slide 312

98

100

100

Douglas Hoffman

Count

38

Histogram
% OF TOTAL DEFECTS 50 40 30 20 10 0 5 10 15 20 25 30 35 40 45

Days the Defect Report is Open

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 313

Graphs
Line Graph 40 B 30 C A 20 10 0 1 3 5 7 9 D E F
Pie Chart

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 314

Scatter Diagram
35 30 25 20 15 10 5 0 0 1 2 3 4 5 6

Defects

Complexity Index

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 315

Control Chart
20 # Failures 15 10 5
LCL UCL

0 0 2 Time 4 6

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 316

Cause and Effect Diagram

Poor Planning
Lack of resources Insufficient time Wrong participants Incomplete documents No traceability Poor standards All defects not recorded No root cause analysis Defects not categorized

Insufficient Preparation
Graphs not available

Poor Overview

Lack of prep time No background

Ineffective Inspection
No process for defect certification Defect opened > 120 days

No follow-up Not trained No checklist

Moderator

Inspection Meeting

Follow-up

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 317

Other Quality Tools

Flow Charts Problem solving Root cause analysis Plan-Do-Check-Act (PDCA) 7 new tools
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 318

Example Flow Chart Symbols

Process Alternate Process Decision Data

Predefined Process

Document

Terminator

Preparation

Manual Input

Manual Operation Connector Merge

Display Off page Connector Data Storage

Delay Extract

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 319

Problem Solving Process

Principles

Douglas Hoffman

Clearly defined problem Participation w/out criticism Address root causes (not symptoms) Data-driven decisions State problem from customer view Generate a list of root causes Solutions to address largest causes Prioritize solutions
Copyright 1997-2004, SQM, LLC. Slide 320

Process

Root Cause Analysis

Identify Initial Cause Repeated Questioning
When was the problem found? Where could the problem have been found? Where should the problem have been found?
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 321

Demings PDCA cycle (Walter Shewhart Cycle)

Plan:
What to accomplish? What changes? What data? New observations? New test? How to use planned observations

Do:
Carry out the change or test

P A C D

Check:
Observe the effects of the change or test

Act:

Study the results. What lessons? What predictions? What improvements or changes (if any)?
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 322

Seven New Tools

Japanese names
1. Relations Diagram 2. Affinity diagram 3. Systematic diagram 4. Matrix diagram 5. Matrix analysis 6. Process decision program chart (PDPC) 7. Arrow diagram
Douglas Hoffman

American counterpart
2. Affinity diagram 3. Tree diagram 6. Process decision program chart (PDPC) 5. Matrix diagram 1. Interrelationship digraph (I.D.) 4. Prioritization matrices 7. Activity network diagram
Copyright 1997-2004, SQM, LLC. Slide 323

Affinity Diagrams
Exam Prep Example
Resources
CSQE Primer Literature Group study ASQ BOK Textbooks Mentor Self study

Learn
CSQE class Watch video SQE Classes Tutor

Preparation
Teach BOK Experience Start early Practice exams
Douglas Hoffman

Motivation
Bonus $ Listen to CSQEs Motivate self Develop pride Improve work
Slide 324

Study intensively Study subjects separately

Copyright 1997-2004, SQM, LLC.

Interrelationship Digraph Example

Cause Effect CSQE workshop Tutor

Get Primer

Prepare for CSQE exam

Attend class

Group study

College classes

Bonus

Get BOK

Call-in source Study tests Promotion takes CSQE Intensive study

Peers have CSQE

Self motivate

Job needs CSQE

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 325

Tree Diagrams
CSQE Test Prep Pass CSQE Exam

Recources

Obtain Knowledge Videos CSQE Prep Class Tutor SQE Classes

Prepare

Motivate

Primer CSQE BOK Other books

Experience Teach CSQE Study BOK Practice Exams

Self Ask for $ Listen to CSQEs Pride

Slide 326

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Criteria Weighting Matrix Example

Weighted Criteria
List Criteria Each person assigns percentages Sum to Totals Totals are composite weights
Douglas Hoffman

Criteria Work Experience Have a Tutor Study in Group Attend CSQE Refresher Study Old Tests High Motivation Total

Perosn 1 Perosn 2 Perosn 3 Perosn 4 Total Ranking Ranking Ranking Ranking 0.05 0.10 0.15 0.10 0.20 0.10 0.10 0.30 0.05 0.20 0.10 0.20 0.45 0.70 0.50

0.25 0.15 0.30 1.00

0.20 0.15 0.25 1.00

0.20 0.25 0.10 1.00

0.30 0.10 0.10 1.00

0.95 0.65 0.75 4.00

Slide 327

Copyright 1997-2004, SQM, LLC.

Criteria/Factor Ranking Matrix

Criteria/Factor Ranking
List Factors (facets) being weighed Consensus rank (order) Factors for each Criteria
Criteria Factors Knowledge and Ethics Quality Management Audits Engineering Processes Program Management Software Metrics Verification & Validation Configuration Management Work Experience 1 2 4 3 7 6 5 8 Have a Tutor 2 3 1 4 7 8 6 5 Study in Group 3 1 2 4 8 7 5 6 Attend Study Old High CSQE Tests Motivation Refresher 2 3 1 4 7 8 6 5 1 3 2 5 6 8 7 4 3 4 2 5 8 7 6 1

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 328

Prioritization Matrix Example

Prioritize
Multiply Weighted Criteria by Factor Ranking Compute Total for Each Factor
Weight Criteria
Factors

0.45 Work Experience 0.45 0.90 1.80 1.35 3.15 2.70 2.25 3.60

0.70 Have a Tutor 1.40 2.10 0.70 2.80 4.90 5.60 4.20 3.50

0.50 Study in Group 1.50 0.50 1.00 2.00 4.00 3.50 2.50 3.00

0.95 0.65 0.75 Attend Total Study Old High CSQE Tests Motivation Refresher 1.90 2.85 0.95 3.80 6.65 7.60 5.70 4.75 0.65 1.95 1.30 3.25 3.90 5.20 4.55 2.60 2.25 3.00 1.50 3.75 6.00 5.25 4.50 0.75 8.2 11.3 7.3 17.0 28.6 29.9 23.7 18.2

Knowledge and Ethics Quality Management Audits Engineering Processes Program Management Software Metrics Verification & Validation Configuration Management

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 329

Matrix Diagrams
L-type
Standard, two factor matrix

T-type
Two factors on Y-axis, one on X-axis

X-type
Two factors on each Y-axis and X-axis

Y-type
Two L-type matrices with common Y-axis Standard, three dimensional matrix

C-type
Two L-type matrices with common Y-axis Three dimensional matrix with unique values
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 330

X-type Matrix Example

O X ? Cause 1 Cause 2 Cause 3 Cause 4 Dept 1 Problem 1 Problem 2 Problem 3 Problem 4 Problem 5 Problem 6 ? ? O Dept 2 Dept 3 Dept 4 O Dept 5 O ? X X ? X

Team 1

Team 2 O

Team 3

X O ?

- Presence documented - Presence likely - Presence possible

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 331

Process Decision Program Charts (PDPC) Example 1

Major Categories 2nd Level Last Level Last Level What-ifs
Result RA4

Solutions to What-ifs
Contingency

A2 A1 A3 Start Goal B2 B1 B3

A4

A5

Result RA5 Result RB4

Contingency

B4

Contingency

B5

Result RB5

Contingency

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 332

Process Decision Program Chart Example 2

Enroll in CSQE Refresher Need for the CSQE Study With Class Have Friends Support Loss of Motivation Get Pumped Up

Get a Tutor Obtain Resources No CSQE Classes Study Alone Find Others

Study via Tutor Study via Tutor Find a CSQE Call Expert

Pass the Test

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 333

Activity Network Diagram Link

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 334

Sampling Theory and Techniques

Sampling is data collection Consumers risk () Producers risk () Auditing sampling Random sampling Sequential sampling Stratified sampling

Acceptance sampling Required sample size

Attributes plan Variables plan
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 335

Sampling Errors
Actual Quality
Called Good Called Bad Good 1-
producer's confidence

Bad
Type II Error

The Decision Made

Type I Error

1
consumer's confidence
Slide 336

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Verification and Validation

General Knowledge, Conduct, and Ethics Software Quality Management Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods

Software Verification and Validation

Software Configuration Management
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 337

V&V Theory
V&V planning procedures and tasks V&V program Evaluating software products and processes Interfaces
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 338

V & V Quality Attributes*

Douglas Hoffman

Accuracy Completeness Consistency Efficiency Expandability Flexibility Interoperability Maintainability Manageability

Portability Readability Reusability Reliability Safety Security Survivability Testability Usability * IEEE 1059
Slide 339

Copyright 1997-2004, SQM, LLC.

Vocabulary
Inspection Walkthrough Review Test V&V Regression
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 340

V & V Planning
Scope for V & V Establish objectives Analyze project inputs Select techniques and tools Develop the plan
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 341

V & V Methods
Evaluating products and processes Requirements traceability Evaluating requirements Evaluating interfaces Evaluating test plans Evaluating the severity of anomalies Assessing proposed modifications Which V & V tasks should be iterated
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 342

Reviewing a V & V Program

Evaluation criteria Requirements traceability Evaluate documents Evaluate test plans Evaluate test results Audit results and corrective actions
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 343

V & V Interfaces

Requirements Design Source code

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 344

Inspection Types
Desk check (walkthrough) Walkthrough (evaluate) Inspection (debug) Management review Technical review
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 345

Measures for Reviewing Inspection Efforts

Inspection Rate Defect Rate Effectiveness Ratio Trends Technical Accomplishments Resource Utilization
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 346

Review and Inspection Items

Project items Project change items Software tool items Software process items
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 347

Review and Inspection Process

Objectives Entry criteria Exit criteria Techniques and methods Participant roles
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 348

Inspection Data Collection

Defect type Defect class Defect severity Inspection measures Phase containment Data management Reporting
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 349

V&V Test planning and design

Types of tests Test tools Test strategies Test design Test coverage of specifications Test environments Supplier components and products Test plans
Copyright 1997-2004, SQM, LLC. Slide 350

Douglas Hoffman

Types of Tests

Douglas Hoffman

Functional Performance Regression Load Worst case Perfective Exploratory Random-Input Certification

Stress Usability Real Time Life* Collision* Security* Installation* Recovery*

* Not included in Primer
Slide 351

Copyright 1997-2004, SQM, LLC.

Test Levels
Unit Component Integration System Field
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 352

Test Strategies
Top down Bottom Up Black box White box Simulation I/O first Alpha/Beta testing Fault insertion
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 353

Fault-Error handling Equivalence class partitioning Boundary value analysis Cause effect graphing Error guessing Customer defects

Test Design
Excellent test cases Equivalence classes Scenarios Customer defects Test coverage Fault insertion
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 354

An Excellent Test Case

Reasonable probability of catching an error* Not redundant with other tests* Best of breed* Exercise to stress the area of interest Minimal use of other areas Neither too simple nor too complex* Makes failures obvious* Allows isolation and identification of errors
*

Kaner (1993)
Slide 355

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Test Coverage of Specifications

Functions States Data Time Localization Internationalization
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 356

Test Environments
Test Libraries Drivers and Stubs Harnesses Compatibility Labs Simulators Other tools
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 357

Supplier Quality
Testing supplier components
COTS MOTS Fully Developed

Customer deliverables specifications

User documentation Marketing materials Training materials
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 358

Test Planning
Types of plans
System Acceptance Validation

Implementation Scheduling Freezing Dependencies V-model Error repair models Acceptance testing Resources Analysis of test results
Slide 359

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Test Documentation
Defect recording Defect tracking Test Plan Test Log Test Design Test Case Test Procedure
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 360

Test Reports
Defect report Test result Test report Traceability matrix Test completion metrics Trouble reports
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 361

Test Management
Scheduling Freezing Resource Management Dependencies Analysis of Results

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 362

Reviewing Testing Efforts

Tasks
Technical results Planning Risk management Resource use

Methods
Management review Technical review Inspection Walkthrough Test Audit

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 363

Test Coverage of Code

Douglas Hoffman

Branch Condition Domain Cyclomatic complexity Boundary Path Individual predicates Data Functions
Copyright 1997-2004, SQM, LLC. Slide 364

Cyclomatic Complexity
R-1 R(G) = 6 N(G) = 7 E(G) = 11 V(G) = R(G) = 6 V(G) = E - N + 2 = 11 - 7 + 2 = 6 E-8 R-6 N-6 E-11 N-7 E-5 N-2 R-3 N-5 E-9 E-10 R-4 E-6 N-3 E-7 E-1 R-2 E-3 E-4 R-5 N-4 N-1 E-2

V(G) = 6
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 365

Severity of Anomalies*
Criticality
High Major Moderate

Description
Selected function affects critical performance of the system. Selected function affects important system performance. Selected function affects critical system performance, but workaround strategies can be implemented to compensate for loss of performance. Selected function has a noticeable effect on system performance, but only creates inconvenience to the user if the function does not perform in accordance with requirements.
*

Level
4 3 2

Low

Example from IEEE 1012-1998

Slide 366

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Configuration Management
General Knowledge, Conduct, and Ethics Software Quality Management Software Engineering Processes Program and Project Management Software Metrics, Measurement, and Analytical Methods Software Verification and Validation (V&V)

Software Configuration Management

Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 367

Configuration Management

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 368

Benefits from CM
Product attributes defined Documented product configuration Labeled and correlated Change evaluation Change management Configuration captured Configuration verification
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 369

Rationale for CM
Change inevitable Complexity increases Simultaneous updates Double maintenance Shared code Common code Versions evolve
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 370

CM Terms1
Baseline: A uniquely identified consistent group of related SCIs captured as a milestone. A specification or product that has been reviewed and agreed upon as a basis for future development, which can be changed only through procedures CASE: Computer Aided Software Engineering; Use of software tools to design, requirements tracing, code production, testing, doc generation, and other software activities Configuration: Functional and physical characteristics of hardware or software as set forth in technical docs or achieved in a product Configuration Control: Element of CM; Evaluation, coordination, approval (or dis-), and implementation of changes to CItems after establishment of C ID. Configuration Control Board (CCB): Group responsible for evaluating and approving (or dis-) proposed changes to C Items and ensuring implementation of approved changes
Copyright 1997-2004, SQM, LLC. Slide 371

Douglas Hoffman

CM Terms2
Configuration Identification: selecting the C Items for a system and recording their functional and physical characteristics in technical docs Configuration Item (CI): Hardware or Software (or both) designated for CM and treated as a single entity in the CM process Configuration Management (CM): Discipline applying technical and admin direction and surveillance to ID and doc the functional and physical characteristics of a C Item, control changes, record and report change processing and implementation status, and verify compliance with requirements Configuration Status Accounting: Element of CM: recording and reporting info to effectively manage a config; includes config ID, status of proposed changes, and implementation status of approved changes Patch: 1) Modification to object program by replacing code; 2) Modification to object program without recompiling from source Release: Formal notification and distribution of approved version
Copyright 1997-2004, SQM, LLC. Slide 372

Douglas Hoffman

CM Terms3
Software Configuration: programs, documents, data structures; all items produced as part of software engineering process Software Configuration Item (SCI): Information produced, such as source code, design documents, test suite, or program modules. The software subset of CI. That which is managed together. (SCI often includes tools so versions can be reproduced) Software Development Life Cycle (SDLC): Period of time begins at concept and ends when software is no longer available for use Software Engineering environment: Hardware, software, and firmware used for SE effort; includes equipment, compilers, assemblers, OS, debuggers, simulators, emulators, test tools, documentation tools, and DBMS Software Library: Controlled collection of software and related documents to aid Software development, use, or maintenance. Types: master, production, library, software development library, software repository, and system library Versions: named SCI set with a defined set of functional capabilities
Copyright 1997-2004, SQM, LLC. Slide 373

Douglas Hoffman

Configuration Infrastructure

Configuration management Library/repository processes Defect tracking and library tools

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 374

SCM Responsibilities
Configuration identification Configuration control Status accounting Co-chair audits and reviews Co-chair CCB Maintain baselines Implement SCM plan
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 375

SCM Tools
SEIs 15 concepts SEIs four SCM models
Check out/in Composition Long transactions Change sets
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 376

UNIX SCM Utilities

SCCS and RCS CVS Make

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 377

SCM tool features

Version Control Configurations Processes Change control Team support Library/Repository Security/Protection
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 378

Reporting/Query Tool integration Builds Releases Customization GUI

Configuration Identification

Douglas Hoffman

Configuration items Structure Identification of items Baselines Accessibility Traceability Software builds Methods
Copyright 1997-2004, SQM, LLC. Slide 379

Software Product Partitioning

Managerial factors Technical factors
Hierarchy External objects Internal objects Environment and Tools
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 380

Configuration Control
Version control Naming Configuration ID Library control Release process Patching
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 381

Configuration Control
Item and baseline control Proposed modifications Review and configuration control boards (CCBs) Concurrent development Traceability Version control Configuration item interfaces
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 382

Configuration Status Accounting

Status reporting Changes to configuration items and baselines Documentation control Configuration Audits
Functional configuration audit Physical configuration audit
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 383

Change Control
Change process Change review Change approval Baselines Impact analysis Change management Verification
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 384

CM Issues
Definition of configuration Status of configuration Change control Communication of changes Release and distribution issues
Product release process issues Packaging, production, and distribution
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 385

Exam Preparation
Get good rest Be prepared to focus Bring indexed library Clear your head Relax and stay cool Soft pencils, calculator, munchies
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 386

About The Exam

Know the rules
Score is number right Time is main enemy

Take care of the answer sheet

Verify name and exam information Check the numbering layout How to mark/clear answers

Write notes in the test book

Eliminate wrong answers Mark probably right Where will you find the answer?
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 387

Taking The Exam

Answer easy questions first Make two + passes Re-read every question Parse hard questions Batch research Answer all questions at the end
Leave it until the last second Guess smart - use your notes
Douglas Hoffman Copyright 1997-2004, SQM, LLC. Slide 388

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 389

Douglas Hoffman

Copyright 1997-2004, SQM, LLC.

Slide 390