Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

How to Verify and Validate a System or Process

•Download as PPTX, PDF•
•

Does your product or system meet the requirements? Find out in this webinar. Your host will discuss a verification and validation process that has worked for hundreds of our clients to answer this question. Then you will learn how to use a model-based systems engineering tool, Innoslate, to develop your own V&V process document. Then your host will dive into Test Center. Test Center allows for easy test case capture and traceability to requirements and provides the ability to run test cases within one easy-to-use view. This webinar will cover: 1. V&V process 2. Test Center 3. Documents View 4. Traceability Matrix

1 of 26

1

HowtoVerifyandValidateaSystemorProcess Presented by: Steven H. Dam, Ph.D., ESEP – Part Four of the “How to MBSE” webinar series.

2

AskUsYourQuestions • Ask us your questions using the panel on the right • This presentation is being recorded and will be made available to you. • Contact us after the webinar through • support@Innoslate.com • Call 571.485.7800 • LinkedIn Innoslate User Group • Twitter Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

3

MeetYourHost • President and Founder of SPEC Innovations • Participated in the development of C4ISR and the DoDAF • Expert Systems Engineering Professionals Certificate • steven.dam@specinnovations.com • @stevenhdam Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

4

WhatYouWillLearnToday • How Different Customers View V&V • How to Develop Your V&V Requirements, V&V Plans, and Test Suites from the Beginning • How to Capture and Report Test Results in Innoslate® • How to Evaluate Processes using Sopatra® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

5

V&VinDoD • DoD focuses on “operational effectiveness, operational suitability and survivability (including cybersecurity), or lethality.” [Defense Acquisition Guidebook (DAG), Chapter 8] • Organizationally, DoD divides into Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E) • Development of a Test and Evaluation Master Plan (TEMP) is considered critical • DoD supports major test ranges and facilities, worldwide • An entire command focuses on interoperability (Joint Interoperability Test Command – JITC) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

6

V&VinNASA • According to NASA: • “Product verification proves that an end product (whether built, coded, bought, or reused) for any element within the system structure conforms to its requirements or specifications.” • “While verification proves whether ‘the product was done right,’ validation proves whether ‘the right product was done.’” • These definitions then map back to the DoD approach: • DT&E == Verification • OT&E == Validation • So, frequently different customers will frame V&V in different terms and have different processes and procedures, but the end goals are the same Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam From NASA Systems Engineering Handbook

7

EachOrganizationWillDefineTheirOwnProcessesforCM • Our process has many of the same elements as the DoD and NASA processes • We include the use of our “SE Knowledgebase” as a key part of our process • We currently use Innoslate® as our primary SE Knowledgebase and tool • It can ingest or capture all the work products and may even be the source of many of those products • We can also use it to conduct analyses and create any necessary reports Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam SE Knowledgebase Review Test Results with ERB Develop Deployment Plan and Conduct Beta Testing Assess Risk and Plan Mitigation Updated SE Knowledgebase Return Item to “Integrate Solution” (if necessary) Beta Test Report Deficiencies? YES NO Workaround Identified Verify/Validate to Requirements Verification/ Validation Requirements From Design & Analysis Phase Perform Regression Testing Acceptable Regression Tests? YES NO Deployment Plan Submit Changes to CM Process Regression Test Report Inspection and Test Report Updated Test Plan

8

“HowtoMBSE”WebinarSeries: WhereDoesV&VFitintotheSystemLifecycle? Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam Documents View Requirements View Modeling Simulation Risk Analysis Timeline WBS Design Engineering Integrations Test Center Sopatra EVERYWHERE!

9

HowtoDevelopYourV&VRequirements,V&VPlans, andTestSuitesfromtheBeginning Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

10

CreateVerification/ValidationRequirementsandDocuments • Start with Validation Requirements • Customer needs get translated into implementation independent requirements • Determine then how you will validate that these requirements have been met • May require a separate set of requirements • Once system requirements have been decomposed to specification level, create the verification requirements as well Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

11

CreateTestPlansandTestSuites • Test Plans can be developed using built-in templates or create your won • Essential to be done early in the lifecycle to identify long lead items, such as test facilities and automated test equipment (ATE) • Preliminary Test Suites can also be created as verification requirements are developed • Just about everything can be planned well ahead of the actual V&V execution activities • This upfront planning ensures better requirements and much quicker V&V Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

12

EnsureRequirementsTraceabilityEarly • The Traceability Matrix provides a way to trace requirements to Test Cases • The traceability assist provides a check on the probability that a requirement and test case should be connected Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

13

IncludeVerificationMethodsforRequirements • Verification Methods are identified as labels in Innoslate® • Analysis • Demonstration • Inspection • Modeling & Simulation • Test • Additional labels can be added, if needed by ensuring that the “Verification Method” label is used as a parent • These labels are provided at the Organization level Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

14

HowtoCaptureandReportTestResultsinInnoslate® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

15

UseTestCenterandTestSuitestoCaptureResults • Test Center provides a means to define Test Suites for each major V&V activity, using all methods (analysis, demonstration, inspection, simulation, test) • Innoslate® can support the simulation portion of V&V as well • Data recorded in files can be stored as Artifacts or a “File” attribute can be added to the Test Case class to associate files with each Test Case Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

16

UseAlltheTestCaseAttributesandDiagrams • Test Cases are a subclass of Actions • This feature enables the Test case to be use in Action Diagrams, Timelines, and other project planning and execution activities • Test Cases have extra attributes to capture V&V related information • Expected Result • Actual Result • Setup • Add others as needed, e.g. • Result Link • File Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

17

CreateTestReportsandVerificationMatrices • Test Reports are another document type in Innoslate® • No templates are currently available • Add your own custom templates • Verification Matrices are reports in the Document View or can be developed in Database View Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

18

HowtoEvaluateProcessesusingSopatra® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

19

Sopatra®WasDesignedtoEvaluateProcessesandProcedures • Sopatra® translates a text-based Standard Operating Procedure into an Action Diagram • It uses other information (input devices, output displays, acronyms, and time distribution databases) to better develop the process diagram • Best practice actions (Hear, See, Decide) are automatically added • Special metrics were defined to identify the Available Operational Time Window (AOTW) and Time of Procedure (ToP) to calculate the Procedure Buffer Time (PBT) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

20

Sopatra®’sMonteCarloSimulatorProvidestheanalysisResults • A well developed procedure will show “all green” • Any failed runs will show up in “red” • Some procedures may be able to have an acceptable failure rate and the check for that will highlight the fail rate in red when exceeded Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

21

LiveDemo Let’s dive on in. Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

22

PredicttheFuture! A well calibrated simulation can do this • Remember what we do and how we do it matters • High-quality analyses save projects, products, systems, and lives • Develop high-quality simulations by using Innoslate’s discrete event and Monte Carlo Simulators • Reduce operational errors by using Sopatra Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

23

Questions and Answers Use the panel on the right to ask your questions Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

24

Next Webinar in “How to MBSE” “Webinar: How to Verify and Validate” May 26, 2022, 11:00 am ET Go To Webinar Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

25

MoreResources SPEC Innovations offers training, books, videos, documentation, trials, and more Training: specinnovations.com/training Books: “Real MBSE” textbook and lab manual available on Amazon Videos: Visit the SPEC Innovations YouTube channel LinkedIn: Innoslate and Systems Engineers User Group Documentation: help.Innoslate.com Trial: cloud.Innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

26

SPEC Innovations offers training, books, videos, documentation, trials, and more Thank you! Visit cloud.innoslate.com for a trial. SPEC Innovations @Innoslate Innoslate User Group Innoslate.com/blog 571.485.7800 innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

More Related Content

How to Verify and Validate a System or Process

  • 1. HowtoVerifyandValidateaSystemorProcess Presented by: Steven H. Dam, Ph.D., ESEP – Part Four of the “How to MBSE” webinar series.
  • 2. AskUsYourQuestions • Ask us your questions using the panel on the right • This presentation is being recorded and will be made available to you. • Contact us after the webinar through • support@Innoslate.com • Call 571.485.7800 • LinkedIn Innoslate User Group • Twitter Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 3. MeetYourHost • President and Founder of SPEC Innovations • Participated in the development of C4ISR and the DoDAF • Expert Systems Engineering Professionals Certificate • steven.dam@specinnovations.com • @stevenhdam Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 4. WhatYouWillLearnToday • How Different Customers View V&V • How to Develop Your V&V Requirements, V&V Plans, and Test Suites from the Beginning • How to Capture and Report Test Results in Innoslate® • How to Evaluate Processes using Sopatra® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 5. V&VinDoD • DoD focuses on “operational effectiveness, operational suitability and survivability (including cybersecurity), or lethality.” [Defense Acquisition Guidebook (DAG), Chapter 8] • Organizationally, DoD divides into Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E) • Development of a Test and Evaluation Master Plan (TEMP) is considered critical • DoD supports major test ranges and facilities, worldwide • An entire command focuses on interoperability (Joint Interoperability Test Command – JITC) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 6. V&VinNASA • According to NASA: • “Product verification proves that an end product (whether built, coded, bought, or reused) for any element within the system structure conforms to its requirements or specifications.” • “While verification proves whether ‘the product was done right,’ validation proves whether ‘the right product was done.’” • These definitions then map back to the DoD approach: • DT&E == Verification • OT&E == Validation • So, frequently different customers will frame V&V in different terms and have different processes and procedures, but the end goals are the same Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam From NASA Systems Engineering Handbook
  • 7. EachOrganizationWillDefineTheirOwnProcessesforCM • Our process has many of the same elements as the DoD and NASA processes • We include the use of our “SE Knowledgebase” as a key part of our process • We currently use Innoslate® as our primary SE Knowledgebase and tool • It can ingest or capture all the work products and may even be the source of many of those products • We can also use it to conduct analyses and create any necessary reports Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam SE Knowledgebase Review Test Results with ERB Develop Deployment Plan and Conduct Beta Testing Assess Risk and Plan Mitigation Updated SE Knowledgebase Return Item to “Integrate Solution” (if necessary) Beta Test Report Deficiencies? YES NO Workaround Identified Verify/Validate to Requirements Verification/ Validation Requirements From Design & Analysis Phase Perform Regression Testing Acceptable Regression Tests? YES NO Deployment Plan Submit Changes to CM Process Regression Test Report Inspection and Test Report Updated Test Plan
  • 8. “HowtoMBSE”WebinarSeries: WhereDoesV&VFitintotheSystemLifecycle? Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam Documents View Requirements View Modeling Simulation Risk Analysis Timeline WBS Design Engineering Integrations Test Center Sopatra EVERYWHERE!
  • 10. CreateVerification/ValidationRequirementsandDocuments • Start with Validation Requirements • Customer needs get translated into implementation independent requirements • Determine then how you will validate that these requirements have been met • May require a separate set of requirements • Once system requirements have been decomposed to specification level, create the verification requirements as well Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 11. CreateTestPlansandTestSuites • Test Plans can be developed using built-in templates or create your won • Essential to be done early in the lifecycle to identify long lead items, such as test facilities and automated test equipment (ATE) • Preliminary Test Suites can also be created as verification requirements are developed • Just about everything can be planned well ahead of the actual V&V execution activities • This upfront planning ensures better requirements and much quicker V&V Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 12. EnsureRequirementsTraceabilityEarly • The Traceability Matrix provides a way to trace requirements to Test Cases • The traceability assist provides a check on the probability that a requirement and test case should be connected Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 13. IncludeVerificationMethodsforRequirements • Verification Methods are identified as labels in Innoslate® • Analysis • Demonstration • Inspection • Modeling & Simulation • Test • Additional labels can be added, if needed by ensuring that the “Verification Method” label is used as a parent • These labels are provided at the Organization level Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 14. HowtoCaptureandReportTestResultsinInnoslate® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 15. UseTestCenterandTestSuitestoCaptureResults • Test Center provides a means to define Test Suites for each major V&V activity, using all methods (analysis, demonstration, inspection, simulation, test) • Innoslate® can support the simulation portion of V&V as well • Data recorded in files can be stored as Artifacts or a “File” attribute can be added to the Test Case class to associate files with each Test Case Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 16. UseAlltheTestCaseAttributesandDiagrams • Test Cases are a subclass of Actions • This feature enables the Test case to be use in Action Diagrams, Timelines, and other project planning and execution activities • Test Cases have extra attributes to capture V&V related information • Expected Result • Actual Result • Setup • Add others as needed, e.g. • Result Link • File Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 17. CreateTestReportsandVerificationMatrices • Test Reports are another document type in Innoslate® • No templates are currently available • Add your own custom templates • Verification Matrices are reports in the Document View or can be developed in Database View Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 18. HowtoEvaluateProcessesusingSopatra® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 19. Sopatra®WasDesignedtoEvaluateProcessesandProcedures • Sopatra® translates a text-based Standard Operating Procedure into an Action Diagram • It uses other information (input devices, output displays, acronyms, and time distribution databases) to better develop the process diagram • Best practice actions (Hear, See, Decide) are automatically added • Special metrics were defined to identify the Available Operational Time Window (AOTW) and Time of Procedure (ToP) to calculate the Procedure Buffer Time (PBT) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 20. Sopatra®’sMonteCarloSimulatorProvidestheanalysisResults • A well developed procedure will show “all green” • Any failed runs will show up in “red” • Some procedures may be able to have an acceptable failure rate and the check for that will highlight the fail rate in red when exceeded Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 21. LiveDemo Let’s dive on in. Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 22. PredicttheFuture! A well calibrated simulation can do this • Remember what we do and how we do it matters • High-quality analyses save projects, products, systems, and lives • Develop high-quality simulations by using Innoslate’s discrete event and Monte Carlo Simulators • Reduce operational errors by using Sopatra Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 23. Questions and Answers Use the panel on the right to ask your questions Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 24. Next Webinar in “How to MBSE” “Webinar: How to Verify and Validate” May 26, 2022, 11:00 am ET Go To Webinar Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 25. MoreResources SPEC Innovations offers training, books, videos, documentation, trials, and more Training: specinnovations.com/training Books: “Real MBSE” textbook and lab manual available on Amazon Videos: Visit the SPEC Innovations YouTube channel LinkedIn: Innoslate and Systems Engineers User Group Documentation: help.Innoslate.com Trial: cloud.Innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
  • 26. SPEC Innovations offers training, books, videos, documentation, trials, and more Thank you! Visit cloud.innoslate.com for a trial. SPEC Innovations @Innoslate Innoslate User Group Innoslate.com/blog 571.485.7800 innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam

Editor's Notes

  1. Exercise: Quality