research-article

On the benefits of using dedicated models in validation processes for behavioral specifications

Authors:

Jennifer Brings,

Lisa Krajinski,

Thorsten WeyerAuthors Info & Claims

ICSSP '19: Proceedings of the International Conference on Software and System Processes

Pages 44 - 53

https://doi.org/10.1109/ICSSP.2019.00016

Published: 25 May 2019 Publication History

Abstract

[Background] In model-based engineering models need to be regularly validated by manual assessment. Therefore, review processes have established. This commonly means that a visual inspection of the respective models is conducted. [Aims] In this paper, we report a study that aims at investigating whether automatically generated review models can aid the manual review of model-based specifications. We investigate this for the case of embedded systems' functional design. [Method] For that purpose, we compared the manual review of the functional design with the review of an automatically generated review model. In this paper, we report on a controlled experiment to compare effectiveness, efficiency, user confidence, and subjective supportiveness of both review artifacts. [Results] The experiment results show that the use of the review model as review artifact for the functional design is significantly more effective, leads to a significantly higher user confidence in decision making and is valued as significantly more supportive than the review of the original functional design. [Conclusions] Our experiment provides evidences that reviewing a generated review model instead of the original model-based specification of the functional design increases the quality of the reviews. Our findings also indicate that the use of generated review models have the potential to improve the review of model-based specifications in general.

References

[1]

M. E. Fagan, "Advances in software inspections," IEEE Trans. Softw. Eng., vol. SE-12, no. 7, pp. 744--751, Jul. 1986.

Digital Library

[2]

A. A. Porter, L. G. Votta, and V. R. Basili, "Comparing detection methods for software requirements inspections: a replicated experiment," IEEE Trans. Softw. Eng., vol. 21, no. 6, pp. 563--575, Jun. 1995.

Digital Library

[3]

S. Liu, Y. Chen, F. Nagoya, and J. A. McDermid, "Formal Specification-Based Inspection for Verification of Programs," IEEE Trans. Softw. Eng., vol. 38, no. 5, pp. 1100--1122, Sep. 2012.

Digital Library

[4]

A. Bacchelli and C. Bird, "Expectations, Outcomes, and Challenges of Modern Code Review," in 2013 International Conference on Software Engineering, Piscataway, NJ, USA, 2013, pp. 712--721.

Digital Library

[5]

T. Gilb and D. Graham, Software Inspection. Addison-Wesley, 1993.

Digital Library

[6]

Z. Zhang, V. Basili, and B. Shneiderman, "An Empirical Study of Perspective-Based Usability Inspection," Proc. Hum. Factors Ergon. Soc. Annu. Meet., vol. 42, no. 19, pp. 1346--1350, Oct. 1998.

[7]

J. C. S. P. Leite and P. A. Freeman, "Requirements validation through viewpoint resolution," IEEE Trans. Softw. Eng., vol. 17, no. 12, pp. 1253--1269, Dec. 1991.

Digital Library

[8]

B. Boehm and V. R. Basili, "Software Defect Reduction Top 10 List," IEEE Comput., vol. 34, pp. 135--137, Jan. 2001.

Digital Library

[9]

G. Travassos et al., "Detecting Defects in Object-oriented Designs: Using Reading Techniques to Increase Software Quality," in 14th ACM SIGPLAN Conf. on Object-oriented Programming, Systems, Languages, and Applications, 1999, pp. 47--56.

Digital Library

[10]

E. M. Clarke, E. A. Emerson, and J. Sifakis, "Model Checking: Algorithmic Verification and Debugging," Commun ACM, vol. 52, no. 11, pp. 74--84, Nov. 2009.

Digital Library

[11]

O. Balci, "Validation, verification, and testing techniques throughout the life cycle of a simulation study," in Winter Simulation Conference, 1994, pp. 215--220.

Digital Library

[12]

"ISO/IEC/IEEE International Standard - Systems and software engineering - Life cycle processes -Requirements engineering," ISOIECIEEE 291482011E, pp. 1--94, Dec. 2011.

[13]

M. Daun, J. Höfflinger, and T. Weyer, "Function-centered engineering of embedded systems: Evaluating industry needs and possible solutions," in 2014 9th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), 2014, pp. 1--9.

[14]

"ISO 26262-1:2011 - Road vehicles - Functional safety - Part 1: Vocabulary." {Online}. https://www.iso.org/standard/43464.html.

[15]

"ARP4761: Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment - SAE International." {Online}.http://standards.sae.org/arp4761/.

[16]

A. Jedlitschka, M. Ciolkowski, and D. Pfahl, "Reporting Experiments in Software Engineering," in Guide to Advanced Empirical Software Engineering, Springer, London, 2008, pp. 201--228.

[17]

C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering: An Introduction. Norwell, MA, USA: Kluwer Academic Publishers, 2000.

Digital Library

[18]

M. E. Fagan, "Design and code inspections to reduce errors in program development," IBM Syst. J., vol. 15, no. 3, pp. 182--211, 1976.

Digital Library

[19]

R. Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools. 1999.

Digital Library

[20]

V. R. Basili et al., "The empirical investigation of Perspective-Based Reading," Empir. Softw. Eng., vol. 1, no. 2, pp. 133--164, Jan. 1996.

[21]

Y. WANG et al., "Process Improvement of Peer Code Review and Behavior Analysis of Its Participants," in 39th SIGCSE Technical Symp. on Computer Science Education, 2008, pp. 107--111.

Digital Library

[22]

C. Denger and M. Ciolkowski, "High quality statecharts through tailored, perspective-based inspections," in 29th Euromicro Conference, 2003, pp. 316--323.

Digital Library

[23]

J. Miller, M. Wood, and M. Roper, "Further Experiences with Scenarios and Checklists," Empir. Softw. Eng., vol. 3, no. 1, pp. 37--64, Mar. 1998.

Digital Library

[24]

L. He and J. Carver, "PBR vs. checklist: a replication in the n-fold inspection context.," 2006, pp. 95--104.

Digital Library

[25]

J. C. Maldonado et al., "Perspective-Based Reading: A Replicated Experiment Focused on Individual Reviewer Effectiveness," Empir. Softw. Eng., vol. 11, no. 1, pp. 119--142, Mar. 2006.

Digital Library

[26]

A. Porter and L. Votta, "Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects," Empir. Softw. Eng., vol. 3, no. 4, pp. 355--379, Dec. 1998.

Digital Library

[27]

O. Laitenberger, K. E. Emam, T. G. Harbich, "An internally replicated quasi-experimental comparison of checklist and perspective based reading of code documents," IEEE Trans. Softw. Eng., vol. 27, no. 5, pp. 387--421, 2001.

Digital Library

[28]

T. Berling and P. Runeson, "Evaluation of a perspective based review method applied in an industrial setting," IEE Proc. - Softw., vol. 150, no. 3, pp. 177--184, Jun. 2003.

[29]

G. Sabaliauskaite, S. Kusumoto, and K. Inoue, "Assessing defect detection performance of interacting teams in object-oriented design inspection," Inf. Softw. Technol., vol. 46, pp. 875--886, Oct. 2004.

[30]

M. Daun, A. Salmon, T. Weyer, and K. Pohl, "The impact of students' skills and experiences on empirical results: a controlled experiment with undergraduate and graduate students," in 19th Int. Conf. on Evaluation and Assessment in Software Engineering, 2015, Paper 29.

Digital Library

[31]

M. Daun, T. Weyer, and K. Pohl, "Improving manual reviews in function-centered engineering of embedded systems using a dedicated review model," Softw. Syst. Model., Feb. 2019.

[32]

M. Daun, J. Brings, and T. Weyer, "On the Impact of the Model-Based Representation of Inconsistencies to Manual Reviews," in Conceptual Modeling, 2017, pp. 466--473.

[33]

M. Broy, "Challenges in Automotive Software Engineering," in Proceedings of the 28th International Conference on Software Engineering, New York, NY, USA, 2006, pp. 33--42.

[34]

K. Beetz and W. Böhm, "Challenges in Engineering for Software-Intensive Embedded Systems," in Model-Based Engineering of Embedded Systems, Springer, Berlin, Heidelberg, 2012, pp. 3--14.

[35]

A. Pretschner, M. Broy, I. H. Kruger, and T. Stauner, "Software Engineering for Automotive Systems: A Roadmap," in Future of Software Engineering, 2007. FOSE '07, 2007, pp. 55--71.

Digital Library

[36]

S. Brinkkemper and S. Pachidi, "Functional Architecture Modeling for the Software Product Industry," in Software Architecture, 2010, pp. 198--213.

Digital Library

[37]

A. Jantsch and I. Sander, "On the roles of functions and objects in system specification," in 8<sup>th</sup> Int. WS on Hardw./Softw. Codesign. 2000, pp. 8--12.

Digital Library

[38]

A. Vogelsang and S. Fuhrmann, "Why feature dependencies challenge the requirements engineering of automotive systems: An empirical study," in 21st IEEE Int. Requirements Engineering Conference, 2013, pp. 267--272.

[39]

X. Blanc, I. Mounier, A. Mougenot, and T. Mens, "Detecting model inconsistency through operation-based model construction," in ACM/IEEE 30th Int. Conf. on Software Engineering, 2008, pp. 511--520.

Digital Library

[40]

J. Grundy, J. Hosking, and W. B. Mugridge, "Inconsistency management for multiple-view software development environments," IEEE Trans. Softw. Eng., vol. 24, no. 11, pp. 960--981, Nov. 1998.

Digital Library

[41]

P. Fradet, D. L. Métayer, and M. Périn, "Consistency Checking for Multiple View Software Architectures," in Software Engineering --- ESEC/FSE '99, Springer, Berlin, Heidelberg, 1999, pp. 410--428.

Digital Library

[42]

R. F. Paige, P. J. Brooke, and J. S. Ostroff, "Metamodel-based Model Conformance and Multiview Consistency Checking," ACM Trans Softw Eng Methodol, vol. 16, no. 3, Jul. 2007.

Digital Library

[43]

D. Milicev, "Automatic model transformations using extended UML object diagrams in modeling environments," IEEE Trans. Softw. Eng., vol. 28, no. 4, pp. 413--431, Apr. 2002.

Digital Library

[44]

H. Giese and R. Wagner, "From model transformation to incremental bidirectional model synchronization," Softw. Syst. Model., vol. 8, no. 1, pp. 21--43, Feb. 2009.

[45]

H. Giese, S. Hildebrandt, and S. Neumann, "Model Synchronization at Work: Keeping SysML and AUTOSAR Models Consistent," in Graph Transformations and Model-Driven Engineering, 2010, pp. 555--579.

Digital Library

[46]

M. Daun, T. Weyer, and K. Pohl, "Validating the functional design of embedded systems against stakeholder intentions," in 2nd Int. Conf. on Model-Driven Engineering and Softw. Development, 2014, pp. 333--339.

[47]

ITU, Recommendation Z. 120: Message Sequence Charts, 2011.

[48]

S. Mauw and M. A. Reniers, "Operational Semantics for MSC'96," Comput Netw, vol. 31, no. 17, pp. 1785--1799, Jun. 1999.

Digital Library

[49]

L. Hélouët and P. L. Maigat, Decomposition of Message Sequence Charts. 2000.

[50]

V. Venkatesh and H. Bala, "Technology Acceptance Model 3 and a Research Agenda on Interventions," Decis. Sci., vol. 39, no. 2, pp. 273--315, May 2008.

[51]

M. Höst, B. Regnell, and C. Wohlin, "Using Students as Subjects---A Comparative Study of Students and Professionals in Lead-Time Impact Assessment," Empir. Softw. Eng., vol. 5, no. 3, pp. 201--214, Nov. 2000.

Digital Library

[52]

M. Svahnberg, A. Aurum, and C. Wohlin, "Using Students As Subjects -an Empirical Evaluation," in 2<sup>nd</sup> ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 2008, pp. 288--290.

Digital Library

[53]

P. Runeson, "Using Students as Experiment Subjects - An Analysis on Graduate and Freshmen Student Data," 7th Int. Conf. Empir. Assess. Eval. Softw. Eng., pp. 95--102, 2003.

[54]

W. F. Tichy, "Hints for Reviewing Empirical Work in Software Engineering," Empir. Softw Engg, vol. 5, no. 4, pp. 309--312, Dec. 2000.

Digital Library

[55]

J. Siegmund, N. Siegmund, and S. Apel, "Views on Internal and External Validity in Empirical Software Engineering," in 37th IEEE Int. Conf. on Software Engineering, 2015, vol. 1, pp. 9--19.

Digital Library

[56]

I. Salman, A. T. Misirli, and N. Juristo, "Are Students Representatives of Professionals in Software Engineering Experiments?," in 37th Int. Conf. on Software Engineering, 2015, pp. 666--676.

Digital Library

[57]

F. Faul, E. Erdfelder, A.-G. Lang, and A. Buchner, "G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences," Behav. Res. Methods, vol. 39, no. 2, pp. 175--191, May 2007.

Cited By

Daun MBrings J(2023)Investigating Factors Influencing Students’ Assessment of Conceptual ModelsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593960(470-474)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593960
Daun MBrings J(2023)Aggregating N-fold Requirements Inspection ResultsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593465(339-347)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593465
Daun MBrings JWeyer T(2020)Do Instance-level Review Diagrams Support Validation Processes of Cyber-Physical System SpecificationsProceedings of the International Conference on Software and System Processes10.1145/3379177.3388893(11-20)Online publication date: 26-Jun-2020
https://dl.acm.org/doi/10.1145/3379177.3388893

Recommendations

Model inspections in the engineering of collaborative cyber‐physical systems with instance‐level review diagrams
Abstract
Model inspections are important to ensure high‐quality software and to satisfy legal obligations in model‐based engineering processes. As model‐based specifications are typically documented on type‐level, errors concerning the interactions between ...

This paper contributes a definition of instance‐level review diagrams in message sequence charts notation and reports results of a controlled experiment to evaluate instance‐level review diagrams. Results show that instance‐level review models support the ...
Do Instance-level Review Diagrams Support Validation Processes of Cyber-Physical System Specifications: Results from a Controlled Experiment
ICSSP '20: Proceedings of the International Conference on Software and System Processes

In the field of safety-critical systems, manual reviews are important to ensure high-quality software and to satisfy legal obligations. When applying model-based engineering approaches, no longer are only textual requirements specifications or software ...
The impact of students' skills and experiences on empirical results: a controlled experiment with undergraduate and graduate students
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

In empirical software engineering research, graduate students are often seen as legitimate substitutes for industry professionals. It has been also argued in the literature that the generalizability of empirical results from experiments with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSSP '19: Proceedings of the International Conference on Software and System Processes

May 2019

181 pages

General Chair:
Stanley M Sutton,
Program Chairs:
Ove Armbrust,
Regina Hebig

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 25 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 25, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
28
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Daun MBrings J(2023)Investigating Factors Influencing Students’ Assessment of Conceptual ModelsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593960(470-474)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593960
Daun MBrings J(2023)Aggregating N-fold Requirements Inspection ResultsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593465(339-347)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593465
Daun MBrings JWeyer T(2020)Do Instance-level Review Diagrams Support Validation Processes of Cyber-Physical System SpecificationsProceedings of the International Conference on Software and System Processes10.1145/3379177.3388893(11-20)Online publication date: 26-Jun-2020
https://dl.acm.org/doi/10.1145/3379177.3388893

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten