research-article

Software Architecture Design Assistants Need Controlled Efficiency Experiments: Lessons Learned from a Survey

Authors:

Sebastian Lehrig,

Steffen BeckerAuthors Info & Claims

FoSADA '15: Proceedings of the 1st International Workshop on Future of Software Architecture Design Assistants

Pages 19 - 24

https://doi.org/10.1145/2751491.2751492

Published: 06 May 2015 Publication History

Abstract

Software architects use so-called software architecture design assistants to get tool-based, (semi-)automated support in engineering software systems. Compared to manual engineering, the main promise of such a support is that architects can create high-quality architectural designs more efficiently. Yet, current practice in evaluating whether this promise is kept is based on case studies conducted by the original authors of respective design assistants. The downside of such evaluations is that they are neither generalizable to third-party software architects nor can be used for quantitative efficiency comparisons between competing design assistants. To tackle this problem, we investigate how researchers can apply controlled experiments for evaluating the impact of software architecture design assistants on the efficiency of architects. For our investigation, we survey related controlled experiments. Based on this survey, we derive lessons learned in terms of best practices and challenges for such experiments.

References

[1]

V. Basili. The role of controlled experiments in software engineering research. In V. Basili, D. Rombach, K. Schneider, B. Kitchenham, D. Pfahl, and R. Selby, editors, Empirical Software Engineering Issues. Critical Assessment and Future Directions, volume 4336 of Lecture Notes in Computer Science, pages 33--37. Springer Berlin Heidelberg, 2007.

Digital Library

[2]

V. R. Basili, G. Caldiera, and H. D. Rombach. The Goal Question Metric Approach. In J. J. Marciniak, editor, Encyclopedia of Software Engineering, pages 578--583. John Wiley & Sons, 2 edition, 2002.

[3]

B. Boehm and V. R. Basili. Software defect reduction top 10 list. Computer, 34(1):135--137, Jan. 2001.

Digital Library

[4]

F. Brosig, S. Kounev, and C. Paclat. Using WebLogic Diagnostics Framework to Enable Performance Prediction for Java EE Applications. Oracle Technology Network (OTN) Article, September 2009.

[5]

D. Falessi, M. A. Babar, G. Cantone, and P. Kruchten. Applying empirical software engineering to software architecture: Challenges and lessons learned. Empirical Softw. Engg., 15(3):250--276, June 2010.

Digital Library

[6]

K. Farias, A. Garcia, J. Whittle, C. Chavez, and C. Lucena. Evaluating the effort of composing design models: A controlled experiment. In Proceedings of the 15th International Conference on Model Driven Engineering Languages and Systems, MODELS'12, pages 676--691, Berlin, Heidelberg, 2012. Springer-Verlag.

Digital Library

[7]

E. Golden, B. E. John, and L. Bass. The value of a usability-supporting architectural pattern in software architecture design: A controlled experiment. In Proceedings of the 27th International Conference on Software Engineering, ICSE '05, pages 460--469, New York, NY, USA, 2005. ACM.

Digital Library

[8]

T. Haitzer and U. Zdun. Controlled experiment on the supportive effect of architectural component diagrams for design understanding of novice architects. In Proceedings of the 7th European Conference on Software Architecture, ECSA'13, pages 54--71, Berlin, Heidelberg, 2013. Springer-Verlag.

Digital Library

[9]

S. Herold, H. Klus, Y. Welsch, C. Deiters, A. Rausch, R. Reussner, K. Krogmann, H. Koziolek, R. Mirandola, B. Hummel, M. Meisinger, and C. Pfaller. CoCoME - The Common Component Modeling Example. In A. Rausch, R. Reussner, R. Mirandola, and F. Pléil, editors, The Common Component Modeling Example, volume 5153 of Lecture Notes in Computer Science, pages 16--53. Springer Berlin Heidelberg, 2008.

Digital Library

[10]

M. Höst, B. Regnell, and C. Wohlin. Using students as subjects -- a comparative study of students and professionals in lead-time impact assessment. Empirical Softw. Engg., 5(3):201--214, Nov. 2000.

Digital Library

[11]

N. Huber, S. Becker, C. Rathfelder, J. Schweinghaus, and R. Reussner. Performance modeling in industry: a case study on storage virtualization. In Software Engineering, 2010 ACM/IEEE 32nd International Conference on, volume 2, pages 1--10, May 2010.

Digital Library

[12]

A. Jedlitschka, M. Ciolkowski, and D. Pfahl. Reporting experiments in software engineering. In F. Shull, J. Singer, and D. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 201--228. Springer London, 2008.

[13]

S. Lehrig. Architectural templates: Engineering scalable SaaS applications based on architectural styles. In Proceedings of the MODELS 2013 Doctoral Symposium co-located with the 16th International ACM/IEEE Conference on Model Driven Engineering Languages and Systems (MODELS 2013), volume 1071, pages 48--55, Miami, USA, 2013. CEUR-WS.org.

[14]

S. Lehrig and T. Zolynski. Performance prototyping with protocom in a virtualised environment: A case study. In Proceedings to Palladio Days 2011, 17--18 November 2011, FZI Forschungszentrum Informatik, Karlsruhe, Germany, 2011.

[15]

A. Martens, H. Koziolek, L. Prechelt, and R. Reussner. From monolithic to component-based performance evaluation of software architectures. Empirical Softw. Engg., 16(5):587--622, Oct. 2011.

Digital Library

[16]

J. D. McGregor, F. Bachman, L. Bass, P. Bianco, and M. Klein. Using an architecture reasoning tool to teach software architecture. In Software Engineering Education Training, 2007. CSEET '07. 20th Conference on, pages 275--282, July 2007.

Digital Library

[17]

K. Ng, J. Kramer, and J. Magee. A case tool for software architecture design. Automated Software Engineering, 3(3--4):261--284, 1996.

[18]

K. Ng, J. Kramer, J. Magee, and N. Dulay. The software architect's assistant-a visual environment for distributed programming. In System Sciences, 1995. Proceedings of the Twenty-Eighth Hawaii International Conference on, volume 2, pages 254--263 vol.2, Jan 1995.

Digital Library

[19]

M. Shahin, P. Liang, and Z. Li. Do architectural design decisions improve the understanding of software architecture? two controlled experiments. In Proceedings of the 22nd International Conference on Program Comprehension, ICPC 2014, pages 3--13, New York, NY, USA, 2014. ACM.

Digital Library

[20]

C. U. Smith and L. G. Williams. Performance solutions: a practical guide to creating responsive, scalable software. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 2002.

Digital Library

[21]

C. Wohlin, P. Runeson, M. Hst, M. C. Ohlsson, B. Regnell, and A. Wessln. Experimentation in Software Engineering. Springer Publishing Company, Incorporated, 2012.

Digital Library

[22]

Y. Yang, M. He, M. Li, Q. Wang, and B. Boehm. Phase distribution of software development effort. In Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '08, pages 61--69, New York, NY, USA, 2008. ACM.

Digital Library

Cited By

Lehrig SBecker S(2016)Using Performance Models for Planning the Redeployment to Infrastructure-as-a-Service Environments: A Case Study2016 12th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA)10.1109/QoSA.2016.17(11-20)Online publication date: Apr-2016
https://doi.org/10.1109/QoSA.2016.17
Galster MMirakhorli MKoziolek A(2015)Twin Peaks goes AgileACM SIGSOFT Software Engineering Notes10.1145/2815021.281503840:5(47-49)Online publication date: 14-Sep-2015
https://dl.acm.org/doi/10.1145/2815021.2815038

Index Terms

Software Architecture Design Assistants Need Controlled Efficiency Experiments: Lessons Learned from a Survey
1. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Exploratory data analysis
2. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Software implementation planning
        Software design techniques
    2. Software development process management
      1. Software development methods

Recommendations

Do architectural design decisions improve the understanding of software architecture? two controlled experiments
ICPC 2014: Proceedings of the 22nd International Conference on Program Comprehension

Architectural design decision (ADD) and its design rationale, as a paradigm shift on documenting and enriching architecture design description, is supposed to facilitate the understanding of architecture and the reasoning behind the design rationale, ...
Two controlled experiments on model-based architectural decision making

We conduct controlled experiments concerning reusable architectural decision models.We measure the effect of decision models on decision making and documentation.Reusable architectural decision models increase the effectiveness of users.Reusable ...
The value of a usability-supporting architectural pattern in software architecture design: a controlled experiment
ICSE '05: Proceedings of the 27th international conference on Software engineering

Design patterns have been claimed to facilitate modification and improve understanding in software design. A controlled experiment was performed to assess the usefulness of portions of a Usability-Supporting Architectural Pattern (USAP) in modifying the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FoSADA '15: Proceedings of the 1st International Workshop on Future of Software Architecture Design Assistants

May 2015

34 pages

ISBN:9781450334389

DOI:10.1145/2751491

General Chair:
Philippe Kruchten
University of British Columbia, Canada
,
Program Chairs:
Roger Champagne
ÉTS, Canada
,
Anne Koziolek
Karlsruhe Institute of Technology, Germany

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Commission

Conference

CompArch '15

Sponsor:

SIGSOFT
IEEE-CS\DATC

CompArch '15: Federated Events on Component-Based Software Engineering and Software Architecture

May 6, 2015

QC, Montréal, Canada

Acceptance Rates

FoSADA '15 Paper Acceptance Rate 4 of 4 submissions, 100%;

Overall Acceptance Rate 4 of 4 submissions, 100%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lehrig SBecker S(2016)Using Performance Models for Planning the Redeployment to Infrastructure-as-a-Service Environments: A Case Study2016 12th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA)10.1109/QoSA.2016.17(11-20)Online publication date: Apr-2016
https://doi.org/10.1109/QoSA.2016.17
Galster MMirakhorli MKoziolek A(2015)Twin Peaks goes AgileACM SIGSOFT Software Engineering Notes10.1145/2815021.281503840:5(47-49)Online publication date: 14-Sep-2015
https://dl.acm.org/doi/10.1145/2815021.2815038

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten