research-article

Comparing trace visualizations for program comprehension through controlled experiments

Authors:

Florian Fittkau,

Wilhelm Hasselbring,

Jan WallerAuthors Info & Claims

ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

Pages 266 - 276

Published: 16 May 2015 Publication History

Abstract

For efficient and effective program comprehension, it is essential to provide software engineers with appropriate visualizations of the program's execution traces. Empirical studies, such as controlled experiments, are required to assess the effectiveness and efficiency of proposed visualization techniques.

We present controlled experiments to compare the trace visualization tools Extravis and ExplorViz in typical program comprehension tasks. We replicate the first controlled experiment with a second one targeting a differently sized software system. In addition to a thorough analysis of the strategies chosen by the participants, we report on common challenges comparing trace visualization techniques. Besides our own replication of the first experiment, we provide a package containing all our experimental data to facilitate the verifiability, reproducibility and further extensibility of our presented results.

Although subjects spent similar time on program comprehension tasks with both tools for a small-sized system, analyzing a larger software system resulted in a significant efficiency advantage of 28 percent less time spent by using ExplorViz. Concerning the effectiveness (correct solutions for program comprehension tasks), we observed a significant improvement of correctness for both object system sizes of 39 and 61 percent with ExplorViz.

References

[1]

S. P. Reiss and A. Tarvo, "What is my program doing? program dynamics in programmer's terms," in Runtime Verification, ser. LNCS. Springer Berlin Heidelberg, 2012, vol. 7186, pp. 245--259.

Digital Library

[2]

B. Cornelissen, A. Zaidman, A. van Deursen, L. Moonen, and R. Koschke, "A systematic survey of program comprehension through dynamic analysis," IEEE TSE, vol. 35, no. 5, pp. 684--702, 2009.

Digital Library

[3]

R. Koschke, "Software visualization in software maintenance, reverse engineering, and re-engineering: A research survey," Journal of Software Maintenance and Evolution: Research and Practice, vol. 15, no. 2, pp. 87--109, 2003.

Digital Library

[4]

M.-A. Storey, K. Wong, and H. A. Müller, "How do program understanding tools affect how programmers understand programs?" in Proc. of the 4h Working Conference on Reverse Engineering (WCRE 1997). IEEE, 1997, pp. 12--21.

Digital Library

[5]

V. R. Basili, "The role of controlled experiments in software engineering research," in Empirical Software Engineering Issues: Critical Assessment and Future Directions. Springer, 2007, pp. 33--37.

Digital Library

[6]

W. F. Tichy, "Should computer scientists experiment more?" IEEE Computer, vol. 31, no. 5, pp. 32--40, May 1998.

Digital Library

[7]

D. I. Sjøberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahasanovic, N.-K. Liborg, and A. C. Rekdal, "A survey of controlled experiments in software engineering," IEEE TSE, vol. 31, no. 9, pp. 733--753, 2005.

Digital Library

[8]

W. F. Tichy, "Where's the science in software engineering?: Ubiquity symposium: The science in computer science," Ubiquity, vol. 2014, no. March, pp. 1:1--1:6, Mar. 2014.

Digital Library

[9]

B. Cornelissen, A. Zaidman, A. van Deursen, and B. van Rompaey, "Trace visualization for program comprehension: A controlled experiment," in Proc. of the 17th IEEE International Conference on Program Comprehension (ICPC 2009), May 2009, pp. 100--109.

[10]

B. Cornelissen, A. Zaidman, and A. van Deursen, "A controlled experiment for program comprehension through trace visualization," IEEE TSE, vol. 37, no. 3, pp. 341--355, May 2011.

Digital Library

[11]

B. Cornelissen, D. Holten, A. Zaidman, L. Moonen, J. J. Van Wijk, and A. Van Deursen, "Understanding execution traces using massive sequence and circular bundle views," in Proc. of the 15th IEEE International Conference on Program Comprehension (ICPC 2007). IEEE, 2007, pp. 49--58.

Digital Library

[12]

W. F. Tichy, "Hints for reviewing empirical work in software engineering," Empirical Software Engineering, vol. 5, no. 4, pp. 309--312, 2000.

Digital Library

[13]

V. R. Basili, F. Shull, and F. Lanubile, "Building knowledge through families of experiments," IEEE TSE, vol. 25, no. 4, pp. 456--473, 1999.

Digital Library

[14]

F. Fittkau, J. Waller, C. Wulf, and W. Hasselbring, "Live trace visualization for comprehending large software landscapes: The ExplorViz approach," in Proc. of the 1st International Working Conference on Software Visualization (VISSOFT 2013), Sep. 2013.

[15]

R. Wettel and M. Lanza, "Visualizing software systems as cities," in Proc. of the 4th International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT 2007), Jun. 2007.

[16]

F. Fittkau, S. Finke, W. Hasselbring, and J. Waller, "Experimental data for: Comparing trace visualizations for program comprehension through controlled experiments," May 2015.

[17]

B. Shneiderman, "Software psychology: Human factors in computer and information systems," Winthrop Publishers, Inc., 1980.

Digital Library

[18]

R. Wettel, "Software systems as cities," Ph.D. dissertation, University of Lugano, 2010.

[19]

R. Brooks, "Towards a theory of the comprehension of computer programs," International Journal of Man-Machine Studies, vol. 18, no. 6, pp. 543--554, 1983.

[20]

B. Shneiderman, "The eyes have it: A task by data type taxonomy for information visualizations," in Proc. of the IEEE Symposium on Visual Languages. IEEE, 1996, pp. 336--343.

Digital Library

[21]

J. Ehlers, A. van Hoorn, J. Waller, and W. Hasselbring, "Self-adaptive software system monitoring for performance anomaly localization," in Proc. of the 8th IEEE/ACM International Conference on Autonomic Computing (ICAC 2011). ACM, Jun. 2011, pp. 197--200.

Digital Library

[22]

R. M. Lindsay and A. S. Ehrenberg, "The design of replicated studies," The American Statistician, vol. 47, no. 3, pp. 217--228, 1993.

[23]

V. Rajlich and G. S. Cowan, "Towards standard for experiments in program comprehension," in Proc. of the 5th International Workshop on Program Comprehension (IWPC 1997). IEEE, 1997, pp. 160--161.

Digital Library

[24]

B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg, "Preliminary guidelines for empirical research in software engineering," IEEE TSE, vol. 28, no. 8, pp. 721--734, 2002.

Digital Library

[25]

A. Jedlitschka and D. Pfahl, "Reporting guidelines for controlled experiments in software engineering," in Proc. of the International Symposium on Empirical Software Engineering (ISESE 2005). IEEE, 2005.

[26]

G. A. Di Lucca and M. Di Penta, "Experimental settings in program comprehension: Challenges and open issues," in Proc. of the 14th IEEE International Conference on Program Comprehension (ICPC 2006). IEEE, 2006, pp. 229--234.

Digital Library

[27]

M. Di Penta, R. E. K. Stirewalt, and E. Kraemer, "Designing your next empirical study on program comprehension," in Proc. of the 15th IEEE International Conference on Program Comprehension (ICPC 2007), June 2007, pp. 281--285.

Digital Library

[28]

M. Sensalire, P. Ogao, and A. Telea, "Evaluation of software visualization tools: Lessons learned," in Proc. of the 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT 2009), Sep. 2009, pp. 19--26.

[29]

R. Wettel, M. Lanza, and R. Robbes, "Software systems as cities: A controlled experiment," in Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011). ACM, 2011, pp. 551--560.

Digital Library

[30]

V. R. Basili and D. M. Weiss, "A methodology for collecting valid software engineering data," IEEE TSE, vol. SE-10, no. 6, Nov. 1984.

Digital Library

[31]

M. J. Pacione, M. Roper, and M. Wood, "A novel software visualisation model to support software comprehension," in Proc. of the 11th Working Conference on Reverse Engineering (WCRE 2004), Nov. 2004.

Digital Library

[32]

R. Likert, "A technique for the measurement of attitudes," Archives of Psychology, vol. 22, no. 140, pp. 5--55, 1932.

[33]

K. Wong, "Rigi user's manual -- version 5.4.4," last accessed 14-09-05. {Online}. Available: http://www.rigi.cs.uvic.ca/downloads/rigi/doc/rigi-5.4.4-manual.pdf

[34]

S. S. Shapiro and M. B. Wilk, "An analysis of variance test for normality (complete samples)," Biometrika, pp. 591--611, 1965.

[35]

N. Razali and Y. B. Wah, "Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests," Journal of Statistical Modeling and Analytics, vol. 2, no. 1, pp. 21--33, 2011.

[36]

E. Pearson and H. Hartley, Biometrika Tables for Statisticians, 2nd ed. Cambridge University Press, 1972.

[37]

H. Levene, "Robust tests for equality of variances," Contributions to probability and statistics: Essays in honor of Harold Hotelling, vol. 2, pp. 278--292, 1960.

[38]

W. R. Shadish, T. D. Cook, and D. T. Campbell, Experimental and quasi-experimental designs for generalized causal inference. Wadsworth -- Cengage Learning, 2002.

[39]

N. Juristo and A. M. Moreno, Basics of software engineering experimentation. Springer, 2010.

Digital Library

[40]

C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering. Springer, 2012.

Digital Library

[41]

A. Marcus, D. Comorski, and A. Sergeyev, "Supporting the evolution of a software visualization tool through usability studies," in Proc. of the 13th International Workshop on Program Comprehension (IWPC 2005), May 2005, pp. 307--316.

Digital Library

[42]

J. Quante, "Do dynamic object process graphs support program understanding? -- A controlled experiment." in Proc. of the 16th IEEE International Conference on Program Comprehension (ICPC 2008), June 2008, pp. 73--82.

Digital Library

[43]

B. Sharif, G. Jetty, J. Aponte, and E. Parra, "An empirical study assessing the effect of SeeIT 3D on comprehension," in Proc. of the 1st IEEE Working Conference on Software Visualization (VISSOFT 2013), Sep. 2013, pp. 1--10.

[44]

C. Lange and M. R. V. Chaudron, "Interactive views to improve the comprehension of UML models -- An experimental validation," in Proc. of the 15th IEEE International Conference on Program Comprehension (ICPC 2007), June 2007, pp. 221--230.

Digital Library

[45]

T. Crick, B. A. Hall, and S. Ishtiaq, "Can I implement your algorithm?: A model for reproducible research software," in Proc. of the 2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). arXiv, Nov. 2014, pp. 1--4.

[46]

J. Trümper, J. Bohnet, and J. Döllner, "Understanding complex multithreaded software systems by using trace visualization," in Proc. of the 5th International Symposium on Software Visualization (SOFTVIS 2010). ACM, 2010, pp. 133--142.

Digital Library

Cited By

Krüger JÇalıklı GBerger TLeich TSaake GDumas MPfahl DApel SRusso A(2019)Effects of explicit feature traceability on program comprehensionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338968(338-349)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338968
Hasselbring WAvritzer AIosup AZhu XBecker S(2016)Microservices for ScalabilityProceedings of the 7th ACM/SPEC on International Conference on Performance Engineering10.1145/2851553.2858659(133-134)Online publication date: 12-Mar-2016
https://dl.acm.org/doi/10.1145/2851553.2858659

Index Terms

Comparing trace visualizations for program comprehension through controlled experiments
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Recommendations

Comparing Trace Visualizations for Program Comprehension through Controlled Experiments
ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

For efficient and effective program comprehension, it is essential to provide software engineers with appropriate visualizations of the program's execution traces. Empirical studies, such as controlled experiments, are required to assess the ...
A Controlled Experiment for Program Comprehension through Trace Visualization

Software maintenance activities require a sufficient level of understanding of the software at hand that unfortunately is not always readily available. Execution trace visualization is a common approach in gaining this understanding, and among our own ...
Visualizing Visualizations: User Interfaces for Managing and Exploring Scientific Visualization Data

The process of scientific visualization is inherently iterative. A good visualization comes from experimenting with visualization, rendering, and viewing parameters to bring out the most relevant information in the data. A good data visualization system ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

May 2015

325 pages

General Chair:
Andrea De Lucia
University of Salerno, Italy
,
Program Chairs:
Christian Bird
Microsoft Research
,
Rocco Oliveto
University of Molise, Italy

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

IEEE Press

Publication History

Published: 16 May 2015

Check for updates

Qualifiers

Research-article

Conference

ICSE '15

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '15: 37th International Conference on Software Engineering

May 16 - 24, 2015

Florence, Italy

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
145
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Krüger JÇalıklı GBerger TLeich TSaake GDumas MPfahl DApel SRusso A(2019)Effects of explicit feature traceability on program comprehensionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338968(338-349)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338968
Hasselbring WAvritzer AIosup AZhu XBecker S(2016)Microservices for ScalabilityProceedings of the 7th ACM/SPEC on International Conference on Performance Engineering10.1145/2851553.2858659(133-134)Online publication date: 12-Mar-2016
https://dl.acm.org/doi/10.1145/2851553.2858659

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents