research-article

Revisit of automatic debugging via human focus-tracking analysis

Authors:

Baowen XuAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Pages 808 - 819

https://doi.org/10.1145/2884781.2884834

Published: 14 May 2016 Publication History

Abstract

In many fields of software engineering, studies on human behavior have attracted a lot of attention; however, few such studies exist in automated debugging. Parnin and Orso conducted a pioneering study comparing the performance of programmers in debugging with and without a ranking-based fault localization technique, namely Spectrum-Based Fault Localization (SBFL). In this paper, we revisit the actual helpfulness of SBFL, by addressing some major problems that were not resolved in Parnin and Orso's study. Our investigation involved 207 participants and 17 debugging tasks. A user-friendly SBFL tool was adopted. It was found that SBFL tended not to be helpful in improving the efficiency of debugging. By tracking and analyzing programmers' focus of attention, we characterized their source code navigation patterns and provided in-depth explanations to the observations. Results indicated that (1) a short "first scan" on the source code tended to result in inefficient debugging; and (2) inspections on the pinpointed statements during the "follow-up browsing" were normally just quick skimming. Moreover, we found that the SBFL assistance may even slightly weaken programmers' abilities in fault detection. Our observations imply interference between the mechanism of automated fault localization and the actual assistance needed by programmers in debugging. To resolve this interference, we provide several insights and suggestions.

References

[1]

R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. An evaluation of similarity coefficients for software fault localization. In Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing, pages 39--46, Riverside, USA, 2006.

Digital Library

[2]

R. Bednarik. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies, 70(2):143--155, 2012.

Digital Library

[3]

D. Binkley, M. Davis, D. Lawrie, J. I. Maletic, C. Morrell, and B. Sharif. The impact of identifier style on effort and comprehension. Empirical Software Engineering, 18(2):219--276, 2013.

Digital Library

[4]

C. Chen, H.-G. Gross, and A. Zaidman. Improving service diagnosis through increased monitoring granularity. In Proceedings of the 7th IEEE International Conference on Software Security and Reliability, pages 129--138, Gaithersburg, Maryland, USA, 2013.

Digital Library

[5]

M. Chen and V. Lim. Eye gaze and mouse cursor relationship in a debugging task. In Proceedings of HCI International Posters' Extended Abstracts, pages 468--472. Springer, 2013.

[6]

M. E. Crosby and J. Stelovsky. How do we read algorithms? a case study. Computer, 23(1):25--35, 1990.

Digital Library

[7]

M. A. Francel and S. Rugaber. The value of slicing while debugging. Science of Computer Programming, 40(2):151--169, 2001.

Digital Library

[8]

J. Freeman, R. Dale, and T. Farmer. Hand in motion reveals mind in motion. Frontiers in Psychology, 2:59, 2011.

[9]

J. D. Gannon. Human factors in software engineering. Computer, 12(12):6--7, 1979.

Digital Library

[10]

C. Gouveia, J. Campos, and R. Abreu. Using html5 visualizations in software fault localization. In Proceedings of the First IEEE Working Conference on Software Visualization, pages 1--10, Eindhoven, Netherlands, 2013.

[11]

J. Huang, R. W. White, and S. Dumais. No clicks, no problem: Using cursor movements to understand and improve search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1225--1234, Vancouver, BC, Canada, 2011.

Digital Library

[12]

J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering, pages 467--477, Florida, USA, 2002.

Digital Library

[13]

A. Ko, B. Myers, M. Coblenz, and H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on Software Engineering, 32(12):971--987, 2006.

Digital Library

[14]

A. J. Ko and B. A. Myers. Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 151--158, Vienna, Austria, 2004.

Digital Library

[15]

B. Korel and J. Laski. Dynamic program slicing. Information Processing Letters, 29(3):155--163, 1988.

Digital Library

[16]

J. Lawrance, C. Bogart, M. Burnett, R. Bellamy, K. Rector, and S. Fleming. How programmers debug, revisited: An information foraging theory perspective. IEEE Transactions on Software Engineering, 39(2):197--215, 2013.

Digital Library

[17]

T.-D. B. Le, D. LO, and F. Thung. Should i follow this fault localization tool's output? automated prediction of fault localization effectiveness. Empirical Software Engineering, pages 1--38, 2015.

Digital Library

[18]

C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead. Does bug prediction support human developers? findings from a google case study. In Proceedings of 35th IEEE/ACM International Conference on Software Engineering, pages 372--381, San Francisco, USA, 2013.

Digital Library

[19]

Mooctest. http://mooctest.net/.

[20]

L. Naish, H. J. Lee, and K. Ramamohanarao. A model for spectra-based software diagnosis. ACM Transactions on Software Engineering and Methodology, 20(3):11:1--11:32, 2011.

Digital Library

[21]

C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proceedings of the International Symposium on Software Testing and Analysis, pages 199--209, Toronto, Canada, 2011.

Digital Library

[22]

D. Piorkowski, S. Fleming, C. Scaffidi, L. John, C. Bogart, B. John, M. Burnett, and R. Bellamy. Modeling programmer navigation: A head-to-head empirical evaluation of predictive models. In Proceedings of 2011 IEEE Symposium on Visual Languages and Human-Centric Computing, pages 109--116, Pittsburgh, PA, USA, 2011.

[23]

P. Rodeghero, C. McMillan, P. W. McBurney, N. Bosch, and S. D'Mello. Improving automated source code summarization via an eye-tracking study of programmers. In Proceedings of the 36th International Conference on Software Engineering, pages 390--401, Hyderabad, India, 2014.

Digital Library

[24]

R. Saha, M. Lease, S. Khurshid, and D. Perry. Improving bug localization using structured information retrieval. In Proceedings of the 28th International Conference on Automated Software Engineering, pages 345--355, Palo Alto, California, USA, 2013.

Digital Library

[25]

I. Salman, A. T. Misirli, and N. Juristo. Are students representatives of professionals in software engineering experiments? In Proceedings of 37th IEEE/ACM International Conference on Software Engineering, pages 666--676, Florence, Italy, 2015.

Digital Library

[26]

B. Sharif, M. Falcone, and J. I. Maletic. An eye-tracking study on the role of scan time in finding source code defects. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 381--384, Santa Barbara, California, 2012.

Digital Library

[27]

B. Sharif, J. Maletic, et al. The effects of layout on detecting the role of design patterns. In Proceedings of the IEEE Conference on Software Engineering Education and Training, pages 41--48, Pittsburgh, USA, 2010.

Digital Library

[28]

J. Silva and O. Chitil. Combining algorithmic debugging and program slicing. In Proceedings of the 8th ACM SIGPLAN international conference on Principles and practice of declarative programming, pages 157--166, Venice, Italy, 2006.

Digital Library

[29]

I. Sommerville. Software Engineering. Pearson, 9 edition, 2009.

Digital Library

[30]

R. Stein and S. E. Brennan. Another person's eye gaze as a cue in solving programming problems. In Proceedings of the 6th international conference on Multimodal interfaces, pages 9--15, State College, USA, 2004.

Digital Library

[31]

Y. Tao, J. Kim, S. Kim, and C. Xu. Automatically generated patches as debugging aids: a human study. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 64--74, Hong Kong, China, 2014.

Digital Library

[32]

F. Thung, D. Lo, L. Jiang, et al. Are faults localizable? In Proceedings of the IEEE 9th Working Conference on Mining Software Repositories, pages 74--77, Zurich, Switzerland, 2012.

Digital Library

[33]

H. Uwano, M. Nakamura, A. Monden, and K.-i. Matsumoto. Analyzing individual performance of source code review using reviewers' eye movement. In Proceedings of the 2006 symposium on Eye tracking research & applications, pages 133--140, San Diego, USA, 2006.

Digital Library

[34]

Q. Wang, C. Parnin, and A. Orso. Evaluating the usefulness of ir-based fault localization techniques. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, pages 1--11, Baltimore, Maryland, USA, 2015.

Digital Library

[35]

M. Weiser and J. Lyle. Experiments on slicing-based debugging aids. In Proceedings of the Workshop on Empirical Studies of Programmers, pages 187--197, USA, 1986.

Digital Library

[36]

W. E. Wong, V. Debroy, and B. Choi. A family of code coverage-based heuristics for effective fault localization. Journal of Systems and Software, 83(2):188--208, 2010.

Digital Library

[37]

X. Xie, T. Y. Chen, F.-C. Kuo, and B. Xu. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering and Methodology, 22(4):31:1--31:40, 2013.

Digital Library

[38]

J. Xuan and M. Monperrus. Learning to combine multiple ranking metrics for fault localization. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution, pages 191--200, Victoria, Canada, 2014.

Digital Library

[39]

J. Xuan and M. Monperrus. Test case purification for improving fault localization. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 52--63, Hong Kong, China, 2014.

Digital Library

[40]

X. Zhang, R. Gupta, and Y. Zhang. Precise dynamic slicing algorithms. In Proceedings of the 25th International Conference on Software Engineering, pages 319--329, Portland, Oregon, 2003.

Digital Library

Cited By

Rafi MKim DChen AChen TWang S(2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660793
Eladawy HLe Goues CBrun YRoychoudhury APaiva AAbreu RStorey M(2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639095
Shen CYang WJia HPan MZhou Y(2024)Richen: Automated enrichment of Git documentation with usage examples and scenariosJournal of Software: Evolution and Process10.1002/smr.2662Online publication date: 13-Mar-2024
https://doi.org/10.1002/smr.2662
Show More Cited By

Index Terms

Revisit of automatic debugging via human focus-tracking analysis
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Understanding the use of spectrum‐based fault localization
Summary
Developers spend significant time locating and fixing bugs, which is often performed manually. Although spectrum‐based fault localization (SFL) techniques aim at helping developers to locate faults, they are not yet used in practice. Recent ...

This paper presents a user study of spectrum‐based fault localization (SFL), showing that SFL can improve the developers' debugging effectiveness, leading them close to faulty code excerpts. SFL was well‐accepted by the participants of our study but ...
Contextualizing spectrum-based fault localization

Context: Fault localization is among the most expensive tasks in software development. Spectrum-based fault localization (SFL) techniques seek to pinpoint faulty program elements (e.g., statements), by sorting them only by their suspiciousness scores. ...
Efficient statistical debugging via hierarchical instrumentation
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

Debugging is known to be a notoriously painstaking and time-consuming task. As one major family of automated debugging, statistical debugging approaches have been well investigated over the past decade to assist in debugging. All these approaches ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

May 2016

1235 pages

ISBN:9781450339001

DOI:10.1145/2884781

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key Basic Research and Development Program of China

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
477
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rafi MKim DChen AChen TWang S(2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660793
Eladawy HLe Goues CBrun YRoychoudhury APaiva AAbreu RStorey M(2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639095
Shen CYang WJia HPan MZhou Y(2024)Richen: Automated enrichment of Git documentation with usage examples and scenariosJournal of Software: Evolution and Process10.1002/smr.2662Online publication date: 13-Mar-2024
https://doi.org/10.1002/smr.2662
Yan YJiang SZhang YZhang C(2024)Improving fault localization via weighted execution graph and graph attention networkJournal of Software: Evolution and Process10.1002/smr.261936:6Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1002/smr.2619
Callaghan DFischer BJust RFraser G(2023)Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite ReductionProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598148(1445-1457)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598148
Badihi SAhmed KLi YRubin J(2023)Responsibility in Context: On Applicability of Slicing in Semantic Regression Analysis2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00057(563-575)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00057
Sun WQian WLuo BChen Z(2023)MuTCR: Test Case Recommendation via Multi-Level Signature Matching2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00022(179-190)Online publication date: May-2023
https://doi.org/10.1109/AST58925.2023.00022
Amario de Souza Hde Souza Lauretto MKon FLordello Chaim M(2023)Understanding the use of spectrum‐based fault localizationJournal of Software: Evolution and Process10.1002/smr.2622Online publication date: 11-Oct-2023
https://doi.org/10.1002/smr.2622
Wong WGao RLi YAbreu RWotawa FLi D(2023)Software Fault Localization: an Overview of Research, Techniques, and ToolsHandbook of Software Fault Localization10.1002/9781119880929.ch1(1-117)Online publication date: 21-Apr-2023
https://doi.org/10.1002/9781119880929.ch1
Lopes Da Silva DLordello Chaim MAmario De Souza H(2022)Data flow Subsumption and its Impact on Spectrum-based Fault LocalizationProceedings of the 7th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3559744.3559750(39-48)Online publication date: 3-Oct-2022
https://dl.acm.org/doi/10.1145/3559744.3559750
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents