research-article

Questions developers ask while diagnosing potential security vulnerabilities with static analysis

Authors:

Brittany Johnson,

Emerson Murphy-Hill,

Heather Richter LipfordAuthors Info & Claims

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Pages 248 - 259

https://doi.org/10.1145/2786805.2786812

Published: 30 August 2015 Publication History

Abstract

Security tools can help developers answer questions about potential vulnerabilities in their code. A better understanding of the types of questions asked by developers may help toolsmiths design more effective tools. In this paper, we describe how we collected and categorized these questions by conducting an exploratory study with novice and experienced software developers. We equipped them with Find Security Bugs, a security-oriented static analysis tool, and observed their interactions with security vulnerabilities in an open-source system that they had previously contributed to. We found that they asked questions not only about security vulnerabilities, associated attacks, and fixes, but also questions about the software itself, the social ecosystem that built the software, and related resources and tools. For example, when participants asked questions about the source of tainted data, their tools forced them to make imperfect tradeoffs between systematic and ad hoc program navigation strategies.

References

[1]

1 research.csc.ncsu.edu/dlf/ 9. REFERENCES

[2]

N. Ammar and M. Abi-Antoun. Empirical evaluation of diagrams of the run-time structure for coding tasks. In Reverse Engineering (WCRE), 2012 19th Working Conference on, pages 367–376. IEEE, 2012.

Digital Library

[3]

A. Austin and L. Williams. One technique is not enough: A comparison of vulnerability discovery techniques. In Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on, pages 97–106. IEEE, 2011.

Digital Library

[4]

M. Barnett, R. DeLine, A. Lal, and S. Qadeer. Get me here: Using verification tools to answer developer questions. Technical Report MSR-TR-2014-10, February 2014.

[5]

H. Chen and D. Wagner. Mops: an infrastructure for examining security properties of software. In Proceedings of the 9th ACM conference on Computer and communications security, pages 235–244. ACM, 2002.

Digital Library

[6]

L. Dukes, X. Yuan, and F. Akowuah. A case study on web application security testing with tools and manual testing. In Southeastcon, 2013 Proceedings of IEEE, pages 1–6. IEEE, 2013.

[7]

T. Fritz and G. C. Murphy. Using information fragments to answer the questions developers ask. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pages 175–184. ACM, 2010.

Digital Library

[8]

T. Fritz, G. C. Murphy, E. Murphy-Hill, J. Ou, and E. Hill. Degree-of-knowledge: Modeling a developer’s knowledge of code. ACM Trans. Softw. Eng. Methodol., 23(2):14:1–14:42, Apr. 2014.

Digital Library

[9]

B. G. Glaser and A. L. Strauss. The discovery of grounded theory: Strategies for qualitative research. Transaction Publishers, 2009.

[10]

G. Guest, A. Bunce, and L. Johnson. How many interviews are enough? an experiment with data saturation and variability. Field methods, 18(1):59–82, 2006.

[11]

W. Hudson. Card Sorting. The Interaction Design Foundation, Aarhus, Denmark, 2013.

[12]

B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge. Why don’t software developers use static analysis tools to find bugs? In Software Engineering (ICSE), 2013 35th International Conference on, pages 672–681. IEEE, 2013.

Digital Library

[13]

N. Jovanovic, C. Kruegel, and E. Kirda. Pixy: A static analysis tool for detecting web application vulnerabilities. In Security and Privacy, 2006 IEEE Symposium on, pages 6–pp. IEEE, 2006.

Digital Library

[14]

A. J. Ko, R. DeLine, and G. Venolia. Information needs in collocated software development teams. In Proceedings of the 29th international conference on Software Engineering, pages 344–353. IEEE Computer Society, 2007.

Digital Library

[15]

A. J. Ko and B. A. Myers. Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 151–158. ACM, 2004.

Digital Library

[16]

O. Kononenko, D. Dietrich, R. Sharma, and R. Holmes. Automatically locating relevant programming help online. In Visual Languages and Human-Centric Computing (VL/HCC), 2012 IEEE Symposium on, pages 127–134. IEEE, 2012.

[17]

J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):pp. 159–174, 1977.

[18]

T. D. LaToza and B. A. Myers. Developers ask reachability questions. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pages 185–194. ACM, 2010.

Digital Library

[19]

T. D. LaToza and B. A. Myers. Hard-to-answer questions about code. In Evaluation and Usability of Programming Languages and Tools, page 8. ACM, 2010.

Digital Library

[20]

T. D. LaToza and B. A. Myers. Visualizing call graphs. In Visual Languages and Human-Centric Computing (VL/HCC), 2011 IEEE Symposium on, pages 117–124. IEEE, 2011.

[21]

S. Letovsky. Cognitive processes in program comprehension. Journal of Systems and software, 7(4):325–339, 1987.

Digital Library

[22]

V. B. Livshits and M. S. Lam. Finding security vulnerabilities in java applications with static analysis. In Usenix Security, pages 18–18, 2005.

Digital Library

[23]

M. Martin, B. Livshits, and M. S. Lam. Finding application errors and security flaws using pql: a program query language. In ACM SIGPLAN Notices, volume 40, pages 365–383. ACM, 2005.

Digital Library

[24]

A. Mockus, R. T. Fielding, and J. D. Herbsleb. Two case studies of open source software development: Apache and mozilla. ACM Trans. Softw. Eng. Methodol., 11(3):309–346, July 2002.

Digital Library

[25]

E. Murphy-Hill, R. Jiresal, and G. C. Murphy. Improving software developers’ fluency by recommending development environment commands. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, pages 42:1–42:11, New York, NY, USA, 2012. ACM.

Digital Library

[26]

J. Nielsen, T. Clemmensen, and C. Yssing. Getting access to what goes on in people’s heads?: reflections on the think-aloud technique. In Proceedings of the second Nordic conference on Human-computer interaction, pages 101–110. ACM, 2002.

Digital Library

[27]

C. Parnin and S. Rugaber. Programmer information needs after memory failure. In Program Comprehension (ICPC), 2012 IEEE 20th International Conference on, pages 123–132. IEEE, 2012.

[28]

K. Y. Phang, J. S. Foster, M. Hicks, and V. Sazawal. Triaging checklists: a substitute for a phd in static analysis. Evaluation and Usability of Programming Languages and Tools (PLATEAU) PLATEAU 2009, 2009.

[29]

F. Servant and J. A. Jones. History slicing: assisting code-evolution tasks. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, page 43. ACM, 2012.

Digital Library

[30]

Y. Yoon, B. A. Myers, and S. Koo. Visualization of fine-grained code change history. In Visual Languages and Human-Centric Computing (VL/HCC), 2013 IEEE Symposium on, pages 119–126. IEEE, 2013.

[31]

Nist source code security analyzers. http://samate.nist.gov/index.php/Source_ Code_Security_Analyzers.html.

[32]

Codesonar. http://grammatech.com/codesonar.

[33]

Coverity. http://coverity.com/.

[34]

Security questions experimental materials. http://http://www4.ncsu.edu/~bijohnso/ security-questions.html.

[35]

Findbugs. http://findbugs.sourceforge.net.

[36]

Find security bugs. http: //h3xstream.github.io/find-sec-bugs/.

[37]

Hippa statute. http://hhs.gov/ocr/privacy/.

[38]

itrust software system. http://agile.csc.ncsu. edu/iTrust/wiki/doku.php?id=start.

[39]

Otranscribe. http://otranscribe.com.

[40]

Owasp source code analysis tools. http://owasp.org/ index.php/Source_Code_Analysis_Tools.

[41]

Owasp. http://owasp.org/index.php/Main_Page.

[42]

Web application security consortium static code analysis tools. http://projects.webappsec.org/w/ page/61622133/StaticCodeAnalysisList.

Cited By

Groppe JGroppe SSenf DMöller R(2024)There Are Infinite Ways to Formulate Code: How to Mitigate the Resulting Problems for Better Software Vulnerability DetectionInformation10.3390/info1504021615:4(216)Online publication date: 11-Apr-2024
https://doi.org/10.3390/info15040216
Zhang HLi GLi SLiu HWang SLi JLi BXu WChen JZhang YXue JWang SBai GYuan X(2024)How to Efficiently Manage Critical Infrastructure Vulnerabilities? Toward Large Code-graph ModelsProceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis10.1145/3689217.3690622(25-34)Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1145/3689217.3690622
Ami AMoran KPoshyvanyk DNadkarni A(2024)"False negative - that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00019(3979-3997)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00019
Show More Cited By

Index Terms

Questions developers ask while diagnosing potential security vulnerabilities with static analysis
1. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments

Recommendations

How Developers Diagnose Potential Security Vulnerabilities with a Static Analysis Tool
While using security tools to resolve security defects, software developers must apply considerable effort. Success depends on a developer's ability to interact with tools, ask the right questions, and make strategic decisions. To build better security ...
Hard-to-answer questions about code
PLATEAU '10: Evaluation and Usability of Programming Languages and Tools

To build new tools and programming languages that make it easier for professional software developers to create, debug, and understand code, it is helpful to better understand the questions that developers ask during coding activities. We surveyed ...
Mitigating program security vulnerabilities: Approaches and challenges

Programs are implemented in a variety of languages and contain serious vulnerabilities which might be exploited to cause security breaches. These vulnerabilities have been exploited in real life and caused damages to related stakeholders such as program ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

August 2015

1068 pages

ISBN:9781450336758

DOI:10.1145/2786805

General Chair:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Program Chairs:
Mark Harman
University College London, UK
,
Patrick Heymans
University of Namur, Belgium

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ESEC/FSE'15

Sponsor:

SIGSOFT

ESEC/FSE'15: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

August 30 - September 4, 2015

Bergamo, Italy

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

60
Total Citations
View Citations
823
Total Downloads

Downloads (Last 12 months)92
Downloads (Last 6 weeks)9

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Groppe JGroppe SSenf DMöller R(2024)There Are Infinite Ways to Formulate Code: How to Mitigate the Resulting Problems for Better Software Vulnerability DetectionInformation10.3390/info1504021615:4(216)Online publication date: 11-Apr-2024
https://doi.org/10.3390/info15040216
Zhang HLi GLi SLiu HWang SLi JLi BXu WChen JZhang YXue JWang SBai GYuan X(2024)How to Efficiently Manage Critical Infrastructure Vulnerabilities? Toward Large Code-graph ModelsProceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis10.1145/3689217.3690622(25-34)Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1145/3689217.3690622
Ami AMoran KPoshyvanyk DNadkarni A(2024)"False negative - that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00019(3979-3997)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00019
Debeyan FMadeyski LHall TBowes D(2024)The Impact of hard and easy negative training data on vulnerability prediction performanceJournal of Systems and Software10.1016/j.jss.2024.112003(112003)Online publication date: Feb-2024
https://doi.org/10.1016/j.jss.2024.112003
He HWang SWang YLiu KYu L(2024)VulTR: Software Vulnerability Detection Model Based on Multi-layer Key Feature EnhancementComputers & Security10.1016/j.cose.2024.104139(104139)Online publication date: Sep-2024
https://doi.org/10.1016/j.cose.2024.104139
Ma ZMondal SChen TZhang HHassan A(2024)VulNet: Towards improving vulnerability management in the Maven ecosystemEmpirical Software Engineering10.1007/s10664-024-10448-629:4Online publication date: 5-Jun-2024
https://doi.org/10.1007/s10664-024-10448-6
Kim TPark SKim H(2023)Why Johnny Can’t Use Secure Docker Images: Investigating the Usability Challenges in Using Docker Image Vulnerability Scanners through Heuristic EvaluationProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607244(669-685)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3607199.3607244
Rahman AShamim SBose DPandita R(2023)Security Misconfigurations in Open Source Kubernetes Manifests: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/357963932:4(1-36)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3579639
He JVechev MMeng WJensen CCremers CKirda E(2023)Large Language Models for Code: Security Hardening and Adversarial TestingProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security10.1145/3576915.3623175(1865-1879)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3576915.3623175
Plöger SMeier MSmith M(2023)A Usability Evaluation of AFL and libFuzzer with CS StudentsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581178(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581178
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten