Vulnerably (Mis)Configured? Exploring 10 Years of Developers' Q&As on Stack Overflow
Authors: 
Richard May, Christian Biermann, Xenia Marlene Zerweck, Kai Ludwig, Jacob Krüger, Thomas LeichAuthors Info & Claims
Richard May, Christian Biermann, Xenia Marlene Zerweck, Kai Ludwig, Jacob Krüger, Thomas LeichAuthors Info & ClaimsPages 112 - 122
Abstract
The increasing number of attacks exploiting system vulnerabilities in recent years underpins the growing importance of security; especially for software comprising configuration options that may cause unintended vulnerabilities. So, not surprisingly, developers discuss secure software configurations extensively, for instance, via community-question-answering systems like Stack Overflow. In this exploratory study, we analyzed 651 Stack Overflow posts from 2013 until 2022 to investigate what vulnerabilities in the context of configuring software developers discuss. We employed a manual data analysis and automated topic modeling using Latent Dirichlet Allocation to identify and classify relevant topics and contexts. Our results show that vulnerabilities in the context of configuring receive more and more interest, with most posts discussing issues related to faulty security configurations and dependencies causing vulnerabilities that could be or have actually been exploited. Overall, we contribute insights into configuration and security issues that developers experience in the real world. Such insights help researchers and practitioners understand and resolve these issues, thereby guiding future improvements.
References
[1]
I. Abal, C. Brabrand, and A. Wasowski. 2014. 42 variability bugs in the Linux kernel: A qualitative analysis. In Automated Software Engineering Conference. ACM, 421–432.
[2]
I. Abal, J. Melo, Ş. Stănciulescu, C. Brabrand, M. Ribeiro, and A. Wąsowski. 2018. Variability bugs in highly configurable systems: A qualitative analysis. ACM Transactions on Software Engineering and Methodology 26, 3 (2018), 1–34.
[3]
M. Abomhara and G. M. Køien. 2015. Cyber security and the internet of things: Vulnerabilities, threats, intruders and attacks. Journal of Cyber Security and Mobility (2015), 65–88.
[4]
M. Acher, G. Bécan, B. Combemale, B. Baudry, and J.-M. Jézéquel. 2015. Product lines can jeopardize their trade secrets. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 930–933.
[5]
A. Agrawal, W. Fu, and T. Menzies. 2018. What is wrong with topic modeling? And how to fix it using search-based software engineering. Information and Software Technology 98 (2018), 74–88.
[6]
A. Ahmad, C. Feng, S. Ge, and A. Yousif. 2018. A survey on mining Stack Overflow: Question and answering (Q&A) community. Data Technologies and Applications 52, 2 (2018), 190–247.
[7]
A. Albakri, H. Fatima, M. Mohammed, A. Ahmed, A. Ali, A. Ali, and N. M. Elzein. 2022. Survey on reverse-engineering tools for android mobile devices. Mathematical Problems in Engineering 2022 (2022), 1–7.
[8]
S. Apel, D. Batory, C. Kästner, and G. Saake. 2013. Feature-oriented software product lines. Springer.
[9]
A. Bamrara. 2015. Evaluating database security and cyber attacks: A relational approach. The Journal of Internet Banking and Commerce 20, 2 (2015).
[10]
A. Barua, S. W. Thomas, and A. E. Hassan. 2014. What are developers talking about? An analysis of topics and trends in Stack Overflow. Empirical Software Engineering 19 (2014), 619–654.
[11]
R. Beckett, A. Gupta, R. Mahajan, and D. Walker. 2017. A general approach to network configuration verification. In Conference of the Special Interest Group on Data Communication. ACM, 155–168.
[12]
C. Beddiar, I. E. Khelili, N. Bounour, and A.-D. Seriai. 2020. Classification of android APIs posts: An analysis of developer’s discussions on Stack Overflow. In International Conference on Advanced Aspects of Software Engineering. IEEE, 1–5.
[13]
D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3, 1 (2003), 993–1022.
[14]
D. Bringhenti, G. Marchetto, R. Sisto, and F. Valenza. 2023. Automation for network security configuration: State of the art and research Trends. Comput. Surveys (2023).
[15]
M. Calder, M. Kolberg, E. H. Magill, and S. Reiff-Marganiec. 2003. Feature interaction: A critical review and considered Forecast. Computer Networks 41, 1 (2003), 115–141.
[16]
I. Cernica and N. Popescu. 2019. Security evaluation of wordpress backup plugins. In Conference on Control Systems and Computer Science. IEEE, 312–316.
[17]
R. Croft, Y. Xie, M. Zahedi, M. A. Babar, and C. Treude. 2022. An empirical study of developers’ discussions about security challenges of different programming languages. Empirical Software Engineering 27 (2022), 1–52.
[18]
Krzysztof Czarnecki, Paul Grünbacher, Rick Rabiser, Klaus Schmid, and Andrzej Wąsowski. 2012. Cool features and tough decisions: A comparison of variability modeling approaches. In Working Conference on Variability Modelling of Software-Intensive Systems. ACM, 173–182.
[19]
A. A. Elkhail and T. Cerny. 2019. On relating code smells to security vulnerabilities. In BigDataSecurity. IEEE, 7–12.
[20]
S. Fernandes and J. Bernardino. 2015. What is BigQuery?. In International Database Engineering and Applications Symposium. ACM, 202–203.
[21]
D. Fernandez-Amoros, R. Heradio, C. Mayr-Dorn, and A. Egyed. 2022. Scalable sampling of highly-configurable systems: Generating random instances of the Linux kernel. In Automated Software Engineering Conference. 1–12.
[22]
F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes, and S. Fahl. 2017. Stack Overflow considered harmful? The impact of copy&paste on Android application security. In Symposium on Security and Privacy. IEEE, 121–136.
[23]
A. M. Gamundani and L. M. Nekare. 2018. A review of new trends in cyber attacks: A zoom into distributed database systems. In IST-Africa. IEEE, 1–9.
[24]
P. Gazzillo. 2020. Inferring and Securing Software Configurations Using Automated Reasoning. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 1517–1520.
[25]
P. B. Gutgarts and A. Temin. 2010. Security-critical versus safety-critical software. In International Symposium on Technologies for Homeland Security. IEEE.
[26]
A. Haleem, M. Javaid, and R. P. Singh. 2022. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks, Standards and Evaluations 2, 4 (2022), 100089.
[27]
M. U. Haque, L. H. Iwaya, and M. A. Babar. 2020. Challenges in docker development: A large-scale study using Stack Overflow. In Empirical Software Engineering and Measurement. ACM, 1–11.
[28]
M. Humayun, M. Niazi, N. Z. Jhanjhi, M. Alshayeb, and S. Mahmood. 2020. Cyber security threats and vulnerabilities: A systematic mapping study. Arabian Journal for Science and Engineering 45, 4 (2020).
[29]
M. Hussain, A. Al-Haiqi, A. A. Zaidan, B. Bahaa Zaidan, M. Kiah, S. Iqbal, S. Iqbal, and M. Abdulnabi. 2018. A security framework for mHealth apps on Android platform. Computers & Security 75 (2018), 191–217.
[30]
M. S. Iqbal, R. Krishna, M. A. Javidian, B. Ray, and P. Jamshidi. 2022. Unicorn: Reasoning about configurable system performance through the lens of causality. In European Conference on Computer Systems. ACM, 199–217.
[31]
ISO/IEC 25010 2011. Systems and software engineering – SQuaRE - System and software quality. Standard. ISO.
[32]
ISO/IEC 27000 2018. Information technology – Security techniques – Information security management systems. Standard. ISO.
[33]
ISO/IEC 27001 2013. Information Security Management Systems – Requirements. Standard. ISO.
[34]
ISO/IEC 27005 2022. Information security, cybersecurity and privacy protection – Guidance on managing information security risks. Standard. ISO.
[35]
K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. 1990. Feature-oriented domain analysis feasibility study. Technical Report CMU/SEI-90-TR-21. Carnegie Mellon University.
[36]
A. Kenner, S. Dassow, C. Lausberger, J. Krüger, and T. Leich. 2020. Using variability modeling to support security evaluations: Virtualizing the right attack scenarios. In Working Conference on Variability Modelling of Software-Intensive Systems. ACM, 1–9.
[37]
A. Kenner, R. May, J. Krüger, G. Saake, and T. Leich. 2021. Safety, security, and configurable software systems: A systematic mapping study. In Systems and Software Product Line Conference. ACM, 148–159.
[38]
R. A. Khan, S. U. Khan, H. U. Khan, and M. Ilyas. 2021. Systematic mapping study on security approaches in secure software engineering. IEEE Access 9 (2021), 19139–19160.
[39]
B. A. Kitchenham, D. Budgen, and O. P. Brereton. 2015. Evidence-based software engineering and systematic reviews. CRC Press.
[40]
V. Klotzman, F. Farmahinifarahani, and C. Lopes. 2021. Public software development activity during the pandemic. In Empirical Software Engineering and Measurement. 1–12.
[41]
J. Krüger. 2018. Separation of concerns: Experiences of the crowd. In Symposium On Applied Computing. ACM, 2076–2077.
[42]
J. Krüger. 2019. Are you talking about software product lines? An analysis of developer communities. In Working Conference on Variability Modelling of Software-Intensive Systems. ACM, 1–9.
[43]
J. Krüger, I. Schröter, A. Kenner, and T. Leich. 2017. Empirical studies in question-answering systems: A discussion. In International Workshop on Conducting Empirical Studies in Industry. IEEE, 23–26.
[44]
Y. Li and Q. Liu. 2021. A comprehensive review study of cyber-attacks and cyber security: Emerging trends and recent developments. Energy Reports 7 (2021), 8176–8186.
[45]
S. A. Licorish and T. Nishatharan. 2021. Contextual profiling of Stack Overflow Java code cecurity vulnerabilities initial insights from a pilot study. In International Conference on Software Quality, Reliability and Security. IEEE, 1060–1068.
[46]
M. Linares-Vásquez, B. Dit, and D. Poshyvanyk. 2013. An exploratory analysis of mobile development issues using Stack Overflow. In Mining Software Repositories Conference. IEEE, 93–96.
[47]
T. Lopez, T. T. Tun, A. Bandara, M. Levine, B. Nuseibeh, and H. Sharp. 2018. An investigation of security conversations in Stack Overflow: Perceptions of security and community involvement. In International Conference on Information and Communication Technologies for Sustainability. ACM, 26–32.
[48]
K. Ludwig, J. Krüger, and T. Leich. 2019. Covert and phantom features in annotations: Do they impact variability analysis?. In Systems and Software Product Line Conference. ACM, 218–230.
[49]
R. May. 2022. Security and configurable storage systems in Industry 4.0 environments: A systematic literature study. In Open Conference Proceedings, Vol. 2. 151–156.
[50]
R. May, C. Biermann, A. Kenner, J. Krüger, and T. Leich. 2023. A product-line-engineering framework for secure enterprise-resource-planning systems. In International Conference on ENTERprise Information Systems. Elsevier, 1–8.
[51]
R. May, C. Biermann, J. Krüger, G. Saake, and T. Leich. 2022. A systematic mapping study of security concepts for configurable data storages. In Systems and Software Product Line Conference. ACM, 108–119.
[52]
R. May, J. Gautam, C. Sharma, C. Biermann, and T. Leich. 2023. A systematic mapping study on security in configurable safety-critical systems based on product-line concepts. In International Conference on Software Technologies. SciTePress, 217–224.
[53]
J. Meinicke, T. Thüm, R. Schröter, F. Benduhn, T. Leich, and G. Saake. 2017. Mastering software variability with FeatureIDE. Springer.
[54]
J. Meinicke, C.-P. Wong, C. Kästner, T. Thüm, and G. Saake. 2016. On essential configuration complexity: Measuring interactions in highly-configurable systems. In Conference on Automated Software Engineering. ACM, 483–494.
[55]
S. Meldrum, S. A. Licorish, and B. T. R. Savarimuthu. 2017. Crowdsourced knowledge on Stack Overflow: A systematic mapping study. In Conference on Evaluation and Assessment in Software Engineering. ACM, 180–185.
[56]
P. Mell, K. Scarfone, and S. Romanosky. 2006. Common vulnerability scoring system. IEEE Security & Privacy 4, 6 (2006), 85–89.
[57]
D. Mellado, E. Fernández-Medina, and M. Piattini. 2008. Towards security requirements management for software product lines: A security domain requirements engineering process. Computer Standards & Interfaces 30, 6 (2008), 361–371.
[58]
D. Mellado, E. Fernández-Medina, and M. Piattini. 2010. Security requirements engineering framework for software product lines. Information and Software Technology 52, 10 (2010), 1094–1117.
[59]
N. Meng, S. Nagy, D. Yao, W. Zhuang, and G. A. Argoty. 2018. Secure coding practices in Java: challenges and vulnerabilities. In International Conference on Software Engineering. ACM, 372–383.
[60]
O. Mesa, R. Vieira, M. Viana, V. H. S. Durelli, E. Cirilo, M. Kalinowski, and C. Lucena. 2018. Understanding vulnerabilities in pPugin-based web systems: An exploratory study of wordpress. In Systems and Software Product Line Conference. ACM, 149–159.
[61]
S. Nadi, T. Berger, C. Kästner, and K. Czarnecki. 2014. Mining configuration constraints: Static analyses and empirical results. In International Conference on Software Engineering. IEEE, 140–151.
[62]
A. R. Nasab, M. Shahin, S. A. H. Raviz, P. Liang, A. Mashmool, and V. Lenarduzzi. 2023. An empirical study of security practices for microservices systems. Journal of Systems and Software 198 (2023), 111563.
[63]
D. Nešić, J. Krüger, S. Stănciulescu, and T. Berger. 2019. Principles of feature modeling. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 62–73.
[64]
A. Nhlabatsi, R. Laney, and B. Nuseibeh. 2008. Feature interaction: The security threat from within software systems. Progress in Informatics 5, 75 (2008), 1.
[65]
NIST SP 800-154 2016. Guide to data-centric system threat modeling. Standard. National Institute of Standards and Technology.
[66]
NIST SP 800-30r1 2012. Guide for conducting risk assessments. Standard. National Institute of Standards and Technology.
[67]
N. Onumah, S. Attwood, and R. Kharel. 2020. Towards secure application development: A cyber security centred holistic approach. In International Symposium on Communication Systems, Networks and Digital Signal Processing. IEEE, 1–6.
[68]
K. Pohl, G. Böckle, and F. Van Der Linden. 2005. Software product line engineering: Foundations, principles, and techniques. Springer.
[69]
S. Samonas and D. Coss. 2014. The CIA strikes back: Redefining confidentiality, integrity and availability in security. Journal of Information System Security 10, 3 (2014).
[70]
M. Santolucito, E. Zhai, R. Dhodapkar, A. Shim, and R. Piskac. 2017. Synthesizing configuration file specifications with association rule learning. Proceedings of the ACM on Programming Languages 1 (2017), 1–20.
[71]
I. Schaefer, R. Rabiser, D. Clarke, L. Bettini, D. Benavides, G. Botterweck, A. Pathak, S. Trujillo, and K. Villela. 2012. Software diversity: State of the art and perspectives. International Journal on Software Tools for Technology Transfer 14 (2012), 477–495.
[72]
S. Sengupta and C. Haythornthwaite. 2020. Learning with comments: An analysis of comments and community on Stack Overflow. In Hawaii International Conference on System Sciences. IEEE, 2898–2907.
[73]
I. Srba and M. Bieliková. 2016. A comprehensive survey and classification of approaches for community question answering. ACM Transactions on the Web 10, 3 (2016), 1–63.
[74]
K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler. 2012. Exploring topic coherence over many models and many topics. In Conference on Empirical Methods in Natural Language Processing. ACL, 952–961.
[75]
R. Sulatycki and E. B. Fernandez. 2015. Two threat patterns that exploit security misconfiguration and sensitive data exposure vulnerabilities. In European Conference on Pattern Language of Programs. IEEE, 1–11.
[76]
M. Tahaei, J. Bernd, and A. Rashid. 2022. Privacy, permissions, and the health app ecosystem: A stack overflow exploration. In European Symposium on Usable Security. ACM, 117–130.
[77]
M. Tahaei, T. Li, and K. Vaniea. 2022. Understanding privacy-related advice on Stack Overflow. Proceedings on Privacy Enhancing Technologies 2022, 2 (2022), 114–131.
[78]
M. Tahaei, K. Vaniea, and N. Saphra. 2020. Understanding Privacy-related questions on Stack Overflow. In Conference on Human Factors in Computing Systems. ACM, 1–14.
[79]
R. Tartler, D. Lohmann, J. Sincero, and W. Schröder-Preikschat. 2011. Feature consistency in compile-time-configurable system software: Facing the Linux 10,000 feature problem. In European Conference on Computer Systems. ACM, 47–60.
[80]
T. Thüm, S. Apel, C. Kästner, I. Schaefer, and G. Saake. 2014. A classification and survey of analysis strategies for software product lines. Computing Surveys 47, 1 (2014), 1–45.
[81]
H. Trunde and E. Weippl. 2015. Wordpress security: An analysis based on publicly available exploits. In International Conference on Information Integration and Web-based Applications & Services. ACM, 1–7.
[82]
Á. J. Varela-Vaca, D. G. Rosado, L. E. Sánchez, M. T. Gómez-López, R. M. Gasca, and E. Fernandez-Medina. 2021. CARMEN: A framework for the verification and diagnosis of the specification of security requirements in cyber-physical systems. Computers in Industry 132 (2021), 1–14.
[83]
D. Vecchiato, M. Vieira, and E. Martins. 2016. The perils of Android security configuration. Computer 49, 06 (2016), 15–21.
[84]
I. von Nostitz-Wallwitz, J. Krüger, J. Siegmund, and T. Leich. 2018. Knowledge transfer from research to industry: A survey on program comprehension. In International Conference on Software Engineering. ACM, 300–301.
[85]
R. Wang, Y. Zhou, S. Chen, S. Qadeer, D. Evans, and Y. Gurevich. 2013. Explicating { SDKs} : uncovering assumptions underlying secure authentication and authorization. In Security. USENIX, 399–314.
[86]
W. Wang, S. Jian, Y. Tan, Q. Wu, and C. Huang. 2022. Representation learning-based network intrusion detection system by capturing explicit and implicit feature interactions. Computers & Security 112 (2022), 102537.
[87]
S. J. Y. Weamie. 2022. Cross-site scripting attacks and defensive techniques: A comprehensive survey. International Journal of Communications, Network and System Sciences 15, 8 (2022), 126–148.
[88]
Y. Wei, X. Sun, L. Bo, S. Cao, X. Xia, and B. Li. 2021. A comprehensive study on security bug characteristics. Journal of Software: Evolution and Process 33, 10 (2021), e2376.
[89]
J. Xu and G. Russello. 2022. Automated security-focused network configuration management: State of the art, challenges, and future directions. In nternational Conference on Software Engineering and Applications. IEEE, 409–420.
[90]
D. Yang, A. Hussain, and C. V. Lopes. 2016. From query to usable code: An analysis of Stack Overflow code snippets. In Mining Software Repositories Conference. ACM, 391–402.
[91]
J. Yang, C. Hauff, A. Bozzon, and G.-J. Houben. 2014. Asking the right Question in collaborative Q&A systems. In Conference on Hypertext and Social Media. ACM, 179–189.
[92]
X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, and J.-L. Sun. 2016. What security questions do developers ask? A large-scale study of Stack Overflow posts. Journal of Computer Science and Technology 31 (2016), 910–924.
[93]
S. Yuan, Y. Zhang, J. Tang, W. Hall, and J. B. Cabotà. 2020. Expert finding in community question answering: A review. Artificial Intelligence Review 53 (2020), 843–874.
[94]
Y. Zheng and X. Zhang. 2013. Path sensitive static analysis of web applications for remote code execution vulnerability detection. In International Conference on Software Engineering. IEEE, 652–661.
[95]
Y. Zhou and A. Sharma. 2017. Automated identification of security issues from commit messages and bug reports. In Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 914–919.
[96]
T. Zimmermann. 2016. Card-sorting: From text to themes. In Perspectives on Data Science for Software Engineering. Elsevier, 137–141.
[97]
M. Zwilling, G. Klien, D. Lesjak, Ł. Wiechetek, F. Cetin, and H. N. Basim. 2022. Cyber security awareness, knowledge and behavior: A comparative study. Journal of Computer Information Systems 62, 1 (2022), 82–97.
Index Terms
- Vulnerably (Mis)Configured? Exploring 10 Years of Developers' Q&As on Stack Overflow
Recommendations
Are Comments on Stack Overflow Well Organized for Easy Retrieval by Developers?
Continuous Special Section: AI and SEMany Stack Overflow answers have associated informative comments that can strengthen them and assist developers. A prior study found that comments can provide additional information to point out issues in their associated answer, such as the ...
A New Class of Buffer Overflow Attacks
ICDCS '11: Proceedings of the 2011 31st International Conference on Distributed Computing SystemsIn this paper, we focus on a class of buffer overflow vulnerabilities that occur due to the "placement new'' expression in C++. "Placement new'' facilitates placement of an object/array at a specific memory location. When appropriate bounds checking is ...
Mitigating security attacks in kubernetes manifests for security best practices violation
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringKubernetes is an open-source software system that helps practitioners in automatically deploying, scaling, and managing containerized applications. Information technology (IT) organizations, such as IBM, Spotify, and Capital One, use Kubernetes to ...
Comments
Information & Contributors
Information
Published In
February 2024
172 pages
ISBN:9798400708770
DOI:10.1145/3634713
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 February 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
VaMoS 2024
VaMoS 2024: 18th International Working Conference on Variability Modelling of Software-Intensive Systems
February 7 - 9, 2024
Bern, Switzerland
Acceptance Rates
Overall Acceptance Rate 66 of 147 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 105Total Downloads
- Downloads (Last 12 months)105
- Downloads (Last 6 weeks)5
Reflects downloads up to 09 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format