Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3460319.3464799acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Challenges and opportunities: an in-depth empirical study on configuration error injection testing

Published: 11 July 2021 Publication History
  • Get Citation Alerts
  • Abstract

    Configuration error injection testing (CEIT) could systematically evaluate software reliability and diagnosability to runtime configuration errors. This paper explores the challenges and opportunities of applying CEIT technique. We build an extensible, highly-modularized CEIT framework named CeitInspector to experiment with various CEIT techniques. Using CeitInspector, we quantitatively measure the effectiveness and efficiency of CEIT using six mature and widely-used server applications. During this process, we find a fair number of test cases are left unstudied by the prior research work. The injected configuration errors in these cases often indicate latent misconfigurations, which might be ticking time bombs in the system and lead to severe damage. We conduct an in-depth study regarding these cases to reveal the root causes, and explore possible remedies. Finally, we come up with actionable suggestions guided by our study to improve the effectiveness and efficiency of the existing CEIT techniques.

    References

    [1]
    2020. HTTPD configuration manual page. http://httpd.apache.org/docs/2.4/mod/directives.html
    [2]
    2020. MySQL configuration manual page. https://dev.mysql.com/doc/refman/5.6/en/server-option-variable-reference.html
    [3]
    2020. Nginx configuration manual page. http://nginx.org/en/docs/dirindex.html
    [4]
    2020. PostgreSQL configuration manual page. https://www.postgresql.org/docs/11/bookindex.html
    [5]
    2020. Squid configuration manual page. http://www.squid-cache.org/Doc/config/
    [6]
    2020. VSFTPD configuration manual page. https://security.appspot.com/vsftpd.html
    [7]
    George Amvrosiadis and Medha Bhadkamkar. 2016. Getting Back Up: Understanding How Enterprise Data Backups Fail. In Proceedings of 2016 USENIX Annual Technical Conference (ATC’16). Denver, CO. isbn:978-1-931971-30-0
    [8]
    Fahad A. Arshad, Rebecca J. Krause, and Saurabh Bagchi. 2013. Characterizing Configuration Problems in Java EE Application Servers: An Empirical Study with GlassFish and JBoss. In Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE’13). Pasadena, CA, USA. https://doi.org/10.1109/ISSRE.2013.6698919
    [9]
    Mona Attariyan, Michael Chow, and Jason Flinn. 2012. X-ray: Automating Root-Cause Diagnosis of Performance Anomalies in Production Software. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI’12). Hollywood, CA, USA. https://doi.org/10.5555/2387880.2387910
    [10]
    Mona Attariyan and Jason Flinn. 2008. Using Causality to Diagnose Configuration Bugs. In Proceedings of 2008 USENIX Annual Technical Conference (USENIX ATC’08). Boston, MA, USA. https://doi.org/10.5555/1404014.1404037
    [11]
    Mona Attariyan and Jason Flinn. 2010. Automating Configuration Troubleshooting with Dynamic Information Flow Analysis. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation (OSDI’10). Vancouver, BC, Canada. https://doi.org/10.5555/1924943.1924960
    [12]
    Augeas. 2018. Augeas - a configuration API. http://augeas.net/
    [13]
    Luiz André Barroso, Urs Hölzle, and Parthasarathy Ranganathan. 2018. The Datacenter as a Computer: An Introduction to the Design of Warehouse-scale Machines (Third Edition). Morgan and Claypool Publishers. https://doi.org/10.2200/S00874ED3V01Y201809CAC046
    [14]
    Salman Baset, Sahil Suneja, Nilton Bila, Ozan Tuncer, and Canturk Isci. 2017. Usable Declarative Configuration Specification and Validation for Applications, Systems, and Cloud. In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference (Middleware’17), Industrial Track. https://doi.org/10.1145/3154448.3154453
    [15]
    Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed Greybox Fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS’17). https://doi.org/10.1145/3133956.3134020
    [16]
    Jim Gray. 1985. Why Do Computers Stop and What Can Be Done About It? Tandem Technical Report 85.7, Jun., https://doi.org/10.1.1.59.6561
    [17]
    Haryadi S. Gunawi, Mingzhe Hao, Riza O. Suminto, Agung Laksono, Anang D. Satria, Jeffry Adityatama, and Kurnia J. Eliazar. 2016. Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages. In Proceedings of the 7th ACM Symposium on Cloud Computing (SoCC’16). Santa Clara, CA. https://doi.org/10.1145/2987550.2987583
    [18]
    Peng Huang, William J. Bolosky, Abhishek Sigh, and Yuanyuan Zhou. 2015. ConfValley: A Systematic Configuration Validation Framework for Cloud Services. In Proceedings of the 10th European Conference on Computer Systems (EuroSys’15). Bordeaux, France. https://doi.org/10.1145/2741948.2741963
    [19]
    Zhouyang Jia, Shanshan Li, Xiaodong Liu, Xiangke Liao, and Yunhuai Liu. 2018. SMARTLOG: Place error log statement by deep understanding of log intention. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 61–71. https://doi.org/10.1109/SANER.2018.8330197
    [20]
    Weihang Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky. 2008. Are Disks the Dominant Contributor for Storage Failures? A Comprehensive Study of Storage Subsystem Failure Characteristics. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). San Jose, CA, USA. https://doi.org/10.1145/1416944.1416946
    [21]
    Lorenzo Keller, Prasang Upadhyaya, and George Candea. 2008. ConfErr: A Tool for Assessing Resilience to Human Configuration Errors. In Proceedings of the 38th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’08). Anchorage, AK, USA. https://doi.org/10.1109/DSN.2008.4630084
    [22]
    Stuart Kendrick. 2012. What Takes Us Down? USENIX ;login:, 37, 5 (2012), Oct., 37–45. https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down
    [23]
    Caroline Lemieux and Koushik Sen. 2018. FairFuzz: A Targeted Mutation Strategy for Increasing Greybox Fuzz Testing Coverage. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). https://doi.org/10.1145/3238147.3238176
    [24]
    Shanshan Li, Wang Li, Xiangke Liao, Shaoliang Peng, Shulin Zhou, Zhouyang Jia, and Teng Wang. 2018. ConfVD: System Reactions Analysis and Evaluation Through Misconfiguration Injection. IEEE Transactions on Reliability (Early Access), Sep., https://doi.org/10.1109/TR.2018.2865962
    [25]
    Wang Li, Shanshan Li, Xiangke Liao, Xiangyang Xu, Shulin Zhou, and Zhouyang Jia. 2017. ConfTest: Generating Comprehensive Misconfiguration for System Reaction Ability Evaluation. In The International Conference. 88–97. https://doi.org/10.1145/3084226.3084244
    [26]
    Yuekang Li, Bihuan Chen, Mahinthan Chandramohan, Shang-Wei Lin, Yang Liu, and Alwen Tiu. 2017. Steelix: Program-State Based Binary Fuzzing. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). https://doi.org/10.1145/3106237.3106295
    [27]
    Xiangke Liao, Shulin Zhou, Shanshan Li, Zhouyang Jia, Xiaodong Liu, and Haochen He. 2018. Do You Really Know How to Configure Your Software? Configuration Constraints in Source Code May Help. IEEE Transactions on Reliability, 67, 3 (2018), Sep., 832–846. https://doi.org/10.1109/TR.2018.2834419
    [28]
    Valentin J.M. Manés, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering of Fuzzing: A Survey. arXiv:1812.00140, Apr., https://doi.org/10.1109/TSE.2019.2946563
    [29]
    Ben Maurer. 2015. Fail at Scale: Reliability in the Face of Rapid Change. Commun. ACM, 58, 11 (2015), Nov., 44–49. https://doi.org/10.1145/2838344.2839461
    [30]
    Barton P. Miller, Louis Fredriksen, and Bryan So. 1990. An Empirical Study of the Reliability of UNIX Utilities. Commun. ACM, 33, 12 (1990), Dec., 32–44. https://doi.org/10.1145/96267.96279
    [31]
    Barton P. Miller, David Koski, Cjin Pheow Lee, Vivekananda Maganty, Ravi Murthy, Ajitkumar Natarajan, and Jeff Steidl. 1995. Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services. University of Wisconsin-Madison, Computer Sciences Department.
    [32]
    MySQL Parameter. 2019. MySQL parameter from Innodb and myisam. https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html
    [33]
    Kiran Nagaraja, Fábio Oliveira, Ricardo Bianchini, Richard P. Martin, and Thu D. Nguyen. 2004. Understanding and Dealing with Operator Mistakes in Internet Services. In Proceedings of the 6th USENIX Conference on Operating Systems Design and Implementation (OSDI’04). San Francisco, CA, USA. https://doi.org/10.5555/1251254.1251259
    [34]
    Saahil Ognawala, Thomas Hutzelmann, Eirini Psallida, and Alexander Pretschner. 2018. Improving Function Coverage with Munch: A Hybrid Fuzzing and Directed Symbolic Execution Approach. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing (SAC’18). https://doi.org/10.1145/3167132.3167289
    [35]
    David Oppenheimer, Archana Ganapathi, and David A. Patterson. 2003. Why Do Internet Services Fail, and What Can Be Done About It? In Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems (USITS’03). Seattle, WA, USA. https://doi.org/10.5555/1251460.1251461
    [36]
    Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2019. Semantic Fuzzing with Zest. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19). https://doi.org/10.1145/3293882.3330576
    [37]
    Rohan Padhye, Caroline Lemieux, Koushik Sen, Laurent Simon, and Hayawardh Vijayakumar. 2019. FuzzFactory: Domain-Specific Fuzzing with Waypoints. In Proceedings of the ACM on Programming Languages (OOPSLA’19). https://doi.org/10.1145/3360600
    [38]
    Rahul Potharaju, Joseph Chan, Luhui Hu, Cristina Nita-Rotaru, Mingshi Wang, Liyuan Zhang, and Navendu Jain. 2015. ConfSeer: Leveraging Customer Support Knowledge Bases for Automated Misconfiguration Detection. In Proceedings of the 35th International Conference on Very Large Data Bases (VLDB’15). https://doi.org/10.14778/2824032.2824079
    [39]
    Ariel Rabkin and Randy Katz. 2011. Precomputing Possible Configuration Error Diagnosis. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11). Lawrence, KS, USA. https://doi.org/10.1109/ASE.2011.6100053
    [40]
    Ariel Rabkin and Randy Katz. 2011. Static Extraction of Program Configuration Options. In Proceedings of the 33th International Conference on Software Engineering (ICSE’11). Honolulu, HI, USA. https://doi.org/10.1145/1985793.1985812
    [41]
    Ariel Rabkin and Randy Katz. 2013. How Hadoop Clusters Break. IEEE Software Magazine, 30, 4 (2013), Jul., 88–94. https://doi.org/10.1109/MS.2012.73
    [42]
    Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida, and Herbert Bos. 2017. VUzzer: Application-aware Evolutionary Fuzzing. In Proceedings of the 2017 Network and Distributed System Security Symposium (NDSS’17). https://doi.org/10.14722/ndss.2017.23404
    [43]
    Mark Santolucito, Ennan Zhai, Rahul Dhodapkar, Aaron Shim, and Ruzica Piskac. 2017. Synthesizing Configuration File Specifications with Association Rule Learning. In Proceedings of 2017 ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’17). https://doi.org/10.1145/3133888
    [44]
    Mark Santolucito, Ennan Zhai, and Ruzica Piskac. 2016. Probabilistic Automated Language Learning for Configuration Files. In 28th International Conference on Computer Aided Verification (CAV’16). Toronto, Canada. https://doi.org/10.1007/978-3-319-41540-65
    [45]
    Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. In Proceedings of the 2016 Network and Distributed System Security Symposium (NDSS’16). https://doi.org/10.14722/ndss.2016.23368
    [46]
    Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. 2015. Holistic Configuration Management at Facebook. In Proceedings of the 25th Symposium on Operating System Principles (SOSP’15). Monterey, CA, USA. https://doi.org/10.1145/2815400.2815401
    [47]
    Ozan Tuncer, Nilton Bila, Sastry Duri, Canturk Isci, and Ayse K. Coskun. 2018. ConfEx: Towards Automating Software Configuration Analytics in the Cloud. In Proceedings of the 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). https://doi.org/10.1109/DSN-W.2018.00019
    [48]
    Ozan Tuncer, Nilton Bila, Canturk Isci, and Ayse K. Coskun. 2018. ConfEx: An Analytics Framework for Text-Based Software Configurations in the Cloud. IBM Research.
    [49]
    Whoosh. 2018. Whoosh 2.7.4 documentation. https://whoosh.readthedocs.io/en/latest
    [50]
    Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early Detection of Configuration Errors to Reduce Failure Damage. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). Savannah, GA. https://doi.org/10.5555/3026877.3026925
    [51]
    Tianyin Xu, Han Min Naing, Le Lu, and Yuanyuan Zhou. 2017. How Do System Administrators Resolve Access-Denied Issues in the Real World? In Proceedings of the 35th Annual CHI Conference on Human Factors in Computing Systems (CHI’17). Denver, CO. https://doi.org/10.1145/3025453.3025999
    [52]
    Tianyin Xu, Jiaqi Zhang, Peng Huang, Jing Zheng, Tianwei Sheng, Ding Yuan, Yuanyuan Zhou, and Shankar Pasupathy. 2013. Do Not Blame Users for Misconfigurations. In Proceedings of the 24th Symposium on Operating System Principles (SOSP’13). Farmington, PA, USA. https://doi.org/10.1145/2517349.2522727
    [53]
    Zuoning Yin, Xiao Ma, Jing Zheng, Yuanyuan Zhou, Lakshmi N. Bairavasundaram, and Shankar Pasupathy. 2011. An Empirical Study on Configuration Errors in Commercial and Open Source Systems. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP’11). Cascais, Portugal. https://doi.org/10.1145/2043556.2043572
    [54]
    Ding Yuan, Jing Zheng, Soyeon Park, Yuanyuan Zhou, and Stefan Savage. 2012. Improving software diagnosability via log enhancement. ACM Transactions on Computer Systems (TOCS), 30 (2012), 4. https://doi.org/10.1145/1950365.1950369
    [55]
    Jiaqi Zhang, Lakshmi Renganarayana, Xiaolan Zhang, Niyu Ge, Vasanth Bala, Tianyin Xu, and Yuanyuan Zhou. 2014. EnCore: Exploiting System Environment and Correlation Information for Misconfiguration Detection. In Proceedings of the 19th International Conference on Architecture Support for Programming Languages and Operating Systems (ASPLOS’14). Salt Lake City, UT, USA. https://doi.org/10.1145/2644865.2541983
    [56]
    Sai Zhang and Michael D. Ernst. 2015. Proactive Detection of Inadequate Diagnostic Messages for Software Configuration Errors. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA’15). Baltimore, MD, USA. https://doi.org/10.1145/2771783.2771817
    [57]
    Shulin Zhou, Xiaodong Liu, Shanshan Li, Wei Dong, Xiangke Liao, and Yun Xiong. 2016. Confmapper: Automated variable finding for configuration items in source code. In 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). 228–235. https://doi.org/10.1109/QRS-C.2016.35
    [58]
    Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R Lyu, and Dongmei Zhang. 2015. Learning to Log: Helping Developers Make Informed Logging Decisions. In Proc. of ACM/IEEE ICSE. https://doi.org/10.5555/2818754.2818807

    Cited By

    View all
    • (2023)Understanding and Detecting On-the-Fly Configuration BugsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00062(628-639)Online publication date: 14-May-2023
    • (2023)ConfTainter: Static Taint Analysis For Configuration Options2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00067(1640-1651)Online publication date: 11-Sep-2023
    • (2023)WMWatcher: Preventing Workload-Related Misconfigurations in Production Environment2023 30th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC60848.2023.00038(279-288)Online publication date: 4-Dec-2023
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis
    July 2021
    685 pages
    ISBN:9781450384599
    DOI:10.1145/3460319
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 July 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    Author Tags

    1. Configuration
    2. Empirical Study
    3. Testing

    Qualifiers

    • Research-article

    Conference

    ISSTA '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 58 of 213 submissions, 27%

    Upcoming Conference

    ISSTA '24

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)86
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 11 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Understanding and Detecting On-the-Fly Configuration BugsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00062(628-639)Online publication date: 14-May-2023
    • (2023)ConfTainter: Static Taint Analysis For Configuration Options2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00067(1640-1651)Online publication date: 11-Sep-2023
    • (2023)WMWatcher: Preventing Workload-Related Misconfigurations in Production Environment2023 30th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC60848.2023.00038(279-288)Online publication date: 4-Dec-2023
    • (2023)Improvement of AFL’s Seed Deterministic Mutation AlgorithmEmerging Networking Architecture and Technologies10.1007/978-981-19-9697-9_28(347-357)Online publication date: 1-Feb-2023
    • (2022)Learning from the history: Identifying configuration change in software development2022 4th International Academic Exchange Conference on Science and Technology Innovation (IAECST)10.1109/IAECST57965.2022.10062268(716-720)Online publication date: 9-Dec-2022
    • (2022)Troubleshooting Configuration Errors via Information Retrieval and Configuration Testing2022 4th International Academic Exchange Conference on Science and Technology Innovation (IAECST)10.1109/IAECST57965.2022.10062229(422-426)Online publication date: 9-Dec-2022

    View Options

    Get Access

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media