Unearthing Semantic Checks for Cloud Infrastructure-as-Code Programs
Pages 574 - 589
Abstract
Cloud infrastructures are increasingly managed by Infrastructure-as-Code (IaC) frameworks (e.g., Terraform). IaC frameworks enable cloud users to configure their resources in a declarative manner, without having to directly work with low-level cloud API calls. However, with today's IaC tooling, IaC programs that pass the compilation phase may still incur errors at deployment time, resulting in significant disruption. We observe that this stems from a fundamental semantic gap between IaC-level programs and cloud-level requirements---even a syntactically-correct IaC program may violate cloud-level expectations. To bridge this gap, we develop Zodiac, a tool that can unearth IaC-level semantic checks on cloud-level requirements. It provides an automated pipeline to mine these checks from online IaC repositories and validate them using deployment-based testing. We have applied Zodiac to Terraform resources offered by Microsoft Azure---a leading IaC framework and a leading cloud vendor---where it found 500+ semantic checks where violation would produce deployment failures. With these checks, we have identified 200+ buggy Terraform projects and helped fix errors within official Azure provider usage examples.
References
[1]
AWS cloud development kit. https://github.com/aws/aws-cdk.
[2]
AWS CloudFormation. https://aws.amazon.com/cloudformation/.
[3]
Azure Bicep. https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/.
[4]
Azure site-to-site VPN connection. https://learn.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal.
[5]
Azure SQL Managed Instance management. https://learn.microsoft.com/en-us/azure/azure-sql/managed-instance/management-operations-overview?view=azuresql.
[6]
Azure Storage Account Redundancy. https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview.
[7]
Azure subscription and service limits, quotas, and constraints. https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits.
[8]
Checking for IaC Programs where APPGW uses Basic IP Address with Dynamic Allocation Method. https://github.com/search?q=azurerm_public_ip+allocation_method++Dynamic+NOT+static+azurerm_application_gateway+resource+NOT+each+language%3AHCL+&type=code.
[9]
Checking for IaC Programs where APPGW with non-WAF v2 sku uses Web Application Firewall. https://github.com/search?q=waf_configuration+NOT+WAF_v2+azurerm_application_gateway++NOT+variable+NOT+output+language%3AHCL+&type=code.
[10]
Checking for IaC Programs where the Request Routing Rule of Standard v2 APPGW does not Specify Priority. https://github.com/search?q=+azurerm_application_gateway+request_routing_rule+NOT+priority+Standard_v2+language%3AHCL&type=code.
[11]
Checkov: ship code that's secure by default. https://bridgecrew.io/checkov/.
[12]
Cloud development kit for terraform. https://developer.hashicorp.com/terraform/cdktf.
[13]
Example Usage within azurerm_application_gateway Documentation Cannot be Deployed Successfully. https://github.com/hashicorp/terraform-provider-azurerm/issues/27065.
[14]
Example Usage within azurerm_dedicated_hardware_security_moduleDocumentation Cannot be Deployed Successfully. https://github.com/hashicorp/terraform-provider-azurerm/issues/27078.
[15]
Example Usage within azurerm_mssql_database Documentation Cannot be Deployed Successfully. https://github.com/hashicorp/terraform-provider-azurerm/issues/27194.
[16]
Example Usage within azurerm_network_interface_application_gateway_backend_address_pool_association Documentation Cannot be Deployed Successfully. https://github.com/hashicorp/terraform-provider-azurerm/issues/27222.
[17]
Microsoft Azure Fsv2-series VM. https://learn.microsoft.com/en-us/azure/virtual-machines/fsv2-series.
[18]
Opa's native query language rego. https://www.openpolicyagent.org/docs/latest/policy-language/.
[19]
Open world assumptions. https://www.sciencedirect.com/topics/computer-science/open-world-assumption.
[20]
OpenTofu: The open source infrastructure as code tool. https://opentofu.org/.
[21]
Pulumi ai. https://www.pulumi.com/ai.
[22]
Pulumi: Infrastructure as code in any programming language. https://www.pulumi.com/.
[23]
Regions for virtual machines in azure. https://learn.microsoft.com/en-us/azure/virtual-machines/regions#special-azure-regions.
[24]
Shift testing left with unit tests. https://learn.microsoft.com/en-us/devops/develop/shift-left-make-testing-fast-reliable.
[25]
Terraform by Hashicorp. https://www.terraform.io/.
[26]
Terraform Resource: Manages the association between a Network Interface and a Application Gateway's Backend Address Pool. https://registry.terraform.io/providers/hashicorp/azurerm/3.97.0/docs/resources/network_interface_application_gateway_backend_address_pool_association.
[27]
Terraform v.s. alternatives. https://developer.hashicorp.com/terraform/intro/vs/cloudformation.
[28]
Terrascan: Detect compliance and security violations across Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure. https://runterrascan.io/.
[29]
TFLint: A Pluggable Terraform Linter. https://github.com/terraform-linters/tflint.
[30]
TFSec: Security Scanner for Your Terraform Code. https://github.com/aquasecurity/tfsec.
[31]
Zodiac: Unearthing Semantic Checks for Cloud Infrastructure-as-Code Programs. https://github.com/824728350/Zodiac.
[32]
Glenn Ammons, Rastislav Bodík, and James R Larus. Mining specifications. ACM Sigplan Notices, 37(1):4--16, 2002.
[33]
Mina Tahmasbi Arashloo, Ryan Beckett, and Rachit Agarwal. Formal methods for network performance analysis. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023.
[34]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. Restler: Stateful REST API fuzzing. In IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019.
[35]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. Checking security properties of cloud service rest apis. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020.
[36]
Ryan Beckett, Aarti Gupta, Ratul Mahajan, and David Walker. A general approach to network configuration verification. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM), 2017.
[37]
Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. A review on fact extraction and verification. ACM Computing Surveys (CSUR), 55(1):1--35, 2021.
[38]
Eric Hayden Campbell, William T Hallahan, Priya Srikumar, Carmelo Cascone, Jed Liu, Vignesh Ramamurthy, Hossein Hojjat, Ruzica Piskac, Robert Soulé, and Nate Foster. Avenir: Managing data plane diversity with control plane synthesis. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), 2021.
[39]
Claudia Cauli, Meng Li, Nir Piterman, and Oksana Tkachuk. Predeployment security assessment for cloud services through semantic reasoning. In Computer Aided Verification (CAV 21), 2021.
[40]
Michael D Ernst, Jeff H Perkins, Philip J Guo, Stephen McCamant, Carlos Pacheco, Matthew S Tschantz, and Chen Xiao. The daikon system for dynamic detection of likely invariants. Science of computer programming, 69(1-3):35--45, 2007.
[41]
Alexandros Evangelidis, David Parker, and Rami Bahsoon. Performance modelling and verification of cloud-based auto-scaling policies. Future Generation Computer Systems, 87:629--638, 2018.
[42]
Grigory Fedyukovich and Rastislav Bodík. Accelerating syntax-guided invariant synthesis. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2018.
[43]
Grigory Fedyukovich, Sumanth Prabhu, Kumar Madhukar, and Aarti Gupta. Quantified invariants via syntax-guided synthesis. In Computer Aided Verification (CAV 19), 2019.
[44]
Zheng Guo, David Cao, Davin Tjong, Jean Yang, Cole Schlesinger, and Nadia Polikarpova. Type-directed program synthesis for restful apis. In 43rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 22), 2022.
[45]
Travis Hance, Marijn Heule, Ruben Martins, and Bryan Parno. Finding invariants of distributed systems: It's a small (enough) world after all. In 18th USENIX symposium on networked systems design and implementation (NSDI 21), 2021.
[46]
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
[47]
Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. Adaptive REST API Testing with Reinforcement Learning. In IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023.
[48]
Ivo Krka, Yuriy Brun, and Nenad Medvidovic. Automatic mining of specifications from invocation traces and method invariants. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 14), 2014.
[49]
Nayeon Lee, Yejin Bang, Andrea Madotto, Madian Khabsa, and Pascale Fung. Towards few-shot fact-checking via perplexity. arXiv preprint arXiv:2103.09535, 2021.
[50]
Nayeon Lee, Belinda Z Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. Language models as fact checkers? arXiv preprint arXiv:2006.04102, 2020.
[51]
Caroline Lemieux. Mining temporal properties of data invariants. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE 15), volume 2. IEEE, 2015.
[52]
Julien Lepiller, Ruzica Piskac, Martin Schäf, and Mark Santolucito. Analyzing infrastructure as code to prevent intra-update sniping vulnerabilities. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS 21), 2021.
[53]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459--9474, 2020.
[54]
Wang Li, Zhouyang Jia, Shanshan Li, Yuanliang Zhang, Teng Wang, Erci Xu, Ji Wang, and Xiangke Liao. Challenges and opportunities: an in-depth empirical study on configuration error injection testing. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 21), 2021.
[55]
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with Alphacode. Science, 378(6624):1092--1097, 2022.
[56]
Jed Liu, William Hallahan, Cole Schlesinger, Milad Sharif, Jeongkeun Lee, Robert Soulé, Han Wang, Călin Caşcaval, Nick McKeown, and Nate Foster. P4v: Practical verification for programmable data planes. In Proceedings of the 2018 Conference of the ACM Special Interest Group on data communication (SIGCOMM 18), 2018.
[57]
Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Maddila, Balasubramanyan Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. Rex: Preventing bugs and misconfiguration in large services using correlated change analysis. In USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), 2020.
[58]
Yiming Qiu, Ryan Beckett, and Ang Chen. Synthesizing runtime programmable switch updates. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023.
[59]
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
[60]
Mark Santolucito, Ennan Zhai, Rahul Dhodapkar, Aaron Shim, and Ruzica Piskac. Synthesizing configuration file specifications with association rule learning. Proceedings of the ACM on Programming Languages, 1(OOPSLA 17):1--20, 2017.
[61]
Mark Santolucito, Ennan Zhai, and Ruzica Piskac. Probabilistic automated language learning for configuration files. In Computer Aided Verification: 28th International Conference, (CAV 16), Toronto, ON, Canada, July 17--23, 2016, Proceedings, Part II 28. Springer, 2016.
[62]
Alireza Souri, Nima Jafari Navimipour, and Amir Masoud Rahmani. Formal verification approaches and standards in the cloud computing: a comprehensive and systematic review. Computer Standards & Interfaces, 58:1--22, 2018.
[63]
Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. Resttestgen: automated black-box testing of RESTful APIs. In IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020.
[64]
Helen J. Wang, John C. Platt, Yu Chen, Ruyun Zhang, and Yi-Min Wang. Automatic misconfiguration troubleshooting with PeerPressure. In 6th Symposium on Operating Systems Design & Implementation (OSDI 04), 2004.
[65]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
[66]
Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. Combinatorial testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE 22), 2022.
[67]
Jianan Yao, Runzhou Tao, Ronghui Gu, and Jason Nieh. DuoAI: Fast, automated inference of inductive invariants for verifying distributed protocols. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022.
[68]
Jianan Yao, Runzhou Tao, Ronghui Gu, Jason Nieh, Suman Jana, and Gabriel Ryan. Distai:data-driven automated invariant learning for distributed protocols. In 15th USENIX symposium on operating systems design and implementation (OSDI 21), 2021.
[69]
Jialu Zhang, Ruzica Piskac, Ennan Zhai, and Tianyin Xu. Static detection of silent misconfigurations with deep interaction analysis. Proceedings of the ACM on Programming Languages, 5(OOPSLA 21):1--30, 2021.
[70]
Jiaqi Zhang, Lakshminarayanan Renganarayana, Xiaolan Zhang, Niyu Ge, Vasanth Bala, Tianyin Xu, and Yuanyuan Zhou. Encore: Exploiting system environment and correlation information for misconfiguration detection. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems (ASPLOS 14), 2014.
[71]
Sai Zhang and Michael D Ernst. Proactive detection of inadequate diagnostic messages for software configuration errors. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 15), 2015.
Index Terms
- Unearthing Semantic Checks for Cloud Infrastructure-as-Code Programs
Recommendations
Simplifying Cloud Management with Cloudless Computing
HotNets '23: Proceedings of the 22nd ACM Workshop on Hot Topics in NetworksCloud computing has transformed the IT industry, but managing cloud infrastructures remains a difficult task. We make a case for putting today's management practices, known as "Infrastructure-as-Code," on a firmer ground via a principled design. We call ...
SLA Approach for "Cloud as a Service"
CLOUD '13: Proceedings of the 2013 IEEE Sixth International Conference on Cloud ComputingOne of slogans of cloud computing is to pay only what you use and in the highly concurrencial today context, it is important to properly identify the offer of cloud provider. The offer should not only encompass the offering services, but also the ...
Comments
Information & Contributors
Information
Published In
November 2024
765 pages
ISBN:9798400712517
DOI:10.1145/3694715
- Chair:
- Emmett Witchel,
- Co-chair:
- Christopher J Rossbach,
- Program Chair:
- Andrea Arpaci-Dusseau,
- Program Co-chair:
- Kimberly Keeton
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- USENIX
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 November 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SOSP '24
Sponsor:
SOSP '24: ACM SIGOPS 30th Symposium on Operating Systems Principles
November 4 - 6, 2024
TX, Austin, USA
Acceptance Rates
SOSP '24 Paper Acceptance Rate 43 of 245 submissions, 18%;
Overall Acceptance Rate 174 of 961 submissions, 18%
Upcoming Conference
SOSP '25
- Sponsor:
- sigops
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 78Total Downloads
- Downloads (Last 12 months)78
- Downloads (Last 6 weeks)46
Reflects downloads up to 13 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in