Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security (Raw Slides)
•
0 likes•428 views
The SolarWinds attack brought additional scrutiny software supply chain security, but concerns about organizations’ software supply chains have been discussed for a number of years. Development organizations’ shift to DevOps or DevSecOps has pushed teams to adopt new technologies in the build pipeline – often hosted by 3rd parties. This has resulted in build pipelines that expose a complicated and often uncharted attack surface. In addition, modern products also incorporate code from a variety of contributors – ranging from in-house developers, 3rd party development contractors, as well as an array open source contributors. This talk looks at the challenge of developing secure build pipelines. This is done via the construction of a threat model for an example software build pipeline that walks through how the various systems and communications along the way can potentially be misused by malicious actors. Coverage of the major components of a build pipeline – source control, open source component management, software builds, automated testing, and packaging for distribution – is used to enumerate likely attack surface exposed via the build process and to highlight potential controls that can be put in place to harden the pipeline against attacks. The presentation is intended to be useful both for evaluating internal build processes as well as to support the evaluation of critical external vendors’ processes.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security (Raw Slides)
Similar to Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security (Raw Slides) (20)
More from Denim Group
More from Denim Group (20)
Recently uploaded
Recently uploaded (20)
Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security (Raw Slides)
- 1. Threat Modeling the CI/CD Pipeline to Improve Software Supply Chain Security Dan Cornell CTO, Denim Group
- 2. Agenda • Risk Exposure from Software Build Pipeline • Constituents and Stakeholders • Drivers for Change • Threat Modeling Overview • Supporting Infrastructure • Pipeline Threat Model • Using the Threat Model • Questions
- 3. Notable Incidents • SolarWinds • Compromised server in build infrastructure • CodeCov • Compromised container server • Potentially spawning Executive Order, other government activity
- 4. Fundamental Disconnect • We have developed tremendous tooling and automation that allows us to create secure, reliable software at a scale not previously considered • The way we deploy and deliver this tooling puts us in a situation where we can't actually trust anything that comes out of it • Oops
- 5. Fractal Attack Surface The closer you look, the more you find
- 6. Confidentiality Impact • IP disclosure • Leaked secrets • Vulnerability disclosure
- 7. Integrity Impact • Backdoors • Other unwanted behaviors • Compromised cryptography
- 8. Availability Impact • Development pipeline infrastructure is hyper-connected • Release delays • Absence or misbehavior of supporting services
- 9. Threat Modeling Overview • Full overview beyond the scope of this talk • Some background for reference • https://www.denimgroup.com/resources/blog/2014/07/modeling-builders-breakers/ • Let's learn as we go
- 10. Supporting Infrastructure • Fixed assets vs ephemeral assets • Scanning/patching regimen • Who is responsible for this infrastructure? • "Corporate" or a DevOps team?
- 11. The Perimeter is the Problem • Critical part of threat modeling process – what is in/what is out? • For modern CI/CD pipelines: turns out to be maddeningly hard (impossible) to determine the boundary • Many steps with lots of moving parts • Dynamic attack surface: tomorrow may be different than today • DevOps team members can often add stages/tools/elements with minimal supervision • Fractal complexity • Approach: Start with code commit and follow the workflow
- 12. Example CI/CD Pipeline Dataflow
- 13. General/Overarching Concerns • Lots of network traffic • Protect with TLS to guard Confidentiality, Integrity • Lots of authentication points / IAM concerns • Often multiple methods/avenues per system: interactive + API (+ other?) • User <-> System • System <-> System • Is there a comprehensive IAM paradigm in place, or a combination of overlapping approaches?
- 14. General/Overarching Concerns • Data storage concerns • How is data at rest protected? (Especially file/block storage) • What are your unknown-unknowns • External services and provides that are (generally) unknown but part of the process • Any developer with a GitHub account is now their own purchasing agent
- 15. Source Repository and Workflow Engine
- 16. Source Repository and Workflow Engine • Sources of (custom) code • In-house developers • 3rd party development teams • Likely different risk profiles • Do contributions go through different workflows?
- 17. Source Repository and Workflow Engine • Authentication • Developers authenticating to push/pull code • Developers authenticating to evaluate/approve merge requests • External system API access • Webhooks • Additionally for Hosted Repositories/Workflows • GitHub: Apps and Actions • GitLab: Integrations • Both: External collaborators
- 18. Open Source Component Management
- 19. Open Source Component Management • Are you: • Directly pulling from Central repositories • Proxying requests through a Remote repository • Ability to enforce policies • Known security vulnerabilities in open source components • License restrictions on open source components
- 20. Open Source Backdoor Concerns • PHP example • https://arstechnica.com/gadgets/2021/03/hackers-backdoor-php-source-code-after-breaching-internal-git-server/ • Linux kernel example • https://www.techrepublic.com/article/linux-kernel-security-uproar-what-some-people-missed/ • Not realistic to detect • Maintainers don't even have the bandwidth • Have to be able to respond • Pulling from Remote vs directly from Central • Policies on acceptable (and unacceptable) versions
- 21. Build Management
- 22. Build Management • Combining source code and open source components to create a new build • How often run: • Every merge request • Accepted merge requests • Specific branches/tags • Maven plugins – yet another source of code you can't trust that runs in your environment!
- 24. Automated Testing • Most common types of testing: • Unit testing • Acceptance testing • Other common analysis: • Code quality ("smells") • Code metrics (complexity, etc) • Code coverage
- 25. Automated Testing: Security Testing • SAST • What is being analyzed: source or binary • For a given language: is "binary" even a thing? • Where is the analysis being performed: on-network, in the cloud • Where are the results being stored: on-network, in the cloud • DAST • Where is the server being tested: on-network, in the cloud • Where is the test traffic being generated: on-network, in the cloud, proxied onto-network • Where are the results being storoed: on-network, in the cloud
- 26. Automated Testing: Security Testing (cont'd) • IAST • Where is the server being tested: on-network, in the cloud • Where is the test traffic being generated: on-network, in the cloud, proxied onto-network • Where are the results being stored: on-network, in the cloud • SCA • Where is the analysis being performed: on-network, in the cloud • Where are the results being stored: on-network, in the cloud
- 27. Security Testing and Backdoors • Best backdoor: Normal-looking security vulnerability • "Oops, sorry I introduced that [SQL injection|missing auth check|etc] • Other types of backdoor detection • Static analysis – see the theoretical behavior of the application • Look for suspicious behavior patterns • References: • https://www.acsac.org/2007/workshop/Wysopal.pdf • https://www.veracode.com/sites/default/files/Resources/Whitepapers/static-detection-of- backdoors-1.0.pdf • https://owasp.org/www-pdf-archive/Protecting_the_Enterprise_-_Software_Backdoors.pdf • https://owasp.org/www-pdf-archive/Protecting_Your_Applications_From_Backdoors.pdf
- 28. Software Packaging and Distribution
- 29. Software Packaging • Monolithic applications vs. Microservice applications • Application binary vs. Application binaries combined with containers • Source container management is very similar to open source component management • Code signing is important • But signed malicious/vulnerable code is still malicious/vulnerable code
- 30. Software Distribution • Now the software should be ready too distribute to customers • Internal • External • How is the software distributed? • Binary • Binary + Container • Binaries + Containers + orchestration • How are builds verified? • Checksums
- 31. Two Exercises • Passive: Grab netflow data from your build environment and see what it talks to • Active: Follow code change from checkout to release • Spoiler alert: the active exercise where I engaged the development team was more productive
- 32. Reviewing Netflow Data • Pull netflow logs from network where development infrastructure is hosted • Look at what the build infrastructure talks to • Code repositories • Open source component repositories • Testing services • Other services <- Area of concern • Container repositories • In practice • Identified some interesting communications • "Expensive" to run down results • "High" false positive rate
- 33. Netflow Data Review [Run through cleaned up data from build infrastructure netflow review]
- 34. Follow a Code Change • Change code • Check out code (and make changes) • Create merge request • Review merge request • Run build • Pull code • Pull open source components • Automated testing • Unit tests • Acceptance tests • Security tests • Other tests • Accept build • Push to distribution • Distribute software Questions to ask at each stage • How does this user/system authenticate to the next step • Where is the server/service that performs this step • Where possible: show, don't tell • Look at the UI developers look at • Ask questions about where data comes from
- 38. Parallel Build Pipelines • Have two or more separate, parallel build pipelines • Compare artifacts at the end • Goal is to force attackers to compromise multiple pathways • Best way to get budget for this is to have a public breach of epic proportions and suggest that your successor try it out • How do you compare the outputs from multiple toolchains? • Compare SBOM • Asset enumeration • Reverse engineering • Live/behaviral testing
- 39. Using the Threat Model • Identify gaps in your own pipeline security • What can you shore up? • What do you have to live with? • How do you respond to incidents? • Have meaningful conversations with suppliers of your software • Understand their practices • Push for change – vote with your wallet • Be better able to detect malicious behavior
- 41. Questions