Sans A Devsecops Playbook
Sans A Devsecops Playbook
Sans A Devsecops Playbook
A SANS Whitepaper
Written by Dave Shackleford
March 2016
Sponsored by
CloudPassage
Today, the dramatic shift toward the dynamic provisioning, shared resources and open
standards of cloud computing models has, as with client/server, been driven largely by
the benefits to be gained, in this case in speed, agility and cost through the judicious
use of cloud systems and services. As before, changes in hardware and computing
Security teams
architecture are the most outwardly visible. But what promises to deliver the most
are often seen significant payback, in terms of higher-quality software that is delivered faster than was
as roadblocks to possible before, are the less obvious benefits—in speed, flexibility and deployment
rapid development methods—that are possible using DevOps and other rapid- or continuous application-
delivery models.
or operations
implementations, For business leaders, moving to the cloud is all about speed of service delivery, flexibility,
scalability and new capabilities and features that would take too long to develop
slowing down
in-house. The ability to deploy applications and services into cloud environments
production code is pushing all facets of IT to move faster than ever before. Unfortunately, despite
pushes. interdependence on one another, development teams, IT operations and information
security personnel often operate in their own silos. The alignment of development
and operations teams has made it possible to build customized software and business
functions far more quickly than before, but security teams continue to be left out of the
DevOps conversation.
This paper will argue that DevOps and information security can co-exist through the
application of a new approach referred to as DevSecOps.
Organizations can find it hard to resist becoming more agile by accelerating the rollout
of innovative new applications. But when an organization speeds up development
by skipping steps meant to identify flaws in untested code or resisting efforts to add
security controls to the code before it ships, it exposes itself in ways that would have
When an been unthinkable under traditional development models.
organization speeds
The cloud itself is rapidly scalable, but that does not mean the development of
up development by applications to run on it can match the pace. For example, even perfunctory testing
skipping steps meant might reveal that a poorly designed application cannot scale on its own, a problem
to identify flaws in that can drive up costs because the application will use the cloud’s endless supply of
compute cycles to keep itself running at an acceptable pace.
untested code or
Most continuous development methods include steps designed to allow developers to
resisting efforts to add
fix coding errors or small bugs as they happen rather than as they are executed in the
security controls to the
field. They are not, however, designed to catch major architectural or security flaws.
code before it ships,
And even if rapidly developed code is perfect, data, applications and systems designed
it exposes itself in
for closed, trusted data center environments can still be vulnerable to configuration
ways that would have errors, especially those resulting from uncertainty about the version of a product or
been unthinkable technology being used in the cloud—a frequent problem in an environment designed
to make such details invisible. The virtual servers that support Amazon Web Services’
under traditional
Elastic Compute Cloud (EC2), for example, run on the open-source Xen hypervisor. Not
development models.
many software-based security products have been successfully ported to formats that
allow them to run as virtual machines on Xen hypervisors, and those that have been are
probably going to run only on plain-vanilla Xen hypervisors, not the customized versions
that AWS uses for EC2.
When a flaw does appear, such as the one that forced a reboot of about 10 percent of
AWS’ Xen servers in September 2014, Amazon notifies customers that the servers are
going down but cannot put other organizations at risk by announcing why, according to
Network World.1 Without that information, it is impossible to tune new code to run more
smoothly or add controls that address weaknesses that developers cannot see.
1
“What happens inside Amazon when there’s a Xen vulnerability,” Network World, March 3, 2015,
www.networkworld.com/article/2892313/cloud-computing/what-happens-inside-amazon-when-there-s-a-xen-vulnerability.html
SANS ANALYST PROGRAM
2 A DevSecOps Playbook
Delivering Speed—and Control
If organizations are not going to let information security slow down the business, then
information security needs to find a way to embed security controls and monitoring
into the deployment cycle. The answer to this may come from automation, which many
enterprises are focusing on more heavily than in the past in their data centers.
The rise in automation has come as development and operations teams have started to
collaborate more often and with much more rigor. This trend has led to a movement of
sorts known as DevOps, which strives to foster open dialogue and intense collaboration
between development and operations teams, leading to the possibility of what is
referred to as the continuous delivery of code, or much more frequent code promotion
than has been traditional. This collaboration can have a major effect in condensed data
centers and cloud environments because it allows for many new features to be rolled out
much more quickly. What are continuous delivery and DevOps exactly? The following
basic definitions may help to clarify things:
• “Fail fast and often.” The sooner code flaws can be detected, the less effect they
will have in a working production environment. Rapid and almost constant testing
is needed for this to happen.
• Automated builds and testing. More automation in the testing and quality
assurance (QA) processes helps speed things up and improve delivery times.
Large cloud environments such as Amazon and Netflix push hundreds, even
thousands, of code changes per day—a practice that would be considered insane
under traditional IT operational standards, which required thorough testing of
every patch before deployment. But that pace is required by the working model of
commercial cloud services.
Information security is building on the push within DevOps teams for collaboration
and automation by introducing another concept, DevSecOps. DevSecOps strives to
automate core security tasks by embedding security controls and processes into the
DevOps workflow. DevSecOps originally focused primarily on automating code security
and testing, but now it also encompasses more operations-centric controls. Security can
benefit from automation by incorporating logging and event monitoring, configuration
and patch management, user and privilege management, and vulnerability assessment
into DevOps processes. In addition, DevSecOps provides security practitioners with the
ability to script and monitor security controls at a much larger and more dynamic scale
than traditional in-house data centers.
DevSecOps strives
to automate core Step 1: Assess Your Current Security Controls for Cloud
security tasks by When planning for DevSecOps, security teams first need to perform threat modeling
and risk assessment for the deployment types that they envision. By performing a threat
embedding security
modeling exercise, security teams can better understand the types and sensitivity levels
controls and processes
of the assets they are protecting, how they will be managed and monitored in the cloud,
into the DevOps and what the most likely threat vectors are for those assets. Once the threat landscape is
workflow. understood, security and compliance teams should perform a risk assessment to better
clarify what the real risks are in cloud deployments, what risks should be considered
priorities, and what the potential impact and likelihood of those risks are.
The following is a short list of things to consider during threat modeling and risk
assessment:
• Most likely threats. A comprehensive threat modeling exercise should take both
insider threats (within the organization and the cloud provider environment) and
external adversaries into account.
• Data types and sensitivity. The type of data that is stored, transmitted and
processed will make a difference when assessing the risk of systems and
applications in the cloud. Some data types will dictate specific security controls, as
well as provisioning into compliant cloud provider environments. There may also
be more (and more serious) threats in play with certain types of data (credit card,
health care and personal information, for example), and the risk severity may be
greater as well.
SANS ANALYST PROGRAM
4 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
• Cloud environment security posture. A major part of the risk inherent in any
cloud scenario is the security posture of the cloud provider. Most reputable cloud
providers offer a variety of controls attestation documents, such as the SSAE 16
SOC 2, ISO 27001 and ISO 27002 reports, or a report on the Cloud Security Alliance
Cloud Controls Matrix (CCM). Security teams should review this documentation
carefully when choosing a cloud provider, if at all possible.
• Existing controls in place. The types of controls currently in use will make
a difference in how cloud operations will function. Some controls may be
absolutely required, while others may not be. Making a list of controls (network
access controls, host monitoring, etc.) will help security teams evaluate the cloud
providers’ control offerings as well as compatible third-party options.
• Controls we lose in the cloud. Not all controls will be available in the cloud;
others will need to be adapted to function correctly in a new environment. Some
controls will be available from the cloud providers themselves—AWS and other
large providers offer many native encryption and key management options, for
example. Other controls may have to come from new vendors or service providers
or be provided and managed entirely by the cloud provider staff.
After performing a risk assessment, security teams should have a better understanding
of what controls they currently have available, which ones they will have to modify when
moving to the cloud and which of their concerns are the most pressing. It is almost a
guarantee that some security controls will not operate the way they did in-house or
will not be available in the environment of the cloud service provider. For example,
many network security controls are not available to cloud tenants, and hypervisor
configuration and patching is entirely up to the service providers as well.
Table 1 provides a brief breakdown (not a complete list by any means) of some
of the major things to consider for risk assessment purposes in various cloud
deployment scenarios.
In a nutshell, different cloud models afford tenants access to and control over different
layers of the stack and the components therein. Tenants should thoroughly assess the
risks of every layer that they can configure and consider controls that align with the
design and operational model they have chosen for that cloud environment.
The notion of DevSecOps first appeared in a January 2012 blog entry by Neil MacDonald
of Gartner. He argued that DevOps efforts should not be curtailed in favor of security,
while acknowledging that security was needed. His answer was that security should
be integrated into the DevOps process.2 (MacDonald called this DevOpsSec, but most
people now refer to it as DevSecOps.)
2
“ DevOps Needs to Become DevOpsSec,” Gartner Blog Network, Jan. 17, 2012,
blogs.gartner.com/neil_macdonald/2012/01/17/devops-needs-to-become-devopssec
SANS ANALYST PROGRAM
6 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
MacDonald’s idea was interesting, though contrary to most security organizations’ long-
held practices. Most promising, it seemed to hold out hope for circumventing an ever-
starker choice between speed and security.
Security practitioners should focus on certain areas when looking to adapt in support of
a DevSecOps practice.
Development
Internal security teams remain the primary testers of application security (see Table 2).
If security teams are going to be a core component of DevSecOps, they must impress
upon development and operations that they can bring a series of tests and quality
conditions to bear on production code pushes without slowing the process. If security
parameters and metrics are incorporated into development and test qualifications,
then the chance for security to be involved in the processes for DevOps is much higher.
Security teams should work with QA and development to define certain parameters and
key qualifiers that need to be met before any code can be promoted.
In addition, security teams should look to integrate automated dynamic and static code
testing throughout the development and promotion life cycle to help development and
security teams detect and fix any major code flaws discovered as rapidly as possible.
3
“ 2015 State of Application Security: Closing the Gap,” SANS Institute InfoSec Reading Room, May 2015,
www.sans.org/reading-room/whitepapers/analyst/2015-state-application-security-closing-gap-35942
SANS ANALYST PROGRAM
7 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
Inventory Management
Defining configuration baselines and solidifying those into practical policies that can
be applied to all systems are vital steps in the proper implementation and monitoring
of secure configurations. One challenge that many large organizations have had with
this over the years is the sheer diversity of system types, as well as the disparity in
tools that will not work on all platforms. As organizations move to the cloud, finding
some homogeneity across tools and consolidating systems will prove invaluable in
maintaining system security and patch levels over time, especially in a dynamic, DevOps-
driven architecture.
4
Center for Internet Security, CIS Controls for Effective Cyber Defense Version 6.0, www.cisecurity.org/critical-controls.cfm
SANS ANALYST PROGRAM
8 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
Logs and events generated by services, applications and operating systems within cloud
instances should be automatically collected and sent to a central platform. Automated
and remote logging is something many security teams are already comfortable with,
so organizations implementing DevSecOps really just need to ensure that they are
collecting the appropriate logs, sending them to secure central logging services or
cloud-based event management platforms, and monitoring them closely using security
information and event management (SIEM) and/or analytics tools.
One major goal of both DevOps and DevSecOps is to facilitate much more rapid and
scalable IT, with security controls built in and rapid execution of security processes.
To scale in a large hybrid or public cloud, security will need to embrace automation,
a concept that many security practitioners have been loath to embrace. For true
DevSecOps to take hold, security teams will need to embed automated tests and
validation of controls into the deployment cycle and monitor applications continuously
in production with triggered responses that can roll controls back to a known good
state, among other outcomes.
Microsegmentation
While most network services and controls are unavailable in cloud provider
environments, one technique for network isolation and policy control that is gaining
significant traction is microsegmentation.
With this technique, each cloud instance adopts a “zero trust” policy model that allows
for very granular network interaction controlled at the virtual machine network interface
controller. This allows each cloud system to essentially take its network access control
and interaction policy with it as it migrates through virtual and cloud environments,
minimizing disruption and reliance on physical and hypervisor-based network
technology, although software-defined networking and automation techniques can
definitely play a role in microsegmentation operations.
70%
60%
50%
40%
30%
20%
10%
0%
OWASP Top 10
ISO/IEC 27034
NIST 800-53/64
Microsoft SDL
BSIMM
Open SAMM
SAFECode
Other
Figure 1. Application Security Standards in Use5
One of the better-known tools is Chef, a Ruby-driven framework that is used to create
configuration “recipes” and “cookbooks.” The goal is an increased number of automated
configurations of controls across multiple servers in virtualized and cloud data centers.
Chef is ideal for building automated “blueprints” of systems for rapid template-based
builds, as well as recovery and cloning of environments in very short time frames.
5
“ 2015 State of Application Security: Closing the Gap,” SANS Institute InfoSec Reading Room, May 2015,
www.sans.org/reading-room/whitepapers/analyst/2015-state-application-security-closing-gap-35942, Figure 7, p. 12.
SANS ANALYST PROGRAM
11 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
One other toolkit that is becoming more popular is Salt, which strives to make the
process and management of infrastructure configuration much simpler. It leverages the
ZeroMQ messaging library and a central configuration library and uses Python-based
modules for handling all aspects of configuration queries, responses, rapid system state
gathering for immediate use, remote execution of commands and more.
There are also commercial products and third-party services that can help security teams
implement automated DevSecOps by embedding simple agents into virtual machine
instance templates, which then fetch policy updates upon instantiation. These systems
are under the control of the operations team, but security can continuously monitor the
state of the systems as well.
In 2012, researchers Josh Corman and Gene Kim,6 both of whom are DevOps-focused
analysts formerly with the 451 Group, and James Wickett of the Open Web Application
Security Project (OWASP)7 suggested an approach they called “rugged DevOps.” Wickett
laid out a series of steps to implement the concept in a presentation he called the
“Survival Guide Pyramid.” It was designed to illustrate a practical approach to secure
development, at speed.
6
“ Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed,” Gene Kim LinkedIn slideshow, March 9, 2012,
www.slideshare.net/realgenekim/security-is-dead-long-live-rugged-devops-it-at-ludicrous-speed
7
“ Rugged DevOps: Bridging Security and DevOps,” James Wickett LinkedIn slideshow, March 31, 2012,
www.slideshare.net/wickett/rugged-devops-bridging-security-and-devops
SANS ANALYST PROGRAM
12 A DevSecOps Playbook
Delivering Speed—and Control (CONTINUED)
DevOps workflow. • P
ay careful attention to privilege allocation and user, group and role management.
This can easily creep over time in a dynamic environment.
• C
ommit to a culture of continuous monitoring, helping to automate detection and
scripted response activities that minimize manual intervention wherever possible.
• D
iscuss vulnerabilities detected in cloud deployments with all team members, and
make sure DevOps teams are involved in vulnerability, patch and configuration
management discussions and policy creation.
• E nsure that you are gathering adequate security and operations logs and event
data, sending it to a remote monitoring and collection platform.
• D
iscuss the changing threat landscape with DevOps teams, and solicit their
feedback on practical measures that can be taken to implement the most effective
security without impeding progress or slowing down the pace of business activities.
All in all, security teams can be valuable team players in the DevOps movement.
However, they need to adapt to more rapid changes, more flexibility and the willingness
to concede “perfect security” for business benefits, with advanced monitoring and
awareness after the fact.
Sponsor
SANS would like to thank this paper’s sponsor: