Devsecguide Kubernetes
Devsecguide Kubernetes
to Kubernetes
Kubernetes is the de facto container orchestration system,
offering development teams incredible scale, flexibility, and speed
when deploying and managing cloud-native applications. For all its
benefits, however, it also brings new complexity and risk.
Table of contents
Conclusion..................................................................................................................12
• Cluster: Securing Kubernetes deployments requires configurations. Isolating non-dependent pods from talking
securing the underlying infrastructure (nodes, load to one another using network policies is important to
balancers, etc.), configurable components, and the prevent lateral movement across containers in the event
applications which run in the cluster, including maintaining of a breach. Kubernetes also has built-in features to define
the posture of underlying nodes and controlling access how pods behave and what they have the ability to access.
to the API and kubelet. It’s also important to prevent
• Containers: Container security depends on the security
malicious workloads from running in the cluster and isolate
of the images that make them up. This requires using
workload communication through strict network controls.
trusted registries and libraries and scanning them for
• Control plane: Also known as master nodes, control planes vulnerabilities in build-time. It’s also important to monitor
include schedulers, API servers, and other components runtime containers for suspicious activity such as a shell
that make global decisions for worker nodes within running inside or sensitive data being leaked.
clusters. Minimizing admin-level access to control planes
• Code: Application and infrastructure code is almost entirely
and ensuring your API server isn’t publicly exposed are the
controlled by the end-user and thus often presents
most important security basics.
attackers with the most penetrable parts of the attack
• Pod: Kubernetes nodes host pods or workloads, which surface. Minimizing known vulnerabilities and attack
are collections of containers that share common vectors reduces the ways an application can be exploited.
Cluster Code
Application code
Infrastructure code
etcd API server
Nodes
Pods
kubelet
Containers
Scheduler Controllers Container registries
• PodSecurityPolicies: PSPs let you control pod and reducing complexity and duplication.
container behavior by specifying a set of requirements that
pods must follow in order to be accepted by the cluster; The biggest security advantage with IaC is that it enables you
if a request to create or update a Pod fails to match the to scan earlier in the development lifecycle to catch easy-to-
requirements, the request is rejected, and an error is miss misconfigurations before they’re deployed.
returned. PSP is far from comprehensive, which is why it is
currently being deprecated.
That said, a truly secure Kubernetes environment requires a huge component of DevSecOps and is crucial for
These are some of the most common Kubernetes IAM security for Kubernetes clusters
security errors and misconfigurations: In addition to managing internal access controls within
• Leaving host infrastructure vulnerable clusters using Kubernetes RBAC and depending on where
and how you run Kubernetes, you may use an external cloud
• Granting overly permissive access to clusters and registries
service to manage access controls for your Kubernetes
• Running containers in privileged mode and allowing environment. For example, with AWS EKS, you’ll use AWS
privilege escalation
IAM to grant varying access levels to Kubernetes clusters
• Pulling “latest” container images based on individual users’ needs.
• Failing to isolate pods and encrypt internal traffic
Using a least-privilege approach to manage IAM roles and
• Forgetting to specify resource limits and enable audit
policies in IaC minimizes the risk of manual configuration
logging
errors that could grant overly permissive access to the
• Using the default namespace
wrong user or service. You can also scan your configurations
• Incorporating insecure open-source components with IaC scanning tools (such as our open-source tool,
Checkov) to automatically catch overly permissive or
unused IAM roles and policies.
Kubernetes host infrastructure security
Let’s start at the most basic layer of a Kubernetes
environment: the host infrastructure. This is the bare metal Container registry security
and/or virtual servers that serve as Kubernetes nodes. Although they’re not part of native Kubernetes, container
Securing this infrastructure starts with ensuring that registries are widely used as part of a Kubernetes-based
each node (whether it’s a worker or a master) is hardened application deployment pipeline to store and host the
against security risks. An easy way to do this is to provision images deployed into a Kubernetes environment. Access
each node using IaC templates that enforce security best control frameworks vary between container registries.
practices at the configuration and operating system level. With some, you can manage access via public cloud IAM
frameworks. Regardless, you can typically define, apply,
When you write your IaC templates, ensure that the image and manage them using IaC.
template and/or the startup script for your nodes are
configured to run only the strictly necessary software to In doing so, you’ll want to ensure that container images
serve as nodes. Extraneous libraries, packages, and services are only accessible by registry users who need to access
should be excluded. You may also want to provision nodes them. You should also prevent unauthorized users from
with a kernel-level security hardening framework, as well as uploading images to a registry, as insecure registries are an
employ basic hygiene like encrypting any attached storage. excellent way for threat actors to push malicious images
into your environment.
Avoid pulling “latest” container images Isolate pods at the network level
When you tell Kubernetes to pull an image from a container By default, any pod running in Kubernetes can talk to any
registry, it will automatically pull the version of the specified other pod over the network. That’s why, unless your pods
image labeled with the “latest” tag in the registry. actually need to talk to each other (which is usually only
the case if they are part of a related workload), you should
While this may seem logical—after all, you typically want isolate them to achieve a higher level of segmentation
the latest version of an application—it can be risky from between workloads.
a security perspective because relying on the “latest”
tag makes it more difficult to track the specific version As long as you have a Kubernetes networking layer that
of a container that you are using. In turn, you may not supports Network Policies (most do, but some Kubernetes
know whether your containers are subject to security distributions default to CNIs that lack this support), you
vulnerabilities or image poisoning attacks that impact can write a Network Policy to specify which other pods the
specific images on an application. selected pod can connect to for both ingress and egress.
The value of defining all of this in code is that you can scan
To avoid this mistake, specify image versions, or better yet, the code to check for configuration mistakes or oversights
the image manifest when pulling images, and audit your that may grant more network access than intended.
Kubernetes configurations to detect instances that lack
specific version selection for images. It’s a little more work,
but it’s worth it from a Kubernetes security perspective. Encrypt internal traffic
Another small but critical setting is the --kubelet-https=...
flag. It should be set to “true.” If it’s not, traffic between
Avoid privileged containers and escalation your API server and kubelets won’t be encrypted.
At the container level, a critical security consideration
is ensuring that containers can’t run in privileged mode. Omitting the setting entirely also usually means that traffic
Running containers in privileged mode—which gives them will be encrypted by default. But because this behavior
unfettered access to host-level resources—grants way too could vary depending on which Kubernetes version and
much power but, unfortunately, is an easy mistake to make. distribution you use, it’s best practice to be explicit about
requiring Kubelet encryption, at least until the Kubernetes
Disallowing privilege escalation is easy to do using IaC. developers encrypt traffic universally by default.
Simply write a security context that denies privilege
escalation, and make sure to include the context when
defining a pod: Specifying resource limits
You may think of resource limits in Kubernetes as a way to
control infrastructure costs and prevent “noisy neighbor”
securityContext:
allowPrivilegeEscalation: false issues between pods by restricting how much memory
and CPU a pod can consume. In addition, resource limits
are crucial from a security perspective, helping to mitigate
Then, ensure that privilege isn’t granted directly with the the risk of denial-of-service (DoS) attacks. A compromised
“privilege” flag or by granting CAP_SYS_ADMIN. Here again, pod can do more damage when it can suck up the
you can use IaC scanning tools to check for the absence entire cluster’s resources, depriving other workloads of
of this security context and to catch any other privilege functioning properly. From a security perspective, then, it’s
escalation settings within pod settings. a good idea to define resource limits.
• Hard-coded secrets
In addition to avoiding misconfigurations and vulnerabilities,
You can check for misconfigurations using IaC scanners it’s crucial to avoid hard coding secrets such as passwords
which can be deployed from the command line or via IDE or API keys that threat actors can leverage to gain privileged
extensions. IaC scanners work by identifying missing or access. Secrets scanning tools are key to check for
incorrect security configurations that may later lead to sensitive data like passwords and access keys inside source
security risks when the files are applied. code. They can also be used to check for secrets data in
configuration files that developers write to govern how their
Container scanning checks for vulnerabilities inside application will behave once it is compiled and deployed.
container images. You can run scanners on individual
images directly from the command line; however, most The key to getting feedback in this stage is to integrate with
container registries also feature built-in scanners. The developers’ local tools and workflows—either via integrated
major limitation of container image scanning tools is that development environments (IDEs) or command lines. As
they can only detect known vulnerabilities—meaning code gets integrated into shared repositories, it’s also
those that have been discovered and recorded in public important to have guardrails in place that allow for team-
vulnerability databases. wide feedback to collaboratively address.
People Processes
Having the right people sets the foundation for DevSecOps. The DevSecOps paradigm necessitates new processes
Security training and fostering security champions has been or perhaps improvements to existing ones that prioritize
the go-to solution for making security matter, but you can’t security at each step.
stop there. DevSecOps requires bi-directional knowledge
sharing to build true shared accountability for security. • Development: As code is being written and updated,
securing feedback should be incorporated into workflows
Building your team based on formal titles isn’t necessary via IDE extensions or CLI tools. By surfacing security best
for building the right culture; whether you already have the practices earlier, it’s easier to address issues with the
right building blocks or are looking to build out your teams, right context and quickly prevent issues from progressing
these are some of the skills you should look out for: further. This is also a great way to foster continuous
security education through actionable insights.
• A knack for efficiency: Regardless of department, • Build and deploy: As you add checks to your pull/merge
efficiency and automation are key to DevSecOps success. requests and CI/CD pipelines, ensure that all stakeholders
When manual work inevitably crops up, teammates with are aware and expectations are aligned. That way, friction
productivity mindsets will invest the time to make that won’t arise when a build fails or a deployment is blocked
repeatable in the future despite the temptation to just due to a critical misconfiguration or vulnerability. When
complete the task at hand. issues do arise, make sure you have individuals responsible
and on-call to help things keep running smoothly.
• Balance individual focus and greater goals: DevOps aims
to break down the development process into smaller • Runtime: Even with the most mature proactive security
components and processes, isolating individual outcomes guardrails in place, the work doesn’t stop at deployment.
at each phase. DevSecOps requires striking the right Having the right visibility and processes for when security
balance between security and efficiency. To do that issues are exposed in runtime is a significant part of a
in practice, priorities need to be set, recognized, and comprehensive DevSecOps strategy.
constantly evaluated from the organization level to the
individual contributors. • Feedback and Planning: It’s important for all stakeholders
to understand the security impact new features and
• Continuous learning: Although Kubernetes has been around updates may have. Security training and awareness are
for a while now, it’s valuable for everyone to be constantly also crucial at this phase, as work done in this phase will
learning new things when it comes to building the most determine the security coverage throughout the rest of the
performant and innovative products. The same goes for development lifecycle.
security. Staying on top of the latest vulnerabilities and
policies is essential to keep your applications secure.
Having natural curiosity is ideal, but with consistent Setting the right processes in place ensures that everyone
processes for training and education, you can achieve the is on the same page and sets the foundation for security
same outcome. consistency and cohesiveness.
When researching and implementing security tooling, make Here are some sample KPIs that touch all development,
sure to keep these criteria in mind: operations, and security teams:
• In code, for code: Whether you’re looking for workload • Volume of production issues over time and by severity:
protection or cloud security posture management, having Ideally, the number of misconfigurations in runtime should
visibility at the code level is crucial. This is especially true go down over time if issues are addressed earlier. By
if you develop and manage any infrastructure in code— having end-to-end visibility, it should also be easier to
which you likely do if you’re leveraging Kubernetes. On a prioritize higher severity issues, leading to fewer alerts and
similar note, being able to address issues at the source hardening infrastructure over time.
is the best way to prevent issues from resurfacing and
• Mean time to remediation (MTTR): Related, as the volume
snowballing into hundreds of security alerts in runtime.
of issues goes down over time, identified vulnerabilities
• Integrated into dev tools and workflows: For code and misconfigurations should be resolved faster over
feedback and fixes to be truly actionable, it needs to be time. A shorter MTTR also indicates a stronger CI/CD
surfaced at the right time in the right place. Whether pipeline and institutional knowledge when it comes
that’s on a developer’s local workstation, in a pull/merge to infrastructure being deployed and its security
request, or during a CI/CD build, continuously enforcing expectations.
security best practices is easiest when layered on top of
• Deployment speed and frequency: As you bake security
existing tools and workflows.
measures into your DevOps lifecycle, be sure to monitor
• Code to cloud coverage: One of the big challenges of how frequently and quickly you’re deploying. At the end of
securing cloud-native environments is that the state and the day, security checks are only valuable if you’re able to
connections of their components change throughout the deliver updates, so striking the right balance by tweaking
development lifecycle. That’s why coverage across stages levels of control is key.
is important and why having unified policies and visibility
from code to cloud is so important. Only then can you
address issues at the source and detect drift. Having that Because Kubernetes is such a dynamic and complex
complete coverage also helps to bridge the gap between system, it’s even more crucial to implement a solid set
engineering, operations, and security. of KPIs to help you assess your organization’s success
internally and externally. DevSecOps is getting more
• Don’t forget about compliance: While many of the issues
popular as a means to avoid costly (both in resources and
we’ve addressed so far relate to security best practices,
getting compliance feedback and enforcing rules for
reputation) breaches.
By leveraging an IaC-based approach to defining security rules in Kubernetes, you can more easily configure
and minimize the risk of configuration issues that will lead to security breaches within any layer of your
Kubernetes stack. Leveraging Kubernetes’ built-in security features along with a holistic IaC security strategy is
a solid foundation for building security in from the start. DevSecOps is the strategy that makes it all possible,
integrating teams with common processes, metrics, and tools under a common goal—to deploy secure
applications without being hindered by security best practices.