Built-in Kubernetes Policies for Preventing Misconfigurations

{{text-cta}}

Kubernetes - Power and Responsibility 

Kubernetes provides a powerful interface for managing and distributing resources. Whether you're a developer, SecOps team member, or a DevOps engineer working with CI/CD pipelines, Kubernetes gives you great power in deploying, scaling, and managing your containers across multiple hosts.

But with this power comes responsibility—and the potential for misconfiguration. With Kubernetes you can still do everything you can do in traditional virtual machine or bare metal implementations, but you now share a space with dozens or even hundreds of other functions, each with their own resources and capabilities. In such an environment, a misconfiguration could cause overutilization, vulnerabilities, or even downtime.

So let's look at four Kubernetes-native ways to prevent misconfigurations and protect users from themselves.

1. Protecting Hosts From Pods With Pod Security Policies

‍

Typically, a container should not interact with the host. This is an anti-pattern under the Kubernetes ephemeral paradigm. However, there are times when an application legitimately does need this level of access, and Kubernetes provides features to restrict this capability to only those services that truly need it. These Pod Security Policies define who can decide how a container within a pod can interact with the host. 

Why would you want a pod to interact with the host's networking stack, or access and mount directories on a host?  

One reason is DaemonSets. Deploying monitoring DaemonSets is a common flow that allows services like Telegraf, Prometheus, Fluentd, and many other metrics and log aggregators to access system-level performance data and logs across the cluster.

However, special permissions are required to access the hosts' components and pass the data to a non-privileged datastore. System owners may want to delegate this capability to a group of trusted administrators.

Another reason is to secure traffic between two services on different clusters with a function like Wireguard - this would ensure a VPN between your source container and a destination service in the wild. You might need this for HIPAA compliance measures, or retransmitting stored data to a third-party destination.

By allowing access to your Linux modules and system capabilities like NET_ADMIN, you can use this elevated tool that operates at the kernel level to permit compliance for your application.

To implement such designs, there are two steps.

1) First, create the Pod Security Policy itself:

‍

2) Then, grant user access by role, either per namespace or throughout the whole cluster:

‍

By implementing this feature, any pods created by user-generated resources will be validated to ensure they are complying with the policies set by the business and cluster owners. This prevents users from deploying pods that might have been found in online resources.

These pods might otherwise do things like manipulate TransparentHugePage settings on the host, which could then impact neighboring Redis services, or even inadvertently expose other pod runtime components by incorrectly utilizing host devices. Pod Security Policies can prevent this mayhem while still allowing appropriate groups the elevated permissions required for their own configurations. 

‍

"You don’t curl weirdscript.sh | sudo bash, so why would you trust externally-sourced YAMLs?"

2. Empower Capacity Planning With ResourceQuotas

‍

When a system is shared across multiple tenants, it's important to ensure that the Organization and Service Level Agreements are still being met for each tenant. Resource Quotas help you meet these capacity planning needs by giving you granular control over the allocation of CPU, memory, storage, and other valuable system resources.

Without this configuration in place, you may find yourself with insufficient CPU for your new customer's toolsets, or not enough nodes with GPUs for that monthly data science reporting for your business leaders. 

Here’s an example of a ResourceQuota assigned to a coolapplications namespace, ensuring that this namespace can only be allotted 100 CPUs and 200 GB of RAM: 

‍

This simple configuration prevents this namespace from consuming resources beyond its allotted amounts, guaranteeing that sufficient space is left for other tenants. If a user or service attempts to create a pod that exceeds these limits, the admission controller blocks the creation of the pod and logs an appropriate error. 

By using this configuration to set these limits, a business unit can intelligently coordinate infrastructure expansion and costs, and limit runaway budgets at an infrastructure level rather than deploying wasted compute resources to segment organizations by hardware.

{{text-cta}}

3. Enable Capacity Availability With LimitRanges

‍

Limit Ranges are a complimentary resource allocation mechanism to Resource Quotas. Where Resource Quotas control the exact level allowed of system resources, Limit Ranges allow development teams to define the expected amount of resource consumption by their application.

If the application exceeds these ranges, the server will self-course correct to throttle CPU, reap back memory, and prevent additional deployment of resources. This allows teams to coordinate deployment of that application within the broader ecosystem.

The most common example of Limit Ranges is a pod-based specification that defines both the minimum and maximum required resources for an application to run. This gives a well-defined method of determining what quantity of resources each pod and container will consume within its namespace.

‍

In this example, we’ve set the minimum resources to schedule for the container as well as maximum resources the containers are permitted to use. It's important to note that if either of these is also defined in the earlier Resource Quota, then it must also be defined per container to be permitted for scheduling.

Kubernetes will use these Limit Ranges to monitor and stabilize our system. In the case above, if our application uses more than the expected CPU utilization, the system will simply throttle additional CPU usage of the container. The system is more stringent on memory usage, however.

In the case of memory overutilization, an Out of Memory (OOM) error will kill your container, forcing a pod restart. This limit is included for directives like emptydir.medium: memory, where some of your memory utilization will be in the form of ramdisk.

Limit Ranges is a powerful tool for the application owner to manage their own resources within the confines of capacity limits. No longer will a memory leak lead to the death of a host and elongated downtime as the pod’s memory limit will now do it for them!

4. Infrastructure Driven Consistency with Custom Admission Controllers

‍

Admission Controllers are services that monitor changes in the Kubernetes ecosystem and, based on configuration, take action to customize the infrastructure's behavior. One example is the ImagePolicyWebhook which, every time a pod is created, calls a custom endpoint to determine if the image is permitted in that environment.

Here's an example of the webhook request to the remote service:

‍

Using such a customizable function can be leveraged to curate exactly how the environment responds when the defaults just aren’t enough. In this example, we can determine if myrepo/myimage:v1 is in an approved list of images for the cluster, and ensure that it comes from a trusted repository before returning something like the following:

‍

There are two other examples of provided admission controllers that are worth noting: The Validating Admission Webhook and the Mutating Admission Webhook. These make similar calls to custom APIs for either approve/deny responses (like our provided ImagePolicyWebhook above), or entirely changing the submitted resources before they are officially created in the cluster. 

Utilizing any of these webhooks can add a great amount of control in the environment, helping you to conform Kubernetes to the business needs rather than force the business needs to conform to Kubernetes. 

‍

Conclusion - Built-in K8s policies

‍

Kubernetes is a wide-reaching orchestrator capable of managing both the smallest resources and massive web-scale infrastructures. To handle both these scenarios, and everything in between, it provides many useful functionalities out of the box, but by design cannot account for every use case. So Kubernetes gives you the power to customize your environments as you see fit. Understanding these configurations, and how you can use native ways to prevent misconfiguration, is a key to using Kubernetes effectively and safely.