EKS Security Best Practices - Practical Enforcement Guide

Amazon released an official guide called Amazon EKS Best Practices Guide for Security:

“This guide provides advice about protecting information, systems, and assets that are reliant on EKS while delivering business value through risk assessments and mitigation strategies. 

The guidance herein is part of a series of best practices guides that AWS is publishing to help customers implement EKS in accordance with best practices.”

In this post, I will go over the best practices listed in the guide and show you how to make sure you are following those with the help of the open-source project Datree.

📜 Translating the guide into policy rules

AWS's guide is fantastic, it is very comprehensive, but it can take a minute to understand how to turn it into practical guidelines for your engineers to follow. So, we at Datree have compiled it into practical configuration guidelines.

Here are 14 rules that we’ve compiled from the best practices guide:

1. Prevent containers from having unnecessary system call privileges
2. Ensure each container has a read-only root filesystem
3. Prevent containers from escalating privileges
4. Prevent service account token auto-mounting on pods
5. Ensure each container fully utilizes CPU with no limitations
6. Ensure container memory request and memory limit are equal
7. Ensure containers have limited capabilities
8. Ensure multiple replicas run on different nodes
9. Prevent pods from becoming unschedulable
10. Prevent Windows containers from running with unnecessary privileges
11. Prevent SELinux containers from running with unnecessary privileges
12. Prevent SELinux containers from setting a user
13. Prevent SELinux containers from setting a role
14. Ensure hostPath volume mounts are read-only

The full list of rules and details about how each rule and how to remediate is available within our docs: EKS Security Best Practices

🔭 Step 1: Get your cluster posture score 

The first step is understanding your status: Are your clusters currently compliant with the EKS Security Best Practices Guide? To check that, we will install Datree (in monitor mode) on our EKS cluster.

First, add the repo:

helm repo add datree-webhook https://datreeio.github.io/admission-webhook-datree
helm repo update

And now install Datree:

helm install -n datree datree-webhook datree-webhook/datree-admission-webhook --debug \
--create-namespace \
--set datree.token=<DATREE_TOKEN> \
--set datree.clusterName=$(kubectl config current-context) \
--set datree.policy=EKS 

(get your token from app.datree.io)

Once you finish the installation of Datree on your cluster, you will be able to get your Security Posture Score and see which rules in the EKS Security Best Practices policy are being violated in your cluster.

In the example above, you can see that 48% of our resources fail to meet AWS’ guidelines due to 6 failed policy rules.

🛠️ Step 2: Remediate and repair

Now we can drill down into each namespace and resource type to see which misconfigurations were discovered. When clicking on the rule description, we get instructions on how to fix an issue.

Let's take a look at one of the policy rules and follow the steps to fix them: ☑️ Prevent containers from escalating privileges

When analyzing the Argo namespace, we can see 3 violations of this best practice recommended by AWS, across 3 different deployments. We then click on the rule description and find out that to fix it all, we need to do is to add to the SecurityContext of each deployment the line: AllowPrivilegeEscalation: False

🚓 Step 3: Policy Enforcement - block misconfigured resources from being deployed on the cluster

Now, remediation will only take you so far. What’s even more important is prevention: You want to ensure all future deployments meet the EKS Security Best Practices Guide. For this reason, we also offer to use Datree in enforcement mode. In enforcement mode, Datree uses an admission webhook to scan any configuration change made to the cluster and block changes that do not meet your chosen policy.

To do it, we will first switch Datree from monitor to enforcement mode by running the following command:

helm upgrade -n datree datree-webhook datree-webhook/datree-admission-webhook --reuse-values --set 
datree.enforce="true"

Output:

From now on any resource that does not meet the policy will be rejected by the cluster and will not be created/modified.

Below you can see the Policy Enforcement in action. 

A workload was blocked from being deployed onto the cluster due to being noncompliant with the EKS Security Best Practices

Resources:

• https://aws.github.io/aws-eks-best-practices/security/docs/
• https://github.com/datreeio/datree
• https://hub.datree.io/built-in-rules/rules/#EKS
• https://app.datree.io/

⭐(Bonus) Step: Enable enforcement on CI/CD

A best practice is to shift left and block the misconfigurations before they reach the cluster!

In order to do so, you can use the datree CLI and run it during your development process and in your CI/CD

First install datree:

curl https://get.datree.io | /bin/bash

Now run use datree to test your manifest:

datree test ~/.datree/k8s-demo.yaml

And here are the results:

You can use the CI/CD examples for popular platforms.