Kubernetes Troubleshooting - Fixing ValidationError

{{text-cta}}

Kubernetes Error Codes: ValidationError 

With Kubernetes becoming the gold standard for container orchestration, managing enterprise applications has become simpler than ever. It’s made it an incredibly fast and easy process for organizations to spin up a cluster without worrying about complex deployments, services, and other resources. 

While there are many contexts where Kubernetes has diminished deployment pains for organizations with its automated provisioning, limitless configuration, and scaling capabilities, there are situations—especially the validation of deployment and service configurations—where it’s still a challenge. 

Applying deployment configuration to a Kubernetes cluster is an essential aspect of the development lifecycle, and it can become very tough to implement in situations where organizations wish to deploy multiple applications in Kubernetes. A slight change to the environmental variables in configuration files can return unforeseen errors. 

Before applying deployment configurations to a Kubernetes cluster, it’s crucial to grasp the workings of error codes in Kubernetes, as they help developers determine and debug the cause for policy violations in the cluster while ensuring that best practices are followed consistently across the cluster. 

There are a large number of error codes available that can be grouped into various categories for better understanding. In this article, however, we will be taking a specific look at the `ValidationError` messages. Let’s explore their significance and discuss how they can be resolved. 

What Is a `ValidationError`?

To deploy an application in Kubernetes, a configuration file is specified, known as a *manifest*, which contains the description/desired state for a Kubernetes API object (i.e., Deployment, ConfigMap, Secret, DaemonSet, etc.) deployed on the cluster. 

Written in YAML or JSON, a manifest can easily be filled with simple syntax typos or other mistakes, especially when writing them for complex deployments. When Kubernetes is unable to read and validate from the manifest, it displays a response message: `error validating data ValidationError `. 

Let’s dig into this with an example. We have taken a simple manifest for a sample deployment on a Kubernetes cluster below. 

The manifest is named **test.yaml** and will deploy two container instances of a web app that displays the "manifest sample" message on port 3000 if applied correctly. The manifest also contains configuration for service and selector that will route traffic and target a set of objects by their labels to the application pods.

bash
##test.yaml 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
 name: echo-server 
spec: 
 replicas: 2 
 selector: 
  matchLabels: 
   app: echo-server 
  template: 
   metadata: 
    labels: 
     app: echo-server 
  spec: 
   containers: 
   - names: echo-server 
    image: hm90121/http-echo 
     args: ["-text", "manifest sample"] 
      ports: 
      - containerPort: 3000
---
apiVersion: v1 
kind: Service
metadata: 
 name: echo-server 
spec: 
 ports: 
  - port: 3000 
   protocol: TCP 
    targetPort: 3000 
selector: 
 app: echo-server

To apply this manifest, use the `kubectl apply` command: 

kubectl apply -f test.yaml

Unfortunately, you will end up with an `error validating data ValidationError` error stating that `names` in the `containers` column of the manifest has a typo (it should be `name`): 

error: error validating "test.yml": error validating data: 
[ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "names" in io.k8s.api.core.v1.Container, ValidationError(Deployment.spec.template.spec.containers[0]): ...

**Note:** we have created a slightly broken deployment manifest above simply to illustrate a `ValidationError` example. Typically, the error does not occur in small manifests and is more prevalent for complex configurations. 

This is not the only situation where the`ValidationError` message code pops up. In cases where the manifest has indentation problems, the `ValidationError` comes to the rescue as well. 

To understand this scenario, consider the example manifest below:

##test.yaml 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: httpd-deployment 
  labels:
   app: httpd 
spec: 
  replicas: 1 
  selector: 
   matchLabels: 
    app: httpd 
   template: 
    metadata: 
     labels: 
      app: httpd 
spec: 
 containers: 
 - name: httpd 
  image: httpd:latest 
  ports: 
  - containerPort: 80 
   resources: 
    requests: 
    cpu: "0.5" 
    memory: "700Mi

Now, try applying the manifest using the `kubectl apply -f test.yaml ` command. You will probably get the following error response: 

error: error validating "test.yaml": error validating data: 
[ValidationError(Deployment.spec.template.spec.containers[0].resources): unknown field "cpu" in io.k8s.api.core.v1.ResourceRequirements, 
ValidationError(Deployment.spec.template.spec.containers[0].resources): unknown field "memory" in io.k8s.api.core.v1.ResourceRequirements]; if you choose to ignore these errors, turn validation off with --validate=false

The reason this results in a `ValidationError` is that there is an error in the indentation, as CPU and memory properties should be nested within `requests` and not in `resources`. To resolve the error, we will correct the indentation for the CPU and memory fields under the resources column. The corrected manifest will come out to be: 

##test-corrected.yaml 
apiVersion: apps/v1
kind: Deployment 
metadata: 
 name: httpd-deployment 
 labels: 
  app: httpd 
spec: 
 replicas: 1 
 selector: 
  matchLabels: 
   app: httpd 
  template: 
   metadata: 
    labels: 
     app: httpd 
  spec:
   containers: 
    - name: httpd 
     image: httpd:latest 
      ports: 
      - containerPort: 80 
      resources:
       requests: 
        cpu: "0.5" 
         memory: "700Mi"

 Now, try applying the corrected manifest using the `kubectl apply -f test-corrected.yaml ` command. The error should now be resolved: 

deployment.apps/httpd-deployment created 

{{text-cta}}

Kubernetes Tools for Solving `ValidationError` 

The Kubernetes ecosystem provides a wide variety of tools that can perform static checking on configuration files (YAML or JSON) for validation, security, and best practices. In this section, we will be comparing some of the most popular ones: 

kube-score 

kube-score is a validation tool that analyses Kubernetes resource configurations through static code analysis and built-in checks. The analysis list

provides recommendations for best practices, health checks, and missing limits for resources. Issues are mentioned using OK, WARNING, or CRITICAL tags, depending upon the context. 

kube-score checks manifests for: 

- Pod Health Checks 

- Containers running with escalated privileges 

- Resource limits and requests 

- Anti-affinity rules for high availability 

And many more. 

Installation instructions for kube-score can be found 

here. Demo with a sample output is also available via this link. 

Kubeval 

Kubeval is another validation tool that checks Kubernetes configuration files through a REST API server. The specified configuration file is matched against a specified API schema for missing fields or specifiers. Kubeval is supported with both YAML or JSON configuration files and has the ability to run offline for resource validations. Developers do not require access to the cluster to perform checking against a list of available schema. 

In addition to supporting various API schemas, Kubeval is compatible with multiple Kubernetes versions and provides validation output in various formats such as JSON, Test Anything Protocol (TAP), and plain text. Installation is easy via a single binary that can be [downloaded](https://www.kubeval.com/installation/) and run without any configuration. 

Conftest 

Developed by Stelligent, Conftest is a Kubernetes CLI validation tool that checks configuration files for missing resource limits or violations via tests. The tests are written using the rego query language, which is also used by the Kubernetes Open Policy Agent (OPA) for custom policy enforcement. 

Conftest tests are a collection of rules that returns a boolean based on the desired logic. The result can be formatted into many formats such as JSON, table, or TAP. This is helpful in scenarios where tests are integrated with CI/CD pipelines.

Installation instructions for Conftest can be found here. Conftest does not include pre-built tests, but there are examples in this GitHub repo. 

KubeLinter 

Developed by Stackrox, KubeLinter is a popular validation tool that provides automated analysis of Kubernetes resource configurations for readiness and security in production. 

KubeLinter integrates security-as-a-code for autonomous enforcement of policies according to the defined security practices. The tool also provides recommendations on how to resolve warnings or errors to comply with best practices. 

Installation instructions for KubeLinter can be found here. Usage instructions are available on their documentation site. 

Datree 

Datree is another open-source Kubernetes CLI validation tool that helps in correcting Kubernetes misconfigurations early in the development lifecycle. Datree provides an autonomous resource validation solution that monitors code changes for rule violations and missing limits. The deficiencies in configuration files are provided as alerts to developers, which contain the reason for the violation. 

Datree is open-source, so you don’t have to worry about project stability. Installation instructions are available for all three platforms—Windows, Mac OS, and Linux—to easily get started. 

Conclusion 

We have incorporated the most popular Kubernetes validation solutions in this article. While there are many more tools available for Kubernetes resource validation, it’s important for an organization to determine its needs before choosing one. Organizations that only want to validate resource configurations against a specific set of rules should prefer tools such as Conftest and Kubeval. 

KubeLinter and Kube-score are more geared towards typical best practices, which is a good starting point for organizations who are looking to integrate validation workflows for avoiding common errors in production. 

For organizations that are falling in between these categories, we strongly suggest using Datree. Datree provides support for both built-in schema validation and custom checks, which will help you cover a variety of Kubernetes use cases while specifying custom rules for violations.