Kubernetes Guide: Graceful Shutdown with Lifecycle preStop Hook

Pods are constantly terminated in Kubernetes. It happens when the amount of traffic the app receives changes, when deploying new versions or when the node runs out of resources. In order to ensure those terminations will not create downtime for your users, you need to make sure the app handles termination gracefully.

And while many K8s admins rely on the app catching a SIGTERM, we at Datree believe that you also need to use a preStop hook to ensure your apps are shutting down gracefully. Actually, we believe it so much that we recently added a policy rule that verifies that you have a preStop hook in all your containers.

Why is that? Read on to find out.

* The post is based on this video by Datree’s CEO and AWS Hero, Shimon Tolts.

What is a Lifecycle preStop Hook? 

Lifecycle preStop hook is a hook that runs before a container is terminated. It can be either a HTTP signal or an EXEC command. 

HTTP Signal:

lifecycle:
  preStop:
    httpGet:
      port: 8080
      path: shutdown

EXEC Command:

lifecycle:
  preStop:
    exec:
      command:
        -sh

Why Do You Need a Lifecycle preStop Hook?

When it comes to graceful shutdown, many K8s admins rely on the application handling a SIGTERM. Yet, often this is not enough. In many cases, even though the application handles the SIGTERM, the load balancer still ends up sending traffic to a terminated pod, which translates to downtime.

Why does this happen?

Let’s review how a SIGTERM works. Our process receives a SIGTERM from our K8s in order to start a graceful shutdown process.

In this process the Kubernetes will tell the ingress controller to deregister the pod from receiving additional traffic, while the pod finishes processing all of the tasks it has.

While usually it works, in some cases there might be a race condition where our pod will finish draining connections and terminate itself before the controller finishes the deregistration process.

In this situation, the ingress controller will keep sending traffic to the pod even though the pod is terminated, which will result in downtime.

In order to prevent this situation, you should use a preStop hook.

Unlike SIGTERM which runs at the level of the process, the preStop hook runs at the level of the container and is executed before the SIGTERM is sent to the process. 

Therefore, you can implement a preStop hook that will start the deregistration process, verify that it has completed, and only then proceed to sending the SIGTERM and terminating the pod.

Lifecycle preStop Hook Common Mistakes

Here are 4 common mistakes you should avoid when configuring your preStop hook:

Exposing the HTTP Endpoint 

If you are using an HTTP endpoint for the preStop hook, make sure only the internal network can access it. The last thing you want is some bot crawling your website and initiating a shutdown process.

Using a ‘Sleep’ Command

Don’t just use ‘sleep’ as your EXEC command. You don’t know how much time the ingress controller will take before deregistration is complete. Instead, you should configure a command that initiates the deregistration process and verifies it had completed.

Using the Default TerminationGracePeriodSeconds

This property defines the timeout for the shutdown process, and it includes both the preStop hook and the SIGTERM. Make sure to configure it so both processes will have enough time to complete.

Not Following a Single Responsibility Model

Don’t take the logic that verifies the deregistration process has completed and add it to your application. This does not follow the single responsibility model: Apps should control the process, lifecycle hooks should control the container.