{{text-cta}}
The ImagePullBackOff is an error state that occurs when a Pod startup fails due to an inability to pull the image. The state is displayed when running a $ kubectl get pods command for viewing/finding resources, as shown here:
A request to pull an image may include multiple elements in the path, each with a potential to be misconfigured. For each type of misconfiguration, Kubernetes returns a slightly different error, as described below. In short, there are two types of requests:
- Full path request (e.g. gcr.io/google-samples/pubsub-sample:v1) - this type of request specifies the registry (gcr.io), repository (google-samples), image name (pubsub-sample), and image tag (v1). Images in a private repository require the full path.
- Partial path request (e.g. pubsub-sample:v1) - in this case, the only elements included in the request are the image name (pubsub-sample) and the image tag (v1). In terms of the missing registry, if you don’t define a registry, the default registry is Docker hub. And in terms of the repository, there are two options:
- 1. For verified images in a public repository - you don’t need to specify the registry and the repository (e.g. nginx)
- 2. For non-verified images in a public repository - you need to specify the repository and the image name, but you don't need to specify the registry (e.g. rancher/nginx).
Reasons for ImagePullBackOff error
The ImagePullBackOff state can happen for the following reasons:
- Misconfiguration of path elements - whether for a full path or a partial path, the request itself may have errors in any of the elements. However, as described below, the error messages vary for each type of mistake and it often doesn’t indicate the specific issue.
- Access configuration issue - the registry/repository is private, and you are not authorized to access it.
- The registry is inaccessible - it’s possible that there’s a network issue between your Kubernetes node and the registry.
How to fix the ImagePullBackOff error?
Fixing misconfiguration of path elements
To make sense of the error messages concerning misconfigured path elements, I have generated examples for each misconfiguration option.
Misconfigured image name
Private registry
In the above example, we can see a request to pull - gcr.io/google-samples/fake-app:1.0. In this case the registry (gcr.io), exists. The repository (google-samples) exists. But the image name (fake-app) has been misconfigured.
Now let's look at the error that was returned:
You would expect that the error would indicate that the problem is in the image-name level, but, as you can see, the error message indicates that it has failed to fetch the image tag, although the misconfiguration is in the image-name level. This means that when this error appears, it could be a misconfiguration of both the image-name and the image-tag.
Public registry
In the above example, the request fake-app:1.0 did not indicate the registry or the repository. In this case, the system assumes that this is a verified image from a public repository.
Now let's look at the error that was returned:
The error indicates that either the repository does not exist or that you don't have access to Docker. This means that when this error appears:
- It’s possible that this is not a verified image in a public repository, so you should have indicated the registry and repository before the image name and image tag;
- It’s possible that that image tag does not exist;
- Or it’s possible that you don’t have access to the repository.
{{text-cta}}
Misconfigured image tag
Private registry
In the above example, we can see a request to pull - gcr.io/google-samples/hello-app:0.0.1. In this case the registry (gcr.io), exists. The repository (google-samples) exists. The image name (hello-app) exists. But the image tag (0.0.1) has been misconfigured.
Now let's look at the error that was returned:
In this case, the error rightfully indicates that it has failed to fetch the specific image tag (0.0.1).
Public registry
The short version of the request Pulling image nginx:fake-tag doesn't indicate the registry or the repository. In this case, the error has rightfully indicated that the manifest for the tag was not found.
Misconfigured repository
Private registry
In the above example, we can see the following request - gcr.io/samples/hello-app:1.0. The request lists an incorrect repository, samples, instead of google-samples. In this case the error message indicates the project:samples was not found. This error is used to indicate that the repository was not found.
Public registry
In this example, the request nginxx/nginx:1.14.0 includes a misconfigured repository - nginxx. The error message correctly indicates that the repository either doesn’t exist or requires login.
Misconfigured registry
Private registry
In this example, the request has a misconfigured registry name – wrong-registry. The error indicates, "Service Unavailable". When this type of error appears, it means the system was unable to find the registry because it doesn’t exist.
Public registry
When checking it against the public image registry, kubernetes returns the same "Service Unavailable" message, indicating that the registry itself was not found.
Tips for troubleshooting path misconfigurations
The examples of error messages mentioned above should help you identify whether the ImagePullBackOff state was caused by a path misconfiguration. But in any case there are some additional things you can do to solve this “mystery”:
Try pulling the docker image locally using a generic tag - if you were not able to pull the docker image, try pulling it using a generic latest tag. If you’re successful, it means that the registry is correct.
Do you see any errors?
👉 [YES] There are two possible reasons:
- There’s an access configuration issue between your local machine and the registry.
- The image doesn’t exist on the repository. If your registry has a web GUI, you can use it to see the valid images.
👉 [NO] Add the image tag:
Do you see any errors?
👉 [YES] you now know that your image tag doesn’t exist in your registry. If your registry has a web GUI, you can use it to see the valid tags.
👉 [NO] You now know it is an access configuration issue between the pod and the registry because the image and tag are valid.
Fixing access configuration issue
For security reasons the error will not specifically indicate the lack of permissions to access the container registry name. However, if the names in the path are correct, perhaps you don't have permissions to access the registry/repository. In fact, the same error message saying that the repository doesn’t exist or has no pull access, will apply in this case.
- Check if you can access the container locally - as a first step you can try pulling the same image from your local machine.
- Add the secret to Kubernetes - if you’re trying to pull an image from a private repository that requires authentication, you need to add the required registry secret into Kubernetes. In the following example the secret name is registry-secret:
- Refer to the imagePullSecrets on a Pod - after adding the secret, you should add the imagePullSecrets reference so that your pod knows to use it. This needs to be done for each pod using a private registry.
Fixing inaccessible registry
In case there’s an issue that prevents access to the registry itself, you should do the following:
- Log into the Kubernetes node - you’ll need to check that this is the node to which the container was assigned by checking that you file /tmp/runbooks_describe_pod.txt contains the host name.
- Run container runtime to download and run by hand - run the container runtime command to download and run it manually as described above.
Learn from Nana, AWS Hero & CNCF Ambassador, how to enforce K8s best practices with Datree
Headingajsdajk jkahskjafhkasj khfsakjhf
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
%20(1).webp)
.png)
.png)
