Kubernetes Error Codes: Must Be Unique

{{text-cta}}

As a technology, Kubernetes has significantly simplified the DevOps workflow of managing infrastructure, enabling engineers to concentrate on delivering value to the business.

To report errors like _invalid configuration_ or _failure to execute part of the container lifecycle_, Kubernetes uses error codes like `CrashLoopBackOff`, `Init:ErrImagePull`, or `must be unique`. The intention is for errors to be simple enough to fit in a short message yet specific enough for troubleshooting the root cause.

Learning about these error codes improves your general understanding of Kubernetes and the container ecosystem. Once you’re familiar with them, you’ll be able to debug and troubleshoot any issue faster, saving you valuable time.

In this article, you’ll learn in detail what the `must be unique` error is, why your Kubernetes service is receiving it, and how to resolve it.

The `must be unique` Error

The `must be unique` error code is a storage error that occurs when you try to mount multiple source volumes to the same destination path within your container specification. More specifically, it means that every mountPath in the volumeMounts section of each container must be unique.

Kubernetes will throw this particular error code if you attempt to merge distinct secret and configuration sources into the container under a flat structure.

Let’s take a look at an example. The following code snippet creates a default configuration for all environments and sites:

log.level=INFO
api.url=http://api.example.com

The file will be named `default-config.properties:` and will be used to create a development Kubernetes service (described below) separate from the production service. In that service, you want to override the API URL and set the log level to DEBUG:

log.level=DEBUG
api.url=http://dev-api.example.com

Next, create some secrets containing the credentials of the API URL above. The secrets file is called `default-secrets.properties`.

api.username=default_user
api.password=default_secret_password

You’ll also create a secrets file for the development environment, named `dev-secrets.properties`:

api.username=dev_user
api.password=dev_secret_password

The following Kubernetes commands will create the necessary secrets and configMap in the cluster:

kubectl create configmap default-config --from-env-file=default-config.properties

kubectl create configmap dev-config --from-env-file=dev-config.properties

kubectl create secret generic default-secrets --from-env-file=default-secrets.properties

kubectl create secret generic dev-secrets --from-env-file=dev-secrets.properties

You can now create your Kubernetes container in development and load the `default-config`, the `dev-config` (for development, which may override some part of the default config), and the `default-secrets` and `dev-secrets` to load the necessary secrets.

Here is a sample development-service specification file named `dev-service.yaml` that includes the configurations above:

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: example-issue
  name: example-issue
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine:latest
    name: example-issue
    volumeMounts:
    - name: default-config
      mountPath: /var/config/
      readOnly: true
    - name: dev-config
      mountPath: /var/config/
      readOnly: true
    - name: default-secrets
      mountPath: /var/config/
      readOnly: true
    - name: dev-secrets
      mountPath: /var/config/
      readOnly: true
  volumes:
  - name: default-config
    configMap:
      name: default-config
  - name: dev-config
    configMap:
      name: dev-config
  - name: default-secrets
    secret:
      secretName: default-secrets
  - name: dev-secrets
    secret:
      secretName: dev-secrets

The structure of this document allows developers to maintain a safe default and secret configuration (common to all environments) while also allowing them to override specific configuration settings for different environments or customers.

However, when you try to load this configuration in the cluster, the engine will respond with the following error:

The Pod "example-issue" is invalid:
* spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/var/config/": must be unique
* spec.containers[0].volumeMounts[2].mountPath: Invalid value: "/var/config/": must be unique
* spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/var/config/": must be unique

The key point to note here is that when defining the volumeMount configuration for the container, the mountPath can’t be reused, as Kubernetes does not know how to resolve naming conflicts. For example, if both source volumes contain the file _password_, Kubernetes does not know which one will take precedence over the other.

Similarly, if the volumes are mounted in read-write mode and the container writes a file in the shared mountPath, Kubernetes doesn’t know which source volume it should write to.

How to Fix the `must be unique` Error

To resolve this error, you need to ensure that the volumeMounts are unique in your container. To meet this condition, you have several options. Which one you can use depends on your ability to modify the container image.

Using Projected Volumes

In 2017, Kubernetes proposed the concept of projected volumes, which has subsequently been implemented and documented.

In essence, a projected volume is a new volume driver that will take multiple read-only volume configurations and will generate a composite single volume that can be used in the container’s volumeMount configuration.

To use project volumes, the original configuration file can be modified as follows:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: example-projected
  name: example-projected
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine:latest
    name: example-projected
    volumeMounts:
    - name: secret-composite
      mountPath: /var/secrets/
      readOnly: true
  volumes:
  - name: secret-composite
    projected:
      sources:
        - secret:
            name: secret-one
        - secret:
            name: secret-two

There is now a single volumeMount for the container, and the volume is now a projected one, containing both secrets previously defined.

With this approach, if you have naming conflicts when adding multiple sources into a projected volume, the latter key will take precedence and be the one available to the container.

The above configuration will resolve to the following secrets/configuration within your container:

# for f in /var/config/* ; do echo "=== $f ===" ; cat $f ; echo ; done
=== /var/config/api.password ===
dev_secret_password
=== /var/config/api.url ===
http://dev-api.example.com
=== /var/config/api.username ===
dev_user
=== /var/config/log.level ===
DEBUG

In this particular example, the loaded configuration and secrets are those for the development environment, overriding the default ones. This allows the application to load the configuration files on a fixed path and to override specific configurations or secrets on a case-by-case basis.

{{text-cta}}

Using Distinct mountPaths

Another option is to use a distinct mountPath for each volume in the container definition and update the application’s logic to read each path to find the configuration.

The benefit here is that your pod configuration remains simple and explicit; however, the downside is that your application will need some type of logic to read the secret from different paths. This can get out of hand pretty quickly as you add more override folders or conditions per environment, per site, or per customer. That’s why this pattern is most suitable where you have completely separate volumes that need to be mounted or where some of the volumes are writable.

With this approach, the container definition file can be rewritten like this:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: example-issue
  name: example-issue
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine:latest
    name: example-issue
    volumeMounts:
    - name: default-config
      mountPath: /var/config/default-config
      readOnly: true
    - name: dev-config
      mountPath: /var/config/dev-config
      readOnly: true
    - name: default-secrets
      mountPath: /var/config/default-secrets
      readOnly: true
    - name: dev-secrets
      mountPath: /var/config/dev-secrets
      readOnly: true
  volumes:
  - name: default-config
    configMap:
      name: default-config
  - name: dev-config
    configMap:
      name: dev-config
  - name: default-secrets
    secret:
      secretName: default-secrets
  - name: dev-secrets
    secret:
      secretName: dev-secrets

Injecting Secrets as Environment Variables

This option is generally suitable if you know the name of the secrets and configuration you want to make available to the container. It also makes your container more portable to other orchestration services (such as AWS ECS Fargate), which provide secrets via environment variables. This option resolves the `must be unique` error by not using volumes at all.

In this option, instead of providing all configurations and secrets as files within the container and letting the container read those files to load their value, you will configure Kubernetes to convert the content of specific configurations and secrets into targeted environment variable names.

This approach has the benefit of being more portable to other container technologies and third-party applications that have better support for environment variables; however, this option is harder to implement, as there’s no simple way to override a default configuration or secret.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: example-env
  name: example-env
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine:latest
    name: example-env
    env:
    - name: api.username
      valueFrom:
        secretKeyRef:
          name: dev-secrets
          key: api.username
    - name: api.password
      valueFrom:
        secretKeyRef:
          name: dev-secrets
          key: api.password
    - name: api.url
      valueFrom:
        configMapKeyRef:
          name: dev-config
          key: api.url
    - name: log.level
      valueFrom:
        configMapKeyRef:
          name: dev-config
          key: log.level
  volumes:
  - name: default-config
    configMap:
      name: default-config
  - name: dev-config
    configMap:
      name: default-config
  - name: default-secrets
    secret:
      secretName: default-secrets
  - name: dev-secrets
    secret:
      secretName: dev-secrets

Using an initContainer

This last option uses an initContainer with a local volume to give you complete control over how the secrets and configuration are defined and resolved before your main container starts.

The following snippet shows how your container definition file may look like with initContainer:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: example-initcontainer
  name: example-initcontainer
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine:latest
    name: example-initcontainer
    volumeMounts:
    - name: generated-config
      mountPath: /var/config/
      readOnly: true
  initContainers:
  - command:
    - /bin/generate-config.sh
    image: alpine:latest
    name: generate-config
    volumeMounts:
    - name: generated-config
      mountPath: /var/config/generated-config
      readOnly: false
    - name: default-config
      mountPath: /var/config/default-config
      readOnly: true
    - name: dev-config
      mountPath: /var/config/dev-config
      readOnly: true
    - name: default-secrets
      mountPath: /var/config/default-secrets
      readOnly: true
    - name: dev-secrets
      mountPath: /var/config/dev-secrets
      readOnly: true
  volumes:
  - name: generated-config
    emptyDir: {}
  - name: default-config
    configMap:
      name: default-config
  - name: dev-config
    configMap:
      name: default-config
  - name: default-secrets
    secret:
      secretName: default-secrets
  - name: dev-secrets
    secret:
      secretName: dev-secrets

This option gives you the most control over how all secrets and configurations are merged into a shared volume during initialization without requiring you to modify the main application image. It also allows you to fetch secrets and configurations from outside the Kubernetes cluster, such as S3 or other external storage services.

This option is most suitable in the following cases:

• When you have multiple service containers that rely on a shared configuration that’s generated only once
• When you have security requirements to run your main container with read-only file systems. In such cases, you can’t generate the shared configuration as part of the entry point.

More details on how to configure the initContainer can be found here.

Conclusion

In this article, you have learned what the Kubernetes `must be unique` error code means and how you can resolve it with a few different methods. 

You have also learned about different methods for mounting several configurations to a single-target container in your Kubernetes cluster—using projected volumes, separate mount paths, environment variables, or initialization containers.

Validating the Kubernetes configuration in the CI/CD pipeline would prevent this issue from appearing in production. Be sure to consider using a validation tool to help prevent such misconfigurations and catch issues early on.