ImagePullBackOff in Kubernetes: 12 Causes and How to Fix Each

ImagePullBackOff is not the error. It is the kubelet telling you that it already failed to pull your container image several times and is now waiting before it tries again, so the real message is one line above it in the pod events.

The pull happens on the node, using the node's DNS, the node's network path, and the credentials you handed the kubelet - not from the machine you ran kubectl apply on. That single fact explains most of the confusing cases: the image pulls fine on your laptop and still fails in the cluster.

This page gives you the one command that reveals the actual failure, the fastest fix for the common case, then 12 distinct causes - wrong tag, missing or misplaced pull secret, Docker Hub rate limits, expiring registry tokens, architecture mismatch, blocked egress, private CA, full disk - each with a reproduction, the corrected lines, and a way to confirm it is really fixed.

Error	STATUS: ImagePullBackOff, preceded in the pod events by: Warning Failed kubelet Failed to pull image "ghcr.io/acme/api:1.4.2": rpc error: code = NotFound desc = failed to pull and unpack image ... failed to resolve reference ...: not found / Normal BackOff kubelet Back-off pulling image "ghcr.io/acme/api:1.4.2"
Where it happens	Kubernetes - any version, any distribution (EKS, GKE, AKS, kind, minikube, k3s); produced by the kubelet and its container runtime (containerd, CRI-O) on the node, not by your application
What it means	The kubelet could not get the image bytes onto the node and has backed off before retrying, so the container never started.

The Fast Fix

Read the real error first - ImagePullBackOff itself tells you nothing:

kubectl describe pod <pod> | tail -20

If the line says not found or repository does not exist, the reference is wrong: fix the tag or add the registry host and repository path. If it says 401 Unauthorized or pull access denied, the node has no credentials for that registry - create a pull secret in the pod's own namespace and attach it:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=acme-ci \
  --docker-password="$GHCR_TOKEN" \
  -n staging

kubectl -n staging patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

kubectl -n staging delete pod <pod>   # recreate so the pull retries immediately

Anything else (429, x509, no space left on device, no match for platform) maps to one of the causes below.

What Is Actually Causing It

Jump to your case

• 1. The tag does not exist in the registry
• 2. The image only exists on your laptop, never pushed
• 3. Private registry with no imagePullSecrets at all
• 4. Pull secret lives in a different namespace than the pod
• 5. Secret has the wrong type or the wrong server key
• 6. Docker Hub anonymous pull rate limit (429)
• 7. Short-lived registry token in the secret has expired
• 8. Image has no manifest for the node's architecture
• 9. Missing registry host or wrong repository path
• 10. The node cannot reach the registry (DNS, firewall, proxy)
• 11. Private CA: x509 certificate signed by unknown authority
• 12. The node is out of disk space for the image layers

1. The tag does not exist in the registry

Reproduce it

kubectl run web --image=nginx:1.25-alpnie
kubectl describe pod web | tail -6
# Warning  Failed  kubelet  Failed to pull image "nginx:1.25-alpnie":
#   failed to resolve reference "docker.io/library/nginx:1.25-alpnie": not found
# Normal   BackOff kubelet  Back-off pulling image "nginx:1.25-alpnie"

Why it happens — The repository exists and is public, but the tag does not. The registry answers the manifest request with a 404, containerd reports not found, and the kubelet starts backing off. A typo, a tag that was never pushed, or a tag someone deleted or moved all land here.

The fix

docker manifest inspect nginx:1.25-alpnie   # fails -> the tag really is absent
docker manifest inspect nginx:1.25-alpine    # succeeds -> this is the real tag

kubectl set image pod/web web=nginx:1.25-alpine

Changed: the tag in the image reference, checked against the registry before applying instead of after.

Confirm it worked — kubectl describe pod web | grep -i pulled prints Successfully pulled image, and kubectl get pod web shows Running.

---

2. The image only exists on your laptop, never pushed

Reproduce it

docker build -t myapp:dev .
kubectl run myapp --image=myapp:dev --image-pull-policy=Always
# Failed to pull image "myapp:dev": failed to resolve reference
#   "docker.io/library/myapp:dev": pull access denied, repository does not exist

Why it happens — Your local Docker daemon's image store is invisible to the node's container runtime - they are separate caches, even on a single-node kind or minikube cluster. An unqualified name is also expanded to docker.io/library/myapp, so the node goes looking on Docker Hub.

The fix

# kind
kind load docker-image myapp:dev
# minikube
minikube image load myapp:dev

kubectl run myapp --image=myapp:dev --image-pull-policy=IfNotPresent

Changed: the image is now in the node's runtime store, and the policy no longer forces a remote pull. Note that imagePullPolicy defaults to Always whenever the tag is latest or omitted.

Confirm it worked — docker exec -it kind-control-plane crictl images | grep myapp lists the image on the node.

---

3. Private registry with no imagePullSecrets at all

Reproduce it

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: api
      image: ghcr.io/acme/api:1.4.2   # private repository, no credentials given

Failed to pull image "ghcr.io/acme/api:1.4.2": failed to authorize:
  failed to fetch anonymous token: unexpected status: 401 Unauthorized

Why it happens — The kubelet pulls anonymously unless you give it credentials. A private repository answers the token request with 401, which looks identical to a missing repository from the outside - registries deliberately do not distinguish the two.

The fix

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=acme-ci \
  --docker-password="$GHCR_TOKEN"

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: api
      image: ghcr.io/acme/api:1.4.2

Changed: added a kubernetes.io/dockerconfigjson secret and the imagePullSecrets reference to it.

Confirm it worked — kubectl describe pod api | grep -i pulled shows Successfully pulled; the 401 line no longer appears in new events.

---

4. Pull secret lives in a different namespace than the pod

Reproduce it

kubectl create secret docker-registry regcred -n default \
  --docker-server=ghcr.io --docker-username=acme-ci --docker-password="$GHCR_TOKEN"

kubectl apply -f api.yaml -n staging
kubectl -n staging describe pod api | tail -8
# Warning  Failed  kubelet  Unable to retrieve pull secret staging/regcred for
#   staging/api due to secret "regcred" not found. The image pull may not succeed.
# Warning  Failed  kubelet  Failed to pull image "ghcr.io/acme/api:1.4.2": ... 401 Unauthorized

Why it happens — imagePullSecrets is resolved only within the pod's own namespace - there is no cluster-wide secret. The kubelet logs a warning about the missing secret and then pulls anonymously anyway, so you get a 401 and the misleading impression that the credentials are wrong.

The fix

kubectl create secret docker-registry regcred -n staging \
  --docker-server=ghcr.io --docker-username=acme-ci --docker-password="$GHCR_TOKEN"

# attach once per namespace instead of wiring every pod spec
kubectl -n staging patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Changed: the secret now exists in staging, and the namespace's ServiceAccount injects it into every pod it runs.

Confirm it worked — kubectl -n staging get secret regcred returns the secret, and the Unable to retrieve pull secret warning is gone from kubectl -n staging describe pod api.

---

5. Secret has the wrong type or the wrong server key

Reproduce it

# wrong: a generic secret built from a docker config file
kubectl create secret generic regcred \
  --from-file=.dockerconfigjson=$HOME/.docker/config.json

kubectl get secret regcred -o jsonpath='{.type}'   # Opaque
# pod still gets: 401 Unauthorized

Why it happens — The kubelet reads pull credentials only from secrets of type kubernetes.io/dockerconfigjson (key .dockerconfigjson) or the legacy kubernetes.io/dockercfg. An Opaque secret with the same contents is ignored. The second trap is the server key: --docker-server must be the registry host as it appears in the image reference, not a URL with a repository path - the one exception is Docker Hub, whose legacy value is https://index.docker.io/v1/.

The fix

kubectl delete secret regcred
kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=acme-ci --docker-password="$GHCR_TOKEN"

kubectl get secret regcred -o jsonpath='{.type}'   # kubernetes.io/dockerconfigjson

Changed: create secret docker-registry instead of create secret generic, and the bare host ghcr.io instead of https://ghcr.io/acme.

Confirm it worked — ```bash kubectl get secret regcred -o jsonpath='{.data..dockerconfigjson}'
| base64 -d | jq '.auths | keys' # ["ghcr.io"]

The key must match the host prefix of your image reference exactly.


---




<h3 id="cause-6">6. Docker Hub anonymous pull rate limit (429)</h3>

**Reproduce it**

```text
Failed to pull image "docker.io/library/redis:7": rpc error: code = Unknown
  desc = failed to pull and unpack image "docker.io/library/redis:7":
  failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests -
  Server message: toomanyrequests: You have reached your pull rate limit.

Why it happens — Anonymous Docker Hub pulls are metered per client IP. Every node behind the same NAT gateway shares one budget, so a cluster that autoscales or a CI job that churns pods will exhaust it and every subsequent pull turns into ImagePullBackOff - including images that pulled fine an hour ago.

The fix

kubectl create secret docker-registry dockerhub \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=acme --docker-password="$DOCKERHUB_TOKEN"

kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"dockerhub"}]}'

Changed: pulls are authenticated instead of anonymous, which moves you off the shared per-IP budget. The durable fix is to stop pulling from Docker Hub directly at all - point the images at a pull-through cache or mirror in your own registry.

Confirm it worked — Delete one affected pod and check kubectl describe pod - the 429 line must not reappear. If you run a mirror, confirm the image reference in the manifest now points at the mirror host.

---

More causes (6 remaining)

7. Short-lived registry token in the secret has expired

Reproduce it

TOKEN=$(aws ecr get-login-password --region us-east-1)
kubectl create secret docker-registry ecr-creds \
  --docker-server=111122223333.dkr.ecr.us-east-1.amazonaws.com \
  --docker-username=AWS --docker-password="$TOKEN"

# pulls work now; twelve hours later every new pod:
# Failed to pull image "111122223333.dkr.ecr.us-east-1.amazonaws.com/api:1.4.2":
#   failed to authorize: ... 401 Unauthorized

Why it happens — An ECR authorization token is valid for 12 hours; the equivalent tokens from other cloud registries are also short-lived. Baking one into a Secret works until it expires, then every fresh pull fails while already-running pods keep going - which is why this one usually surfaces as "deployments broke overnight".

The fix

# EKS: no secret at all - the node's credential provider mints tokens per pull
aws iam attach-role-policy \
  --role-name eksNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

kubectl delete secret ecr-creds

Changed: credentials come from the node's identity on every pull instead of a static copy. On a cluster without a built-in provider, refresh the secret from a CronJob well inside the token lifetime.

Confirm it worked — kubectl get pod api -o jsonpath='{.spec.imagePullSecrets}' is empty and the pod still pulls; re-check a new pod the next day rather than only immediately after the change.

---

8. Image has no manifest for the node's architecture

Reproduce it

# built on an Apple Silicon laptop, single-arch arm64
docker build -t ghcr.io/acme/api:1.4.2 . && docker push ghcr.io/acme/api:1.4.2

kubectl describe pod api | tail -4
# Failed to pull image "ghcr.io/acme/api:1.4.2":
#   no match for platform in manifest: not found

Why it happens — The reference and the credentials are fine - the registry simply has no linux/amd64 entry for that tag. The image pulls perfectly on the machine that built it, which is what makes this one waste an afternoon.

The fix

docker buildx build --platform linux/amd64,linux/arm64 \
  -t ghcr.io/acme/api:1.4.2 --push .

Changed: a multi-platform build pushed as a manifest list, instead of a single-arch image for the builder's own CPU.

Confirm it worked — ```bash docker manifest inspect ghcr.io/acme/api:1.4.2 | jq '.manifests[].platform' kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'

Every architecture printed by the second command must appear in the first.


---




<h3 id="cause-9">9. Missing registry host or wrong repository path</h3>

**Reproduce it**

```yaml
      image: acme/api:1.4.2   # intended Google Artifact Registry

Failed to pull image "acme/api:1.4.2": failed to resolve reference
  "docker.io/acme/api:1.4.2": pull access denied, repository does not exist
  or may require authorization

Why it happens — An image reference without a host is expanded to docker.io, so the node asks the wrong registry entirely. The same class of mistake is dropping a path segment: Artifact Registry needs <location>/<project>/<repo>/<image>, and most registries need the org or project prefix.

The fix

      image: us-docker.pkg.dev/acme-prod/containers/api:1.4.2

Changed: fully qualified reference - registry host, project, repository, image, tag. Note the giveaway in the event text: the resolved reference containerd printed is not the one you thought you wrote.

Confirm it worked — Run docker manifest inspect us-docker.pkg.dev/acme-prod/containers/api:1.4.2 with the same credentials; it must return a manifest before you re-apply.

---

10. The node cannot reach the registry (DNS, firewall, proxy)

Reproduce it

Failed to pull image "registry.internal:5000/api:1.4.2": rpc error: code = Unavailable
  desc = failed to resolve reference "registry.internal:5000/api:1.4.2":
  failed to do request: Head "https://registry.internal:5000/v2/api/manifests/1.4.2":
  dial tcp: lookup registry.internal: no such host

Why it happens — The pull is executed by the node's runtime, so it uses the node's resolver, routes and egress rules. A registry that resolves from your VPN or your CI runner can be unreachable from a private subnet, and a cluster behind an HTTP proxy needs that proxy configured for containerd specifically - the pull does not inherit your shell environment.

The fix

# /etc/systemd/system/containerd.service.d/http-proxy.conf
[Service]
Environment="HTTPS_PROXY=http://proxy.internal:3128"
Environment="NO_PROXY=10.0.0.0/8,.svc,.cluster.local,registry.internal"

sudo systemctl daemon-reload && sudo systemctl restart containerd

Changed: the runtime now knows the egress proxy, with internal hosts excluded. If the failure is plain DNS instead, fix the node's resolver or use an address the node can resolve.

Confirm it worked — On the node, sudo crictl pull registry.internal:5000/api:1.4.2 succeeds. Until that command works, nothing you change in the pod spec will help.

---

11. Private CA: x509 certificate signed by unknown authority

Reproduce it

Failed to pull image "registry.internal/api:1.4.2": failed to do request:
  Head "https://registry.internal/v2/api/manifests/1.4.2":
  tls: failed to verify certificate: x509: certificate signed by unknown authority

Why it happens — The registry presents a certificate from an internal or self-signed CA that the node does not trust. Trusting it in your laptop's Docker config changes nothing - the trust store that matters belongs to containerd on the node.

The fix

# /etc/containerd/config.toml - enables the per-host directory layout
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

# /etc/containerd/certs.d/registry.internal/hosts.toml
server = "https://registry.internal"

[host."https://registry.internal"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/registry.internal/ca.crt"

Changed: the CA certificate is installed for that specific registry host, and containerd is told to read the certs.d directory. Restart containerd afterwards.

Confirm it worked — sudo crictl pull registry.internal/api:1.4.2 on the node returns successfully with no TLS error.

---

12. The node is out of disk space for the image layers

Reproduce it

Failed to pull image "ghcr.io/acme/api:1.4.2": failed to pull and unpack image:
  failed to extract layer sha256:9c1b...: write /var/lib/containerd/...:
  no space left on device

kubectl describe node ip-10-0-1-23 | grep -i diskpressure
# DiskPressure   True

Why it happens — Credentials and reference are irrelevant here - the layers cannot be unpacked. Large images, a long-lived node with months of accumulated tags, or noisy container logs fill the filesystem that backs the image store, and every pull on that node fails until space is reclaimed.

The fix

sudo crictl rmi --prune          # drop images no container references
df -h /var/lib/containerd

Changed: reclaimed space on the node. For a lasting fix, grow the disk or tune the kubelet's imageGCHighThresholdPercent / imageGCLowThresholdPercent so garbage collection starts earlier.

Confirm it worked — kubectl describe node ip-10-0-1-23 | grep -i diskpressure reports False, and the pod pulls on its next retry.

## None of Those? Narrow It Down

1. Run kubectl describe pod <pod> | tail -20 and read the line above the BackOff event. The substring decides the branch: not found / repository does not exist = reference problem, 401 / pull access denied = credentials, 429 = rate limit, no match for platform = architecture, x509 = trust, dial tcp / i/o timeout = network, no space left on device = node disk. Everything below only matters if this line is ambiguous.
2. Copy the exact reference containerd printed - not the one in your manifest - and run docker manifest inspect <ref> from a machine with the same credentials. Success means the image exists and your creds are valid, so the problem is node-side (secret wiring, network, trust, disk). Failure means the problem is the reference or the registry permissions themselves.
3. Check that credentials actually reach the pod: kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.imagePullSecrets}', then kubectl -n <ns> get sa <serviceAccountName> -o jsonpath='{.imagePullSecrets}', then kubectl -n <ns> get secret <name>. An empty result or a missing secret in that namespace rules in the namespace/wiring causes and rules out everything else.
4. Inspect the secret's shape: kubectl -n <ns> get secret <name> -o jsonpath='{.type}' must be kubernetes.io/dockerconfigjson, and the decoded auths keys must match the registry host in the image reference. A mismatch here explains a 401 that persists even though the credentials are correct.
5. Compare platforms with nodes: docker manifest inspect <ref> | jq '.manifests[].platform' against kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'. If a node architecture is missing from the manifest, stop - it is the architecture cause, regardless of what else looks suspicious.
6. Get on the affected node and run sudo crictl pull <ref>. Succeeding there while the pod fails means the kubelet is not receiving the secret; a DNS or timeout error means egress or proxy; x509 means trust; no space left on device means disk. This step separates "credentials" from "connectivity" for good.
7. Ask when it last worked. Worked, then started returning 401 after some hours = an expiring registry token. Worked, then 429 under load or after scaling = rate limiting. Never worked for a brand-new tag = reference or push problem. Worked everywhere except on one node = that node's disk, network or trust store.

Why This Error Exists At All

Kubernetes deliberately treats an image reference as a late-bound pointer. Your manifest says ghcr.io/acme/api:1.4.2, and nothing resolves that string until a pod is scheduled and the kubelet on a specific node asks a specific registry for it. The API server never checks whether the image exists; admission does not dial the registry. This is what makes the scheduler fast and the control plane independent of every registry your workloads use - and it is also why a one-character typo passes kubectl apply happily and only fails minutes later, on a node, in an event log you have to go looking for.

The second design decision is the backoff itself. ImagePullBackOff is a distinct status from ErrImagePull because the kubelet refuses to hammer a registry that just said no: each failed attempt waits longer than the last, up to a cap (five minutes in the default kubelet configuration). That protects registries from a thousand nodes retrying in lockstep, but it also means your feedback loop gets slower exactly when you are iterating on a fix. This is why you delete the pod after changing a secret instead of waiting - a fresh pod starts with a fresh backoff.

The third thing to internalise is that "pulling an image" is really three independent questions, and the error text tells you which one failed. Does the name resolve - registry host, repository path, tag, and a manifest for this CPU architecture? Am I authorised - a secret of the right type, in the right namespace, keyed to the right host, with a token that has not expired? Can the node physically complete the transfer - DNS, egress, TLS trust, disk space? Once you read every pull failure in that order, the whole family becomes mechanical: CreateContainerConfigError for a missing non-pull secret, ErrImagePull as the un-backed-off sibling, registry 403s versus 401s. The status at the top of kubectl get pods is never the error; it is the kubelet's summary of a loop that already told you what went wrong.

Stop It From Coming Back

• Deploy by digest, not by tag: image: ghcr.io/acme/api@sha256:.... A digest cannot be moved or deleted out from under a running deployment, and Kustomize (images[].digest) or Flux's image automation can write it for you at build time.
• Add a CI gate between push and apply: docker manifest inspect "$IMAGE_REF" (or crane manifest) against the exact reference the manifest will use. It fails the pipeline on typos, missing pushes and wrong repository paths before anything reaches the cluster.
• Build multi-arch in CI with docker buildx build --platform linux/amd64,linux/arm64 --push and assert in the same job that the manifest platforms cover every architecture reported by kubectl get nodes. This kills the whole Apple-Silicon-to-amd64-nodes class.
• Attach pull secrets to the namespace's ServiceAccount in your namespace bootstrap (Helm chart or Kustomize base) rather than in individual pod specs, so a new namespace cannot be created without credentials.
• Use the platform's kubelet image credential provider (EKS node role, GKE/AKS equivalents) instead of static secrets holding short-lived registry tokens; if you must store a token, refresh it from a CronJob at well under half its lifetime.
• Stop pulling third-party images from Docker Hub directly: mirror them through a pull-through cache (Harbor proxy project, Artifact Registry remote repository, ECR pull-through) and alert on any 429/toomanyrequests appearing in kubelet events.

Related Guides

• CrashLoopBackOff

• npm ERR! code ERESOLVE unable to resolve dependency tree

• TS2345: Argument of type is not assignable to parameter of type

• SettingWithCopyWarning: A value is trying to be set on a copy of a slice

Errors You Will Probably Hit Next

• ErrImagePull
• CreateContainerConfigError: secret "regcred" not found
• CrashLoopBackOff
• ImageInspectError

ImagePullBackOff is a status, not a diagnosis - the usable sentence is always one line above it in kubectl describe pod. Read that line and ask the three questions in order: does the reference resolve, am I authorised, can this node complete the transfer. When the answer is not obvious, reproduce the pull where it actually happens with crictl pull on the node, because anything that works on your laptop proves nothing about the kubelet. And remember to delete the pod after a fix, or you will wait out a five-minute backoff and conclude the fix failed.