CrashLoopBackOff in Kubernetes: 10 Causes and Fixes

Your pod is stuck in CrashLoopBackOff and the restart counter keeps climbing. CrashLoopBackOff is not itself an error — it is the kubelet telling you that your container started, exited, and got restarted, repeatedly, and that the kubelet is now waiting longer between each attempt.

That distinction matters because it means the real error is somewhere else: in the container's exit code, in its last log lines, or in the events attached to the pod. The status is a symptom. Your job is to find the thing that made the process die.

This page walks through the two commands that surface the actual failure, then ten causes ordered by how often they bite in production — each with a manifest or command that reproduces it, the fix, and a way to confirm the fix took.

Error	CrashLoopBackOff
Where it happens	Kubernetes — any distribution (EKS, GKE, AKS, k3s, minikube, kind); reported by the kubelet in kubectl get pods STATUS, any container runtime
What it means	The container's main process keeps exiting shortly after start, so the kubelet is restarting it with an exponentially growing back-off delay (10s, 20s, 40s … capped at 5 minutes).

The Fast Fix

There is no single fix — CrashLoopBackOff is a restart loop, not a root cause. But there are two commands that find the root cause in nearly every case:

# 1. Logs from the run that already crashed (not the one starting now)
kubectl logs <pod> -c <container> --previous

# 2. Exit code + reason from the last termination, plus recent events
kubectl describe pod <pod>

Read Last State: Terminated in the describe output. Exit code 137 means OOMKilled (raise resources.limits.memory). Exit code 1 or 2 means your app threw — the answer is in --previous logs. Exit code 127 means the command in command:/args: does not exist in the image.

If --previous prints nothing, the process died before writing a line: check the command path and the failing probe first.

What Is Actually Causing It

Jump to your case

• 1. The application itself throws on startup
• 2. Container exceeded its memory limit (OOMKilled, exit 137)
• 3. Liveness probe fails before the app finishes booting
• 4. The command or entrypoint does not exist in the image (exit 127)
• 5. Missing ConfigMap key or Secret the app reads at startup
• 6. A dependency is unreachable and the app exits instead of retrying
• 7. Non-root user cannot write to a mounted volume or path
• 8. The main process runs in the background and PID 1 exits (exit 0)
• 9. Image is built for the wrong CPU architecture
• 10. A Job or CronJob pod uses restartPolicy: Always
• 11. Read-only root filesystem blocks a temp or cache write

1. The application itself throws on startup

Reproduce it

kubectl run boom --image=python:3.12-alpine --restart=Never -- \
  python -c "import os; raise RuntimeError('DATABASE_URL missing')"
# then, as a Deployment, this loops:
kubectl logs boom --previous
# RuntimeError: DATABASE_URL missing

Why it happens — The container ran your entrypoint, the process raised, and the runtime exited with a non-zero code. Kubernetes' default restartPolicy: Always on Deployment pods restarts it, it throws again, and after a few rounds the kubelet inserts back-off delay — which is what you see as CrashLoopBackOff.

The fix

# Fix the actual exception. To inspect without the loop, run a
# debug pod with the same image and a shell as PID 1:
kubectl run debug --image=<your-image> --restart=Never \
  --command -- sleep 3600
kubectl exec -it debug -- sh
# now run your real entrypoint by hand and read the full trace

What changed: you replaced the crashing entrypoint with sleep so the container stays up long enough to debug interactively.

Confirm it worked — kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}' stops incrementing over 2–3 minutes, and STATUS reads Running.

---

2. Container exceeded its memory limit (OOMKilled, exit 137)

Reproduce it

resources:
  limits:
    memory: "64Mi"   # JVM/Node heap needs far more

kubectl describe pod <pod> | grep -A3 "Last State"
# Last State: Terminated
#   Reason: OOMKilled
#   Exit Code: 137

Why it happens — The cgroup memory limit was hit, so the kernel OOM killer terminated PID 1 with SIGKILL. Exit code 137 is 128 + 9 (SIGKILL). The app never gets a chance to log anything, which is why --previous is usually empty here.

The fix

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"   # raised above real peak usage

What changed: the limit now exceeds actual peak RSS. For JVM containers also set -XX:MaxRAMPercentage=75 so the heap sizes itself from the cgroup limit instead of the node's total RAM.

Confirm it worked — kubectl describe pod <pod> no longer shows Reason: OOMKilled under Last State. Watch live usage with kubectl top pod <pod> and confirm it stays below the new limit.

---

3. Liveness probe fails before the app finishes booting

Reproduce it

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 1     # app needs ~30s to boot
  periodSeconds: 5
  failureThreshold: 3

kubectl describe pod <pod> | grep Unhealthy
# Warning  Unhealthy  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": connection refused

Why it happens — The kubelet starts probing after 1 second, the server is not listening yet, three failures accumulate, and the kubelet kills the container. It restarts, boots slowly again, and gets killed again — a loop the app can never win.

The fix

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30        # allows up to 150s to boot
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3

What changed: a startupProbe now gates the liveness probe — liveness does not run until startup succeeds, so slow boots are no longer fatal.

Confirm it worked — kubectl get events --field-selector reason=Unhealthy returns nothing for this pod, and the restart count freezes after the first successful start.

---

4. The command or entrypoint does not exist in the image (exit 127)

Reproduce it

containers:
  - name: app
    image: alpine:3.20
    command: ["/usr/local/bin/myserver"]   # not in this image

kubectl describe pod <pod> | grep -A2 "Last State"
# Exit Code: 127

Why it happens — The runtime could not exec the binary — wrong path, or the binary was built for a different libc (a glibc binary on Alpine's musl fails the same way). Shell exit code 127 means "command not found", and PID 1 dying immediately gives you a fast, empty-log loop.

The fix

containers:
  - name: app
    image: alpine:3.20
    command: ["/app/myserver"]   # path verified inside the image

# Confirm the path before deploying:
docker run --rm --entrypoint ls alpine:3.20 -l /app

What changed: the command points at a path that actually exists in the image layer.

Confirm it worked — kubectl logs <pod> prints your app's own first line instead of nothing, and the exit-code 127 entry disappears from describe.

---

5. Missing ConfigMap key or Secret the app reads at startup

Reproduce it

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: url        # key is actually named "URL"

kubectl logs <pod> --previous
# panic: DATABASE_URL is empty

Why it happens — When optional is not set, a missing Secret blocks the pod from starting at all (you get CreateContainerConfigError). But a Secret that exists with the wrong key — or one injected as an empty string — starts the container and lets the app crash on its own validation. That path lands you in CrashLoopBackOff.

The fix

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: URL        # matches the actual key

kubectl get secret db-creds -o jsonpath='{.data}' | tr ',' '\n'

What changed: the key now matches the real key name in the Secret — key names are case-sensitive.

Confirm it worked — kubectl exec <pod> -- printenv DATABASE_URL prints a non-empty value, and the app's startup log line appears.

---

6. A dependency is unreachable and the app exits instead of retrying

Reproduce it

kubectl logs <pod> --previous
# Error: dial tcp: lookup postgres.default.svc.cluster.local: no such host
# exit status 1

Why it happens — The app resolved a Service name that does not exist yet (wrong namespace, Service not created, or the DB pod itself is down), then treated the connection failure as fatal. Crash-on-boot is a reasonable design, but under Kubernetes it turns a transient dependency gap into a restart loop.

The fix

# 1. Confirm the Service name and namespace from inside the cluster
kubectl run netcheck --rm -it --image=busybox:1.36 --restart=Never -- \
  nslookup postgres.default.svc.cluster.local

# 2. Gate startup on the dependency instead of crashing on it
initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z postgres 5432; do sleep 2; done']

What changed: an initContainer blocks until the dependency answers, so the main container starts only once the DB is reachable. InitContainer retries do not count toward the app's restart back-off.

Confirm it worked — kubectl get pod <pod> shows Init:0/1 briefly, then Running. kubectl logs <pod> -c wait-for-db exits cleanly.

---

More causes (5 remaining)

7. Non-root user cannot write to a mounted volume or path

Reproduce it

securityContext:
  runAsUser: 1000
volumeMounts:
  - name: data
    mountPath: /var/lib/app

kubectl logs <pod> --previous
# PermissionError: [Errno 13] Permission denied: '/var/lib/app/app.db'

Why it happens — The volume was created owned by root, the container runs as UID 1000, and the first write fails. The app exits, and the kubelet restarts it into exactly the same filesystem state — a loop that never self-heals.

The fix

securityContext:
  runAsUser: 1000
  fsGroup: 1000     # kubelet chowns the volume to this GID
volumeMounts:
  - name: data
    mountPath: /var/lib/app

What changed: fsGroup makes the kubelet set group ownership on the volume at mount time, so the non-root user can write. Note fsGroup applies to volume types that support ownership management — for hostPath, fix the permissions on the node instead.

Confirm it worked — kubectl exec <pod> -- touch /var/lib/app/.probe succeeds, and kubectl exec <pod> -- ls -ld /var/lib/app shows group 1000.

---

8. The main process runs in the background and PID 1 exits (exit 0)

Reproduce it

command: ["sh", "-c", "nginx & echo started"]

kubectl describe pod <pod> | grep -A2 "Last State"
# Reason: Completed
#   Exit Code: 0

Why it happens — The shell backgrounded nginx, printed a line, and exited 0. Kubernetes only tracks PID 1 — when it exits, the container is done, regardless of what else was running. With restartPolicy: Always, even a clean exit 0 gets restarted, and the loop starts.

The fix

command: ["nginx", "-g", "daemon off;"]

What changed: the server now runs in the foreground as PID 1. Same idea elsewhere: httpd -D FOREGROUND, postgres (not pg_ctl start), no trailing &, no -d/--daemon flags.

Confirm it worked — kubectl describe pod <pod> shows no Reason: Completed under Last State, and kubectl exec <pod> -- ps -o pid,comm lists your server at PID 1.

---

9. Image is built for the wrong CPU architecture

Reproduce it

kubectl logs <pod> --previous
# exec /app/server: exec format error

Why it happens — You built on an arm64 machine (Apple Silicon) and pushed a single-arch image, but the node is amd64. The kernel cannot exec the binary, the container dies instantly, and the loop starts with no application logs at all.

The fix

# Build for the node's architecture (or both)
docker buildx build --platform linux/amd64,linux/arm64 \
  -t registry.example.com/app:1.4.0 --push .

# Confirm what you actually pushed
docker buildx imagetools inspect registry.example.com/app:1.4.0

What changed: the tag now resolves to a manifest list containing an image for the node's architecture.

Confirm it worked — kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}' matches one of the platforms listed by imagetools inspect, and kubectl logs shows app output instead of exec format error.

---

10. A Job or CronJob pod uses restartPolicy: Always

Reproduce it

spec:
  template:
    spec:
      restartPolicy: Always   # invalid for Job; on a bare Pod it loops
      containers:
        - name: migrate
          image: migrate:1.0
          command: ["./migrate", "up"]

Why it happens — The container does its work and exits 0, which is correct for a batch task. But restartPolicy: Always tells the kubelet to run it again — so a successful migration re-runs on a loop and shows up as CrashLoopBackOff. (The API server rejects Always on a Job outright; the loop happens on bare pods and on Deployments misused for batch work.)

The fix

apiVersion: batch/v1
kind: Job
spec:
  backoffLimit: 4
  template:
    spec:
      restartPolicy: OnFailure   # only restarts on non-zero exit
      containers:
        - name: migrate
          image: migrate:1.0
          command: ["./migrate", "up"]

What changed: the workload is a Job with restartPolicy: OnFailure, so a clean exit is treated as success instead of something to retry.

Confirm it worked — kubectl get job <job> shows COMPLETIONS 1/1, and the pod settles in Completed rather than restarting.

---

11. Read-only root filesystem blocks a temp or cache write

Reproduce it

securityContext:
  readOnlyRootFilesystem: true

kubectl logs <pod> --previous
# OSError: [Errno 30] Read-only file system: '/tmp/cache.lock'

Why it happens — Hardening the container is correct, but many runtimes write to /tmp on startup (nginx to /var/cache/nginx, JVM to /tmp/hsperfdata). With no writable mount there, the first write fails and the process exits.

The fix

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp
volumes:
  - name: tmp
    emptyDir: {}

What changed: an emptyDir provides a writable /tmp while the rest of the root filesystem stays read-only.

Confirm it worked — kubectl exec <pod> -- touch /tmp/x succeeds while kubectl exec <pod> -- touch /x still fails with a read-only error — hardening intact, app running.

## None of Those? Narrow It Down

1. Run kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.exitCode}{"\n"}'. 137 points at OOMKilled or an external SIGKILL; 127 at a missing command; 1 or 2 at an application-level throw; 0 at a process that exits cleanly and should not be restarted at all.
2. Run kubectl logs <pod> -c <container> --previous. If it prints a stack trace, the cause is in your application code and you can stop here. If it prints nothing, the process died before logging — that rules in OOMKill, exec failure, and wrong architecture, and rules out most app-level bugs.
3. Run kubectl describe pod <pod> and read the Events block. Unhealthy ... Liveness probe failed means the probe is killing a healthy-but-slow app; OOMKilled confirms the memory limit; Back-off restarting failed container alone means the events carry no extra signal and you need the logs.
4. Check whether the container has a command/args override in the manifest. If it does, exec the same image with --command -- sleep 3600 and run that command by hand — this separates a broken entrypoint from a broken application.
5. Temporarily remove the livenessProbe (keep readiness) and redeploy. If the pod now stays up but never becomes Ready, the app is genuinely slow or unhealthy; if it crashes anyway, the probe was never the cause.
6. Compare kubectl top pod <pod> against resources.limits.memory during the first 60 seconds. Usage climbing to the limit before the crash confirms memory; flat low usage rules it out.
7. Run the exact image outside Kubernetes with the same env vars: docker run --rm --env-file ./env.list <image>. Failing there too means the problem is the image or config, not the cluster — which rules out probes, volumes, RBAC, and networking in one step.
8. If the pod only fails on some nodes, run kubectl get pod <pod> -o wide and compare the node's architecture and available memory against a node where it works. A node-specific failure points at architecture mismatch, node pressure, or a missing hostPath.

Why This Error Exists At All

Kubernetes is a level-triggered system: it does not execute a startup script, it continuously drives the cluster from its observed state toward the declared state. Your Deployment declares "one running container of this image." When the container exits, observed state no longer matches, so the kubelet does the only thing that reduces the gap — it starts the container again.

That loop needs a brake. A container that fails instantly would otherwise be restarted thousands of times a minute, burning CPU on image pulls and cgroup setup and drowning the node in events. So the kubelet applies exponential back-off: 10 seconds, then 20, 40, 80, doubling up to a 5-minute cap, and it resets the delay once the container has stayed up for a while. CrashLoopBackOff is the name of the state where that brake is engaged. It is a rate limiter's status, not a diagnosis.

This is why the standard debugging move — read the error message — fails you here. There is no error message. CrashLoopBackOff tells you the loop is happening and nothing about why, because the kubelet genuinely does not know why; it only sees a process that exited. The information you need lives one layer down, in the container's exit code and the stdout of the run that already ended. That is the whole reason --previous exists: by the time you type the command, the crashed container has been replaced, and the logs of the run that failed would otherwise be gone.

Recognise the family. ImagePullBackOff is the same brake applied to image pulls. CreateContainerConfigError is the kubelet refusing to start at all because a referenced ConfigMap or Secret is missing. Any Kubernetes status ending in BackOff means "I tried, it failed, I am waiting before trying again" — always look for the underlying failure, never at the back-off itself.

Stop It From Coming Back

• Set a startupProbe on every service that takes more than a few seconds to boot, and keep livenessProbe narrow — it should check that the process is wedged, not that its dependencies are up. A liveness probe that pings the database turns a DB blip into a restart storm.
• Set both requests and limits for memory on every container, and derive the limit from measured peak RSS (kubectl top pod under load) rather than a guess. For JVM workloads add -XX:MaxRAMPercentage; for Node add --max-old-space-size — otherwise the runtime sizes its heap from the node's RAM and ignores the cgroup limit.
• Validate manifests in CI with kubeconform or kubectl apply --dry-run=server, which catches missing ConfigMap and Secret references and invalid restartPolicy values before they reach the cluster.
• Make the app fail loudly and early on missing configuration — validate every required env var at startup and print the variable's name. It does not stop the crash loop, but it puts the answer in kubectl logs --previous on the first look.
• Build multi-arch images with docker buildx build --platform linux/amd64,linux/arm64 in CI, or pin the build platform to the node architecture. This removes exec format error as a possibility entirely.
• Add a smoke stage to CI that runs the built image with production-shaped env vars and asserts it stays up for 30 seconds. Most crash loops are reproducible with docker run alone and never need to reach a cluster.

Errors You Will Probably Hit Next

• ImagePullBackOff / ErrImagePull — the kubelet cannot fetch the image (wrong tag, private registry, missing imagePullSecret).
• CreateContainerConfigError — a referenced ConfigMap or Secret does not exist, so the container is never created.
• OOMKilled — the specific termination reason behind exit code 137, visible in Last Stateinsidekubectl describe pod.
• Error: exec format error — the image binary was built for a different CPU architecture than the node.

Treat CrashLoopBackOff as a pointer, never as the bug. Two commands resolve nearly every occurrence: kubectl logs --previous for what the app said before it died, and kubectl describe pod for the exit code and events when it said nothing. Learn the three exit codes that carry the most signal — 137 is memory, 127 is a missing command, 0 is a process that was never meant to keep running — and you will skip straight to the cause instead of restarting the deployment and hoping.