Kubernetes Probes Without the Outages

Probes are the cheapest reliability feature in Kubernetes and one of the easiest ways to cause an outage. A missing readiness probe sends traffic to pods that aren’t ready yet. A badly designed liveness probe can restart every pod of a service at the same moment. This post covers what each probe does and the settings that avoid both problems.

Three probes, three different actions

Probe	When it fails, Kubernetes…	Use it to answer
Readiness	removes the pod from Service endpoints. It keeps running.	“Should this pod get traffic right now?”
Liveness	kills the container and restarts it	“Is this process stuck beyond recovery?”
Startup	holds off the other two until it passes, then restarts the container if it never does	“Has the app finished booting?”

Readiness is about traffic. Liveness is about restarts. Mixing up the two causes most probe incidents.

Mistake 1: liveness checks a dependency

livenessProbe:
  httpGet:
    path: /health   # returns 500 when the database is unreachable
    port: http

The database has a 30-second hiccup. Every pod’s /health returns 500, and every liveness probe fails. Kubernetes restarts all the pods at once. They come back cold, the connection pools reconnect together, the database gets slower, and the restarts continue.

Restarting your app doesn’t fix the database. A liveness probe should only fail when this process is broken: a deadlock, a stuck event loop, memory corruption. Keep it trivial, for example a handler that returns 200 if the HTTP server can respond at all.

Dependency checks belong in the readiness probe, if anywhere. Even there, be careful: if every pod depends on the same database, they all go unready together and the Service has no endpoints. Often the better choice is to stay ready and return errors quickly, so callers can retry or degrade gracefully.

Mistake 2: the same endpoint and timings for both probes

When readiness and liveness share an endpoint and thresholds, they fail at the same moment. The pod is taken out of rotation and restarted at once, before it has a chance to recover. Make liveness slower to give up than readiness:

readinessProbe:
  httpGet: { path: /ready, port: http }
  periodSeconds: 5
  failureThreshold: 3        # out of rotation after ~15 s
livenessProbe:
  httpGet: { path: /live, port: http }
  periodSeconds: 10
  failureThreshold: 6        # restarted only after ~60 s

Mistake 3: the default one-second timeout

timeoutSeconds defaults to 1. Under load, a garbage-collection pause or a busy thread pool can push a health endpoint past one second. The probe then fails exactly when the pod is busiest, which is the worst time to remove it or restart it. Two to three seconds is a more realistic starting point. Make sure the health handler doesn’t share a queue with slow requests.

Mistake 4: initialDelaySeconds instead of a startup probe

A JVM app that sometimes takes 90 seconds to boot tempts you to set initialDelaySeconds: 120 on the liveness probe. That delays failure detection for every start, including fast ones, and it still fails on the rare boot that takes 130 seconds. A startup probe handles slow boots properly:

startupProbe:
  httpGet: { path: /live, port: http }
  periodSeconds: 5
  failureThreshold: 36       # allows up to 180 s to boot

Liveness and readiness don’t run until the startup probe passes once. After that, the liveness probe can stay strict.

The budget arithmetic

The time it takes a probe to act is roughly periodSeconds × failureThreshold, plus up to timeoutSeconds for each attempt. Write the budget down for each probe and check that it makes sense:

• Readiness: how long are you willing to send traffic to a broken pod? Usually 10–20 s.
• Liveness: how long can a stuck process stay stuck before a restart is clearly better? Usually 30–60 s, and always longer than readiness.
• Startup: the worst boot time you have seen, plus some margin.

A safe starting point

The Kubernetes manifest generator creates a Deployment with a readiness probe that gates traffic and a slower liveness probe on a named port. It also adds resource requests, a memory limit and a restricted security context. Use it as a baseline and adjust the paths and timings to your app.

When you edit existing manifests, run them through the Kubernetes YAML validator. It flags workloads with no readiness or liveness probe, together with the other reliability gaps that tend to show up in the same files: missing requests, single replicas and no PodDisruptionBudget.

Summary

• Readiness controls traffic, liveness controls restarts, and startup protects slow boots.
• Liveness checks only the process itself, never a shared dependency.
• Make liveness slower to give up than readiness.
• Raise timeoutSeconds above the default of 1 s.
• Use a startup probe instead of a long initialDelaySeconds.