Debugging a Kubernetes 503: From Ingress to Pod

The deploy went green, the pods are Running, and the site returns 503 Service Temporarily Unavailable. The ingress controller is telling you something simple: it had nowhere to send the request. The hard part is finding which link in the chain is broken.

A request travels Ingress → Service → EndpointSlice → Pod. Each hop is matched by a name, a label or a port, and a typo in any of them breaks the chain without an error at apply time. Walk the chain in order and check each link.

Hop 1: does the Ingress point at a Service that exists?

kubectl -n shop describe ingress web

Look at the Rules section. Every backend should show a Service name, a port and a list of pod IPs:

Rules:
  Host           Path  Backends
  ----           ----  --------
  shop.example   /     web:80 (10.0.3.17:8080,10.0.4.22:8080)
                 /api  api:80 (<error: endpoints "api" not found>)

<error: endpoints "api" not found> means the Service api does not exist in the Ingress’s namespace. An Ingress can only route to Services in its own namespace, so check the name and the namespace together.

Hop 2: does the Service port match?

The Ingress backend names a Service port, either by number or by name:

backend:
  service:
    name: web
    port:
      number: 80

That has to be a port on the Service, not the container port. A Service that only exposes 8080 will not answer a backend that asks for 80.

kubectl -n shop get svc web -o jsonpath='{.spec.ports}'

Hop 3: does the Service have endpoints?

This is where most 503s are found:

kubectl -n shop get endpointslices -l kubernetes.io/service-name=web

If the ENDPOINTS column is empty or <unset>, the Service matched no ready pods. There are two possible reasons, and they need different fixes.

The selector matches no pods. Compare the Service selector with the pod labels:

kubectl -n shop get svc web -o jsonpath='{.spec.selector}'; echo
kubectl -n shop get pods --show-labels

A selector of app: web and a pod label of app.kubernetes.io/name: web do not match. Neither do app: web and app: Web. Selectors are exact and case-sensitive.

The pods exist but are not ready. A pod only joins the endpoints when its readiness probe passes. kubectl get pods shows 0/1 in the READY column, and describe shows why:

kubectl -n shop describe pod -l app.kubernetes.io/name=web | grep -A3 Readiness

A probe pointed at /healthz on an app that serves /health, or at the wrong port, keeps every pod out of the Service forever. The pods themselves look healthy and never restart, which is why this one is easy to miss.

Hop 4: does targetPort reach the container?

The Service forwards to targetPort. If that doesn’t match the port the process actually listens on, you get a 502 or a connection reset instead of a 503:

# Service
ports:
  - port: 80
    targetPort: http     # must match a named containerPort…
# Pod
ports:
  - name: http
    containerPort: 8080  # …and the app must listen here

Test from inside the cluster to rule out everything above the Service:

kubectl -n shop run curl --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sv http://web.shop.svc.cluster.local/

If that works but the Ingress does not, the problem is in the Ingress or the controller. If it fails too, the problem is in the Service or the pods.

Hop 5: is a NetworkPolicy dropping the traffic?

A default-deny NetworkPolicy in the namespace blocks the ingress controller unless something allows it explicitly. The symptom is a timeout followed by a 504, or a 502 on some controllers. Allow traffic from the controller’s namespace:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: ingress-nginx

Catch it before you apply

Hops 1 to 4 are all mismatches between objects that you can see in the manifests themselves. You don’t need a cluster to find them. Paste the Ingress, Services and Deployments into the Kubernetes YAML validator. It draws the Ingress → Service → workload graph and shows any missing Service as a dashed box. It also flags Services whose selector matches no pod template and Deployments whose selector doesn’t match their own labels, along with probes and security settings worth fixing.

If you are starting a new service, the Kubernetes manifest generator avoids most of these mistakes. It uses one label for the selector, the pod template and the Service, and named ports that the Service and probes refer to.

The checklist

1. describe ingress: is each backend a real Service in the same namespace?
2. Is the backend port a Service port, not a container port?
3. get endpointslices: does the Service have ready endpoints?
4. Do the selector and the pod labels match exactly?
5. Does the readiness probe pass?
6. Does targetPort reach the port the app listens on?
7. Is a NetworkPolicy blocking the ingress controller?