Kubernetes Deployment Troubleshooting Checklist: Diagnose Pods, Services, and Ingress Step by Step

Overview

A Kubernetes application can appear to be “down” for several different reasons: the scheduler may not have placed a pod, the container may be restarting, the image may be unavailable, configuration may be missing, or traffic may be failing after it reaches the cluster. Treating every symptom as an application-code problem usually wastes time.

The most reliable approach is to move from the outside in:

1. Confirm you are connected to the intended cluster and namespace.
2. Check workload and pod status.
3. Inspect events and container logs.
4. Verify configuration, secrets, probes, and resources.
5. Trace traffic through the Service and, if applicable, the Ingress or Gateway.

Run commands against a specific namespace whenever possible. A useful starting point is:

kubectl config current-context
kubectl get namespaces
kubectl -n <namespace> get deploy,rs,pods -o wide
kubectl -n <namespace> get events --sort-by=.lastTimestamp

Events are often the shortest path to an explanation, but they are not a complete history. Capture relevant output while the failure is occurring, then compare it with the Deployment, Pod, Service, and Ingress specifications.

Checklist by scenario

1. Pods remain Pending

A Pending pod has not reached a running container state. First identify the pod and ask the scheduler for its explanation:

kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pod-name>
kubectl get nodes
kubectl describe node <node-name>

In the pod description, inspect the Events section. Common checks include:

• Resource availability: Compare the pod's CPU and memory requests with allocatable capacity on eligible nodes. A request that cannot fit prevents scheduling even if current usage appears low.
• Node selectors and affinity: Confirm that labels, required affinity rules, and topology constraints match the nodes you expect to use.
• Taints and tolerations: A tainted node will reject pods without the corresponding toleration.
• Persistent volumes: Check whether a claim is bound and whether its access mode and storage constraints are compatible with the workload.
• Namespace quotas: A quota or limit range can prevent a new pod from being admitted or scheduled.

Do not solve a scheduling problem by deleting pods at random. Confirm which constraint is blocking placement and change the smallest relevant configuration.

2. Pods are CrashLoopBackOff or restarting

CrashLoopBackOff describes repeated container failures with increasing delays; it is a symptom rather than a root cause. Check current and previous logs, then inspect the pod:

kubectl -n <namespace> logs <pod-name> -c <container-name>
kubectl -n <namespace> logs <pod-name> -c <container-name> --previous
kubectl -n <namespace> describe pod <pod-name>
kubectl -n <namespace> get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState}'

Look for an incorrect command or argument, a missing environment variable, an application binding only to an unexpected address, and a dependency that is unavailable at startup. If the container was terminated with an out-of-memory indication, compare its memory limit with observed workload needs and review application behavior before simply increasing the limit.

For multi-container pods, specify -c. Sidecars, init containers, and service-mesh components can fail independently of the main application.

3. ImagePullBackOff or ErrImagePull

These states indicate that the node could not retrieve or start the referenced image. Inspect the exact image name and the related event:

kubectl -n <namespace> describe pod <pod-name>
kubectl -n <namespace> get deploy <deployment-name> -o jsonpath='{.spec.template.spec.containers[*].image}'

Verify the registry hostname, repository path, tag or digest, and spelling. If the registry is private, check that the pod's imagePullSecrets reference exists in the same namespace and that the credential can access the repository. Also verify that the node can reach the registry through its network path and that the image architecture matches the node environment.

Avoid relying on a mutable tag while diagnosing a release. Record the intended image reference and compare it with the Deployment's pod template.

4. Configuration or Secret errors

Configuration failures commonly appear as failed mounts, missing keys, rejected environment values, or application startup errors. Review the workload definition and referenced objects:

kubectl -n <namespace> get configmap
kubectl -n <namespace> get secret
kubectl -n <namespace> describe configmap <configmap-name>
kubectl -n <namespace> describe pod <pod-name>

Check spelling, namespace boundaries, key names, volume mount paths, and whether an optional reference was intended to be required. Do not print secret values into shared terminals or incident channels. For a broader treatment of secret storage and delivery patterns, see the secrets management tools comparison.

5. Readiness or liveness probes fail

A liveness probe can restart a container, while a readiness probe normally controls whether the pod receives Service traffic. Inspect probe definitions and recent events:

kubectl -n <namespace> describe pod <pod-name>
kubectl -n <namespace> get deploy <deployment-name> -o yaml

Confirm the path, port, scheme, expected response, and probe timing. Test the endpoint from inside the pod when possible:

kubectl -n <namespace> exec -it <pod-name> -c <container-name> -- sh
# From inside the container, use an available HTTP client to test localhost:<port>

Make sure the application is actually listening on the port named by the probe and that startup time is covered by appropriate startup behavior or timing. Avoid weakening probes until they always pass; that can hide a real failure.

6. Service has no endpoints or traffic fails internally

Start by checking the Service selector and the endpoints it produces:

kubectl -n <namespace> get svc <service-name> -o yaml
kubectl -n <namespace> get endpoints <service-name>
kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service-name>
kubectl -n <namespace> get pods --show-labels

If there are no endpoints, compare the Service selector with pod labels exactly. Then check whether pods are Ready; a Service may omit unready pods depending on its configuration and the clients' expectations. If endpoints exist, verify the Service port, target port, and container listening port. Test DNS and connectivity from a temporary diagnostic pod or another existing workload in the same network context.

Network policies, service-mesh rules, and cloud load-balancer behavior can add another layer. If your platform uses a mesh, compare the failure with its sidecar and policy logs rather than assuming the Service is the only component involved. The service mesh comparison provides useful context for where that extra routing layer may sit.

7. Ingress returns 404, 502, or has no external response

Trace external routing in order: Ingress or Gateway class, host and path match, backend Service, and endpoints.

kubectl -n <namespace> get ingress
kubectl -n <namespace> describe ingress <ingress-name>
kubectl get ingressclass
kubectl -n <namespace> get svc <service-name>
kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service-name>

Confirm that the request's host header matches the configured host, the path rule uses the intended matching behavior, and the backend port exists. A 404 can indicate a rule mismatch or an application route mismatch. A 502 or similar upstream error often warrants checking Service endpoints, target ports, readiness, and the ingress controller's logs. Also verify DNS, TLS references, and whether the controller watches the namespace and resource class you are using.

If you are planning a routing change, review Kubernetes Ingress versus Gateway API before mixing resource models during an incident.

What to double-check

• Namespace: Every object reference is namespace-scoped unless explicitly noted. A correctly named Secret in another namespace does not satisfy a pod reference.
• Rendered manifests: Inspect what was actually applied, not only the source template. Compare kubectl get ... -o yaml with the deployment artifact.
• Rollout state: Use kubectl rollout status deployment/<name> and kubectl rollout history deployment/<name> to distinguish a new failure from an older unhealthy ReplicaSet.
• Labels and selectors: A typo can disconnect a Service from healthy pods or cause a Deployment selector mismatch.
• Ports: Check container ports, named ports, Service ports, target ports, probe ports, and ingress backend ports as separate values.
• Resource behavior: Requests affect scheduling; limits affect runtime behavior. Review both rather than treating them as interchangeable.
• Recent changes: Identify the last image, configuration, policy, node, or routing change. Correlation is not proof, but it narrows the investigation.

Common mistakes

Looking only at application logs. Scheduler events, admission errors, image-pull messages, and probe failures may occur before the application writes anything.

Debugging the wrong cluster. Always check the current context and namespace before modifying resources. Use an explicit context and namespace for sensitive commands.

Deleting the evidence. Removing a pod can erase useful state and events. Capture describe, logs, status, and recent events first.

Changing several variables at once. A broad restart, image change, resource change, and probe change makes the result difficult to interpret. Make one controlled change, observe, and record it.

Bypassing readiness. Sending traffic to containers before they are ready can turn a partial deployment into a wider incident.

Exposing sensitive output. Treat logs, environment dumps, manifests, and support bundles as potentially sensitive. Redact tokens, credentials, and private endpoints before sharing them.

When to revisit

Keep this checklist with your deployment runbook and revisit it whenever the workflow or its inputs change. Review it before seasonal planning or a planned release cycle, especially if you are changing cluster capacity, namespaces, registries, ingress controllers, service meshes, or CI/CD tooling.

Update the commands and examples when your Kubernetes distribution, access model, observability stack, or routing approach changes. A troubleshooting guide is most useful when it reflects the commands engineers can actually run and the signals they can actually see.

For the next incident, use this short sequence:

1. Freeze unrelated changes and record the time, context, namespace, and user impact.
2. Run get, describe, events, and logs before restarting anything.
3. Classify the failure as scheduling, startup, configuration, readiness, Service, or ingress-related.
4. Test one hypothesis with the smallest reversible change.
5. Record the root cause, the confirming signal, and the preventive follow-up in the team runbook.

That repeatable loop turns Kubernetes troubleshooting from trial and error into a documented diagnostic process.