Kaushik's Notes

Maintenance & Troubleshooting

6 min read

Overview

  • Chapter 12 of the Kubernetes Cookbook: kubectl autocompletion, parsing status, debugging Pods, cluster snapshots, adding nodes, drain/cordon, and etcd. (Recipes 12.2 and 12.3 are service tricks and live in Services & Ingress.)
  • Plus the modern debugging toolkit the book predates: kubectl debug, ephemeral containers and kubectl events.
  • Back to the hub: Kubernetes Cookbook · Related: Monitoring & Logging, Learning kubectl

Recipes

Recipe 12.1 — kubectl Autocompletion

  • Problem: Typing kubectl get deployment my-long-name is tedious.
  • Solution (zsh, the macOS default):
# ~/.zshrc
source <(kubectl completion zsh)
alias k=kubectl
compdef k=kubectl
  • Gotchas: Bash needs the bash-completion package first. Completion also completes resource names from the live cluster.

Source: K8s Cookbook, Recipe 12.1

Recipe 12.4 — Parse Resource Status (for scripts and CI)

  • Problem: React to an object’s state in a script or pipeline.
  • Solution:
kubectl get pod jump -o jsonpath='{.status.qosClass}'           # Guaranteed | Burstable | BestEffort
kubectl get pod jump -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}'
kubectl get deploy web -o json | jq '.status'
 
# Better than polling: block until a condition is true
kubectl wait --for=condition=Available deploy/web --timeout=120s
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/jump
kubectl rollout status deploy/web --timeout=2m                  # non-zero exit if it fails → fails the CI step
  • Discussion: Built-in JSONPath or go-template output means you don’t need jq, but jq is nicer for complex queries.
  • QoS classes decide who gets evicted first under memory pressure:
    • BestEffort: no requests or limits. Evicted first.
    • Burstable: some requests or limits.
    • Guaranteed: requests = limits for CPU and memory. Evicted last.

Source: K8s Cookbook, Recipe 12.4

Recipe 12.5 — Debug Pods

  • Problem: A Pod won’t start, or keeps dying.
  • Solution: Run an OODA loop. Observe, Orient (list hypotheses), Decide (pick one), Act (test it), repeat. Observe in this order:
kubectl get pod web-abc -o wide                    # STATUS, RESTARTS, NODE
kubectl describe pod web-abc                       # Events at the bottom ← start here
kubectl events --for pod/web-abc                   # (or) kubectl get events --sort-by=.lastTimestamp
kubectl logs web-abc --previous                    # why did the last container die?
kubectl get pod web-abc -o jsonpath='{.status.containerStatuses[*].lastState}'
  • Read the status:
StatusUsual causeLook at
PendingNo node fits (requests too big, taints, PVC unbound, quota)describe → Events (FailedScheduling)
ImagePullBackOff / ErrImagePullTypo in image/tag, private registry without imagePullSecrets, rate limitdescribe → Events
CrashLoopBackOffApp exits on start: bad config, missing env/Secret, liveness too aggressivelogs --previous, exit code
OOMKilled (exit 137)Memory limit too low or a leaklastState.terminated.reason, kubectl top
CreateContainerConfigErrorReferenced ConfigMap/Secret or key missingdescribe
Running but 0/1 ReadyReadiness probe failingdescribe → probe events
Terminating foreverFinalizers, or node unreachableget -o yaml → metadata.finalizers
  • Get inside: use kubectl debug. It attaches an ephemeral debug container that works even for distroless images with no shell:
kubectl exec -it web-abc -- sh                                   # if the image has a shell
kubectl debug -it web-abc --image=busybox:1.36 --target=web      # share the app container's process namespace
kubectl debug web-abc -it --copy-to=web-debug --container=web -- sh   # debug a *copy* (e.g. CrashLoop) with a shell
kubectl debug node/worker-1 -it --image=ubuntu                   # a shell on the node (host fs at /host)
kubectl run tmp --rm -it --image=nicolaka/netshoot --restart=Never -- bash   # network toolbox Pod
  • Pre-flight checklist (from the book, still true):
    • Is the manifest valid? (kubectl apply --dry-run=server)
    • Does the image run locally with docker run?
    • Can nodes pull from the registry?
    • Is DNS working (see Services & Ingress)?
    • Are there enough resources, and are your requests and limits sane?

Changed since the book:

kubectl debug and ephemeral containers (GA 1.25) replace “install curl into the running container”. kubectl events (GA 1.28) gives sorted, filterable events. Validate against your real cluster with --dry-run=server. The book pointed to a JSON schema site.

Source: K8s Cookbook, Recipe 12.5

Recipe 12.6 — Snapshot the Whole Cluster State

  • Problem: Capture everything for an audit, a post-mortem, or a support ticket.
  • Solution:
kubectl cluster-info dump --all-namespaces --output-directory=./cluster-state-$(date +%F)
  • Discussion: You get objects as JSON plus Pod logs, per namespace. For restorable backups (objects + volumes) use Velero. For “what’s in Git vs. live”, GitOps tools show drift.
  • Gotchas: The dump contains Secrets and logs, so treat it as sensitive.

Source: K8s Cookbook, Recipe 12.6

Recipe 12.7 — Add Worker Nodes

  • Problem: The cluster needs more capacity.

  • Solution: Provision a machine and install the three node components:

    1. kubelet: the node agent. It registers the node, runs and probes Pods, mounts volumes, and reports status.
    2. A container runtime speaking CRI: containerd or CRI-O.
    3. kube-proxy: programs Service routing. It’s optional if your CNI replaces it, e.g. Cilium.

    Then run kubeadm join … (see Getting Started & Local Clusters). On managed clouds, just grow the node pool, or let the autoscaler do it (Scaling).

  • Discussion: A Node object is only a record. The node counts once its kubelet reports healthy (kubectl get nodes → Ready). Debug runtimes with crictl on the node, since docker ps won’t show Kubernetes containers anymore.

Changed since the book:

The Docker Engine runtime was removed in 1.24 (dockershim), so containerd or CRI-O it is.

Source: K8s Cookbook, Recipe 12.7

Recipe 12.8 — Drain a Node for Maintenance

  • Problem: Patch, reboot or upgrade a node without dropping traffic.
  • Solution:
kubectl cordon worker-1                     # no new Pods land here
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data --timeout=5m
# … do the maintenance …
kubectl uncordon worker-1                   # schedulable again
  • Protect apps during drains with a PodDisruptionBudget:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2          # or maxUnavailable: 1
  selector:
    matchLabels:
      app: web
  • Discussion: drain = cordon + evict every Pod, respecting PDBs and graceful termination. DaemonSet Pods stay. Bare Pods (no controller) need --force and are gone forever.
  • Gotchas: A PDB with minAvailable equal to the replica count (e.g. 1 of 1) blocks drains forever. Run ≥ 2 replicas for anything you drain around.

Changed since the book:

--delete-local-data → --delete-emptydir-data. PDBs are policy/v1 (GA 1.21).

Source: K8s Cookbook, Recipe 12.8 · Best Practices book, Ch. 8

Recipe 12.9 — Manage etcd

  • Problem: Back up (or inspect) the cluster’s source of truth.
  • Solution (self-managed / kubeadm control plane):
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F).db
 
etcdutl snapshot status /backup/etcd-2026-01-01.db -w table
  • Discussion: Only the API server talks to etcd. Everything else goes through the API. Keys look like /registry/<resource>/<ns>/<name>, stored as protobuf (not readable JSON). Back up regularly and test restores (etcdutl snapshot restore).
  • Gotchas: On managed services (GKE, EKS, AKS) etcd is hidden and backed up for you. Back up your objects with Velero or Git instead.

Changed since the book:

The book’s curl …:2379/v2/keys uses the etcd v2 API, which is gone. Kubernetes uses etcd v3 only, over TLS. Snapshot status/restore moved from etcdctl to etcdutl.

Source: K8s Cookbook, Recipe 12.9

Summary

  • Debug order: get -o wide → describe (Events!) → logs --previous → kubectl debug.
  • Know the statuses: Pending (scheduling), ImagePullBackOff (image), CrashLoopBackOff (app/config/probe), OOMKilled (memory).
  • Script with jsonpath, kubectl wait and rollout status, not sleep 30.
  • Maintenance: cordon → drain --ignore-daemonsets --delete-emptydir-data → uncordon, with PDBs and ≥ 2 replicas.
  • Nodes run kubelet + containerd + kube-proxy, so use crictl on the node. Back up etcd with etcdctl snapshot save (v3 only).

Graph View

Backlinks