Kubernetes Cookbook by Sébastien Goasguen & Michael Hausenblas (O’Reilly, 2018): ~70 recipes in Problem → Solution → Discussion form.
Kubernetes Best Practices by Brendan Burns, Eddie Villalba, Dave Strebel & Lachlan Evenson (O’Reilly, 2019): distilled into one checklist note, Best Practices.
Everything is modernised to Kubernetes 1.31+. The Cookbook was written for Kubernetes ~1.7–1.10, so plenty has changed. Look for the Changed since the book callouts.
Each recipe is condensed into Problem / Solution / Discussion / Gotchas, with its source recipe number.
These notes build on my existing Kubernetes notes rather than repeat them:
Kubernetes: what K8s is, ephemeral Pods, unique Pod IPs
A distillation of Kubernetes Best Practices (Burns, Villalba, Strebel & Evenson, O’Reilly 2019) into checklists, updated for Kubernetes 1.31+. The Cookbook tells you how. This book tells you what good looks like.
Each section cites its chapter (“BP Ch. n”). Deeper how-tos are linked to the Cookbook notes.
Easiest rule: any config change = a new rollout. Use hashed names (Kustomize configMapGenerator) or a Helm checksum/config annotation.
Referenced ConfigMaps/Secrets must exist first (or be optional). A missing configMapKeyRef key blocks Pod start, while invalid keys in envFrom are skipped with an InvalidVariableNames event.
Secrets:
Encrypt at rest (KMS).
Keep the source of truth in an external vault (External Secrets, CSI driver).
Put imagePullSecrets on the ServiceAccount, not in every Pod spec.
RBAC:
Most apps need no RBAC at all. Disable token automount for them.
Apps that call the API get a dedicated ServiceAccount + least-privilege Role.
Humans via OIDC/SSO groups. Just-in-time elevation for SREs.
7. Global Distribution & Staged Rollouts (BP Ch. 7)
Replicate images to registries near every region.
Shift testing left (integration and replay tests) so only confident releases start rolling.
Roll out canary region first, then low-risk regions, then the rest, with health gates between waves.
Write down and rehearse rollback. Nobody remembers runbooks mid-incident.
8. Resource Management (BP Ch. 8)
Requests on every container (scheduling + autoscaling depend on them) and memory limits on every container. CPU limits are debated, since they cause throttling. Many teams set CPU requests only.
QoS: Guaranteed (req = limit) for critical workloads, Burstable for most, never BestEffort in prod.
Pod anti-affinity / topologySpreadConstraints across zones.
PodDisruptionBudgets for drains and upgrades.
Special hardware: taint GPU or special nodes and add tolerations only where needed. Use nodeSelector/affinity to target them. Node pools for mixed workload shapes.
Right-size nodes before production (cost vs. performance).
Scaling: start manual until you know the workload profile. HPA for spiky workloads. VPA recommendations for sizing. Cluster Autoscaler/Karpenter with an eye on spin-up time. Details: Scaling
9. Networking & Service Mesh (BP Ch. 9)
CNI:
Pick for features: NetworkPolicy, eBPF, encryption, multi-cluster. Cilium and Calico are common.
Make sure your cloud supports it.
Plan Pod CIDR and IP space generously (default max 110 Pods per node).
Check that your observability and security tools work with it.
Services: keep most ClusterIP. Expose only what must be public, and for HTTP use one Ingress/Gateway with TLS.
Standardise on one ingress controller (annotations aren’t portable), run it HA, and scale on metrics. API-gateway-style controllers suit API-heavy traffic.
NetworkPolicy: start with ingress rules, add a default-deny per namespace, open explicitly with labels, and keep a permissive test namespace for figuring out flows. Details: Security
Service mesh: only adopt for concrete needs (mTLS, retries, traffic shifting, observability). Weigh the operational cost. Sometimes your CNI gives you mTLS already. Today’s lighter options: Istio ambient mode (no sidecars) and Linkerd.
10. Pod & Container Security (BP Ch. 10)
Enforce pod hardening with Pod Security Admission (baseline everywhere, restricted where possible) instead of PSP.
RuntimeClass sandboxes (gVisor, Kata) for untrusted workloads. Isolation ≠ secure multi-tenancy, since the API is still shared. Mixed runtimes complicate debugging and portability.
Runtime threat detection: Falco, Tetragon.
11. Policy & Governance (BP Ch. 11)
Use a policy engine for org rules: ValidatingAdmissionPolicy (CEL), Kyverno or OPA Gatekeeper. Examples: allowed registries, required labels, no :latest, required probes and resources.
Enforce at the Pod level (the last hop before runtime). The user then sees the error on the ReplicaSet/events, not on kubectl apply. Use warn/audit modes to give early feedback.
Scope constraints tightly (kinds, namespaces, labels). Don’t replicate Secrets into the policy engine’s cache.
Run policies in audit mode first, then enforce.
12. Multiple Clusters (BP Ch. 12)
Use multiple clusters for blast radius, compliance (PCI, HIPAA), hard multi-tenancy, regions, and special workloads (HPC/GPU).
Have a data replication strategy before going multi-region. Stateless is easy, data is hard.
GitOps for fleet consistency (Argo CD ApplicationSets, Flux). Cluster API for lifecycle. Global load balancing between clusters.
Plan cross-cluster discovery/networking (mesh, Cilium ClusterMesh, MCS API) and make CD multi-cluster aware.
13. Integrating External Services (BP Ch. 13)
Calling out:
Selector-less Service + manual EndpointSlice for fixed IPs.
ExternalName (DNS CNAME) for a stable hostname, e.g. a managed DB.
A controller that syncs endpoints if addresses change.
Exposing in: an internal LoadBalancer (cloud annotation) or NodePort for on-prem consumers.
Cluster to cluster: export from one cluster, import as a selector-less Service in the other.
14. Machine Learning Workloads (BP Ch. 14)
Taint GPU nodes (nvidia.com/gpu:NoSchedule) and autoscale GPU pools to zero when idle. GPUs are the expensive bottleneck, so keep them saturated.
Separate, tainted node pools for ML in mixed clusters. Monitor GPU, CPU, network and storage together.
Today: NVIDIA GPU Operator, Kueue for batch/job queueing, Kubeflow/Ray (KubeRay) for training, and Dynamic Resource Allocation (DRA) for flexible accelerators.
15. Building Platforms (BP Ch. 15)
Use admission controllers to validate or mutate (inject sidecars, defaults) and kubectl plugins (krew) to extend the UX.
Design for users’ growth: an easy platform that traps users is a failure. Keep an escape hatch to plain Kubernetes, and support plain container images and normal Service discovery.
16. State & Stateful Apps (BP Ch. 16)
Use volumes mainly to share data between containers (emptyDir). hostPath only for node agents. Logs go to stdout, not a disk.
Default StorageClass on, with WaitForFirstConsumer and zone-aware topology (keep the Pod and its disk in the same zone).
Ask first: can a managed DB service do this instead? How hard is it to make the app stateless?
Back up volumes with CSI VolumeSnapshots or Velero, since Kubernetes doesn’t do it for you. Use Retain reclaim for sensitive or forensic data.
StatefulSets:
Create the headless Service yourself.
Ordinal identity doesn’t always need PVs.
On a dead node, Pods sit in Terminating/Unknown. Force-delete only as a last resort, and only when the node can’t come back.
Mutating runs before validating. Any rejection rejects the whole request.
Don’t let two mutating webhooks edit the same field. Add a validating check after mutation.
Choose failurePolicy deliberately, keep webhooks fast (30 s max), scope with namespaceSelector, skip kube-system, lock webhook configs with RBAC, and don’t send Secrets to webhooks.
Authorization: use RBAC only. ABAC needs file syncing and API server restarts. Authorization webhooks are risky single points of failure.
Summary
Every workload: immutable image tag, requests + memory limit, readiness + liveness (+ startup), labels (app.kubernetes.io/*), non-root restricted securityContext, ≥ 2 replicas spread across zones, a PDB.
Every namespace: quota + LimitRange, Pod Security labels, default-deny NetworkPolicy, least-privilege RBAC.
Every change: through Git and CI/CD (GitOps), traceable tags/labels, rolling or canary rollouts, rehearsed rollback.
Every cluster: metrics + logs + symptom-based alerts, policy engine in audit then enforce, backups of etcd/objects/volumes.
Rule of thumb in 2026: learn on a local cluster (kind, minikube, k3d), deploy to a managed one (GKE, EKS, AKS). Hand-built clusters (kubeadm) are for learning the internals or for on-prem.
Changed since the book (2018 → Kubernetes 1.31+)
Katacoda shut down in 2022, so use Killercoda or Play with Kubernetes instead.
minikube no longer needs VirtualBox. It defaults to the Docker driver (or podman, qemu, vfkit on macOS).
Docker Engine is no longer the node runtime. dockershim was removed in 1.24, and nodes now run containerd or CRI-O. Images built with Docker still run fine because they’re standard OCI images. See docker.
kubectl run now creates a single Pod, not a Deployment (since 1.18). Use kubectl create deployment.
Recipe 1.1 — Try Kubernetes Without Installing Anything
Problem: You want to poke at Kubernetes with zero setup.
Solution: Use a browser sandbox: the interactive tutorials on kubernetes.io, Killercoda, or Play with Kubernetes. Sessions are temporary (roughly an hour).
Gotchas: Nothing persists, so treat it as a scratchpad.
Source: K8s Cookbook, Recipe 1.1
Recipe 1.2 — Install kubectl
Problem: You need the CLI to talk to any cluster.
Solution:
# macOSbrew install kubectl# Linux (official binary)curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"sudo install -m 0755 kubectl /usr/local/bin/kubectlkubectl version --client
Discussion: Keep kubectl within ±1 minor version of the API server (the “version skew” policy).
Gotchas: Docker Desktop also ships a kubectl. Run which -a kubectl if versions look odd.
Changed since the book:
The download URL moved from storage.googleapis.com/kubernetes-release to dl.k8s.io.
Source: K8s Cookbook, Recipe 1.2
Recipe 1.3 / 1.4 — Run and Use minikube Locally
Problem: You want a real single-node (or multi-node) cluster on your laptop.
Solution:
brew install minikubeminikube start --cpus=4 --memory=8g --kubernetes-version=stablekubectl get nodes # context "minikube" is selected for youminikube ip | ssh | stop | delete # everyday sub-commandsminikube addons list # ingress, metrics-server, dashboard…eval $(minikube docker-env) # build images straight into minikube's runtimeminikube image load myapp:dev # …or push a local image into the cluster
Discussion: docker-env (or minikube image load) saves you from pushing images to a registry during dev. With kind, the equivalent is kind load docker-image myapp:dev.
Gotchas: If you use :latest with a locally loaded image, set imagePullPolicy: IfNotPresent. Otherwise Kubernetes tries to pull from Docker Hub and fails.
Changed since the book:
minikube ≥ 1.x picks the Docker driver automatically, and --memory accepts units (8g). You can also run multi-node with minikube start --nodes 3.
Source: K8s Cookbook, Recipes 1.3 & 1.4
Recipe 1.5 — Start Your First Application
Problem: The cluster is up and you want something running and reachable.
Solution:
kubectl create deployment ghost --image=ghost:5 --port=2368kubectl expose deployment ghost --port=2368 --type=NodePortkubectl get pods -wminikube service ghost # opens the NodePort in your browser# (kind/k3d alternative) kubectl port-forward deploy/ghost 2368:2368
Discussion: create deployment and expose are imperative generators. They write the YAML for you. For anything you keep, generate the YAML and commit it: kubectl create deployment ghost --image=ghost:5 --dry-run=client -o yaml > ghost.yaml.
Gotchas: Old blog posts say kubectl run ghost … creates a Deployment. It doesn’t anymore, so nothing would recreate that Pod if it died. See Deployments.
Changed since the book:
kubectl run = bare Pod only. --dry-run must now be --dry-run=client (or =server).
Source: K8s Cookbook, Recipe 1.5
Recipe 1.6 — A GUI for the Cluster
Problem: You want a point-and-click view of Deployments, ReplicaSets and Pods.
Solution: minikube dashboard still works for a quick look. For a maintained desktop/web UI use Headlamp (a Kubernetes SIG UI project). Terminal fans use k9s.
Discussion: A GUI is great for seeing the Deployment → ReplicaSet → Pod chain from Deployments. Create things with YAML, not clicks.
Changed since the book:
The upstream Kubernetes Dashboard project has been archived. Headlamp is now the suggested replacement. Never expose a dashboard publicly with a cluster-admin token.
Source: K8s Cookbook, Recipe 1.6
Recipe 2.1 / 2.2 — Bootstrap a Cluster with kubeadm
Problem: You want a “real” multi-machine cluster on your own VMs or bare metal.
Solution (on every node):
Install a container runtime (containerd), and turn swap off (or configure kubelet’s swap support).
Add the pkgs.k8s.io repo for your minor version and install kubelet kubeadm kubectl.
On the control plane: sudo kubeadm init --pod-network-cidr=10.244.0.0/16.
Copy /etc/kubernetes/admin.conf to ~/.kube/config.
Install a CNI plugin (Cilium, Calico or Flannel). Nodes stay NotReady until you do.
On each worker, run the printed command: kubeadm join <cp-ip>:6443 --token <t> --discovery-token-ca-cert-hash sha256:<h>.
Discussion: kubeadm is the standard building block that many installers wrap. It runs the control plane as static Pods (see 2.6).
Gotchas: Tokens expire after 24h. Make a new one with kubeadm token create --print-join-command.
Changed since the book:
apt.kubernetes.io / packages.cloud.google.com are gone, so use pkgs.k8s.io. Don’t apt-get install docker.io as the runtime. Weave Net is discontinued, so pick Cilium, Calico or Flannel. kubeadm is GA, no longer “beta”.
Source: K8s Cookbook, Recipes 2.1 & 2.2
Recipe 2.3 – 2.6 — Release Binaries, hyperkube and systemd
Problem: You want to understand what a cluster is made of, binary by binary.
Solution / what’s still true:
Release binaries live at https://dl.k8s.io/<version>/…. Always verify the published SHA256, and the signatures, which releases now carry (cosign).
The kubelet is the only component that has to be a systemd service. kubeadm writes /lib/systemd/system/kubelet.service plus a drop-in at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.
The kubelet watches /etc/kubernetes/manifests/ and runs any Pod YAML it finds there as a static Pod. That’s how etcd, kube-apiserver, kube-controller-manager and kube-scheduler run on kubeadm clusters: Kubernetes runs itself.
Gotchas: Static Pods show up in kubectl get pods -n kube-system as mirror pods. Deleting them via kubectl does nothing; edit the file on the node instead.
Changed since the book:
The hyperkube all-in-one image was removed (1.19). Images moved from gcr.io/google_containers / k8s.gcr.io to registry.k8s.io. The insecure API port (--insecure-port=8080) no longer exists. For the “learn every piece by hand” route, see Kubernetes the Hard Way.
Problem: You want a production-grade control plane without running it yourself.
Solution:
# Google GKE (Autopilot = Google also manages nodes)gcloud container clusters create-auto demo --region=europe-west1gcloud container clusters get-credentials demo --region=europe-west1# Azure AKSaz group create -n k8s -l northeuropeaz aks create -g k8s -n demo --node-count 2 --generate-ssh-keysaz aks get-credentials -g k8s -n demo# AWS EKSeksctl create cluster --name demo --region eu-west-1 --nodes 2
Discussion: The control plane (API server, etcd) is invisible to you, so you can’t SSH to it or touch etcd. You pay per cluster-hour plus nodes.
Gotchas: Delete when done (gcloud container clusters delete, az group delete -n k8s, eksctl delete cluster). Forgotten clusters are a classic surprise bill.
Changed since the book:
Azure ACS was retired, so az acs … is now az aks …. GKE gained Autopilot. For EKS, eksctl is the usual CLI.
Source: K8s Cookbook, Recipes 2.7 & 2.8
Summary
Local: kind / minikube / k3d. Cloud: GKE / AKS / EKS. DIY: kubeadm + containerd + a CNI.
kubectl run = Pod, kubectl create deployment = Deployment. Generate YAML with --dry-run=client -o yaml.
Control-plane components on kubeadm clusters are static Pods in /etc/kubernetes/manifests.
Nodes run containerd/CRI-O, not Docker, but your Docker-built images work unchanged.
Verb + resource type + name: kubectl get deployment web, kubectl delete svc web.
Every command hits the API server. kubectl is just a fancy REST client (see API, Namespaces & Labels).
Context & namespace decide where you’re pointing:
kubectl config get-contextskubectl config use-context kind-devkubectl config set-context --current --namespace=my-app # stop typing -n every time
Recipes
Recipe 3.1 — Listing Resources
Problem: You want to see what’s running.
Solution:
kubectl get pods # current namespacekubectl get pods -A # all namespaceskubectl get svc,deploy # several kinds at oncekubectl get deploy web -o wide # extra columnskubectl get pods -o yaml | less # the full objectskubectl api-resources # every kind + short name + namespaced?
Short names worth memorising:
Short
Kind
Short
Kind
po
pods
deploy
deployments
svc
services
rs
replicasets
cm
configmaps
ds
daemonsets
ns
namespaces
sts
statefulsets
no
nodes
ing
ingresses
pv / pvc
persistent volumes / claims
hpa
horizontal pod autoscalers
sa
serviceaccounts
netpol
networkpolicies
Gotchas: kubectl get all is not all. It skips ConfigMaps, Secrets, Ingresses, PVCs, CRDs and more.
Source: K8s Cookbook, Recipe 3.1
Recipe 3.2 — Deleting Resources
Problem: You want to clean up.
Solution:
kubectl delete deploy webkubectl delete svc,deploy -l app=niceone # by labelkubectl delete -f app.yaml # whatever the file createdkubectl delete ns my-app # nukes everything inside itkubectl delete pod stuck --grace-period=0 --force # last resort
Discussion: Delete the owner, not the children. Deleting a Pod owned by a Deployment just makes a new one appear (that’s the self-healing from Pods). Owners cascade-delete their dependents via ownerReferences and garbage collection (--cascade=background|foreground|orphan).
Gotchas: Force-deleting a StatefulSet Pod can leave two Pods with the same identity if the node comes back. A namespace stuck in Terminating almost always means a finalizer is waiting on a controller that’s gone.
Source: K8s Cookbook, Recipe 3.2
Recipe 3.3 — Watching Changes
Problem: You want a live view while things roll out.
Solution: kubectl get pods -w (streams changes), or watch -n1 kubectl get pods (redraws the screen). kubectl rollout status deploy/web blocks until a rollout finishes. kubectl wait --for=condition=Ready pod -l app=web --timeout=60s is the scripting-friendly version.
Gotchas: -w prints a new line per change rather than redrawing. Use watch or k9s for a dashboard feel.
Source: K8s Cookbook, Recipe 3.3
Recipe 3.4 — Editing Resources
Problem: You need to change a live object.
Solution:
KUBE_EDITOR="code --wait" kubectl edit deploy web # opens live YAMLkubectl set image deploy/web web=nginx:1.27 # targeted shortcutskubectl scale deploy/web --replicas=3kubectl patch deploy web -p '{"spec":{"replicas":2}}'
Discussion: edit is great for experiments, but the change isn’t in Git. The next kubectl apply -f of your file will quietly undo it (“drift”). Prefer editing the file and re-applying. More on that in Managing Manifests.
The book’s example starts with kubectl run nginx and then edits deploy/nginx. Today run creates a Pod, so there is no deploy to edit. Use kubectl create deployment nginx --image=nginx.
Source: K8s Cookbook, Recipe 3.4
Recipe 3.5 — kubectl explain
Problem: What does this field mean, what type is it, and is it required?
Chapter 6 of the Kubernetes Cookbook: how the API is organised, what every manifest has in common, and the metadata you organise everything with (namespaces, quotas, labels, annotations).
Key idea: every object has spec (what you want) and status (what is). Controllers spend their whole lives making status match spec.
Recipes
Recipe 6.1 — Discover the API Endpoints
Problem: You want to see which API groups, versions and resources your cluster serves.
Solution:
kubectl api-versions # apps/v1, batch/v1, networking.k8s.io/v1, …kubectl api-resources -o wide # kind, group, namespaced?, verbskubectl get --raw /apis/apps/v1 | jq '.resources[].name'# or browse with plain curl through an authenticated local proxykubectl proxy --port=8001 &curl -s localhost:8001/apis | jq '.groups[].name'
Discussion: The core group lives at /api/v1 (Pods, Services, ConfigMaps, Secrets). Everything else is under /apis/<group>/<version>. That pair is exactly what you write in apiVersion:.
Gotchas: Versions mature as v1alpha1 → v1beta1 → v1. Beta APIs are off by default for new APIs since 1.24, and deprecated versions really do get removed (see the note below).
Changed since the book:
No more unauthenticated port 8080, so use kubectl proxy or kubectl get --raw. Deployments are apps/v1, not extensions/v1beta1. The whole extensions group was removed in 1.16 and 1.22. Check old YAML with kubectl convert (plugin) or the pluto tool.
Source: K8s Cookbook, Recipe 6.1
Recipe 6.2 — The Shape of Every Manifest
Problem: You want to write manifests from scratch, not just generate them.
Solution: Four top-level keys, every time:
apiVersion: v1 # group/version (core group has no prefix)kind: Pod # the typemetadata: # name, namespace, labels, annotations name: mypod labels: app: demospec: # desired state; shape depends on kind containers: - name: app image: nginx:1.27# status: written by the system, never by you
Discussion: Several objects can share one file, separated by ---. JSON works too, since YAML is a superset.
Gotchas: Tabs aren’t allowed in YAML, and indentation errors produce baffling messages. Use kubectl apply --dry-run=server -f x.yaml to have the API server validate a file without creating anything.
Source: K8s Cookbook, Recipe 6.2
Recipe 6.3 — Namespaces to Avoid Name Collisions
Problem: Two teams both want a Deployment called web.
Solution:
kubectl create namespace team-akubectl create deployment web --image=nginx -n team-akubectl create deployment web --image=nginx -n team-b # no clash
apiVersion: v1kind: Namespacemetadata: name: team-a labels: pod-security.kubernetes.io/enforce: baseline # Pod Security Admission, see the Security note
Discussion: Names must be unique per kind, per namespace. Built-in namespaces are default, kube-system (cluster components), kube-public (world-readable), and kube-node-lease (node heartbeats).
Gotchas: Some kinds are cluster-scoped and ignore -n: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CRDs and Namespaces themselves. Namespaces are a soft boundary. They only isolate anything when you add RBAC, quotas and NetworkPolicies.
Source: K8s Cookbook, Recipe 6.3
Recipe 6.4 — Quotas Within a Namespace
Problem: One namespace shouldn’t eat the whole cluster.
Solution: A ResourceQuota caps object counts and total compute. A LimitRange gives defaults to containers that forget their requests and limits.
Discussion: Check usage with kubectl describe quota -n team-a.
Gotchas: Once a quota covers requests.cpu, every Pod must declare a CPU request or it’s rejected. That’s why you pair it with a LimitRange that fills in defaults.
Problem: You want to tag objects and find them again (and so do Services and Deployments).
Solution:
kubectl label pods foobar tier=frontendkubectl label pods foobar tier=backend --overwritekubectl label pods foobar tier- # remove a labelkubectl get pods --show-labelskubectl get pods -l app=cockroachdb # equalitykubectl get pods -l 'env in (production,staging)' # set-basedkubectl get pods -L app,env # labels as columns
Discussion: Labels are free-form key/value pairs that Kubernetes doesn’t interpret. But selectors are how Services find their Pods and how Deployments own their ReplicaSets, so labels are load-bearing.
Gotchas: A Deployment’s spec.selector is immutable in apps/v1. Pick it carefully, and don’t put the version in it.
Source: K8s Cookbook, Recipes 6.5 & 6.6
Recipe 6.7 — Annotations
Problem: You want to attach non-identifying data: build info, tool config, a reason for change.
Solution:
kubectl annotate deploy web kubernetes.io/change-cause="bump to nginx 1.27"kubectl annotate pod foobar description='used by automation' --overwrite
Discussion: Labels are for selecting, annotations are for describing. Tools love annotations (Ingress controllers, Prometheus scrape hints, Helm). kubernetes.io/change-cause fills the CHANGE-CAUSE column in kubectl rollout history.
Gotchas: You can’t select by annotation.
Changed since the book:
--record is deprecated, so set kubernetes.io/change-cause explicitly (as above) after each change.
Source: K8s Cookbook, Recipe 6.7
Summary
apiVersion = group/version. Core is v1, most workloads are apps/v1. Deprecated betas get removed, so keep manifests current.
Every object = apiVersion + kind + metadata + spec (+ status from the system).
Namespaces scope names. Quotas and LimitRanges cap usage per namespace.
Chapters 4 and 7 of the Kubernetes Cookbook: Pods and Deployments (long-running apps), then the specialised controllers for Jobs, CronJobs, DaemonSets, StatefulSets and init containers.
My basics: Pods (ephemeral, unique IP), Deployments (ReplicaSets, thrashing, CrashLoopBackOff). This note covers the how plus the other workload types.
All these controllers are GA in apps/v1 (Deployment, DaemonSet, StatefulSet, ReplicaSet) or batch/v1 (Job, CronJob). extensions/v1beta1, apps/v1beta1/2 and batch/v2alpha1 are gone. In apps/v1, spec.selector is required and must match the template labels.
Recipes — Fundamental Workloads (Ch. 4)
Recipe 4.1 — Create a Deployment from the CLI
Problem: Quickly launch a long-running app.
Solution:
kubectl create deployment ghost --image=ghost:5 --port=2368 --replicas=2kubectl create deployment mysql --image=mysql:8.4kubectl set env deployment/mysql MYSQL_ROOT_PASSWORD=root # demo only, use a Secret for realkubectl expose deployment ghost --port=2368 # Service in one linekubectl run myshell --image=busybox:1.36 -it --rm --restart=Never -- sh # one-off Pod
Gotchas: create deployment has no --env, so add env vars with kubectl set env or in YAML.
Changed since the book:
Every kubectl run … in the book that creates a Deployment (--replicas, --expose, --requests) no longer works that way. run now makes one Pod, and most of those flags were removed.
Source: K8s Cookbook, Recipe 4.1
Recipe 4.2 — Create Objects from Manifests
Problem: You want the object’s properties written down, not typed.
Solution: kubectl apply -f file.yaml, -f dir/, -f https://…/x.yaml, or -k overlay/ for kustomize.
Discussion: create -f fails if the object exists. apply is idempotent (create-or-update), so it’s the one to use in scripts and GitOps.
Discussion: Deployment → ReplicaSet → Pods (the picture in Deployments). Each template change creates a new ReplicaSet, and the old ones are kept (revisionHistoryLimit, default 10) so you can roll back.
Gotchas: Delete the Deployment, not its Pods. Always set requests (the scheduler needs them) and a memory limit (Best Practices).
Changed since the book:
extensions/v1beta1 → apps/v1 with a mandatory selector. To generate this skeleton use kubectl create deployment fancyapp --image=nginx --dry-run=client -o yaml. The book’s plain --dry-run is gone.
Source: K8s Cookbook, Recipe 4.4
Recipe 4.5 — Update (and Roll Back) a Deployment
Problem: Roll out a new version safely, and undo it if it’s broken.
Solution:
kubectl set image deployment/sise sise=mhausenblas/simpleservice:0.5.0kubectl annotate deployment/sise kubernetes.io/change-cause="image 0.5.0"kubectl rollout status deployment/sisekubectl rollout history deployment/sisekubectl rollout undo deployment/sise --to-revision=2kubectl rollout restart deployment/sise # re-create Pods, same spec (e.g. after a ConfigMap change)kubectl rollout pause|resume deployment/sise
Tune the rollout (in the Deployment spec):
spec: strategy: type: RollingUpdate # or Recreate (kill all, then start: has downtime) rollingUpdate: maxSurge: 25% # extra Pods allowed during rollout maxUnavailable: 0 # never drop below desired count minReadySeconds: 10 # Pod must stay Ready this long to count progressDeadlineSeconds: 600 # mark rollout as failed after this
Discussion: Only changes to .spec.template (image, env, ports, resources, annotations on the template) trigger a rollout. Scaling doesn’t.
Gotchas: Rolling updates are only as safe as your readiness probe. Without one, Kubernetes sends traffic to Pods that aren’t ready (see Monitoring & Logging). For canary or blue-green, use Argo Rollouts / Flagger or a second Deployment behind the same Service.
Changed since the book:
--record is deprecated, so use the change-cause annotation. kubectl rollout restart (added 1.15) is the clean way to “bounce” a Deployment.
Source: K8s Cookbook, Recipe 4.5
Recipes — Specialised Workloads (Ch. 7)
Recipe 7.1 — Run a Batch Job
Problem: Run something to completion: a migration, a backup, a conversion.
Solution:
apiVersion: batch/v1kind: Jobmetadata: name: counterspec: completions: 1 parallelism: 1 backoffLimit: 3 # retries before the Job is marked failed activeDeadlineSeconds: 300 # hard timeout ttlSecondsAfterFinished: 600 # auto-clean the finished Job + Pods template: spec: restartPolicy: Never # Jobs need Never or OnFailure containers: - name: counter image: busybox:1.36 command: ["sh", "-c", "for i in 1 2 3; do echo $i; done"]
Discussion: Raise completions/parallelism for work queues. completionMode: Indexed gives each Pod a JOB_COMPLETION_INDEX.
Gotchas: Without ttlSecondsAfterFinished, finished Jobs and their Pods pile up forever.
Changed since the book:
The TTL-after-finished controller is GA, so the manual cleanup the book mentions isn’t needed.
Source: K8s Cookbook, Recipe 7.1
Recipe 7.2 — Run a Task on a Schedule (CronJob)
Problem: Run a Job every hour, night or week.
Solution:
apiVersion: batch/v1kind: CronJobmetadata: name: hourly-datespec: schedule: "0 * * * *" timeZone: "Europe/London" # GA since 1.27 concurrencyPolicy: Forbid # skip if the previous run is still going startingDeadlineSeconds: 120 successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 1 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: date image: busybox:1.36 command: ["sh", "-c", "date"]
Discussion: Trigger a run by hand with kubectl create job --from=cronjob/hourly-date manual-1. Suspend with kubectl patch cronjob hourly-date -p '{"spec":{"suspend":true}}'.
Gotchas: Jobs can occasionally run twice (or be skipped), so make them idempotent.
Changed since the book:
batch/v2alpha1 → batch/v1 (GA in 1.21), plus the timeZone field.
Source: K8s Cookbook, Recipe 7.2
Recipe 7.3 — One Pod per Node (DaemonSet)
Problem: Run an agent (log collector, monitoring exporter) on every node.
Discussion: New nodes get a Pod automatically. Limit to some nodes with nodeSelector or affinity.
Gotchas: kubectl drain needs --ignore-daemonsets, because DaemonSet Pods aren’t evicted.
Changed since the book:
extensions/v1beta1 → apps/v1. The book’s fluentd-elasticsearch image is long gone. Fluent Bit is now the usual node log agent.
Source: K8s Cookbook, Recipe 7.3
Recipe 7.4 — Stateful and Leader/Follower Apps (StatefulSet)
Problem: Your replicas aren’t interchangeable. They need stable names and their own disks (“pets, not cattle”).
Solution:
apiVersion: v1kind: Servicemetadata: name: dbspec: clusterIP: None # headless: gives each Pod a DNS name selector: app: db ports: - port: 5432---apiVersion: apps/v1kind: StatefulSetmetadata: name: dbspec: serviceName: db replicas: 3 selector: matchLabels: app: db template: metadata: labels: app: db spec: containers: - name: postgres image: postgres:16 env: - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password volumeMounts: - name: data mountPath: /var/lib/postgresql/data volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi
Discussion: Pods are named db-0, db-1, db-2 and reachable at db-0.db.<ns>.svc.cluster.local. They’re created in order and deleted in reverse, and each keeps its own PVC across restarts.
Gotchas: You must create the headless Service yourself. Deleting the StatefulSet doesn’t delete the PVCs (by design, unless you set persistentVolumeClaimRetentionPolicy). StatefulSet ≠ replication: Postgres doesn’t magically become a cluster. For real databases use an operator (CloudNativePG, Strimzi for Kafka) or a managed DB. See Extending Kubernetes.
Changed since the book:
apps/v1beta1 → apps/v1. Added persistentVolumeClaimRetentionPolicy and minReadySeconds. The book’s CockroachDB example URL in kubernetes/kubernetes/examples no longer exists.
Discussion: Init containers run sequentially, to completion, before the app containers start. Debug them with kubectl logs <pod> -c checkbackend.
Native sidecars: an init container with restartPolicy: Always starts first and keeps running alongside the app. Use it for proxies or log shippers that must start before the app and stop after it.
Gotchas: Prefer apps that retry their dependencies over hard ordering, because dependencies can go away after startup too.
Changed since the book:
apps/v1beta1 → apps/v1. Native sidecar containers were added (beta, on by default in 1.29, GA in 1.33).
Source: K8s Cookbook, Recipe 7.5
Summary
Deployment for stateless, Job/CronJob for run-to-completion, DaemonSet per node, StatefulSet for identity + storage.
Everything is apps/v1 or batch/v1 now, with a required selector that matches the template labels.
Rollouts trigger on .spec.template changes. Use rollout status/history/undo/restart, and record why with change-cause.
Clean Jobs up with ttlSecondsAfterFinished, and make CronJobs idempotent.
Init containers gate startup. Native sidecars (restartPolicy: Always) run alongside.
Chapter 5 of the Kubernetes Cookbook (plus Recipes 12.2 and 12.3, which are service tricks): giving Pods a stable address, DNS names, service types, and getting HTTP traffic in from outside.
My basics: Services (the four service types, L4 vs L7) and Ingress (my app-ingress.yaml with host-based rules). This note adds the recipes, debugging and the modern pieces (IngressClass, Gateway API).
The control plane finds matching Ready Pods and writes their IPs into EndpointSlices.
kube-proxy on every node programs iptables/IPVS/nftables, so the Service’s virtual ClusterIP forwards to one of those Pod IPs. Some CNIs (Cilium) replace kube-proxy with eBPF.
CoreDNS gives it a name: web.<namespace>.svc.cluster.local.
Changed since the book:
Endpoints are now tracked by EndpointSlices (kubectl get endpointslices). The old Endpoints API is deprecated (since 1.33) but still readable. kube-dns was replaced by CoreDNS. kube-proxy gained an nftables mode.
Recipes
Recipe 5.1 — Create a Service to Expose Your App
Problem: Pods come and go (see Pods), but clients need one stable address.
Solution:
kubectl create deployment nginx --image=nginx:1.27kubectl expose deployment nginx --port=80 --target-port=80kubectl get svc nginxkubectl get endpointslices -l kubernetes.io/service-name=nginx
apiVersion: v1kind: Servicemetadata: name: nginxspec: type: ClusterIP # default selector: app: nginx # must match the Pod labels ports: - name: http port: 80 # the Service's port targetPort: 80 # the container's port (number or named port)
Discussion: Deployments and Services both use labels but do different jobs. The Deployment keeps Pods alive, the Service routes to them.
Gotchas: Service “doesn’t work”? Check the endpoints first. Empty endpoints mean the selector doesn’t match, or no Pod is Ready.
Changed since the book:
kubectl create deployment labels Pods app=nginx. The book’s kubectl run labelled them run=nginx, so old Service YAML with run: selectors won’t match.
Source: K8s Cookbook, Recipe 5.1
Recipe 5.2 — Verify a Service’s DNS Entry
Problem: Is cluster DNS resolving your Service?
Solution:
kubectl run dnstest --image=busybox:1.36 -it --rm --restart=Never -- nslookup nginx# full name: nginx.default.svc.cluster.local
Discussion: From the same namespace, nginx resolves. From another namespace use nginx.<ns>. The answer should be the Service’s ClusterIP (headless Services return Pod IPs instead).
Gotchas: Pin busybox to a known tag, since some busybox builds have a flaky nslookup. registry.k8s.io/e2e-test-images/agnhost or nicolaka/netshoot are nicer debugging images.
Source: K8s Cookbook, Recipe 5.2
Recipe 5.3 — Change a Service’s Type
Problem: You started with ClusterIP but now need NodePort or a cloud LoadBalancer.
Solution:
kubectl patch svc webserver -p '{"spec":{"type":"NodePort"}}'kubectl get svc webserver # PORT(S) shows 80:3xxxx/TCP# or edit the YAML (type: LoadBalancer) and kubectl apply
Discussion: NodePort opens a port (30000–32767) on every node. LoadBalancer asks the cloud for an external LB, which on-prem or locally needs MetalLB, minikube tunnel or cloud-provider-kind.
Gotchas: Each LoadBalancer Service = one paid cloud LB. For many HTTP apps, put one Ingress/Gateway in front instead.
Source: K8s Cookbook, Recipe 5.3
Recipe 5.4 — Deploy an Ingress Controller
Problem: Ingress objects do nothing until a controller watches them.
Solution:
minikube addons enable ingress # minikube# anywhere else: install a controller with Helm, e.g. Traefik, HAProxy,# Contour, Kong, Envoy Gateway, or your cloud's controller (GKE, AWS LB Controller, AGIC)kubectl get ingressclass
Discussion: Each controller registers an IngressClass. Your Ingress points at one with ingressClassName.
Gotchas: Controller-specific annotations (rewrites, auth, rate limits) aren’t portable, so standardise on one controller (Best Practices).
Changed since the book:
The community ingress-nginx project was retired in 2026 (no more fixes). New setups should pick another controller or move to Gateway API (below). My existing Ingress note’s nginx.ingress.kubernetes.io/* annotations are specific to ingress-nginx.
Source: K8s Cookbook, Recipe 5.4
Recipe 5.5 — Make Services Reachable from Outside (Ingress)
Problem: Route http://<host>/web to the nginx Service.
Discussion: The controller watches Ingress objects and reconfigures its proxy. Use cert-manager to get and renew TLS certificates (Let’s Encrypt) automatically.
Gotchas: Forgetting ingressClassName means no controller may pick the Ingress up (unless one is marked as the default class).
Changed since the book:
extensions/v1beta1 → networking.k8s.io/v1. serviceName/servicePort became service.name/service.port.number. pathType is required. The kubernetes.io/ingress.class annotation was replaced by ingressClassName.
Source: K8s Cookbook, Recipe 5.5
Modern Extra — Gateway API (Ingress’s successor)
Why: Ingress only covers basic HTTP routing, and everything else lives in vendor annotations. Gateway API (gateway.networking.k8s.io/v1, GA) splits responsibilities: the platform team owns the Gateway, app teams own their HTTPRoutes. It has first-class header matching, traffic splitting and TCP/gRPC routes.
Gotchas: The Gateway API CRDs plus a controller (Envoy Gateway, Istio, Cilium, Traefik, NGINX Gateway Fabric, cloud implementations) must be installed first.
Recipe 12.2 — Take One Pod Out of a Service (for Inspection)
Problem: One replica is misbehaving. You want it out of rotation but still alive to debug.
Solution: Change its label so the Service selector (and the ReplicaSet) no longer match:
kubectl label pod nginx-7c5d-abcde app=quarantine --overwrite
Discussion: The Service drops it from its endpoints, and the ReplicaSet notices it’s “missing” and starts a replacement. The quarantined Pod keeps running for you to exec into. Delete it when done.
Source: K8s Cookbook, Recipe 12.2
Recipe 12.3 — Reach a ClusterIP Service from Your Laptop
Problem: Test an internal Service without exposing it.
Solution:
kubectl port-forward svc/nginx 8080:80 # simplestcurl localhost:8080kubectl proxy & # via the API servercurl localhost:8001/api/v1/namespaces/default/services/nginx:80/proxy/
Changed since the book:
The old proxy path /api/v1/proxy/namespaces/... was removed. It’s now /api/v1/namespaces/<ns>/services/<name>[:port]/proxy/.
Source: K8s Cookbook, Recipe 12.3
Summary
A Service = selector → EndpointSlices of Ready Pods → a virtual IP plus a DNS name (svc.ns.svc.cluster.local).
Debug Services by checking endpoints and labels first.
ClusterIP (internal) → NodePort (node port) → LoadBalancer (cloud LB). For HTTP, use one Ingress or Gateway.
Ingress needs a controller and networking.k8s.io/v1 with ingressClassName + pathType. Gateway API is the future.
port-forward for quick local access. Relabel a Pod to pull it out of rotation.
Chapter 8 of the Kubernetes Cookbook: sharing files between containers, feeding config and secrets into Pods, and storage that outlives the Pod (PV, PVC, StorageClass).
My basics: ConfigMap (decouple config from images, and ConfigMaps aren’t secure). Remember from Pods: anything written inside a container dies with it.
Storage is now done by CSI drivers. The in-tree cloud plugins (awsElasticBlockStore, gcePersistentDisk, azureDisk…) were migrated or removed. gitRepo volumes are deprecated (use an init container that clones instead).
Recipes
Recipe 8.1 — Share Data Between Containers (emptyDir)
Problem: Two containers in one Pod need to exchange files.
Discussion: Secret types: generic (Opaque), docker-registry (image pull credentials) and tls (cert + key, used by Ingress). Max size is 1 MiB. Secrets are namespaced.
Gotchas: Secrets are base64, not encrypted. Anyone with get secret RBAC, or access to etcd backups, can read them. Do three things:
Turn on encryption at rest (EncryptionConfiguration with a KMS v2 provider). Managed clouds often do this for you.
Lock down RBAC on secrets (list and watch reveal all of them).
Keep real secrets in a vault and sync them in (see Security).
Prefer volume mounts to env vars. Env vars leak into crash dumps and kubectl describe, and they don’t update when the Secret changes.
Changed since the book:
The flag is now --encryption-provider-config (no longer “experimental”), with KMS v2 GA. ServiceAccount token Secrets are no longer auto-created (since 1.24). See Security.
Env vars are read once at start. A ConfigMap change needs a kubectl rollout restart.
Volume files refresh automatically (after up to about a minute), so an app that watches its config file can hot-reload.
subPath mounts never refresh.
Gotchas: The ConfigMap must exist before the Pod starts (or mark it optional: true). Mark stable config immutable: true. That protects it from accidental edits and saves API server watch load.
Trick: Kustomize’s configMapGenerator and Helm’s checksum/config annotation change the Pod template whenever config changes. That triggers a rollout automatically (see Managing Manifests).
Changed since the book:
extensions/v1beta1 → apps/v1. Immutable ConfigMaps and Secrets are GA (1.21).
Recipe 8.4 — Use a PersistentVolume (static provisioning)
Problem: Data must survive Pod restarts.
Solution: The admin creates a PV (the actual disk). The developer creates a PVC (a request for a disk). Kubernetes binds them, and the Pod mounts the PVC.
apiVersion: v1kind: PersistentVolumemetadata: name: hostpathpvspec: storageClassName: manual capacity: storage: 1Gi accessModes: ["ReadWriteOnce"] persistentVolumeReclaimPolicy: Retain hostPath: path: /tmp/pvdata # fine for minikube/kind, never in prod---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: mypvcspec: storageClassName: manual accessModes: ["ReadWriteOnce"] resources: requests: storage: 200Mi
# in the Pod template: volumes: - name: html persistentVolumeClaim: claimName: mypvc
Access modes:
Mode
Meaning
ReadWriteOnce (RWO)
read-write by one node (several Pods on that node can share it)
ReadWriteOncePod (RWOP)
read-write by one Pod only (GA 1.29)
ReadOnlyMany (ROX)
read-only by many nodes
ReadWriteMany (RWX)
read-write by many nodes. Needs NFS, CephFS, EFS, Azure Files…
Gotchas: PVs are cluster-scoped, PVCs are namespaced. A PVC stuck in Pending means no PV or StorageClass matches its size, mode or class (kubectl describe pvc tells you why).
Source: K8s Cookbook, Recipe 8.4
Recipe 8.5 / 8.6 — Dynamic Provisioning with StorageClasses
Problem: You don’t want an admin hand-making PVs for every claim.
Solution: Just create the PVC. The default StorageClass provisions a matching PV on demand.
kubectl get storageclass # the one marked (default) is used when none is named
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: fast-ssdprovisioner: pd.csi.storage.gke.io # ebs.csi.aws.com, disk.csi.azure.com, rancher.io/local-path…parameters: type: pd-ssdreclaimPolicy: Retain # default is Delete!volumeBindingMode: WaitForFirstConsumer # create the disk in the zone where the Pod landsallowVolumeExpansion: true---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: dataspec: storageClassName: fast-ssd accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi
Discussion: minikube’s default class (standard, provisioner k8s.io/minikube-hostpath) and kind’s (standard, local-path) are fine for learning. On cloud you get CSI-backed disks. Grow a volume by editing the PVC’s storage (if allowVolumeExpansion). Back up with VolumeSnapshots or Velero.
Gotchas:
reclaimPolicy: Delete (the default for dynamic PVs) destroys the disk when the PVC is deleted. Use Retain for anything precious.
Use WaitForFirstConsumer, or you’ll get “volume in zone A, Pod in zone B” (Best Practices).
Block disks are zonal, so a Pod can’t follow its disk to another zone.
Changed since the book:
GKE’s default class now uses the PD CSI driver (standard-rwo), not kubernetes.io/gce-pd. The blog-post walkthrough the book links to is outdated.
Chapter 11 of the Kubernetes Cookbook: container logs, liveness/readiness probes, resource metrics, Prometheus and centralised logging. Plus the Best Practices book’s advice on what to monitor and how to alert.
Admins: Are nodes healthy? Is the cluster full? Are teams near quota?
Developers: Is my app healthy? Are requests and limits right? Why did it restart?
Recipes
Recipe 11.1 — Read a Container’s Logs
Problem: See what the app printed.
Solution:
kubectl logs pod/web-abc # single-container Podkubectl logs web-abc -c sidecar # pick a containerkubectl logs deploy/web # some Pod of the Deploymentkubectl logs -l app=web --all-containers --prefix --max-log-requests=10kubectl logs web-abc -f --since=10m --timestampskubectl logs web-abc --previous # the crashed container before the restart ← gold for CrashLoopBackOff
Discussion: Kubernetes only keeps what the container writes to stdout/stderr, rotated on the node. When the Pod is deleted, its logs go too, which is why you want central logging (11.7).
Gotchas: For multi-Pod tailing use stern (stern web) or kubectl logs -l … -f.
Discussion: Readiness makes rolling updates and scaling safe. Liveness rescues deadlocks. The kubelet runs the probes, and a Pod is Ready only when all its containers are Ready.
Gotchas:
Liveness and readiness should usually differ. Readiness can check dependencies (a DB). Liveness should check only the process itself. Otherwise a DB blip restarts every Pod at once, a self-inflicted outage.
Too-aggressive liveness on a slow-booting app = endless restarts (CrashLoopBackOff from Deployments). Use a startupProbe instead of a huge initialDelaySeconds.
The book’s snippet says initialDelaySecond. The real field is initialDelaySeconds.
Changed since the book:
startupProbe (GA 1.20) and grpc probes (GA 1.27) were added, and extensions/v1beta1 → apps/v1.
Problem: kubectl top errors out. You want CPU and memory usage per Pod and node.
Solution:
minikube addons enable metrics-server # minikube# elsewhere:kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml# (kind/local: add --kubelet-insecure-tls to the metrics-server args)kubectl top nodeskubectl top pods -A --sort-by=memorykubectl top pod web-abc --containers
Discussion: metrics-server keeps only current usage in memory (no history) and serves the metrics.k8s.io API that HPA and kubectl top use. For history and dashboards you need Prometheus (11.6).
Changed since the book:
Heapster + InfluxDB + Grafana add-on → metrics-server. Heapster was retired in 2018 and removed from everything.
Source: K8s Cookbook, Recipe 11.5
Recipe 11.6 — Prometheus for Cluster and App Metrics
Problem: Query and alert on system and app metrics in one place.
Solution: Don’t hand-write the ConfigMap, RBAC and Deployment like the book did. Install the kube-prometheus-stack chart: Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, plus ready-made dashboards and alerts.
# Tell the Prometheus Operator to scrape your app's /metrics via its ServiceapiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: web labels: release: mon # must match the operator's selectorspec: selector: matchLabels: app: web endpoints: - port: http path: /metrics interval: 30s
Metric sources:
cAdvisor (inside the kubelet): container CPU and memory.
kube-state-metrics: object state, e.g. “replicas desired vs available”, “Pod restarts”.
node-exporter: node OS metrics.
your app: instrument it with a Prometheus client or OpenTelemetry.
What to watch (Best Practices Ch. 3):
USE for infrastructure: Utilisation, Saturation, Errors.
RED for services: Rate, Errors, Duration.
The four golden signals: latency, traffic, errors, saturation.
Gotchas: Watch label cardinality. A label like user_id explodes Prometheus’s memory.
Changed since the book:
The book’s config targeted 1.7-specific kubelet paths and bound Prometheus to cluster-admin. Use the operator chart’s scoped RBAC instead. Managed Prometheus (GKE, Amazon AMP, Azure Monitor) is a common alternative.
Problem: Search logs from all Pods, including dead ones, in one place.
Solution (modern stack): a node-level agent as a DaemonSet (Fluent Bit, Vector or the OpenTelemetry Collector) reads /var/log/containers/* and ships to a backend: Loki + Grafana (cheap, label-indexed), OpenSearch/Elasticsearch + Dashboards/Kibana (full-text), or a cloud service.
Prefer the DaemonSet agent over a log-shipper sidecar in every Pod (cheaper).
Keep hot logs about 30–45 days and archive older ones to cheap storage.
Correlate logs with metrics and traces (OpenTelemetry).
Alerting: Alert on symptoms that hurt users or SLOs, not on every blip, because alert fatigue makes people ignore pages.
Changed since the book:
EFK on minikube with Kibana’s default kibana/changeme login is purely historical. Fluentd → Fluent Bit is the common agent today, and Loki is popular with Grafana. Everything installs via Helm.
Pods stuck Pending for lack of room, or empty nodes
They stack. HPA adds Pods, and if they don’t fit, the node autoscaler adds nodes.
Everything depends on resource requests. The HPA computes utilisation relative to requests, and node autoscalers decide “doesn’t fit” based on requests.
Recipes
Recipe 9.1 — Scale a Deployment Manually
Problem: Load dropped and five replicas is too many.
Solution:
kubectl scale deployment fancyapp --replicas=3kubectl get deploy fancyapp
Gotchas: If an HPA manages the Deployment, it will overwrite your manual number. And if replicas: is in Git and your GitOps tool re-applies it, it fights the HPA. Leave replicas out of the manifest when an HPA owns it.
Source: K8s Cookbook, Recipe 9.1
Recipe 9.2 / 9.3 — Autoscale the Cluster’s Nodes
Problem: Add nodes when Pods can’t be scheduled, and remove them when they’re idle.
Solution (managed clouds):
# GKE (Autopilot does this for you automatically)gcloud container clusters update demo --enable-autoscaling \ --node-pool=default-pool --min-nodes=1 --max-nodes=5 --region=europe-west1# AKSaz aks update -g k8s -n demo --enable-cluster-autoscaler --min-count 1 --max-count 5# EKS: Karpenter (provisions right-sized nodes directly), or Cluster Autoscaler + ASGs
Discussion:
Cluster Autoscaler grows and shrinks node groups when Pods are Pending for lack of capacity, or nodes are underused.
Karpenter (AWS-born, now also on Azure) skips node groups. It picks an instance type that fits the pending Pods.
Test it by scaling a Deployment past what the current nodes can hold, then kubectl get nodes -w.
Gotchas:
Scale-down evicts Pods, so protect critical apps with a PodDisruptionBudget.
Pods with local storage or safe-to-evict: "false" block scale-down.
New nodes take minutes, so HPA plus slack capacity beats relying on node scaling for sudden spikes.
Watch your cloud quota for max-nodes.
Changed since the book:
The AWS recipe used a Helm 2 chart (Tiller). Today it’s Karpenter, or the Cluster Autoscaler chart via Helm 3. GKE’s gcloud beta is no longer needed.
Problem: Automatically add and remove replicas with load.
Solution:
Make sure metrics-server is running (kubectl top pods works). On minikube: minikube addons enable metrics-server.
Give the target CPU requests.
Create the HPA.
kubectl create deployment appserver --image=registry.k8s.io/hpa-example --port=80kubectl set resources deployment appserver --requests=cpu=200mkubectl expose deployment appserver --port=80kubectl autoscale deployment appserver --cpu-percent=40 --min=1 --max=5kubectl get hpa -w# load generator in another terminalkubectl run -it --rm loadgen --image=busybox:1.36 --restart=Never -- \ sh -c "while true; do wget -q -O- http://appserver; done"
Same thing declaratively, with autoscaling/v2:
apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: appserverspec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: appserver minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 40 # % of the CPU *request* behavior: scaleDown: stabilizationWindowSeconds: 300 # wait 5 min before shrinking (avoid flapping)
Discussion: Roughly, desired = ceil(current × currentMetric / target). The controller checks every ~15 s. autoscaling/v2 also supports memory, custom metrics (Prometheus Adapter: requests/s, queue depth) and external metrics. KEDA extends this to event sources (Kafka lag, SQS depth, cron) and can scale to zero.
Gotchas:
TARGETS <unknown> means no metrics-server, or no CPU request on the containers.
CPU is often a poor proxy for load. App-level metrics (RPS, latency, queue length) scale better, as the book also hints.
Changed since the book:
Heapster is gone → metrics-server (the metrics.k8s.io API). HPA API is autoscaling/v2 (GA 1.23) with a behavior block. kubectl run --requests --expose no longer exists. The hpa-example image moved to registry.k8s.io.
What: Watches real usage and recommends (or applies) better requests/limits. The book skipped it (“still under development, Jan 2018”). It’s now a mature add-on (not built in, so install it from kubernetes/autoscaler).
Gotchas: Don’t let VPA (in Auto mode) and an HPA both act on CPU/memory for the same workload, because they fight. A common combo is VPA Off (for sizing advice) + HPA (for scaling).
In-place Pod resize (resizePolicy, beta and on by default since 1.33) lets requests and limits change without restarting the Pod, so VPA no longer has to evict.
Chapter 10 of the Kubernetes Cookbook (ServiceAccounts, RBAC, security contexts), plus Recipe 14.6 (encrypted secrets in Git) and the modern pieces the book predates: Pod Security Admission, NetworkPolicy and admission policies.
Authentication: who are you? A user (via OIDC, cloud IAM or client certs; Kubernetes has no User object) or a ServiceAccount (an app identity, namespaced).
Authorization: are you allowed? RBAC.
Admission: is this object acceptable, and should it be tweaked? Pod Security Admission, ValidatingAdmissionPolicy, webhooks (Kyverno, Gatekeeper).
Recipes
Recipe 10.1 — Give an App Its Own Identity (ServiceAccount)
Problem: An app talks to the Kubernetes API and needs a fine-grained identity.
Solution:
kubectl create serviceaccount myappsakubectl create token myappsa --duration=1h # mint a short-lived token by hand
apiVersion: v1kind: Podmetadata: name: myappspec: serviceAccountName: myappsa automountServiceAccountToken: true # set false for apps that never call the API containers: - name: main image: busybox:1.36 command: ["sleep", "3600"]
Discussion: The token is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. The identity string is system:serviceaccount:<ns>:<name>. Every namespace has a default SA, which Pods get if you don’t specify one.
Gotchas: Most apps never call the Kubernetes API. Turn the token off for them (automountServiceAccountToken: false), because a leaked token is a foothold.
Changed since the book:
Tokens are now bound, projected, short-lived (auto-rotated, about 1h, tied to the Pod). Since 1.24 no long-lived token Secret is created per ServiceAccount, so the book’s kubectl describe secret myappsa-token-… step no longer exists. Use kubectl create token. Cloud workload identity (GKE Workload Identity, EKS Pod Identity/IRSA, AKS Workload Identity) maps SAs to cloud IAM.
Source: K8s Cookbook, Recipe 10.1
Recipe 10.2 — See What Access You Have
Problem: Can I (or this ServiceAccount) do X?
Solution:
kubectl auth can-i create deployments -n team-akubectl auth can-i list secrets --as=system:serviceaccount:sec:myappsa -n seckubectl auth can-i --list -n seckubectl auth whoami # who does the API server think I am?kubectl get roles,rolebindings -n seckubectl get clusterroles | grep -v system: # admin, edit, view, cluster-adminkubectl describe clusterrole view
Discussion: RBAC’s moving parts:
Subject: a user, group or ServiceAccount.
Role / ClusterRole: a list of verbs on resources (get list watch create update patch delete).
RoleBinding / ClusterRoleBinding: glue a role to subjects.
Gotchas: The book’s tip of binding cluster-admin to kube-system:default to make the dashboard work is a terrible idea. It gives every Pod using that SA full control of the cluster. Never do it.
Changed since the book:
RBAC is on by default everywhere (minikube included, no --extra-config needed). kubectl auth whoami is new (GA 1.28).
Source: K8s Cookbook, Recipe 10.2
Recipe 10.3 — Control Access to Resources (Role + RoleBinding)
Problem: Let an app read Pods in its namespace and nothing else.
Discussion: Three profiles: privileged (anything goes, for system namespaces), baseline (blocks the known escalations: hostPath, privileged, hostNetwork…) and restricted (the hardened list above). Try a label in dry-run first: kubectl label --dry-run=server --overwrite ns payments pod-security.kubernetes.io/enforce=restricted prints which existing Pods would violate it.
Gotchas: Many public images run as root and break under restricted. Use non-root images (distroless, -nonroot tags) or set runAsUser. PSA only covers Pods. For custom rules (“images must come from our registry”, “every Deployment needs an owner label”) use ValidatingAdmissionPolicy (CEL, built in, GA 1.30), Kyverno or OPA Gatekeeper.
Changed since the book:
PodSecurityPolicy (PSP) was removed in 1.25. The book’s PSP pointer (and the Best Practices book’s whole PSP section) is obsolete, and Pod Security Admission is the replacement. seccompProfile became a proper field (it used to be an annotation).
Problem: By default every Pod can talk to every Pod. You want default-deny plus explicit allows.
Solution:
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: default-deny-ingress namespace: paymentsspec: podSelector: {} # all Pods in the namespace policyTypes: ["Ingress"]---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: allow-frontend-to-api namespace: paymentsspec: podSelector: matchLabels: app: api policyTypes: ["Ingress"] ingress: - from: - podSelector: matchLabels: app: frontend - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress # auto-label on every namespace ports: - protocol: TCP port: 8080
Gotchas: Policies are only enforced if your CNI supports them (Calico, Cilium yes, plain Flannel no). Rules are additive, with no “deny” rules. If you add egress default-deny, remember to allow DNS (UDP/TCP 53 to kube-system) or everything breaks mysteriously. Start with ingress rules only (Best Practices book, Ch. 9).
Recipe 14.6 — Encrypted Secrets in Git
Problem: You want everything in Git (GitOps), including Secrets, without leaking them.
Solution options:
Sealed Secrets: encrypt with the controller’s public key. Only the in-cluster controller can decrypt.
kubectl create secret generic db --from-literal=password=s3cr3t \ --dry-run=client -o yaml | kubeseal --format yaml > db-sealed.yamlkubectl apply -f db-sealed.yaml # controller creates the real Secret
External Secrets Operator: Secrets live in AWS Secrets Manager, GCP Secret Manager, Azure Key Vault or HashiCorp Vault. An ExternalSecret CR syncs them in, so nothing secret is in Git at all.
SOPS (with age or KMS keys): encrypt values in YAML files. Flux and Argo CD plugins decrypt at deploy time.
Secrets Store CSI Driver: mount secrets from the vault as files without creating a Secret object.
Gotchas: A SealedSecret is bound to its namespace and name by default. Back up the controller’s private key, or you can’t decrypt after a cluster rebuild.
Changed since the book:
--dry-run → --dry-run=client, kubeseal --format yaml, and install via Helm. The v1alpha1 sealed-secrets API is current but the project is much more mature. ESO didn’t exist in 2018.
Identity: ServiceAccounts with short-lived projected tokens (kubectl create token). Disable automount when an app doesn’t need the API.
Authorization: RBAC v1, least privilege, and check with kubectl auth can-i. Never hand out cluster-admin casually.
Pod hardening: runAsNonRoot, drop ALL capabilities, no privilege escalation, RuntimeDefault seccomp. Enforce with Pod Security Admission labels (PSP is gone).
Network: default-deny NetworkPolicies + explicit allows (needs a policy-capable CNI).
Secrets in Git: Sealed Secrets / External Secrets / SOPS, never plain base64.
Chapter 14 (the “Ecosystem”) of the Kubernetes Cookbook: Helm (Recipes 14.1–14.3) and kompose (14.4). Plus what’s become standard since: Kustomize (built into kubectl), server-side apply and GitOps.
The problem all of these solve: raw YAML copied per environment (dev/staging/prod) drifts and duplicates. You want one source with small, controlled differences.
Installing third-party software, distributing your app
Per-environment tweaks of your own manifests
They combine fine: Argo CD and Flux can render a Helm chart and then apply Kustomize patches on top.
Recipes
Recipe 14.1 — Install Helm
Problem: You don’t want to hand-write manifests for common software.
Solution:
brew install helmhelm version
Discussion: A chart = templated manifests + default values + metadata. Installing one creates a release. Helm 3 stores release state as Secrets in the release’s namespace and uses your kubeconfig permissions.
Changed since the book:
Helm 3 has no Tiller. There’s no helm init, and no server component running with cluster-admin in kube-system (that was a big security hole). The Best Practices book’s “one Tiller per namespace” workaround is obsolete too.
helm upgrade with a missing -f silently resets values to defaults. Keep your values files in Git.
Uninstall keeps PVCs created by StatefulSets.
Changed since the book:
The release name is now required (helm install <name> <chart>, or --generate-name). helm search is now helm search repo / helm search hub. The old stable repo at storage.googleapis.com/kubernetes-charts is dead, so find charts on Artifact Hub. Charts can live in OCI registries.
Source: K8s Cookbook, Recipe 14.2
Recipe 14.3 — Create Your Own Chart
Problem: Package your multi-manifest app so it’s installable with parameters.
Solution:
helm create mychart # Chart.yaml, values.yaml, templates/, charts/helm lint mycharthelm template demo mychart -f values-prod.yaml | less # render locally: no cluster neededhelm install demo ./mychart --dry-run=server # validate against the clusterhelm package mychart # → mychart-0.1.0.tgzhelm push mychart-0.1.0.tgz oci://registry.example.com/charts
Discussion: Bump version (the chart’s version) and appVersion (your app’s version) in Chart.yaml separately. Bundle components that must upgrade and roll back together into one chart (Best Practices, Ch. 6). Hooks (helm.sh/hook: pre-upgrade) run Jobs such as DB backups or migrations at the right moment.
Gotchas: Template files aren’t valid YAML until rendered, so lint with helm lint / helm template, not a YAML linter.
Changed since the book:
No more ~/.helm/repository/local or helm serve. Share charts via OCI registries (GHCR, ECR, ACR, Harbor) or a static chart repo.
kubectl kustomize overlays/prod # render and inspectkubectl diff -k overlays/prod # compare with the live clusterkubectl apply -k overlays/prod
Discussion: configMapGenerator adds a content hash to the ConfigMap name (web-config-7f9k2…) and rewrites references. So changing config automatically rolls the Deployment, the trick the Best Practices book recommends for ConfigMap changes.
Gotchas: kubectl’s bundled kustomize can lag behind the standalone kustomize CLI. Check kubectl version (“Kustomize Version”).
Modern Extra — kubectl apply --server-side
What: The API server (not kubectl) merges your manifest and tracks which manager owns which field (managedFields).
Why:
No more giant last-applied-configuration annotation (which breaks on objects over 256 KB, like big CRDs).
Conflicts are explicit: if the HPA owns replicas and you try to set it, you get an error instead of a silent fight.
Several tools can safely manage different fields of the same object.
Discussion: kompose is a starting point, not a finished product. Review the result and add probes, resources, Secrets, PVCs and an Ingress by hand. Volumes and networks rarely translate cleanly.
Gotchas: Compose’s depends_on has no direct equivalent. Use readiness probes, init containers or retry logic (see Creating & Modifying Workloads).
Changed since the book:
Prefer kompose convert → review → commit → kubectl apply over kompose up, so the result lives in Git. Modern Compose files no longer need the version: key.
Source: K8s Cookbook, Recipe 14.4
Modern Extra — GitOps (the destination)
Idea: Git is the source of truth. A controller in the cluster (Argo CD or Flux) continuously pulls and applies it, and shows or undoes drift. CI builds and pushes images and bumps a tag in Git. CD = the GitOps controller syncing.
Why: auditable (every change is a commit), easy rollback (git revert), no cluster credentials in CI, and consistent multi-cluster rollouts (Best Practices, Ch. 5 & 12).
Summary
Helm 3 (no Tiller) to install and package apps: repo add → install/upgrade --install → history/rollback. Pin versions and keep values in Git.
Kustomize (kubectl apply -k) for per-environment overlays of plain YAML. Hashed configMapGenerator gives automatic rollouts on config change.
Server-side apply for explicit field ownership and conflict detection.
kompose gives you a first draft from Compose. GitOps (Argo CD/Flux) is where it all ends up.
Chapter 12 of the Kubernetes Cookbook: kubectl autocompletion, parsing status, debugging Pods, cluster snapshots, adding nodes, drain/cordon, and etcd. (Recipes 12.2 and 12.3 are service tricks and live in Services & Ingress.)
Plus the modern debugging toolkit the book predates: kubectl debug, ephemeral containers and kubectl events.
Gotchas: Bash needs the bash-completion package first. Completion also completes resource names from the live cluster.
Source: K8s Cookbook, Recipe 12.1
Recipe 12.4 — Parse Resource Status (for scripts and CI)
Problem: React to an object’s state in a script or pipeline.
Solution:
kubectl get pod jump -o jsonpath='{.status.qosClass}' # Guaranteed | Burstable | BestEffortkubectl get pod jump -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}'kubectl get deploy web -o json | jq '.status'# Better than polling: block until a condition is truekubectl wait --for=condition=Available deploy/web --timeout=120skubectl wait --for=jsonpath='{.status.phase}'=Running pod/jumpkubectl rollout status deploy/web --timeout=2m # non-zero exit if it fails → fails the CI step
Discussion: Built-in JSONPath or go-template output means you don’t need jq, but jq is nicer for complex queries.
QoS classes decide who gets evicted first under memory pressure:
BestEffort: no requests or limits. Evicted first.
Burstable: some requests or limits.
Guaranteed: requests = limits for CPU and memory. Evicted last.
Source: K8s Cookbook, Recipe 12.4
Recipe 12.5 — Debug Pods
Problem: A Pod won’t start, or keeps dying.
Solution: Run an OODA loop. Observe, Orient (list hypotheses), Decide (pick one), Act (test it), repeat. Observe in this order:
kubectl get pod web-abc -o wide # STATUS, RESTARTS, NODEkubectl describe pod web-abc # Events at the bottom ← start herekubectl events --for pod/web-abc # (or) kubectl get events --sort-by=.lastTimestampkubectl logs web-abc --previous # why did the last container die?kubectl get pod web-abc -o jsonpath='{.status.containerStatuses[*].lastState}'
Read the status:
Status
Usual cause
Look at
Pending
No node fits (requests too big, taints, PVC unbound, quota)
describe → Events (FailedScheduling)
ImagePullBackOff / ErrImagePull
Typo in image/tag, private registry without imagePullSecrets, rate limit
describe → Events
CrashLoopBackOff
App exits on start: bad config, missing env/Secret, liveness too aggressive
logs --previous, exit code
OOMKilled (exit 137)
Memory limit too low or a leak
lastState.terminated.reason, kubectl top
CreateContainerConfigError
Referenced ConfigMap/Secret or key missing
describe
Running but 0/1 Ready
Readiness probe failing
describe → probe events
Terminating forever
Finalizers, or node unreachable
get -o yaml → metadata.finalizers
Get inside: use kubectl debug. It attaches an ephemeral debug container that works even for distroless images with no shell:
kubectl exec -it web-abc -- sh # if the image has a shellkubectl debug -it web-abc --image=busybox:1.36 --target=web # share the app container's process namespacekubectl debug web-abc -it --copy-to=web-debug --container=web -- sh # debug a *copy* (e.g. CrashLoop) with a shellkubectl debug node/worker-1 -it --image=ubuntu # a shell on the node (host fs at /host)kubectl run tmp --rm -it --image=nicolaka/netshoot --restart=Never -- bash # network toolbox Pod
Pre-flight checklist (from the book, still true):
Is the manifest valid? (kubectl apply --dry-run=server)
Are there enough resources, and are your requests and limits sane?
Changed since the book:
kubectl debug and ephemeral containers (GA 1.25) replace “install curl into the running container”. kubectl events (GA 1.28) gives sorted, filterable events. Validate against your real cluster with --dry-run=server. The book pointed to a JSON schema site.
Source: K8s Cookbook, Recipe 12.5
Recipe 12.6 — Snapshot the Whole Cluster State
Problem: Capture everything for an audit, a post-mortem, or a support ticket.
Discussion: You get objects as JSON plus Pod logs, per namespace. For restorable backups (objects + volumes) use Velero. For “what’s in Git vs. live”, GitOps tools show drift.
Gotchas: The dump contains Secrets and logs, so treat it as sensitive.
Source: K8s Cookbook, Recipe 12.6
Recipe 12.7 — Add Worker Nodes
Problem: The cluster needs more capacity.
Solution: Provision a machine and install the three node components:
kubelet: the node agent. It registers the node, runs and probes Pods, mounts volumes, and reports status.
A container runtime speaking CRI: containerd or CRI-O.
kube-proxy: programs Service routing. It’s optional if your CNI replaces it, e.g. Cilium.
Discussion: A Node object is only a record. The node counts once its kubelet reports healthy (kubectl get nodes → Ready). Debug runtimes with crictl on the node, since docker ps won’t show Kubernetes containers anymore.
Changed since the book:
The Docker Engine runtime was removed in 1.24 (dockershim), so containerd or CRI-O it is.
Source: K8s Cookbook, Recipe 12.7
Recipe 12.8 — Drain a Node for Maintenance
Problem: Patch, reboot or upgrade a node without dropping traffic.
Solution:
kubectl cordon worker-1 # no new Pods land herekubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data --timeout=5m# … do the maintenance …kubectl uncordon worker-1 # schedulable again
Protect apps during drains with a PodDisruptionBudget:
apiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: web-pdbspec: minAvailable: 2 # or maxUnavailable: 1 selector: matchLabels: app: web
Discussion: drain = cordon + evict every Pod, respecting PDBs and graceful termination. DaemonSet Pods stay. Bare Pods (no controller) need --force and are gone forever.
Gotchas: A PDB with minAvailable equal to the replica count (e.g. 1 of 1) blocks drains forever. Run ≥ 2 replicas for anything you drain around.
Changed since the book:
--delete-local-data → --delete-emptydir-data. PDBs are policy/v1 (GA 1.21).
Problem: Back up (or inspect) the cluster’s source of truth.
Solution (self-managed / kubeadm control plane):
ETCDCTL_API=3 etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /backup/etcd-$(date +%F).dbetcdutl snapshot status /backup/etcd-2026-01-01.db -w table
Discussion: Only the API server talks to etcd. Everything else goes through the API. Keys look like /registry/<resource>/<ns>/<name>, stored as protobuf (not readable JSON). Back up regularly and test restores (etcdutl snapshot restore).
Gotchas: On managed services (GKE, EKS, AKS) etcd is hidden and backed up for you. Back up your objects with Velero or Git instead.
Changed since the book:
The book’s curl …:2379/v2/keys uses the etcd v2 API, which is gone. Kubernetes uses etcd v3 only, over TLS. Snapshot status/restore moved from etcdctl to etcdutl.
Chapter 13 of the Kubernetes Cookbook (building Kubernetes, the Python client, CRDs), plus Recipes 14.5 (kubicorn) and 14.7 (kubeless) from the Ecosystem chapter. Then the modern picture: operators, admission control and platforms (Best Practices Ch. 15–17).
Big idea: Kubernetes is a database of desired state + control loops. Extending it = adding new kinds of desired state (CRDs) + new control loops (controllers/operators).
Problem: You want your own binaries (to patch a bug or test a PR).
Solution:
git clone https://github.com/kubernetes/kubernetes && cd kubernetesmake WHAT=cmd/kubectl # one component → _output/bin/kubectlmake # everything (needs Go at the version in .go-version)make quick-release # containerised build (needs Docker with ≥ 8 GB RAM)
Discussion: Few people need this. It’s for contributors. To test a local build, kind build node-image packages your source into a kind node image.
Changed since the book:
hyperkube is gone, so make hyperkube no longer exists. Per-component targets use make WHAT=cmd/<component>.
Source: K8s Cookbook, Recipes 13.1 & 13.2
Recipe 13.3 — Talk to the API from Python
Problem: Script against the cluster in Python.
Solution:
from kubernetes import client, configconfig.load_kube_config() # in-cluster: config.load_incluster_config()core = client.CoreV1Api()apps = client.AppsV1Api()for pod in core.list_pod_for_all_namespaces().items: print(pod.metadata.namespace, pod.metadata.name, pod.status.phase)apps.patch_namespaced_deployment_scale( "web", "default", {"spec": {"replicas": 3}})
Discussion: The client is generated from the OpenAPI spec, with one class per API group/version: CoreV1Api for core, AppsV1Api for Deployments, BatchV1Api, NetworkingV1Api, CustomObjectsApi for CRDs. Its version tracks Kubernetes minors.
Gotchas: In a Pod, the client uses the Pod’s ServiceAccount, so give it a minimal Role (Security). Use watch.Watch() instead of polling. Go’s client-go is the reference client.
Changed since the book:
Deployments are AppsV1Api, not ExtensionsV1beta1Api, and the module is no longer “incubator”.
Source: K8s Cookbook, Recipe 13.3
Recipe 13.4 — Extend the API with a CustomResourceDefinition
Problem: None of the built-in kinds fit your workload, and you want kubectl get functions to just work.
Solution:
apiVersion: apiextensions.k8s.io/v1kind: CustomResourceDefinitionmetadata: name: functions.example.com # must be <plural>.<group>spec: group: example.com scope: Namespaced names: kind: Function plural: functions singular: function shortNames: ["fn"] versions: - name: v1 served: true storage: true schema: # a structural schema is REQUIRED in v1 openAPIV3Schema: type: object properties: spec: type: object required: ["code"] properties: code: type: string ram: type: string default: "128Mi" # defaulting is supported additionalPrinterColumns: - name: RAM type: string jsonPath: .spec.ram subresources: status: {}
Discussion: A CRD alone just stores data in etcd. To make something happen, you need a controller that watches Function objects and reconciles real resources (Deployments, Services) to match. Controller + CRD + domain knowledge = an operator.
Gotchas:
Deleting a CRD deletes every instance of it (cascade).
Changing a CRD’s schema across versions needs a conversion webhook.
Keep CRD installs separate from the CRs that use them (CRDs first).
Changed since the book:
apiextensions.k8s.io/v1beta1 was removed in 1.22, so use v1. The book’s example also had bugs: the name must be functions.example.com, and scope belongs under spec, not names. Every limitation the book lists is now solved: multiple versions with conversion webhooks, defaulting, schema validation, the status/scale subresources, plus CEL validation rules (x-kubernetes-validations).
Source: K8s Cookbook, Recipe 13.4
Modern Extra — Operators
What: A controller that encodes an ops runbook (install, upgrade, back up, fail over) for one piece of software, driven by its CRDs.
Solution today: eksctl (EKS), Terraform/OpenTofu modules, Crossplane, or Cluster API (Kubernetes-style CRDs that create clusters on any cloud). kops is still around for self-managed AWS clusters.
Changed since the book:
kubicorn was abandoned and its repo archived. Cluster API carries its “clusters as Kubernetes objects” idea forward.
Source: K8s Cookbook, Recipe 14.5
Recipe 14.7 — Functions on Kubernetes (was: kubeless)
Problem: Deploy functions (HTTP- or event-triggered) without writing Deployments.
Solution today: Knative Serving (request-driven, scale to zero, revisions and traffic splitting) + Knative Eventing. Or OpenFaaS. For “scale my normal Deployment on events, down to zero”, use KEDA (see Scaling).
Changed since the book:
kubeless was archived (2021), and its Python 2.7 runtime is long dead. It was a nice example of CRD + controller, though: a Function CR got turned into a ConfigMap (code) + Deployment + Service. That’s exactly the operator pattern above.
Source: K8s Cookbook, Recipe 14.7
Summary
Kubernetes = desired-state store + control loops. Extend with CRDs (apiextensions.k8s.io/v1, structural schema) + controllers = operators.
Script with the official clients (Python: CoreV1Api, AppsV1Api, CustomObjectsApi). Build with Kubebuilder or Operator SDK.
Enforce rules at admission time: ValidatingAdmissionPolicy (CEL) first, webhooks (Kyverno, Gatekeeper) when you need more.