Kaushik's Notes

Best Practices

10 min read

Overview

  • A distillation of Kubernetes Best Practices (Burns, Villalba, Strebel & Evenson, O’Reilly 2019) into checklists, updated for Kubernetes 1.31+. The Cookbook tells you how. This book tells you what good looks like.
  • Each section cites its chapter (“BP Ch. n”). Deeper how-tos are linked to the Cookbook notes.
  • Back to the hub: Kubernetes Cookbook · Basics: Kubernetes, Deployments, Services, ConfigMap

Changed since the book (2019):

  • PodSecurityPolicy (all of Ch. 10’s first half) was removed in 1.25 → Pod Security Admission.
  • Helm 2/Tiller advice is obsolete → Helm 3.
  • PodPresets never left alpha and were removed.
  • KubeFed (federation) was archived → GitOps fleets (Argo CD ApplicationSets, Flux), Cluster API.
  • The TTL controller is GA.
  • Gatekeeper is still around, now joined by Kyverno and built-in ValidatingAdmissionPolicy.

1. Deploying a Service (BP Ch. 1)

  • Deployments for almost everything stateless, Services for a stable address, and an Ingress/Gateway for HTTP with TLS and routing.
  • Images:
    • Build from trusted bases or from scratch/distroless. Supply-chain attacks are real, so sign images (cosign) and scan them.
    • Treat tags as immutable: v1.0.1-bfeda01f (semver + git SHA). Never :latest in prod. Pinning a digest (@sha256:…) is even stricter.
  • Parameterise per environment with Helm or Kustomize (Managing Manifests), not copy-paste.

A “Production-Ready” Deployment, All in One

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/version: "1.4.2"
    app.kubernetes.io/part-of: shop
    app.kubernetes.io/managed-by: argocd
spec:
  replicas: 3                                   # drop this if an HPA owns scaling
  revisionHistoryLimit: 5
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
        app.kubernetes.io/version: "1.4.2"
    spec:
      serviceAccountName: web
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:                # spread across zones
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: web
      terminationGracePeriodSeconds: 30
      containers:
        - name: web
          image: registry.example.com/shop/web:1.4.2-bfeda01f
          ports:
            - name: http
              containerPort: 8080
          envFrom:
            - configMapRef:
                name: web-config
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 256Mi                     # memory limit = request → no OOM surprises from overcommit
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: http
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              sleep:
                seconds: 5                      # let endpoints update before shutdown
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

2. Developer Workflows (BP Ch. 2)

  • Design for three phases: onboarding → developing → testing.
  • One shared dev cluster (namespace per developer) usually beats a cluster each. Give each person their own identity + namespace + ResourceQuota.
  • Reap stale namespaces automatically (TTL labels, cleanup jobs), because developers forget.
  • Provide shared cluster services (logging, monitoring, maybe databases via Helm) for everyone.
  • Inner-loop tools: Skaffold, Tilt, DevSpace, kind load docker-image (Getting Started & Local Clusters).

3. Monitoring, Logging, Alerting (BP Ch. 3)

  • Monitor infrastructure with USE (utilisation, saturation, errors) and services with RED (rate, errors, duration).
  • Black-box checks catch symptoms, white-box instrumentation explains them.
  • Prometheus-style, label-rich time series. Watch cardinality.
  • Logs: stdout + a DaemonSet shipper (not a sidecar per Pod). Keep hot logs ~30–45 days, then archive cheaply.
  • Alert on SLO-impacting symptoms, fight alert fatigue, and keep improving alerts.
  • Details: Monitoring & Logging

4. Configuration, Secrets, RBAC (BP Ch. 4)

  • ConfigMaps/Secrets:
    • Mount as volumes for hot reload (not subPath).
    • Env vars need a restart to pick up changes.
    • Easiest rule: any config change = a new rollout. Use hashed names (Kustomize configMapGenerator) or a Helm checksum/config annotation.
    • Referenced ConfigMaps/Secrets must exist first (or be optional). A missing configMapKeyRef key blocks Pod start, while invalid keys in envFrom are skipped with an InvalidVariableNames event.
  • Secrets:
    • Encrypt at rest (KMS).
    • Keep the source of truth in an external vault (External Secrets, CSI driver).
    • Put imagePullSecrets on the ServiceAccount, not in every Pod spec.
  • RBAC:
    • Most apps need no RBAC at all. Disable token automount for them.
    • Apps that call the API get a dedicated ServiceAccount + least-privilege Role.
    • Humans via OIDC/SSO groups. Just-in-time elevation for SREs.
    • Separate ServiceAccounts for CI/CD.
    • Avoid granting list/watch on Secrets.
  • Details: Volumes & Configuration, Security

5. CI/CD (BP Ch. 5)

  • Fast, reliable builds and tests, because quick feedback = productive devs.
  • Pipelines as code, versioned with the app.
  • Multi-stage builds for small images with a small attack surface.
  • Tags traceable to a commit/build ID. No latest.
  • Start CD with rolling updates, then graduate to blue/green and canary (Argo Rollouts, Flagger, Gateway API weights).
  • Test client-connection draining and DB schema migrations (expand → migrate → contract) as part of rollouts.
  • Test in production carefully: chaos experiments with a small blast radius and good monitoring.
  • Today: CI builds, signs and pushes the image, then bumps Git. GitOps (Argo CD/Flux) deploys (Managing Manifests).

6. Versioning, Releases, Rollouts (BP Ch. 6)

  • Version the app as a whole separately from each container and each Deployment spec, because they change independently.
  • Put release name/number labels on objects that match your CD system’s release IDs, for traceability and easy rollback targeting.
  • Bundle things that must roll back together into one Helm chart/release. Use hooks for ordered steps (e.g. a pre-upgrade DB backup Job).
  • Keep release channels simple: stable / canary / alpha.

7. Global Distribution & Staged Rollouts (BP Ch. 7)

  • Replicate images to registries near every region.
  • Shift testing left (integration and replay tests) so only confident releases start rolling.
  • Roll out canary region first, then low-risk regions, then the rest, with health gates between waves.
  • Write down and rehearse rollback. Nobody remembers runbooks mid-incident.

8. Resource Management (BP Ch. 8)

  • Requests on every container (scheduling + autoscaling depend on them) and memory limits on every container. CPU limits are debated, since they cause throttling. Many teams set CPU requests only.
  • QoS: Guaranteed (req = limit) for critical workloads, Burstable for most, never BestEffort in prod.
  • ResourceQuota per namespace/team + LimitRange defaults (API, Namespaces & Labels).
  • High availability:
    • Pod anti-affinity / topologySpreadConstraints across zones.
    • PodDisruptionBudgets for drains and upgrades.
  • Special hardware: taint GPU or special nodes and add tolerations only where needed. Use nodeSelector/affinity to target them. Node pools for mixed workload shapes.
  • Right-size nodes before production (cost vs. performance).
  • Scaling: start manual until you know the workload profile. HPA for spiky workloads. VPA recommendations for sizing. Cluster Autoscaler/Karpenter with an eye on spin-up time. Details: Scaling

9. Networking & Service Mesh (BP Ch. 9)

  • CNI:
    • Pick for features: NetworkPolicy, eBPF, encryption, multi-cluster. Cilium and Calico are common.
    • Make sure your cloud supports it.
    • Plan Pod CIDR and IP space generously (default max 110 Pods per node).
    • Check that your observability and security tools work with it.
  • Services: keep most ClusterIP. Expose only what must be public, and for HTTP use one Ingress/Gateway with TLS.
  • Standardise on one ingress controller (annotations aren’t portable), run it HA, and scale on metrics. API-gateway-style controllers suit API-heavy traffic.
  • NetworkPolicy: start with ingress rules, add a default-deny per namespace, open explicitly with labels, and keep a permissive test namespace for figuring out flows. Details: Security
  • Service mesh: only adopt for concrete needs (mTLS, retries, traffic shifting, observability). Weigh the operational cost. Sometimes your CNI gives you mTLS already. Today’s lighter options: Istio ambient mode (no sidecars) and Linkerd.

10. Pod & Container Security (BP Ch. 10)

  • Enforce pod hardening with Pod Security Admission (baseline everywhere, restricted where possible) instead of PSP.
  • RuntimeClass sandboxes (gVisor, Kata) for untrusted workloads. Isolation ≠ secure multi-tenancy, since the API is still shared. Mixed runtimes complicate debugging and portability.
  • Runtime threat detection: Falco, Tetragon.

11. Policy & Governance (BP Ch. 11)

  • Use a policy engine for org rules: ValidatingAdmissionPolicy (CEL), Kyverno or OPA Gatekeeper. Examples: allowed registries, required labels, no :latest, required probes and resources.
  • Enforce at the Pod level (the last hop before runtime). The user then sees the error on the ReplicaSet/events, not on kubectl apply. Use warn/audit modes to give early feedback.
  • Scope constraints tightly (kinds, namespaces, labels). Don’t replicate Secrets into the policy engine’s cache.
  • Run policies in audit mode first, then enforce.

12. Multiple Clusters (BP Ch. 12)

  • Use multiple clusters for blast radius, compliance (PCI, HIPAA), hard multi-tenancy, regions, and special workloads (HPC/GPU).
  • Have a data replication strategy before going multi-region. Stateless is easy, data is hard.
  • GitOps for fleet consistency (Argo CD ApplicationSets, Flux). Cluster API for lifecycle. Global load balancing between clusters.
  • Plan cross-cluster discovery/networking (mesh, Cilium ClusterMesh, MCS API) and make CD multi-cluster aware.

13. Integrating External Services (BP Ch. 13)

  • Calling out:
    • Selector-less Service + manual EndpointSlice for fixed IPs.
    • ExternalName (DNS CNAME) for a stable hostname, e.g. a managed DB.
    • A controller that syncs endpoints if addresses change.
  • Exposing in: an internal LoadBalancer (cloud annotation) or NodePort for on-prem consumers.
  • Cluster to cluster: export from one cluster, import as a selector-less Service in the other.

14. Machine Learning Workloads (BP Ch. 14)

  • Taint GPU nodes (nvidia.com/gpu:NoSchedule) and autoscale GPU pools to zero when idle. GPUs are the expensive bottleneck, so keep them saturated.
  • Separate, tainted node pools for ML in mixed clusters. Monitor GPU, CPU, network and storage together.
  • Today: NVIDIA GPU Operator, Kueue for batch/job queueing, Kubeflow/Ray (KubeRay) for training, and Dynamic Resource Allocation (DRA) for flexible accelerators.

15. Building Platforms (BP Ch. 15)

  • Use admission controllers to validate or mutate (inject sidecars, defaults) and kubectl plugins (krew) to extend the UX.
  • Design for users’ growth: an easy platform that traps users is a failure. Keep an escape hatch to plain Kubernetes, and support plain container images and normal Service discovery.

16. State & Stateful Apps (BP Ch. 16)

  • Use volumes mainly to share data between containers (emptyDir). hostPath only for node agents. Logs go to stdout, not a disk.
  • Default StorageClass on, with WaitForFirstConsumer and zone-aware topology (keep the Pod and its disk in the same zone).
  • Ask first: can a managed DB service do this instead? How hard is it to make the app stateless?
  • Back up volumes with CSI VolumeSnapshots or Velero, since Kubernetes doesn’t do it for you. Use Retain reclaim for sensitive or forensic data.
  • StatefulSets:
    • Create the headless Service yourself.
    • Ordinal identity doesn’t always need PVs.
    • On a dead node, Pods sit in Terminating/Unknown. Force-delete only as a last resort, and only when the node can’t come back.
    • Use preStop hooks for graceful leader hand-off.
  • Prefer a mature operator for complex data systems (see Extending Kubernetes).

17. Admission Control & Authorization (BP Ch. 17)

  • Mutating runs before validating. Any rejection rejects the whole request.
  • Don’t let two mutating webhooks edit the same field. Add a validating check after mutation.
  • Choose failurePolicy deliberately, keep webhooks fast (30 s max), scope with namespaceSelector, skip kube-system, lock webhook configs with RBAC, and don’t send Secrets to webhooks.
  • Authorization: use RBAC only. ABAC needs file syncing and API server restarts. Authorization webhooks are risky single points of failure.

Summary

  • Every workload: immutable image tag, requests + memory limit, readiness + liveness (+ startup), labels (app.kubernetes.io/*), non-root restricted securityContext, ≥ 2 replicas spread across zones, a PDB.
  • Every namespace: quota + LimitRange, Pod Security labels, default-deny NetworkPolicy, least-privilege RBAC.
  • Every change: through Git and CI/CD (GitOps), traceable tags/labels, rolling or canary rollouts, rehearsed rollback.
  • Every cluster: metrics + logs + symptom-based alerts, policy engine in audit then enforce, backups of etcd/objects/volumes.

Graph View

Backlinks