Kaushik's Notes

Monitoring & Logging

6 min read

Overview

  • Chapter 11 of the Kubernetes Cookbook: container logs, liveness/readiness probes, resource metrics, Prometheus and centralised logging. Plus the Best Practices book’s advice on what to monitor and how to alert.
  • Related: Deployments (CrashLoopBackOff is often a liveness probe killing a slow starter), Maintenance & Troubleshooting. Back to the hub: Kubernetes Cookbook

Who Cares About What

  • Admins: Are nodes healthy? Is the cluster full? Are teams near quota?
  • Developers: Is my app healthy? Are requests and limits right? Why did it restart?

Recipes

Recipe 11.1 — Read a Container’s Logs

  • Problem: See what the app printed.
  • Solution:
kubectl logs pod/web-abc                      # single-container Pod
kubectl logs web-abc -c sidecar               # pick a container
kubectl logs deploy/web                       # some Pod of the Deployment
kubectl logs -l app=web --all-containers --prefix --max-log-requests=10
kubectl logs web-abc -f --since=10m --timestamps
kubectl logs web-abc --previous               # the crashed container before the restart ← gold for CrashLoopBackOff
  • Discussion: Kubernetes only keeps what the container writes to stdout/stderr, rotated on the node. When the Pod is deleted, its logs go too, which is why you want central logging (11.7).
  • Gotchas: For multi-Pod tailing use stern (stern web) or kubectl logs -l … -f.

Source: K8s Cookbook, Recipe 11.1

Recipe 11.2 – 11.4 — Liveness, Readiness (and Startup) Probes

  • Problem: Restart wedged apps automatically, and only send traffic to Pods that are actually ready.
  • Solution:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webserver
  template:
    metadata:
      labels:
        app: webserver
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          startupProbe:              # gives slow starters time; other probes wait for it
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
            failureThreshold: 30     # up to 150 s to start
          readinessProbe:            # fail → removed from Service endpoints (no restart)
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:             # fail → container is restarted
            httpGet:
              path: /
              port: 80
            periodSeconds: 10
            failureThreshold: 3
            timeoutSeconds: 2
  • The three probes:
ProbeQuestionOn failure
startupProbeHas it finished booting?Keep waiting, then restart if it never starts. Other probes are paused until it passes.
readinessProbeCan it take traffic right now?Pulled out of Service endpoints. Not restarted.
livenessProbeIs it hopelessly stuck?Container restarted by the kubelet.
  • Probe mechanisms: httpGet, tcpSocket, exec (a command’s exit code), grpc (GA 1.27).
  • Discussion: Readiness makes rolling updates and scaling safe. Liveness rescues deadlocks. The kubelet runs the probes, and a Pod is Ready only when all its containers are Ready.
  • Gotchas:
    • Liveness and readiness should usually differ. Readiness can check dependencies (a DB). Liveness should check only the process itself. Otherwise a DB blip restarts every Pod at once, a self-inflicted outage.
    • Too-aggressive liveness on a slow-booting app = endless restarts (CrashLoopBackOff from Deployments). Use a startupProbe instead of a huge initialDelaySeconds.
    • The book’s snippet says initialDelaySecond. The real field is initialDelaySeconds.

Changed since the book:

startupProbe (GA 1.20) and grpc probes (GA 1.27) were added, and extensions/v1beta1 → apps/v1.

Source: K8s Cookbook, Recipes 11.2, 11.3 & 11.4 · Best Practices book, Ch. 1

Recipe 11.5 — Resource Metrics with kubectl top

  • Problem: kubectl top errors out. You want CPU and memory usage per Pod and node.
  • Solution:
minikube addons enable metrics-server          # minikube
# elsewhere:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# (kind/local: add --kubelet-insecure-tls to the metrics-server args)
 
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl top pod web-abc --containers
  • Discussion: metrics-server keeps only current usage in memory (no history) and serves the metrics.k8s.io API that HPA and kubectl top use. For history and dashboards you need Prometheus (11.6).

Changed since the book:

Heapster + InfluxDB + Grafana add-on → metrics-server. Heapster was retired in 2018 and removed from everything.

Source: K8s Cookbook, Recipe 11.5

Recipe 11.6 — Prometheus for Cluster and App Metrics

  • Problem: Query and alert on system and app metrics in one place.
  • Solution: Don’t hand-write the ConfigMap, RBAC and Deployment like the book did. Install the kube-prometheus-stack chart: Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, plus ready-made dashboards and alerts.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install mon prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
kubectl port-forward -n monitoring svc/mon-grafana 3000:80
# Tell the Prometheus Operator to scrape your app's /metrics via its Service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  labels:
    release: mon            # must match the operator's selector
spec:
  selector:
    matchLabels:
      app: web
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
  • Metric sources:
    • cAdvisor (inside the kubelet): container CPU and memory.
    • kube-state-metrics: object state, e.g. “replicas desired vs available”, “Pod restarts”.
    • node-exporter: node OS metrics.
    • your app: instrument it with a Prometheus client or OpenTelemetry.
  • What to watch (Best Practices Ch. 3):
    • USE for infrastructure: Utilisation, Saturation, Errors.
    • RED for services: Rate, Errors, Duration.
    • The four golden signals: latency, traffic, errors, saturation.
  • Gotchas: Watch label cardinality. A label like user_id explodes Prometheus’s memory.

Changed since the book:

The book’s config targeted 1.7-specific kubelet paths and bound Prometheus to cluster-admin. Use the operator chart’s scoped RBAC instead. Managed Prometheus (GKE, Amazon AMP, Azure Monitor) is a common alternative.

Source: K8s Cookbook, Recipe 11.6 · Best Practices book, Ch. 3

Recipe 11.7 — Centralised Logging

  • Problem: Search logs from all Pods, including dead ones, in one place.
  • Solution (modern stack): a node-level agent as a DaemonSet (Fluent Bit, Vector or the OpenTelemetry Collector) reads /var/log/containers/* and ships to a backend: Loki + Grafana (cheap, label-indexed), OpenSearch/Elasticsearch + Dashboards/Kibana (full-text), or a cloud service.
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki -n logging --create-namespace      # configure for single-binary/dev
helm install alloy grafana/alloy -n logging                       # or fluent-bit / promtail-style agent
  • Best Practices:
    • Log to stdout in a structured format (JSON).
    • Prefer the DaemonSet agent over a log-shipper sidecar in every Pod (cheaper).
    • Keep hot logs about 30–45 days and archive older ones to cheap storage.
    • Correlate logs with metrics and traces (OpenTelemetry).
  • Alerting: Alert on symptoms that hurt users or SLOs, not on every blip, because alert fatigue makes people ignore pages.

Changed since the book:

EFK on minikube with Kibana’s default kibana/changeme login is purely historical. Fluentd → Fluent Bit is the common agent today, and Loki is popular with Grafana. Everything installs via Helm.

Source: K8s Cookbook, Recipe 11.7 · Best Practices book, Ch. 3

Summary

  • kubectl logs (with -c, -l, -f, --previous) for one-off looks. A DaemonSet agent → Loki/OpenSearch for history.
  • startup = booted? readiness = take traffic? liveness = restart me? Keep liveness simple and dependency-free.
  • metrics-server powers kubectl top and HPA. Prometheus (kube-prometheus-stack) for history, dashboards and alerts.
  • Monitor USE/RED/golden signals, and alert on user-facing symptoms.

Graph View

Backlinks