Kaushik's Notes

Volumes & Configuration

6 min read

Overview

  • Chapter 8 of the Kubernetes Cookbook: sharing files between containers, feeding config and secrets into Pods, and storage that outlives the Pod (PV, PVC, StorageClass).
  • My basics: ConfigMap (decouple config from images, and ConfigMaps aren’t secure). Remember from Pods: anything written inside a container dies with it.
  • Back to the hub: Kubernetes Cookbook

Volume Types at a Glance

TypeLives as long asUse it for
emptyDirthe Podscratch space, sharing files between containers
configMap / secret / projectedthe Pod (data from API objects)config files, credentials
hostPaththe nodenode agents only (log shippers). Avoid for apps.
persistentVolumeClaimindependently of Podsdatabases, uploads, anything you can’t lose
ephemeral (generic)the Pod, but provisioned by a CSI driverbig scratch disks

Changed since the book:

Storage is now done by CSI drivers. The in-tree cloud plugins (awsElasticBlockStore, gcePersistentDisk, azureDisk…) were migrated or removed. gitRepo volumes are deprecated (use an init container that clones instead).

Recipes

Recipe 8.1 — Share Data Between Containers (emptyDir)

  • Problem: Two containers in one Pod need to exchange files.
  • Solution:
apiVersion: v1
kind: Pod
metadata:
  name: sharevol
spec:
  containers:
    - name: c1
      image: busybox:1.36
      command: ["sh", "-c", "echo hello > /xchange/data; sleep 3600"]
      volumeMounts:
        - name: xchange
          mountPath: /xchange
    - name: c2
      image: busybox:1.36
      command: ["sh", "-c", "sleep 5; cat /data/data; sleep 3600"]
      volumeMounts:
        - name: xchange
          mountPath: /data
  volumes:
    - name: xchange
      emptyDir: {}              # add medium: Memory for a tmpfs
  • Discussion: This is the basis of the sidecar/adapter patterns. One container writes, another ships or transforms.
  • Gotchas: Data is gone when the Pod is deleted or moves node. Set sizeLimit so a runaway writer can’t fill the node’s disk.

Source: K8s Cookbook, Recipe 8.1

Recipe 8.2 — Pass an API Key with a Secret

  • Problem: Give an app a credential without putting it in the image or plain YAML.
  • Solution:
kubectl create secret generic pp --from-literal=passphrase='open sesame'
kubectl get secret pp -o jsonpath='{.data.passphrase}' | base64 -d   # it's only base64!
apiVersion: v1
kind: Pod
metadata:
  name: ppconsumer
spec:
  containers:
    - name: shell
      image: busybox:1.36
      command: ["sh", "-c", "cat /tmp/access/passphrase; sleep 3600"]
      volumeMounts:
        - name: passphrase
          mountPath: /tmp/access
          readOnly: true
      env:
        - name: PASSPHRASE                 # alternative: as an env var
          valueFrom:
            secretKeyRef:
              name: pp
              key: passphrase
  volumes:
    - name: passphrase
      secret:
        secretName: pp
        defaultMode: 0400
  • Discussion: Secret types: generic (Opaque), docker-registry (image pull credentials) and tls (cert + key, used by Ingress). Max size is 1 MiB. Secrets are namespaced.
  • Gotchas: Secrets are base64, not encrypted. Anyone with get secret RBAC, or access to etcd backups, can read them. Do three things:
    1. Turn on encryption at rest (EncryptionConfiguration with a KMS v2 provider). Managed clouds often do this for you.
    2. Lock down RBAC on secrets (list and watch reveal all of them).
    3. Keep real secrets in a vault and sync them in (see Security).
  • Prefer volume mounts to env vars. Env vars leak into crash dumps and kubectl describe, and they don’t update when the Secret changes.

Changed since the book:

The flag is now --encryption-provider-config (no longer “experimental”), with KMS v2 GA. ServiceAccount token Secrets are no longer auto-created (since 1.24). See Security.

Source: K8s Cookbook, Recipe 8.2 · Best Practices book, Ch. 4

Recipe 8.3 — Provide Configuration Data (ConfigMap)

  • Problem: Configure an app without rebuilding the image (ConfigMap).
  • Solution:
kubectl create configmap siseconfig --from-literal=siseversion=0.9
kubectl create configmap configfile --from-file=example.cfg
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cmapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cmapp
  template:
    metadata:
      labels:
        app: cmapp
    spec:
      containers:
        - name: sise
          image: mhausenblas/simpleservice:0.5.0
          envFrom:
            - configMapRef:
                name: siseconfig          # every key becomes an env var
          volumeMounts:
            - name: cfg
              mountPath: /etc/app         # every key becomes a file
      volumes:
        - name: cfg
          configMap:
            name: configfile
  • Discussion:
    • Env vars are read once at start. A ConfigMap change needs a kubectl rollout restart.
    • Volume files refresh automatically (after up to about a minute), so an app that watches its config file can hot-reload.
    • subPath mounts never refresh.
  • Gotchas: The ConfigMap must exist before the Pod starts (or mark it optional: true). Mark stable config immutable: true. That protects it from accidental edits and saves API server watch load.
  • Trick: Kustomize’s configMapGenerator and Helm’s checksum/config annotation change the Pod template whenever config changes. That triggers a rollout automatically (see Managing Manifests).

Changed since the book:

extensions/v1beta1 → apps/v1. Immutable ConfigMaps and Secrets are GA (1.21).

Source: K8s Cookbook, Recipe 8.3 · Best Practices book, Ch. 4

Recipe 8.4 — Use a PersistentVolume (static provisioning)

  • Problem: Data must survive Pod restarts.
  • Solution: The admin creates a PV (the actual disk). The developer creates a PVC (a request for a disk). Kubernetes binds them, and the Pod mounts the PVC.
apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpathpv
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /tmp/pvdata                 # fine for minikube/kind, never in prod
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
spec:
  storageClassName: manual
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 200Mi
# in the Pod template:
      volumes:
        - name: html
          persistentVolumeClaim:
            claimName: mypvc
  • Access modes:
ModeMeaning
ReadWriteOnce (RWO)read-write by one node (several Pods on that node can share it)
ReadWriteOncePod (RWOP)read-write by one Pod only (GA 1.29)
ReadOnlyMany (ROX)read-only by many nodes
ReadWriteMany (RWX)read-write by many nodes. Needs NFS, CephFS, EFS, Azure Files…
  • Gotchas: PVs are cluster-scoped, PVCs are namespaced. A PVC stuck in Pending means no PV or StorageClass matches its size, mode or class (kubectl describe pvc tells you why).

Source: K8s Cookbook, Recipe 8.4

Recipe 8.5 / 8.6 — Dynamic Provisioning with StorageClasses

  • Problem: You don’t want an admin hand-making PVs for every claim.
  • Solution: Just create the PVC. The default StorageClass provisions a matching PV on demand.
kubectl get storageclass          # the one marked (default) is used when none is named
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: pd.csi.storage.gke.io      # ebs.csi.aws.com, disk.csi.azure.com, rancher.io/local-path…
parameters:
  type: pd-ssd
reclaimPolicy: Retain                    # default is Delete!
volumeBindingMode: WaitForFirstConsumer  # create the disk in the zone where the Pod lands
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  storageClassName: fast-ssd
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi
  • Discussion: minikube’s default class (standard, provisioner k8s.io/minikube-hostpath) and kind’s (standard, local-path) are fine for learning. On cloud you get CSI-backed disks. Grow a volume by editing the PVC’s storage (if allowVolumeExpansion). Back up with VolumeSnapshots or Velero.
  • Gotchas:
    • reclaimPolicy: Delete (the default for dynamic PVs) destroys the disk when the PVC is deleted. Use Retain for anything precious.
    • Use WaitForFirstConsumer, or you’ll get “volume in zone A, Pod in zone B” (Best Practices).
    • Block disks are zonal, so a Pod can’t follow its disk to another zone.

Changed since the book:

GKE’s default class now uses the PD CSI driver (standard-rwo), not kubernetes.io/gce-pd. The blog-post walkthrough the book links to is outdated.

Source: K8s Cookbook, Recipes 8.5 & 8.6 · Best Practices book, Ch. 16

Summary

  • emptyDir for scratch and sharing, ConfigMap/Secret for config, PVC for anything that must survive.
  • ConfigMap and Secret as env vars = restart to pick up changes. As volume = auto-refresh (except subPath).
  • Secrets are base64, not encrypted: enable encryption at rest, tighten RBAC, use an external secret store.
  • PV (disk) ↔ PVC (request) ↔ StorageClass (on-demand provisioning). Watch reclaimPolicy and WaitForFirstConsumer.

Graph View

Backlinks