Kaushik's Notes

Extending Kubernetes

6 min read

Overview

  • Chapter 13 of the Kubernetes Cookbook (building Kubernetes, the Python client, CRDs), plus Recipes 14.5 (kubicorn) and 14.7 (kubeless) from the Ecosystem chapter. Then the modern picture: operators, admission control and platforms (Best Practices Ch. 15–17).
  • Big idea: Kubernetes is a database of desired state + control loops. Extending it = adding new kinds of desired state (CRDs) + new control loops (controllers/operators).
  • Back to the hub: Kubernetes Cookbook

Recipes

Recipe 13.1 / 13.2 — Build Kubernetes from Source

  • Problem: You want your own binaries (to patch a bug or test a PR).
  • Solution:
git clone https://github.com/kubernetes/kubernetes && cd kubernetes
make WHAT=cmd/kubectl          # one component → _output/bin/kubectl
make                           # everything (needs Go at the version in .go-version)
make quick-release             # containerised build (needs Docker with ≥ 8 GB RAM)
  • Discussion: Few people need this. It’s for contributors. To test a local build, kind build node-image packages your source into a kind node image.

Changed since the book:

hyperkube is gone, so make hyperkube no longer exists. Per-component targets use make WHAT=cmd/<component>.

Source: K8s Cookbook, Recipes 13.1 & 13.2

Recipe 13.3 — Talk to the API from Python

  • Problem: Script against the cluster in Python.
  • Solution:
from kubernetes import client, config
 
config.load_kube_config()            # in-cluster: config.load_incluster_config()
core = client.CoreV1Api()
apps = client.AppsV1Api()
 
for pod in core.list_pod_for_all_namespaces().items:
    print(pod.metadata.namespace, pod.metadata.name, pod.status.phase)
 
apps.patch_namespaced_deployment_scale(
    "web", "default", {"spec": {"replicas": 3}}
)
  • Discussion: The client is generated from the OpenAPI spec, with one class per API group/version: CoreV1Api for core, AppsV1Api for Deployments, BatchV1Api, NetworkingV1Api, CustomObjectsApi for CRDs. Its version tracks Kubernetes minors.
  • Gotchas: In a Pod, the client uses the Pod’s ServiceAccount, so give it a minimal Role (Security). Use watch.Watch() instead of polling. Go’s client-go is the reference client.

Changed since the book:

Deployments are AppsV1Api, not ExtensionsV1beta1Api, and the module is no longer “incubator”.

Source: K8s Cookbook, Recipe 13.3

Recipe 13.4 — Extend the API with a CustomResourceDefinition

  • Problem: None of the built-in kinds fit your workload, and you want kubectl get functions to just work.
  • Solution:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: functions.example.com          # must be <plural>.<group>
spec:
  group: example.com
  scope: Namespaced
  names:
    kind: Function
    plural: functions
    singular: function
    shortNames: ["fn"]
  versions:
    - name: v1
      served: true
      storage: true
      schema:                           # a structural schema is REQUIRED in v1
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["code"]
              properties:
                code:
                  type: string
                ram:
                  type: string
                  default: "128Mi"      # defaulting is supported
      additionalPrinterColumns:
        - name: RAM
          type: string
          jsonPath: .spec.ram
      subresources:
        status: {}
apiVersion: example.com/v1
kind: Function
metadata:
  name: myfaas
spec:
  code: "http://src.example.com/myfaas.js"
  ram: 100Mi
kubectl apply -f functions-crd.yaml
kubectl apply -f myfaas.yaml
kubectl get fn
kubectl explain function.spec
  • Discussion: A CRD alone just stores data in etcd. To make something happen, you need a controller that watches Function objects and reconciles real resources (Deployments, Services) to match. Controller + CRD + domain knowledge = an operator.
  • Gotchas:
    • Deleting a CRD deletes every instance of it (cascade).
    • Changing a CRD’s schema across versions needs a conversion webhook.
    • Keep CRD installs separate from the CRs that use them (CRDs first).

Changed since the book:

apiextensions.k8s.io/v1beta1 was removed in 1.22, so use v1. The book’s example also had bugs: the name must be functions.example.com, and scope belongs under spec, not names. Every limitation the book lists is now solved: multiple versions with conversion webhooks, defaulting, schema validation, the status/scale subresources, plus CEL validation rules (x-kubernetes-validations).

Source: K8s Cookbook, Recipe 13.4

Modern Extra — Operators

  • What: A controller that encodes an ops runbook (install, upgrade, back up, fail over) for one piece of software, driven by its CRDs.
  • Examples: Prometheus Operator, cert-manager, CloudNativePG (Postgres), Strimzi (Kafka), Argo CD, External Secrets.
  • Build your own: Kubebuilder or Operator SDK (Go, controller-runtime). Kopf for Python. Find existing operators on OperatorHub or Artifact Hub.
  • Rules of thumb (Best Practices Ch. 12 & 16):
    • Before running a complex stateful system yourself, check whether a mature operator, or a managed service, exists.
    • Reconcile loops must be idempotent and level-triggered: “make it so”, not “do this once”.

Modern Extra — Admission Control (Best Practices Ch. 17)

  • Mutating webhooks change objects (inject sidecars, add defaults). Validating ones accept or reject them. Mutating always runs before validating.
  • Built-in, no webhook needed:
    • ValidatingAdmissionPolicy (CEL, GA 1.30).
    • MutatingAdmissionPolicy (CEL, newer).
    • Pod Security Admission (Security).
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-team-label
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "has(object.metadata.labels) && 'team' in object.metadata.labels"
      message: "Deployments must carry a 'team' label"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-team-label
spec:
  policyName: require-team-label
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: ["kube-system"]
  • Webhook best practices:
    • Scope webhooks with a namespaceSelector, and exclude kube-system.
    • Choose failurePolicy deliberately: Fail can lock out the whole cluster if the webhook is down, while Ignore silently skips policy.
    • Keep webhooks fast (30 s timeout max), with no external DB lookups.
    • Never have two mutating webhooks touch the same field.
    • Lock webhook configurations down with RBAC, because creating one is effectively root.

Recipe 14.5 — Create Cloud Clusters Declaratively (was: kubicorn)

  • Problem: Create and manage clusters as code.
  • Solution today: eksctl (EKS), Terraform/OpenTofu modules, Crossplane, or Cluster API (Kubernetes-style CRDs that create clusters on any cloud). kops is still around for self-managed AWS clusters.

Changed since the book:

kubicorn was abandoned and its repo archived. Cluster API carries its “clusters as Kubernetes objects” idea forward.

Source: K8s Cookbook, Recipe 14.5

Recipe 14.7 — Functions on Kubernetes (was: kubeless)

  • Problem: Deploy functions (HTTP- or event-triggered) without writing Deployments.
  • Solution today: Knative Serving (request-driven, scale to zero, revisions and traffic splitting) + Knative Eventing. Or OpenFaaS. For “scale my normal Deployment on events, down to zero”, use KEDA (see Scaling).

Changed since the book:

kubeless was archived (2021), and its Python 2.7 runtime is long dead. It was a nice example of CRD + controller, though: a Function CR got turned into a ConfigMap (code) + Deployment + Service. That’s exactly the operator pattern above.

Source: K8s Cookbook, Recipe 14.7

Summary

  • Kubernetes = desired-state store + control loops. Extend with CRDs (apiextensions.k8s.io/v1, structural schema) + controllers = operators.
  • Script with the official clients (Python: CoreV1Api, AppsV1Api, CustomObjectsApi). Build with Kubebuilder or Operator SDK.
  • Enforce rules at admission time: ValidatingAdmissionPolicy (CEL) first, webhooks (Kyverno, Gatekeeper) when you need more.
  • Book-era tools replaced: kubicorn → Cluster API/eksctl/Terraform, kubeless → Knative/OpenFaaS/KEDA.

Graph View

Backlinks