Kubernetes on

Lesson 5: Debugging in Kubernetes — kubectl tricks that save hours

Tue, 18 Mar 2025 00:00:00 +0000

The most stressful hours of my on-call life have been in Kubernetes clusters where something was wrong but nothing was obviously broken. Pods in Pending for reasons that weren’t clear. Services returning 503s but all pods showing as Running. Memory usage climbing slowly across a fleet for two days before things started dying. Over years of debugging production Kubernetes issues, I’ve accumulated a set of commands and mental models that cut through the noise. This article is that collection.

Lesson 4: Operators and CRDs — Extending Kubernetes with your own resources

Fri, 03 Jan 2025 00:00:00 +0000

I used to manage PostgreSQL on Kubernetes with a collection of shell scripts and Helm hooks. Provisioning a new database instance meant running a script that created a StatefulSet, a Service, a ConfigMap, a Secret, and set up replication. Failover meant manually triggering another script. Backups were a cron job that required careful coordination with the stateful set. Every operational task was a script that a human had to run, remember, and maintain. Then I discovered the Zalando Postgres Operator, and I understood what the Operator pattern is actually for: encoding human operational knowledge into the Kubernetes control plane itself.

Lesson 3: ConfigMaps and Secrets — Configuration without rebuilding

Wed, 16 Oct 2024 00:00:00 +0000

We had a bug where the same Docker image was running in staging and production but behaving differently. I spent an hour diffing the code before I checked the environment variables. The staging pod had CACHE_TTL=60 and the production pod had CACHE_TTL=3600. The image was identical. The behavior was totally different. That kind of environment-specific configuration — the values that shouldn’t be baked into the image — is exactly what ConfigMaps and Secrets are for. But they have subtleties that bite you if you treat them as simple key-value stores.

Lesson 2: Deployments and Scaling — Rolling updates, HPA, VPA

Wed, 07 Aug 2024 00:00:00 +0000

The week before a major product launch, our Kubernetes cluster started evicting pods. Traffic was spiking as we ran load tests, but instead of scaling up, the HPA was oscillating — scaling up, then the new pods were getting OOMKilled, then scaled down, then up again. Pods were in CrashLoopBackOff, the HPA metrics were lagging behind the actual load, and I was watching health check failures cascade. It turned out our resource requests were wildly inaccurate — set once during initial deployment and never updated as the service’s actual usage changed. That day I learned that Deployments and autoscaling aren’t fire-and-forget configurations.

Lesson 1: Pod Design Patterns — Sidecar, ambassador, adapter

Fri, 24 May 2024 00:00:00 +0000

The first time I deployed a service to Kubernetes, I put everything in a single container. Logging, metrics, TLS termination, the actual application — all in one Docker image. It worked, but the image was massive, updating any single concern meant rebuilding and redeploying the whole thing, and the application team had to understand infrastructure concerns they shouldn’t need to care about. The sidecar pattern was the thing that changed how I thought about container composition.