Deployment on

Lesson 8: Zero-Downtime Deploys — Rolling updates without dropping requests

Sun, 15 Jun 2025 00:00:00 +0000

The first rolling deployment I did without graceful shutdown handling produced about 200 errors for in-flight requests. Kubernetes sent SIGTERM to the old pods, the Go process exited immediately, and every active request was terminated mid-flight. The users got 502s. The monitoring dashboard lit up red for about 30 seconds per deploy, every time.

Kubernetes’s rolling update strategy replaces pods one at a time and can achieve zero request drops — but only if your application cooperates. The contract is: Kubernetes sends SIGTERM, your application finishes in-flight requests, then exits. If you ignore SIGTERM and exit immediately, Kubernetes kills you with SIGKILL after the grace period anyway, and you drop the requests. The work is in your application code, not in the Kubernetes config.

Lesson 8: Release Profiles and Build Optimization — Shipping fast binaries

Thu, 08 May 2025 13:50:00 +0000

I was benchmarking two builds of the same service — one with default release settings, one with a tuned profile. Same code. Same hardware. The tuned build was 22% faster on our hot path and 40% smaller. I didn’t change a single line of Rust. Just Cargo.toml settings.

Most Rust developers know about cargo build --release. Fewer know that --release is just a starting point — there’s a whole set of knobs in the release profile that trade compile time for runtime performance, or binary size, or debuggability. Let me walk you through every one that matters.

Lesson 7: Configuration — Environment, files, feature flags

Mon, 05 May 2025 10:15:00 +0000

I once shipped a service to production with the staging database URL hardcoded. Not in an environment variable — literally in the source code, in a const. It ran for two hours writing production data to the staging database before anyone noticed. The fix was easy. The data migration to clean up the mess took three days.

Configuration is one of those things that seems trivial until it bites you. And in Rust, we have the type system to make configuration bulletproof — but only if we structure things right. Let me walk you through the approach I’ve converged on after making every possible configuration mistake.

Lesson 6: Graceful Shutdown — Draining connections cleanly

Fri, 02 May 2025 19:45:00 +0000

I deployed a new version of a payment service once, and for about 3 seconds during the rollout, a handful of transactions just… vanished. They weren’t in the database. They weren’t in the error logs. The old pods received the requests, started processing them, and then Kubernetes killed the pods before they finished. SIGKILL doesn’t ask nicely — it just terminates the process. Those transactions were gone.

Graceful shutdown is the solution. When your service receives SIGTERM (Kubernetes’s way of saying “please stop”), it should stop accepting new requests, finish processing in-flight requests, flush any buffered data, close connections cleanly, and then exit. Get this right and you get zero-downtime deployments. Get it wrong and you get data loss.

Lesson 5: Health Checks and Readiness Probes — Production liveness

Wed, 30 Apr 2025 14:05:00 +0000

A service I maintained once passed all its health checks while silently dropping 30% of incoming requests. The health endpoint returned 200 OK every time Kubernetes asked. The database connection pool was exhausted, the service couldn’t process anything, but that little /health endpoint — which didn’t touch the database — happily reported everything was fine.

That’s when I learned the difference between a health check that checks health and one that just says “the process is running.” They’re very different things, and getting this wrong means your orchestrator keeps sending traffic to a broken instance instead of replacing it.

Lesson 4: Observability — tracing, metrics, OpenTelemetry

Sun, 27 Apr 2025 08:22:00 +0000

I once spent six hours debugging a production issue where requests were randomly timing out. No errors in the logs. CPU and memory looked fine. Response times were normal — except for the 2% that took 30 seconds. Without distributed tracing, I was blind. I ended up bisecting the problem by adding println! statements, deploying them one at a time, and watching CloudWatch. It was miserable.

That experience permanently changed how I build services. Observability isn’t something you bolt on when things break. It’s something you build in from day one, or you pay for it later — with your time, your sleep, and your sanity.

Lesson 3: CI/CD for Rust — GitHub Actions, caching, cargo-nextest

Thu, 24 Apr 2025 11:30:00 +0000

My first Rust CI pipeline took 45 minutes. Forty-five. Every push triggered a full dependency build, tests ran sequentially, and clippy ran as a separate job that also built everything from scratch. I was burning through GitHub Actions minutes like they were free — which they are for open source, but my patience certainly wasn’t.

Getting Rust CI right is mostly about caching. The compilation model means a clean build downloads and compiles hundreds of crates, and without caching, every single CI run pays that cost. Let me show you the pipeline I’ve landed on after iterating through dozens of projects.

Lesson 2: Static Linking with musl — Single binary deploys

Tue, 22 Apr 2025 16:40:00 +0000

Last year I had to deploy a Rust service to a hardened environment — no package manager, no shared libraries, no internet access. Just a bare Linux kernel and my binary. If my binary depended on libc, libssl, or anything else in /usr/lib, it simply wouldn’t start. That constraint forced me to learn static linking properly, and it turned out to be one of the best deployment patterns I’ve ever used.

Lesson 1: Docker for Rust — Multi-stage builds, minimal images

Sun, 20 Apr 2025 09:15:00 +0000

The first time I shipped a Rust service in Docker, my image was 2.1 GB. Two. Point. One. Gigabytes. For a binary that was 8 MB. I’d used rust:latest as my base, ran cargo build --release inside it, and called it a day. The image had GCC, LLVM, every system library known to humanity, and my tiny HTTP server somewhere in the corner.

That was the day I learned about multi-stage builds. And honestly, getting Docker right for Rust is one of those things that separates “I deployed it” from “I deployed it well.”

Lesson 7: Profiling in Containers — pprof works in Kubernetes too

Sat, 12 Apr 2025 00:00:00 +0000

The first time I needed to profile a Go service in production, I assumed I’d have to deploy a special build, reproduce the problem locally, or use some heavyweight APM product. Then I learned that Go’s net/http/pprof package can serve live profiling data from a running process via HTTP — in a container, in Kubernetes, right now, without redeployment.

pprof is the built-in Go profiler. It can capture CPU profiles (what functions are consuming CPU time), heap profiles (what’s allocated in memory and by whom), goroutine profiles (how many goroutines exist and what they’re doing), and block/mutex profiles (where goroutines are waiting). All of this is available as HTTP endpoints that you can scrape with go tool pprof from your laptop.

Lesson 6: Race Detector in CI — Run -race on every PR or ship bugs

Wed, 12 Feb 2025 00:00:00 +0000

A data race is a memory safety bug. Two goroutines access the same variable without synchronization, at least one is writing, and the Go memory model makes no guarantees about what happens. In practice: values get corrupted, counters drift, maps panic at runtime. These bugs are intermittent — they appear under load, on fast machines, during deploys — and they’re nearly impossible to reproduce deterministically.

The Go race detector is one of the most powerful debugging tools in any language ecosystem. It instruments every memory access at the compiler level and reports races precisely: the exact goroutine stacks at the moment of the conflicting accesses. But it only helps if you run it. Most teams run it once after a bug report. The right approach is running it on every PR, in CI, before the code is ever merged.

Lesson 5: CI/CD for Go — GitHub Actions that actually catch bugs

Sun, 15 Dec 2024 00:00:00 +0000

I’ve set up CI pipelines for Go services about a dozen times, and every iteration taught me something about what the pipeline should actually catch. The first version ran go test ./... and called it done. That caught compilation errors and test failures, but not data races, not staticcheck warnings, not formatting drift, not license issues, not security vulnerabilities in dependencies. Each of those failure categories has caused a production incident at some point. The current version catches most of them before the PR is merged.

Lesson 4: Config Injection — Environment variables, flags, files — in that order

Sat, 02 Nov 2024 00:00:00 +0000

Configuration management is one of those topics that seems solved until you maintain a service that runs in four different environments, has twenty configurable parameters, and needs its secrets rotated without a redeployment. I’ve gone through several evolutionary stages on this: hardcoded values (the naive phase), a single config.yaml file, then twelve config-{env}.yaml files, and finally landing on what the 12-factor app methodology describes — environment variables as the source of truth for deployment-specific configuration.

Lesson 3: Health Checks — Readiness vs liveness, and why both matter

Sun, 22 Sep 2024 00:00:00 +0000

I’ve seen two types of health check implementations in production: the ones that always return 200 OK, and the ones that actually check something. The first kind is theater — Kubernetes thinks the pod is healthy and routes traffic to it, but the pod is actually deadlocked or its database connection pool is exhausted. The second kind is what saves you at 3am.

Health checks are Kubernetes’s mechanism for deciding when a pod is ready to receive traffic and when it needs to be restarted. Get them right and Kubernetes becomes a genuinely reliable self-healing system. Get them wrong and you have an elaborate system that sends traffic to broken pods and restarts healthy ones.

Lesson 2: Docker for Go — Multi-stage builds that produce tiny images

Sat, 10 Aug 2024 00:00:00 +0000

The first Dockerfile I wrote for a Go service produced a 1.2 GB image. It was based on golang:1.22, which includes the full Go toolchain, build cache, test infrastructure, and a complete Debian system. The compiled binary was 18 MB. I was shipping 1.2 GB to run 18 MB.

The second version, after learning about multi-stage builds, produced a 22 MB image. Same binary. No compiler. No package manager. No shell. Just the binary and the absolute minimum needed to run it. The difference matters not just for storage: smaller images pull faster, have smaller attack surfaces, and fail more obviously when a required file is missing.

Lesson 1: Static Binaries — One file, zero dependencies, deploy anywhere

Sun, 30 Jun 2024 00:00:00 +0000

The first time I deployed a Go binary to a production server, I was braced for the usual ceremony: SSH in, install the right runtime version, copy over config files, fiddle with symlinks. Instead, I ran scp and then the binary, and it just worked. No installation. No dependency hell. No “but it works on my machine.” The binary contained everything it needed.

This is Go’s single greatest operational advantage: the compiler produces a self-contained executable. Understanding exactly how this works — and how it can break — makes the difference between a deploy that takes five minutes and one that takes an afternoon.