Go Deployment & Operations on Atharva Pandey

Lesson 8: Zero-Downtime Deploys — Rolling updates without dropping requests

Sun, 15 Jun 2025 00:00:00 +0000

The first rolling deployment I did without graceful shutdown handling produced about 200 errors for in-flight requests. Kubernetes sent SIGTERM to the old pods, the Go process exited immediately, and every active request was terminated mid-flight. The users got 502s. The monitoring dashboard lit up red for about 30 seconds per deploy, every time.

Kubernetes’s rolling update strategy replaces pods one at a time and can achieve zero request drops — but only if your application cooperates. The contract is: Kubernetes sends SIGTERM, your application finishes in-flight requests, then exits. If you ignore SIGTERM and exit immediately, Kubernetes kills you with SIGKILL after the grace period anyway, and you drop the requests. The work is in your application code, not in the Kubernetes config.

Lesson 7: Profiling in Containers — pprof works in Kubernetes too

Sat, 12 Apr 2025 00:00:00 +0000

The first time I needed to profile a Go service in production, I assumed I’d have to deploy a special build, reproduce the problem locally, or use some heavyweight APM product. Then I learned that Go’s net/http/pprof package can serve live profiling data from a running process via HTTP — in a container, in Kubernetes, right now, without redeployment.

pprof is the built-in Go profiler. It can capture CPU profiles (what functions are consuming CPU time), heap profiles (what’s allocated in memory and by whom), goroutine profiles (how many goroutines exist and what they’re doing), and block/mutex profiles (where goroutines are waiting). All of this is available as HTTP endpoints that you can scrape with go tool pprof from your laptop.

Lesson 6: Race Detector in CI — Run -race on every PR or ship bugs

Wed, 12 Feb 2025 00:00:00 +0000

A data race is a memory safety bug. Two goroutines access the same variable without synchronization, at least one is writing, and the Go memory model makes no guarantees about what happens. In practice: values get corrupted, counters drift, maps panic at runtime. These bugs are intermittent — they appear under load, on fast machines, during deploys — and they’re nearly impossible to reproduce deterministically.

The Go race detector is one of the most powerful debugging tools in any language ecosystem. It instruments every memory access at the compiler level and reports races precisely: the exact goroutine stacks at the moment of the conflicting accesses. But it only helps if you run it. Most teams run it once after a bug report. The right approach is running it on every PR, in CI, before the code is ever merged.

Lesson 5: CI/CD for Go — GitHub Actions that actually catch bugs

Sun, 15 Dec 2024 00:00:00 +0000

I’ve set up CI pipelines for Go services about a dozen times, and every iteration taught me something about what the pipeline should actually catch. The first version ran go test ./... and called it done. That caught compilation errors and test failures, but not data races, not staticcheck warnings, not formatting drift, not license issues, not security vulnerabilities in dependencies. Each of those failure categories has caused a production incident at some point. The current version catches most of them before the PR is merged.

Lesson 4: Config Injection — Environment variables, flags, files — in that order

Sat, 02 Nov 2024 00:00:00 +0000

Configuration management is one of those topics that seems solved until you maintain a service that runs in four different environments, has twenty configurable parameters, and needs its secrets rotated without a redeployment. I’ve gone through several evolutionary stages on this: hardcoded values (the naive phase), a single config.yaml file, then twelve config-{env}.yaml files, and finally landing on what the 12-factor app methodology describes — environment variables as the source of truth for deployment-specific configuration.

Lesson 3: Health Checks — Readiness vs liveness, and why both matter

Sun, 22 Sep 2024 00:00:00 +0000

I’ve seen two types of health check implementations in production: the ones that always return 200 OK, and the ones that actually check something. The first kind is theater — Kubernetes thinks the pod is healthy and routes traffic to it, but the pod is actually deadlocked or its database connection pool is exhausted. The second kind is what saves you at 3am.

Health checks are Kubernetes’s mechanism for deciding when a pod is ready to receive traffic and when it needs to be restarted. Get them right and Kubernetes becomes a genuinely reliable self-healing system. Get them wrong and you have an elaborate system that sends traffic to broken pods and restarts healthy ones.

Lesson 2: Docker for Go — Multi-stage builds that produce tiny images

Sat, 10 Aug 2024 00:00:00 +0000

The first Dockerfile I wrote for a Go service produced a 1.2 GB image. It was based on golang:1.22, which includes the full Go toolchain, build cache, test infrastructure, and a complete Debian system. The compiled binary was 18 MB. I was shipping 1.2 GB to run 18 MB.

The second version, after learning about multi-stage builds, produced a 22 MB image. Same binary. No compiler. No package manager. No shell. Just the binary and the absolute minimum needed to run it. The difference matters not just for storage: smaller images pull faster, have smaller attack surfaces, and fail more obviously when a required file is missing.

Lesson 1: Static Binaries — One file, zero dependencies, deploy anywhere

Sun, 30 Jun 2024 00:00:00 +0000

The first time I deployed a Go binary to a production server, I was braced for the usual ceremony: SSH in, install the right runtime version, copy over config files, fiddle with symlinks. Instead, I ran scp and then the binary, and it just worked. No installation. No dependency hell. No “but it works on my machine.” The binary contained everything it needed.

This is Go’s single greatest operational advantage: the compiler produces a self-contained executable. Understanding exactly how this works — and how it can break — makes the difference between a deploy that takes five minutes and one that takes an afternoon.