Go Networking & Distributed Systems on Atharva Pandey

Lesson 8: gRPC Basics and Streaming — Protobuf on the wire, types in your code

Thu, 15 May 2025 00:00:00 +0000

My team migrated a set of internal service APIs from JSON over HTTP/1.1 to gRPC roughly two years ago. The motivating factors were type safety across service boundaries, binary serialization that was 5-10x smaller on the wire, and bidirectional streaming that HTTP/1.1 can’t do at all. The migration took about a week per service and the performance improvements were immediately visible in our latency percentiles.

gRPC is a Remote Procedure Call framework that runs over HTTP/2. The interface is defined in Protocol Buffers — a language-neutral schema language — and the gRPC toolchain generates client and server stubs in Go. You call a method; the framework handles serialization, connection management, and streaming.

Lesson 7: Outbox Pattern — Reliable events without distributed transactions

Thu, 20 Mar 2025 00:00:00 +0000

Here’s a bug that’s bitten almost every distributed system I’ve worked on: a service saves a record to the database and then publishes an event to Kafka. The record is saved. The process crashes before the publish. Now the database says the order exists, but the fulfillment service never heard about it. The order sits in limbo forever.

The naive fix — wrapping both operations in a transaction — doesn’t work because Kafka isn’t a participant in your database transaction. You can’t two-phase commit across Postgres and Kafka without a distributed transaction coordinator, and nobody wants to operate one of those in production.

Lesson 6: Eventual Consistency — Your data will be wrong, temporarily

Sat, 25 Jan 2025 00:00:00 +0000

I used to believe that eventual consistency was an exotic property of distributed databases that I’d only encounter at Google scale. Then I added a Redis cache to a simple CRUD service and immediately had a bug where users couldn’t see their own updates. Welcome to eventual consistency — it shows up the moment you have two places that store the same data.

Eventual consistency means that if you stop writing to a system, all replicas will eventually converge to the same value. The word “eventually” can mean milliseconds or minutes, depending on the system. The hard part isn’t the definition — it’s designing application code that works correctly even when replicas haven’t converged yet.

Lesson 5: Message Queues in Go — NATS, RabbitMQ, Kafka — pick your tradeoff

Wed, 20 Nov 2024 00:00:00 +0000

I’ve integrated all three of these systems in production Go services, and the question I get most often is: “Which one should I use?” The honest answer is that it depends on what guarantee you actually need — and most engineers pick based on familiarity or hype rather than requirements. NATS, RabbitMQ, and Kafka solve meaningfully different problems, and choosing the wrong one creates operational pain that no amount of clever application code can fix.

Lesson 4: Connection Pooling — One connection per request is a performance bug

Sat, 12 Oct 2024 00:00:00 +0000

The first time I benchmarked a Go service against a database, the numbers were embarrassing. Five hundred requests per second, each one taking 30 milliseconds to execute a trivially simple query. The query itself took 2 milliseconds on the database server. The other 28 milliseconds were TCP handshake plus TLS plus PostgreSQL authentication — repeated for every single request because I had no connection pool.

Connection pooling is one of those topics that feels like an advanced optimization until you discover that nearly every database driver in Go already pools connections by default — you just have to configure the pool instead of leaving it at its default settings, which are almost always wrong for your workload.

Lesson 3: Circuit Breaking — Stop calling the service that's already down

Wed, 28 Aug 2024 00:00:00 +0000

There’s a particular kind of production incident that goes like this: Service A calls Service B. Service B starts responding slowly because its database is under pressure. Service A’s goroutines pile up waiting for responses. After a few minutes, Service A is out of memory, and now two services are down instead of one. The postmortem note says “cascading failure.” The fix, which nobody implemented, is a circuit breaker.

The circuit breaker pattern comes from electrical engineering. When too much current flows through a circuit, the breaker trips — it opens the circuit and prevents more current from flowing until someone resets it. In software, the “current” is requests, and “tripping” means refusing to make calls to a failing downstream instead of piling up timeouts and consuming resources.

Lesson 2: Retries with Exponential Backoff — Retry right or retry forever

Sat, 20 Jul 2024 00:00:00 +0000

I once worked on a service that retried failed HTTP requests in a tight loop with no backoff and no jitter. When our upstream had a brief outage, every instance of our service fired retries simultaneously, on the same cadence, forever. The upstream came back online, got immediately hammered by a synchronized retry storm from fifty instances, went down again, and the cycle repeated for forty minutes. The “retry” logic had turned a five-minute outage into a cascading failure.

Lesson 1: HTTP Client Internals — Transport, connection reuse, and the pool you forgot to configure

Fri, 14 Jun 2024 00:00:00 +0000

The first production Go service I wrote hammered our database proxy with thousands of new TCP connections every minute. The proxy started refusing connections. The on-call engineer thought it was the database. It wasn’t. It was me — specifically, my decision to create a new http.Client on every request and then never read the response body to completion. I spent three hours debugging what turned out to be two lines of misunderstood standard library behavior.