Networking for Backend Engineers on Atharva Pandey

Lesson 7: Service Mesh — Sidecar proxies and mTLS without code

Sat, 03 Aug 2024 00:00:00 +0000

A team I worked with had seventeen microservices. Each service had its own implementation of retry logic, circuit breaking, timeout handling, and mutual TLS. Some used libraries, some rolled their own. When we needed to update the TLS certificate rotation policy, it touched eleven different code repositories, four different languages, and took two months. Then we introduced Linkerd and moved all of that to the infrastructure layer. The services still did their jobs. The networking became someone else’s problem — specifically, the platform team’s.

Lesson 6: gRPC and Protobuf — Binary protocols and streaming

Wed, 17 Jul 2024 00:00:00 +0000

The first JSON API I replaced with gRPC was passing []Order objects around — each order had about 40 fields, most of which the caller never used. The JSON payload for a list of 100 orders was around 180KB. After the migration it was 22KB, and the serialization time in benchmarks dropped by 8x. But more than the performance numbers, what I noticed was the schema. Proto files are contracts. When a field changes, you know it. With JSON, you find out when things break in production.

Lesson 5: WebSockets — Upgrade, framing, vs SSE vs polling

Mon, 01 Jul 2024 00:00:00 +0000

When I first built a real-time notification system, I reached for WebSockets because that’s what everyone said to use for “real-time.” Three months later I was debugging connection drops under load, wrestling with proxy timeouts, and fighting with nginx configuration. When I stepped back and actually thought about the access pattern — server pushing notifications to the browser, never the browser sending data to the server — I replaced WebSockets with Server-Sent Events in a weekend. The code got simpler, the proxies stopped complaining, and we had less to maintain. Picking the right tool requires understanding what each one actually does.

Lesson 4: DNS — Resolution, caching, and why changes take time

Sat, 15 Jun 2024 00:00:00 +0000

We pushed a production incident fix at 3am — rotated to a new IP address for a critical service, updated the DNS record, set the TTL to 60 seconds. Thirty minutes later, half our users were still hitting the broken server. The other half were fine. We had updated the DNS record correctly. The TTL had expired. Yet somehow, stale answers were persisting. That night taught me more about DNS than any documentation had.

Lesson 3: TLS Handshake — What happens in those 2 round trips

Thu, 30 May 2024 00:00:00 +0000

A colleague once asked me why adding TLS to a service increased our P99 latency by 50ms. She had measured it carefully, switching between HTTP and HTTPS in a load test. My first instinct was to say “encryption overhead” but that’s wrong — modern CPUs with AES-NI can encrypt gigabytes per second. The actual cost is the handshake. Once I explained what was actually happening in those first few round trips, the answer to “how do we fix it” became obvious: stop creating new connections.

Lesson 2: HTTP/2 and HTTP/3 — Multiplexing and QUIC

Tue, 14 May 2024 00:00:00 +0000

The first time I looked at a waterfall chart in Chrome DevTools and saw a hundred requests stacking up like rush-hour traffic, I knew HTTP/1.1 was the problem. We had a dashboard loading twelve API calls and a handful of assets, and they were all waiting in line. Not because the server was slow — it was idle — but because the browser had a six-connection-per-host limit and every request had to wait its turn. That was my introduction to why HTTP/2 exists, and it changed how I think about protocol design.

Lesson 1: TCP Deep Dive — Three-way handshake, congestion, Nagle

Mon, 29 Apr 2024 00:00:00 +0000

I used to treat TCP as a black box. Data goes in one side, data comes out the other — reliably, in order, no duplicates. That was all I needed to know, right? Then I started debugging latency spikes in a payment service and spent three days chasing a 200ms tail latency that turned out to be Nagle’s algorithm fighting with delayed ACKs. After that, I stopped treating TCP as a black box.