System Design on

Lesson 4: Recommendation Systems — Collaborative filtering, embeddings, and the cold start problem

Fri, 13 Dec 2024 00:00:00 +0000

Recommendation systems are the most economically consequential ML systems most engineers will ever build. Netflix estimates that its recommendation system saves over a billion dollars per year in avoided cancellations. Amazon’s “customers who bought this also bought” drives a significant fraction of its revenue. Spotify’s Discover Weekly has become a user retention flywheel. The stakes are real, and the engineering is genuinely interesting.

I got deep into recommendation systems when I was working on a content platform that had about 500,000 items and needed to surface relevant content for each user without overwhelming the ranking team with engineering requests. What I found was that the architecture of a production recommender is almost always the same shape, regardless of the domain — and the hardest problem is not the ML, it’s the cold start.

Lesson 15: CAP Theorem in Practice — What It Actually Means for Your System

Tue, 19 Nov 2024 00:00:00 +0000

CAP theorem is probably the most cited and most misunderstood concept in distributed systems interviews. Candidates memorize “you can only pick two of consistency, availability, and partition tolerance” and then either over-apply it (treating every design decision as a CAP trade-off) or under-apply it (never relating it to actual design choices). The theorem is real and important, but the way it’s usually taught in 30-second summaries strips out the nuance that makes it actually useful. This final lesson clears that up.

Lesson 5: Design Twitter/X — Tweet fanout, timeline ranking, trending topics at 500M users

Thu, 14 Nov 2024 00:00:00 +0000

Twitter is the classic system design problem for good reason. It looks like a glorified blog until you start pulling on the threads: how does a tweet from a user with 100 million followers appear in every follower’s timeline within seconds? How do you rank timelines without reading millions of tweets per request? How do you identify trending topics across 500 million users in near real-time? Each of these is a genuinely hard problem, and they interact in non-obvious ways.

Lesson 14: Designing for Failure — Circuit Breakers, Bulkheads, Chaos

Mon, 04 Nov 2024 00:00:00 +0000

In distributed systems, failure is not an exceptional case — it’s the default condition. Networks partition. Hard drives fail. Memory fills up. Dependencies have bugs. Every distributed system that’s been running for more than a few years has experienced every kind of failure you can imagine, and many you can’t. The engineers who build resilient systems aren’t smarter than the ones who don’t. They’ve just internalized a single principle: design for when things go wrong, not for when things go right.

Lesson 13: Design a Payment System — Idempotency, Reconciliation, Double-Entry

Mon, 21 Oct 2024 00:00:00 +0000

Payment systems have a property that almost no other software has: the cost of a bug isn’t a bad user experience — it’s a legal liability and a business catastrophe. Charging a customer twice, losing a transfer in a network failure, or crediting the wrong account can result in millions of dollars of loss and destroyed trust. Every other system we’ve covered tolerates a degree of eventual inconsistency. Payment systems, in most cases, do not. This lesson is about building for that level of correctness.

Lesson 3: Model Serving — Latency, batching, A/B testing in production

Mon, 14 Oct 2024 00:00:00 +0000

Training a model is satisfying. Deploying it to serve real traffic is humbling. I remember the first time I pushed a model to production and watched the p99 latency hover at 800ms on what was supposed to be a “fast” model. The benchmark had shown 12ms inference time. What happened? The benchmark ran the model on pre-loaded batches; production served one request at a time, loaded the model fresh on cold starts, and had no GPU batching. The gap between “the model is accurate” and “the model is fast enough to be useful” is where model serving engineering lives.

Lesson 12: Design a Search Engine — Inverted Index and Ranking

Mon, 07 Oct 2024 00:00:00 +0000

Search is the feature that separates usable products from unusable ones at scale. When your application has thousands of documents, a sequential scan works. At millions, it doesn’t. The difference between a search box that works and one that times out is a data structure invented in the 1960s that every modern search engine still fundamentally relies on: the inverted index. Understanding how it works — and how to build a system around it — is what this lesson is about.

Lesson 11: Design a Notification System — Push vs Pull, Priority, Dedup

Sat, 21 Sep 2024 00:00:00 +0000

Notifications are the feature that can make or break user retention — and also destroy it. Done right, they bring users back at exactly the right moment. Done wrong, they’re spam that drives uninstalls. The system design challenge isn’t just the technical plumbing (though that’s interesting). It’s building infrastructure that’s fast for critical alerts, reliable for important messages, and smart enough to not overwhelm users with low-priority noise.

The Core Concept

A notification system has to handle multiple channels with wildly different characteristics:

Lesson 4: Design WhatsApp — End-to-end encryption, message delivery guarantees, presence

Sat, 14 Sep 2024 00:00:00 +0000

WhatsApp is deceptively simple from a user perspective: you send a message, it arrives. But building a messaging system that handles 100 billion messages per day with end-to-end encryption, reliable delivery semantics, and real-time presence for 2 billion users is a genuinely hard engineering problem. I find this problem particularly instructive because it forces you to confront three things simultaneously: cryptographic key management, message delivery guarantees, and the cost of maintaining online/offline state at massive scale.

Lesson 10: Design a News Feed — Fan-out on Write vs Read

Fri, 06 Sep 2024 00:00:00 +0000

The news feed is the product feature that defines social media. Every time you open Instagram or Twitter, you see a personalized, ranked, real-time stream of content from people you follow. Behind that deceptively simple UI is one of the hardest distributed systems problems in consumer tech: how do you compute a personalized feed for hundreds of millions of users, where any piece of content needs to appear in potentially millions of feeds, within seconds of being posted?

Lesson 9: Design a Chat System — WebSocket, Presence, Message Ordering

Wed, 21 Aug 2024 00:00:00 +0000

Building a chat system is the interview problem that catches people on protocol fundamentals. HTTP is a request-response protocol — the client asks, the server answers, and then the connection is idle. For chat, the server needs to push messages to clients the moment they arrive. This inversion of the HTTP model is what makes chat hard, and it’s what drives the entire architecture.

The Core Concept

Why HTTP polling doesn’t work at scale

Lesson 8: Design a URL Shortener — The Classic, Done Properly

Sun, 04 Aug 2024 00:00:00 +0000

The URL shortener is the “Hello World” of system design interviews. It appears deceptively simple: take a long URL, return a short one. But if you treat it superficially, you miss what the interviewer is actually testing: your ability to think through ID generation at scale, read-heavy caching, redirect semantics, analytics storage, and data modeling. Done well, the URL shortener problem touches nearly every fundamental we’ve covered so far.

The Core Concept

A URL shortener has two primary operations:

Lesson 2: Feature Stores — Why feature engineering is 80% of ML

Fri, 02 Aug 2024 00:00:00 +0000

There is a saying in machine learning that is so universally acknowledged it has become a cliché: 80% of the work in any ML project is feature engineering. I spent a long time thinking this referred to the cognitive labor — the domain expertise required to craft meaningful features. It does, in part. But the deeper meaning is operational. The 80% is not just about what features to build; it’s about how to compute them consistently, store them efficiently, retrieve them with sub-millisecond latency at serving time, and keep them synchronized between the training pipeline and the production system.

Lesson 7: Rate Limiting — Token Bucket, Sliding Window, Distributed

Sun, 21 Jul 2024 00:00:00 +0000

Without rate limiting, a single misbehaving client can consume all your resources and deny service to everyone else. A bug in a client that retries in a tight loop. A competitor scraping your API. A DDoS attack. A feature that accidentally calls your endpoint ten times per button click. Rate limiting is the mechanism that prevents any of these from taking down your service — it’s the first line of defense between the internet and your origin.

Lesson 3: Design Google Docs — Real-time collaboration, OT vs CRDT, conflict resolution

Mon, 15 Jul 2024 00:00:00 +0000

Google Docs is the problem I recommend to every engineer who thinks they understand distributed systems. The surface looks trivial: multiple users editing a document simultaneously. The depth is staggering. When two users type at the same position in a document at the same millisecond, what does each user see? How do you converge on a consistent state without a central lock? How do you preserve the intention behind each edit, not just the characters?

Lesson 6: CDNs — Put Your Bytes Close to Your Users

Thu, 04 Jul 2024 00:00:00 +0000

Physics is the enemy of performance. Light travels through fiber optic cables at about 200,000 km/s — roughly two-thirds the speed of light in a vacuum. A round-trip from New York to London is about 11,000 km each way. That means the minimum possible latency for that trip is 55ms. You cannot engineer your way past it. What you can do is stop making the trip at all. Content Delivery Networks are the infrastructure that puts popular content at the edge — closer to users, so the bytes never have to travel far.

Lesson 5: Message Queues — Decoupling Services Without Losing Messages

Wed, 19 Jun 2024 00:00:00 +0000

Synchronous calls are elegant until they’re not. Service A calls Service B. B is slow. A waits. A’s request pool fills up. A becomes slow. The caller of A waits. The whole request chain stalls. Add a few more services in the chain and you have a distributed deadlock in slow motion. Message queues exist to break this dependency — to let a producer say “here’s some work” and move on, without caring whether the consumer is fast, slow, or temporarily down.

Lesson 4: Database Scaling — Read Replicas, Sharding, and When Each Helps

Wed, 05 Jun 2024 00:00:00 +0000

Your startup ships, gains traction, and one day your database becomes the bottleneck. Queries slow down. CPU spikes. The single machine hosting your Postgres instance can’t keep up. This is one of the most predictable problems in engineering, yet I’ve seen teams reach it completely unprepared because they’d never thought carefully about how databases scale. The right solution depends entirely on whether you’re read-heavy or write-heavy, whether your data is relational or not, and whether your workload is even or spiky.

Lesson 1: ML Pipelines — From raw data to deployed model

Sun, 02 Jun 2024 00:00:00 +0000

The first time I built a “production ML pipeline,” it was a cron job that ran a Python script, trained a model, and saved the artifact to disk. It worked, until it didn’t. The model silently degraded over three weeks because the training data schema changed and nobody noticed. There were no tests, no validation, no monitoring. That experience, embarrassing as it was, taught me more about ML system design than any paper on model architecture.

Lesson 2: Design Uber — Real-time matching, geospatial indexing, surge pricing

Sat, 25 May 2024 00:00:00 +0000

Uber is one of the most instructive system design problems because it forces you to think about real-time data pipelines, spatial indexing, and latency-sensitive matching algorithms all at once. I spent a significant amount of time studying this problem specifically because the geospatial angle is something most candidates gloss over. They say “use a database with location queries” and move on. But at Uber’s scale — 5 million trips per day, hundreds of thousands of concurrent drivers and riders — the geospatial indexing strategy is the entire problem.

Lesson 3: Caching — The Hardest Easy Problem in CS

Mon, 20 May 2024 00:00:00 +0000

Phil Karlton supposedly said: “There are only two hard things in Computer Science: cache invalidation and naming things.” He was joking, but also completely serious. Caching is the answer to nearly every “make it faster” problem in system design. It’s also the source of some of the most insidious bugs in production: stale data served with confidence, cache stampedes that take down your database, memory explosions from an unbounded cache. The concept is simple. Getting it right is not.

Lesson 2: Load Balancing — L4 vs L7 and Why It Matters

Sun, 05 May 2024 00:00:00 +0000

The first time I drew a system design diagram in an interview, I drew a box labeled “load balancer” and drew arrows from clients to it, and from it to servers. My interviewer asked, “What kind of load balancer?” I didn’t have an answer. I knew load balancers existed. I didn’t know they made fundamentally different decisions at different network layers — and that the choice between them shapes what your system can and cannot do.

Lesson 1: How the Internet Works — DNS, TCP, HTTP, TLS in 15 Minutes

Sat, 20 Apr 2024 00:00:00 +0000

Every system design interview starts with the same silent assumption: you already know how the internet works. Interviewers won’t ask you to explain DNS. But when you confidently say “the client calls the API” without being able to say what actually happens between those words, the cracks show up in your design. Understanding the layers — DNS, TCP, HTTP, TLS — isn’t trivia. It’s the mental model that tells you where latency hides, why connections are expensive, and what breaks under load.

Lesson 1: Design YouTube — Video upload, transcoding, streaming at scale

Tue, 09 Apr 2024 00:00:00 +0000

YouTube serves over 500 hours of video uploaded every minute and delivers billions of views per day. When I first studied this problem seriously, I made the mistake of treating it as a simple “upload file, store it, serve it” exercise. It isn’t. The interesting engineering is in what happens between the moment a creator hits upload and the moment a viewer’s video starts playing seamlessly on a 3G connection in rural India. That gap is where the system design lives.