ML on

Lesson 4: Recommendation Systems — Collaborative filtering, embeddings, and the cold start problem

Fri, 13 Dec 2024 00:00:00 +0000

Recommendation systems are the most economically consequential ML systems most engineers will ever build. Netflix estimates that its recommendation system saves over a billion dollars per year in avoided cancellations. Amazon’s “customers who bought this also bought” drives a significant fraction of its revenue. Spotify’s Discover Weekly has become a user retention flywheel. The stakes are real, and the engineering is genuinely interesting.

I got deep into recommendation systems when I was working on a content platform that had about 500,000 items and needed to surface relevant content for each user without overwhelming the ranking team with engineering requests. What I found was that the architecture of a production recommender is almost always the same shape, regardless of the domain — and the hardest problem is not the ML, it’s the cold start.

Lesson 3: Model Serving — Latency, batching, A/B testing in production

Mon, 14 Oct 2024 00:00:00 +0000

Training a model is satisfying. Deploying it to serve real traffic is humbling. I remember the first time I pushed a model to production and watched the p99 latency hover at 800ms on what was supposed to be a “fast” model. The benchmark had shown 12ms inference time. What happened? The benchmark ran the model on pre-loaded batches; production served one request at a time, loaded the model fresh on cold starts, and had no GPU batching. The gap between “the model is accurate” and “the model is fast enough to be useful” is where model serving engineering lives.

Lesson 2: Feature Stores — Why feature engineering is 80% of ML

Fri, 02 Aug 2024 00:00:00 +0000

There is a saying in machine learning that is so universally acknowledged it has become a cliché: 80% of the work in any ML project is feature engineering. I spent a long time thinking this referred to the cognitive labor — the domain expertise required to craft meaningful features. It does, in part. But the deeper meaning is operational. The 80% is not just about what features to build; it’s about how to compute them consistently, store them efficiently, retrieve them with sub-millisecond latency at serving time, and keep them synchronized between the training pipeline and the production system.

Lesson 1: ML Pipelines — From raw data to deployed model

Sun, 02 Jun 2024 00:00:00 +0000

The first time I built a “production ML pipeline,” it was a cron job that ran a Python script, trained a model, and saved the artifact to disk. It worked, until it didn’t. The model silently degraded over three weeks because the training data schema changed and nobody noticed. There were no tests, no validation, no monitoring. That experience, embarrassing as it was, taught me more about ML system design than any paper on model architecture.