Engineering Practices on

Lesson 7: On-Call Engineering — Reducing toil, improving reliability

Sun, 11 Aug 2024 00:00:00 +0000

I did 12 months of on-call on a team that hadn’t invested in reliability. The rotation was weekly. In a bad week, I’d get 15-20 pages. A good week was 5. I was exhausted by the end of my shift, and the paging frequency had barely changed over those 12 months. We were fixing incidents, not fixing the causes. The next team I joined approached on-call differently: on-call was treated as a reliability sensor, not a firefighting rotation. Pages were tracked, patterns identified, and root causes fixed. By month 6 I was averaging 2 pages per week on-call. The work we did during on-call made future on-call better.

Lesson 6: Feature Flags — Progressive rollout and kill switches

Sun, 28 Jul 2024 00:00:00 +0000

We deployed a new checkout flow on a Friday afternoon. It had passed code review, passed testing, passed staging load tests. By Friday evening, the error rate was climbing. The new flow had a race condition that only manifested under specific mobile browser timing that we hadn’t tested. Without a feature flag, the fix would have required an emergency deployment — 25 minutes of build time, deployment, and validation. With a feature flag, the rollback was turning off a switch. Thirty seconds.

Lesson 5: Load Testing — k6, vegeta, realistic patterns

Sat, 13 Jul 2024 00:00:00 +0000

We load-tested our new checkout service before launch. 1,000 virtual users, 10 minutes, all hitting /v1/checkout sequentially. It passed with excellent numbers. Launch day: real traffic hit the service, and it fell over at 200 concurrent users. The problem was our test. Real users don’t all call the same endpoint in sequence. They browse, add to cart, apply discount codes, fill in addresses, and then checkout — a session that touches 8 different endpoints over 4 minutes. Our test didn’t model this. Our test was measuring the wrong thing.

Lesson 4: Monitoring and Alerting — SLOs and alert fatigue

Mon, 24 Jun 2024 00:00:00 +0000

The on-call rotation I inherited had 47 active alerts. On a bad week, the on-call engineer got paged 30 times. Most pages were “something might be wrong” noise — high CPU on one instance, a spike in error rate that self-resolved in 30 seconds, disk space at 70% on a server with months of capacity remaining. Engineers stopped taking the pages seriously. Then the one real incident got buried in the noise, and we had a 4-hour outage because no one treated the first alert seriously. Alert fatigue is not a monitoring problem. It’s an architecture-of-trust problem.

Lesson 3: Incident Response — Postmortems and blameless culture

Sat, 08 Jun 2024 00:00:00 +0000

The first major incident I was on-call for, I spent 90 minutes trying to fix the problem and 30 minutes frantically communicating to stakeholders in a panic. The second one, I followed a runbook and spent the 90 minutes coordinating, communicating clearly, and delegating diagnosis — while the problem was resolved in 40 minutes. The difference wasn’t technical skill. It was process. Incident response is a skill you can learn and practice, and it makes a measurable difference in how quickly you restore service and how well the team learns from failures.

Lesson 2: Code Review That Works — What to look for

Wed, 22 May 2024 00:00:00 +0000

I’ve been on the wrong end of bad code reviews in both directions. Reviews that were nitpick sessions about variable naming while missing a race condition. Reviews that rubber-stamped everything because the reviewer was busy. And I’ve given both kinds myself. It took a few years, a few incidents, and a few honest retrospectives to develop a framework for reviews that actually improve code quality without burning out reviewers or demoralizing authors.

Lesson 1: Git Beyond Basics — Rebase, bisect, worktrees

Tue, 07 May 2024 00:00:00 +0000

Most engineers use maybe 15% of Git. add, commit, push, pull, branch, merge, and status covers daily work. That’s fine until you need to find which commit introduced a regression across 300 commits, or you need to untangle a messy history before merging, or you want to work on three features simultaneously without context-switching overhead. The commands I’m covering here don’t come up every day. When they do, they save hours.