Databases on

Lesson 10: Vacuum and Bloat — Why Postgres Tables Grow

Mon, 09 Sep 2024 00:00:00 +0000

I watched a Postgres instance run out of disk space on a 500 GB SSD. The database had 50 GB of actual data. The other 450 GB was table bloat — dead row versions from MVCC that vacuum had failed to clean up. A batch job had been running long-running transactions for weeks, holding a transaction horizon that prevented vacuum from reclaiming anything. By the time we noticed, the disk was nearly full and autovacuum was struggling to catch up. Understanding why this happens — and how to prevent it — is one of the most important operational skills for running Postgres in production.

Lesson 9: Partitioning — Range, Hash, List and When Each Helps

Sat, 24 Aug 2024 00:00:00 +0000

I have worked with a few systems that had grown their main tables to 500 million rows or more. At that scale, even well-indexed queries start slowing down — not because the indexes are wrong, but because the index itself becomes large and the buffer cache can only hold so many pages. Vacuum struggles to keep up. EXPLAIN output looks fine but queries still feel sluggish. Partitioning is the architectural solution to this class of problem: instead of one big table, you have many smaller tables that look like one from the application’s perspective.

Lesson 8: Replication — Streaming, Logical, and Failover

Fri, 09 Aug 2024 00:00:00 +0000

The first time I had to fail over a Postgres primary, it was 2 AM, the primary was not responding, and I genuinely did not know whether the replica was up to date or 10 minutes behind. We recovered, but that experience drove me to actually understand replication — not just “the replica gets the writes somehow” but exactly what data is shipped, when it arrives, and what happens when the primary dies. It turns out the mechanism is elegant and directly connected to the WAL we covered in Lesson 3.

Lesson 7: Connection Pooling — What PgBouncer Actually Does

Sat, 27 Jul 2024 00:00:00 +0000

A Postgres connection is not cheap. When I first scaled a service from a single server to 20 replicas — each running a Go application with database/sql’s default pool settings — the database became the bottleneck almost immediately. Not because the queries were slow. Because we had 20 × 100 = 2,000 open connections, each consuming RAM on the Postgres server, and Postgres was spending more time managing connections than executing queries. This is the problem PgBouncer solves, and understanding how it works makes you a much better architect of backend systems.

Lesson 6: Query Planning and EXPLAIN — Reading Execution Plans

Thu, 11 Jul 2024 00:00:00 +0000

The first time I ran EXPLAIN ANALYZE on a slow query, I stared at the output for five minutes and understood none of it. It looked like a compiler error message crossed with a financial report. Then a senior engineer walked me through it, and what had looked like noise resolved into a clear picture: this node was doing more work than the planner expected, this join strategy was wrong, this sort was spilling to disk. EXPLAIN ANALYZE is the most powerful tool in database performance work, and it takes an hour to learn but pays back every week for the rest of your career.

Lesson 5: Transaction Isolation — Read Committed vs Serializable

Sun, 23 Jun 2024 00:00:00 +0000

I shipped a bug once that allowed a user to spend the same gift card balance twice. Two requests arrived nearly simultaneously, both read the same balance, both decided the balance was sufficient, both deducted it, and both succeeded. The database did exactly what I asked. The problem was what I asked for: I assumed reads were consistent across statements within a transaction, but I was running at the default isolation level. Understanding the four isolation levels — and what each one actually protects you from — is not academic. It is the difference between shipping correct financial code and shipping race conditions.

Lesson 4: MVCC — How Postgres Handles Concurrent Reads and Writes

Sun, 09 Jun 2024 00:00:00 +0000

One thing that puzzled me early on was how Postgres could let me read from a table while someone else was writing to it — without locking me out and without me seeing half-written data. Most systems I had worked with used explicit read locks, which meant readers and writers had to take turns. Postgres doesn’t do that. Reads never block writes, and writes never block reads. The mechanism that makes this possible is called Multiversion Concurrency Control, or MVCC, and it works by keeping multiple versions of every row simultaneously.

Lesson 3: Write-Ahead Log — How Databases Survive Crashes

Thu, 23 May 2024 00:00:00 +0000

Databases promise durability. When INSERT returns successfully, your data is safe — even if the server loses power a millisecond later. For a long time I accepted this as magic. Then I started reading about what actually happens when Postgres writes data, and the mechanism behind that promise is both elegant and counterintuitive: to make writes safe, you write them twice. The first write goes to a sequential log. The second write goes to the actual data file. And the log is what saves you when things go wrong.

Lesson 2: B-Tree Indexes — O(n) to O(log n) in one CREATE INDEX

Wed, 08 May 2024 00:00:00 +0000

I once joined a team that had a users table with 8 million rows. The GET /users/{email} endpoint was consistently timing out under load. When I ran EXPLAIN ANALYZE on the query, I saw it: Seq Scan on users (cost=0.00..180000.00 rows=1 width=120) (actual rows=1 loops=1). It was reading every single row — all 8 million of them — to find one user by email. Adding a single index fixed it in under five minutes. That experience made me want to actually understand what an index is, not just that it makes things faster.

Lesson 1: How a Query Executes — Parser to Planner to Disk

Mon, 22 Apr 2024 00:00:00 +0000

Every time I call db.QueryContext(ctx, "SELECT * FROM orders WHERE user_id = $1", userID) in Go, I used to think the database just “found” the rows. It wasn’t until I started debugging a production slowdown — a query that was fast for months and then suddenly wasn’t — that I actually traced what happens between the moment my application hands off that SQL string and the moment rows come back. Understanding that pipeline changed how I write queries, design schemas, and diagnose performance problems.