Linux on

Lesson 8: Memory-Mapped IO — How Databases Read Files

Tue, 13 Aug 2024 00:00:00 +0000

I was reading about how RocksDB works one evening and kept seeing references to mmap for reads. I had heard of memory-mapped files but thought of them as a niche optimization. Then I realized: SQLite uses mmap. WiredTiger (MongoDB’s storage engine) uses mmap. LMDB is built almost entirely around mmap. Even some Postgres configurations use mmap for WAL. Understanding mmap explains a lot about how high-performance storage works, why some databases are so fast for random reads, and why memory and I/O are so deeply intertwined at the OS level.

Lesson 7: Containers from Scratch — Namespaces, Cgroups, What Docker Does

Mon, 29 Jul 2024 00:00:00 +0000

I spent the first two years of my career using Docker without understanding what it was actually doing. I thought containers were a “lightweight VM” — some kind of virtualization. Then I read Liz Rice’s “Containers from Scratch” talk and wrote a 100-line container runtime in Go. Containers are not virtual machines. They are just Linux processes with restricted views of the system, implemented using two kernel features: namespaces and cgroups. Once you see how simple the underlying mechanism is, you understand exactly what Docker adds and why containers behave the way they do.

Lesson 6: Signals — SIGTERM vs SIGKILL and Graceful Shutdown

Sun, 14 Jul 2024 00:00:00 +0000

The first time I deployed a service to Kubernetes and watched it restart, I noticed that requests in flight were sometimes failing with connection resets. The pod was receiving 502 responses for a few seconds before it disappeared. I had heard of “graceful shutdown” but hadn’t implemented it. Kubernetes sends SIGTERM before killing a process, giving it time to finish in-flight requests — but my service was either ignoring the signal or exiting immediately, cutting off connections mid-request. Learning how signals work at the OS level, and then implementing correct signal handling in Go, fixed the issue permanently.

Lesson 5: Epoll and IO Multiplexing — How Go''s Netpoller Works

Thu, 27 Jun 2024 00:00:00 +0000

I used to wonder how a Go HTTP server could handle 100,000 concurrent connections with only 8 OS threads. If each connection required a dedicated thread, you would need 100,000 threads — which would require roughly 800 GB of stack space and would spend all their time in the kernel scheduler. The answer is epoll: a Linux kernel interface that lets a single thread wait on thousands of file descriptors simultaneously and be notified only when one is ready for I/O. Go’s runtime uses epoll internally as the foundation of its network I/O model. This lesson explains how.

Lesson 4: TCP/IP Stack — SYN Floods, TIME_WAIT, Connection Tuning

Tue, 11 Jun 2024 00:00:00 +0000

A load test I ran against a Go HTTP server one afternoon produced a strange result: throughput leveled off at about 30,000 requests per second and couldn’t go higher, even though CPU was at 30%. Running netstat -an | grep TIME_WAIT | wc -l showed over 28,000 connections in TIME_WAIT state. The kernel was running out of local port numbers. Understanding TCP connection states — and how to tune Linux’s TCP stack — unblocked the test and taught me more about networking than any course had. This lesson covers the TCP mechanics that actually matter for backend engineers running high-throughput services.

Lesson 3: File Descriptors — Why Too Many Open Files Kills Your Server

Mon, 27 May 2024 00:00:00 +0000

I got a 3 AM page for a service that was returning connection errors to every client. The application logs said dial tcp: lookup ...: too many open files. We hadn’t changed anything. Load was normal. But over 48 hours, something had been slowly accumulating open file descriptors and not closing them, and we hit the per-process limit. Restarting the service fixed the immediate problem; understanding why it happened required me to actually learn what file descriptors are and how the kernel manages them.

Lesson 2: Virtual Memory — 1GB RSS but Only 50MB is Real

Sun, 12 May 2024 00:00:00 +0000

Early in my career I deployed a Go service and checked top. The VIRT column showed 1.2 GB. I nearly had a heart attack — our server had 4 GB of RAM and I thought the service was consuming nearly a third of it. A senior engineer laughed and told me to look at RES instead: 52 MB. I had no idea what the difference was. Understanding virtual memory is fundamental to reading memory metrics correctly, debugging out-of-memory kills, and understanding how processes interact with the kernel.

Lesson 1: Processes and Threads — What Goroutines Map To

Thu, 25 Apr 2024 00:00:00 +0000

When I first started using Go seriously, I accepted goroutines as “lightweight threads” without really understanding what that meant. The Go runtime creates them, schedules them, and I launch them with go. Then I got curious: what does the OS actually see? When I run 10,000 goroutines, does the kernel manage 10,000 things? The answer is no — and understanding why requires understanding the difference between processes, kernel threads, and userspace threads. It also explains why goroutines scale so much better than Java threads or Python threads.