Internals on

Lesson 8: Runtime Comparison — Tokio vs async-std vs smol vs glommio

Thu, 02 Oct 2025 16:55:31 +0000

Every few months, someone asks me “which async runtime should I use?” and every time my answer is frustrating: “it depends.” But after spending the last seven lessons understanding how runtimes work from the inside, we can finally have a real conversation about what the differences actually are, why they exist, and when each one is the right choice.

I’ve used all four of these runtimes in production. Tokio for most things, smol for a lightweight embedded project, glommio for a storage engine prototype, and async-std briefly before migrating away from it. Let me share what I’ve learned — not from reading documentation, but from debugging real problems at 2 AM.

Lesson 7: Designing a Custom Async Runtime — When Tokio isn't enough

Tue, 30 Sep 2025 11:20:06 +0000

A colleague once asked me, “Why would anyone build a custom async runtime when Tokio exists?” Fair question. Tokio is battle-tested, well-maintained, and fast. For 95% of use cases, it’s the right answer. But I’ve now been in three situations where it wasn’t — and each one taught me something about what a runtime actually does.

The first was a latency-sensitive trading system where work-stealing’s cache invalidation was unacceptable. The second was an embedded system with no allocator. The third was a specialized database engine where we needed precise control over I/O scheduling. Each time, understanding how to build a runtime from the ground up saved the project.

Lesson 6: Work-Stealing Schedulers — Balancing load across cores

Sat, 27 Sep 2025 13:45:50 +0000

I once spent two days debugging a performance issue where our Tokio application was using four cores but only one was doing any work. Three threads were idle, one was pegged at 100%. The problem wasn’t Tokio’s scheduler — it was ours. We’d accidentally created a pattern where all tasks were spawned from and waking on the same thread, and nothing triggered work-stealing because the tasks completed too quickly.

That experience taught me that you can’t treat the scheduler as a black box. If you understand how work-stealing works, you can design your task topology to take advantage of it. If you don’t, you’ll write code that accidentally defeats it.

Lesson 5: The Reactor Pattern — How Tokio actually works

Wed, 24 Sep 2025 09:27:13 +0000

I read the Tokio source code on a Sunday afternoon, expecting to find something inscrutable. Layers of unsafe code, impenetrable abstractions, the kind of thing that makes you question your career choices. What I actually found was a clean, well-documented reactor implementation that I could follow. Not easily — but followably. The architecture is elegant once you see how the pieces connect.

This lesson is about that architecture. Not Tokio’s API — you already know how to use tokio::spawn and TcpStream. This is about what happens underneath when you call those functions. The reactor pattern, the I/O driver, the timer wheel, and how they all feed into the executor.

Lesson 4: epoll/kqueue — Platform event loops

Sun, 21 Sep 2025 17:08:45 +0000

Before I understood event loops, I thought async I/O was some kind of kernel magic. You register interest in a socket, and somehow the OS tells you when data arrives — without blocking a thread. How? I imagined complex kernel subsystems doing heavy lifting behind the scenes.

Turns out, the mechanism is almost embarrassingly simple. The kernel maintains a list of file descriptors you care about. When something happens on one of them, it flips a bit. You ask “what happened?”, it tells you. That’s it. The entire async I/O ecosystem — Tokio, Node.js, nginx, everything — is built on this one primitive.

Lesson 3: io_uring — Zero-copy async I/O on Linux

Fri, 19 Sep 2025 10:33:08 +0000

I ran a benchmark last year that genuinely surprised me. A simple TCP echo server using io_uring was handling 40% more connections per second than the same server on epoll, with measurably lower tail latencies. Not 5% — forty percent. On the same hardware, same kernel version, same application logic. That’s when I stopped treating io_uring as a curiosity and started treating it as the future of Linux I/O.

If you’ve been building async runtimes on epoll (or even kqueue on macOS), io_uring changes the game completely. Let me show you why.

Lesson 2: Building a Minimal Executor — Your own async runtime

Wed, 17 Sep 2025 14:12:37 +0000

The moment I built my first executor from scratch, async Rust stopped being scary. Not because executors are simple — they’re not — but because once you see the machinery, every “mysterious” behavior has an obvious explanation. Futures hanging? The waker isn’t being called. Tasks not making progress? The executor’s run loop has a bug. Performance terrible? You’re probably polling too aggressively or not enough.

So let’s build one. A real, working executor. Not a production-quality one (that’s Tokio’s job), but one that actually runs futures to completion, handles multiple tasks, and demonstrates every concept from the previous lesson.

Lesson 1: Future Internals — Poll, Waker, Context

Mon, 15 Sep 2025 08:45:22 +0000

I thought I understood Rust futures until I tried to implement one without async/await. Not a toy future that immediately returns Ready. A real future — one that yields, gets woken up, and resumes where it left off. That exercise broke every mental model I had and rebuilt it from scratch.

The async/await syntax is one of Rust’s great lies. It looks simple. It feels like you’re writing sequential code. Under the hood, the compiler is generating state machines, threading waker references through call graphs, and constructing self-referential structs that would make most C++ programmers nervous. Let’s rip the lid off.

Lesson 8: String Internals — Immutable, backed by bytes, cheaper than you think

Tue, 15 Apr 2025 00:00:00 +0000

I used to reach for []byte over string in performance-sensitive code, assuming strings were somehow more expensive because “immutability must cost something.” That intuition was wrong. Understanding what a string actually is — a read-only slice header — corrected my assumptions and simplified a lot of code I had unnecessarily complicated.

The Problem

Strings in Go are everywhere, and most developers use them without thinking about what they are at the memory level. This leads to a few common mistakes:

Lesson 10: How rustc Works — From source code to binary

Sat, 22 Mar 2025 11:30:00 +0000

A junior on my team once asked why Rust compiles so slowly compared to Go. I gave the standard answer about monomorphization and LLVM, but realized I couldn’t actually explain the full pipeline. So I spent a weekend reading the rustc dev guide and poking at compiler internals. What I found was a surprisingly elegant six-stage pipeline, and understanding it changed how I think about Rust’s design tradeoffs.

The Big Picture

When you run cargo build, your source code goes through six major stages before becoming a binary:

Lesson 9: Miri — Your safety net for unsafe Rust

Wed, 19 Mar 2025 08:45:00 +0000

I once shipped a Rust library with an unsafe block that worked perfectly on x86, passed all tests, and ran flawlessly in production for months. Then someone compiled it on ARM and it segfaulted immediately. The undefined behavior had been lurking the whole time — x86 just happened to tolerate the misaligned read that ARM wouldn’t. If I’d run Miri before releasing, I would have caught it in thirty seconds. Lesson learned the hard way.

Lesson 8: Custom Allocators — GlobalAlloc, arena allocation, and beyond

Sun, 16 Mar 2025 13:10:00 +0000

I was building a JSON parser that processed millions of small documents per second. Profiling showed that 40% of the time was spent in malloc and free — not parsing, not I/O, just allocating and deallocating tiny strings and vectors. Switched to an arena allocator that bulk-freed everything after each document, and throughput doubled. That experience taught me that the allocator isn’t just plumbing — it’s a performance lever.

How Rust Allocates Memory

When you write Box::new(42) or Vec::with_capacity(100), Rust asks the global allocator for memory. By default, this is the system allocator — malloc/free on Unix, HeapAlloc/HeapFree on Windows. It’s a general-purpose allocator designed to handle any allocation pattern reasonably well, but it’s not optimized for any specific pattern.

Lesson 7: Drop Order — Deterministic destruction and why it matters

Thu, 13 Mar 2025 07:20:00 +0000

I once had a deadlock in a Rust program that only manifested during shutdown. A mutex guard and a database connection were being dropped in the wrong order — the connection’s destructor tried to acquire the mutex that was already locked by the guard that hadn’t been dropped yet. Rust’s drop order is deterministic, but “deterministic” doesn’t mean “obvious.” Knowing the rules saved me hours of debugging.

The Drop Trait

In Rust, cleanup logic is implemented through the Drop trait. When a value goes out of scope, the compiler calls drop() on it automatically. This is RAII — Resource Acquisition Is Initialization — borrowed from C++ but made more reliable by the ownership system.

Lesson 6: repr(C), repr(transparent), repr(packed) — Taking control of memory layout

Tue, 11 Mar 2025 09:55:00 +0000

The first time I wrote Rust FFI bindings to a C library, I defined a struct, passed it across the boundary, and got garbage data back. The C side was reading fields at fixed offsets, but Rust had silently reordered my fields for padding efficiency. Took me twenty minutes of staring at hex dumps to realize the layouts didn’t match. That’s the day I learned about #[repr(C)].

Default Rust Layout: repr(Rust)

By default, Rust structs use repr(Rust) — the compiler is free to reorder fields, add padding wherever it wants, and generally optimize the layout however it sees fit. The only guarantees are:

Lesson 5: Fat Pointers — &dyn Trait, &[T], and &str under the hood

Sun, 09 Mar 2025 16:40:00 +0000

I remember being confused why &str was 16 bytes on a 64-bit system. A pointer is 8 bytes — what’s the other 8? That question sent me down a rabbit hole that fundamentally changed how I think about Rust’s type system. Turns out, some references carry extra baggage, and that baggage is the entire reason dynamically sized types work.

Thin Pointers vs Fat Pointers

Most references in Rust are “thin” — a single machine word pointing to the data:

Lesson 4: vtables — How dyn Trait actually works under the hood

Fri, 07 Mar 2025 11:15:00 +0000

I was profiling a parser once and found that a hot path using dyn Iterator was 3x slower than the equivalent code using generics. The algorithm was identical. The difference? Dynamic dispatch — every method call went through a vtable lookup instead of being inlined. That day I learned to respect what dyn really costs. Let’s open the hood.

Static vs Dynamic Dispatch

Rust gives you two ways to call methods on trait objects: static dispatch (generics) and dynamic dispatch (dyn Trait).

Lesson 3: Box — Heap allocation by choice

Wed, 05 Mar 2025 08:30:00 +0000

When I first started writing Rust, I used Box everywhere. Came from a Java background where everything lives on the heap, and wrapping things in Box felt natural. Took me a while to realize I was fighting the language — most of the time, you don’t need it. But when you do need Box, nothing else will do.

What Box Actually Is

Box<T> is the simplest smart pointer in Rust. It allocates a value of type T on the heap and gives you ownership of that allocation through a pointer on the stack. When the Box goes out of scope, it frees the heap memory. That’s it. No reference counting, no garbage collection, no magic.

Lesson 2: Stack vs Heap — Where your data actually lives

Mon, 03 Mar 2025 14:45:00 +0000

A colleague once asked me why their Rust program was ten times slower than expected. They were allocating a Vec<u8> inside a tight loop — millions of heap allocations per second. Moved the vec outside the loop, pre-allocated with with_capacity, and the function went from 2 seconds to 80 milliseconds. Stack vs heap isn’t an academic distinction. It’s the difference between fast code and slow code.

Two Memory Regions, Two Personalities

Your program has (at minimum) two regions of memory to work with: the stack and the heap. They behave differently, perform differently, and serve different purposes. Rust makes the choice between them more explicit than most languages, which is one of the reasons it’s fast by default.

Lesson 1: Memory Layout — Size, alignment, and the padding you never asked for

Sat, 01 Mar 2025 10:22:00 +0000

I spent an embarrassing amount of time debugging a networking project where my hand-crafted packet structs were mysteriously three bytes too large. Turns out the compiler was inserting padding I didn’t know about. That’s when I realized most Rust developers — myself included — treat memory layout as a black box. Let’s crack it open.

Why Layout Matters

Every type in Rust has two fundamental properties: size and alignment. The size is how many bytes the value occupies. The alignment is which memory addresses the value is allowed to start at. These two numbers determine everything about how your data sits in memory, how much RAM your program uses, and whether your structs can talk to C code or hardware registers.

Lesson 7: Values Copy vs Share — When Go copies and when it doesn't

Sat, 01 Mar 2025 00:00:00 +0000

Early in my Go career I had a bug where I modified a slice inside a function expecting the caller’s slice to remain unchanged — and it did. Then I had a different bug where I expected the opposite and the caller’s slice was modified. I had no mental model for predicting which would happen. Building that model is what this lesson is about.

The Problem

Go passes everything by value. That’s the official line, and it’s true. But “by value” means different things for different types:

Lesson 6: Memory Alignment and Struct Padding — Field order affects struct size

Wed, 08 Jan 2025 00:00:00 +0000

I once reviewed a PR where someone had defined a struct with a bool field between two int64 fields, resulting in a 24-byte struct instead of the 17 bytes you might naively calculate. When I pointed it out, the response was “the compiler should handle that.” It doesn’t. Go respects the field order you give it, and it inserts padding to satisfy alignment requirements. Knowing the rules means you write efficient structs by default.

Lesson 5: GC Behavior and Tuning — GOGC and GOMEMLIMIT changed the game

Fri, 15 Nov 2024 00:00:00 +0000

For years, tuning Go’s GC meant tweaking GOGC and hoping for the best. I operated on vibes and guesswork. Then Go 1.19 introduced GOMEMLIMIT — a hard memory limit that fundamentally changed how I reason about GC tuning. Suddenly I had a second axis of control that actually matched how production memory is constrained. This lesson is the mental model I wish I had from day one.

The Problem

Go uses a concurrent, tri-color mark-and-sweep garbage collector. “Concurrent” means most of the GC work happens while your program is running, not in stop-the-world pauses. This is generally excellent, but the GC needs to be triggered somehow, and the default trigger can be surprising.

Lesson 4: Escape Analysis Deep Dive — The compiler decides where your data lives

Sat, 05 Oct 2024 00:00:00 +0000

I used to assume that every time I wrote &someStruct{} or returned a pointer from a function, I was creating a heap allocation. I was wrong — and the wrongness mattered for how I was designing APIs. After learning about escape analysis, I stopped guessing and started asking the compiler directly. This changed how I write Go.

The Problem

Go has two places to put data: the stack and the heap. Stack allocations are cheap — they’re just a pointer bump, and the stack frame is reclaimed automatically when the function returns. Heap allocations go through the memory allocator, require garbage collection, and have non-trivial overhead.

Lesson 3: Method Sets and Addressability — Why your value can't satisfy that interface

Sun, 25 Aug 2024 00:00:00 +0000

I spent a solid twenty minutes staring at a compilation error that read “does not implement interface (Write method has pointer receiver)” and thinking: I can see the Write method right there. What is Go complaining about? Once I understood method sets and why addressability matters, the rule became obvious — and I have never needed to look it up since.

The Problem

Go lets you define methods with either a value receiver or a pointer receiver:

Lesson 2: Nil Interface Gotchas — nil is not nil when it has a type

Thu, 18 Jul 2024 00:00:00 +0000

A few months into my first production Go service, I had a bug that cost me two hours. A function returned an error, I checked if err != nil and it passed — but then further down the stack the program panicked trying to call .Error() on a nil pointer. The function had returned a nil *MyError, not a nil error. These are not the same thing, and until you internalize why, you will hit this bug.

Lesson 1: Interface Representation — Two words that carry your abstractions

Mon, 10 Jun 2024 00:00:00 +0000

I remember the first time I hit a bug where err != nil returned true even though the function clearly returned nil. I spent forty minutes staring at perfectly reasonable-looking code before I realized I had no idea what an interface actually is at the memory level. Once I understood the two-word representation, that class of bug never confused me again.

The Problem

Go interfaces are the backbone of polymorphism in the language. You use them constantly — io.Reader, error, fmt.Stringer. But most developers treat them as magic. You assign a concrete value, you call methods, and somehow Go figures out which implementation to dispatch to at runtime.