Concurrency on

Lesson 28: Production Concurrency Architecture — Putting it all together

Fri, 20 Feb 2026 00:00:00 +0000

This is the lesson I wish existed when I started. Not “here’s how channels work” or “here’s a simple worker pool” — but the full picture: how you take everything you’ve learned about goroutines, channels, contexts, rate limiting, idempotency, observability, and graceful shutdown and assemble it into a service that actually belongs in production. One that handles failures, recovers gracefully, doesn’t leak resources, and gives you the visibility to understand what it’s doing.

Lesson 27: Idempotency in Concurrent Systems — Assume everything runs twice

Sun, 08 Feb 2026 00:00:00 +0000

Every distributed system I’ve worked on eventually ran something twice. A retry after a timeout. A duplicate webhook delivery. A job that got processed by two workers simultaneously because of a clock skew issue in the claim timeout logic. A user who double-clicked a submit button. Systems aren’t gentle about this — they don’t ask “are you sure?” before running your code again. They just run it. The question isn’t whether your code will ever run twice. It’s whether running it twice causes a problem.

Lesson 26: Safe Background Jobs in Web Servers — Don''t spawn goroutines in handlers

Fri, 23 Jan 2026 00:00:00 +0000

The first time I saw go func() inside an HTTP handler, I thought it was clever. Fire off the slow work in the background, respond to the client fast, everybody wins. Then I watched what happened when we deployed a new version: the server started shutting down, half the background goroutines got killed mid-operation, we had partial writes in the database and no way to know which ones had completed. The “clever” optimization had created a correctness nightmare. The real problem wasn’t the goroutine — it was that it was invisible to the server’s shutdown lifecycle.

Lesson 25: Observability for Concurrency — You can''t fix what you can''t see

Thu, 15 Jan 2026 00:00:00 +0000

The goroutine leak doesn’t announce itself. It just slowly inflates your memory graph — 50MB, 80MB, 150MB — until either your alerting fires or the OOM killer shows up. Then you’re staring at a core dump or (if you’re lucky) a live process trying to figure out which of the 10,000 goroutines running in your service is the culprit. I’ve been there. It’s not fun. The difference between spending 20 minutes diagnosing a leak and spending 4 hours diagnosing a leak is almost entirely whether you invested in observability before the incident.

Lesson 24: Testing Concurrent Code — Flaky tests mean flaky design

Tue, 30 Dec 2025 00:00:00 +0000

A flaky test is a lie your codebase tells you. It says “this sometimes works and sometimes doesn’t” and if you’re honest with yourself, you already know what that means: there’s a race condition somewhere, and the test is just unlucky enough to expose it occasionally. I used to mark flaky tests with t.Skip("flaky, fix later") and move on. I stopped doing that when a race condition that a flaky test was hinting at caused a double-charge bug in production. Now I treat every flaky test as a production incident waiting to happen.

Lesson 23: Distributed vs Local Concurrency — Channels don''t cross process boundaries

Mon, 22 Dec 2025 00:00:00 +0000

There’s a particular kind of confidence that comes after you’ve spent a few months writing Go. You’ve learned channels, you understand the happens-before guarantees, you’ve built a worker pool that hums along beautifully. Then someone says “we need to scale this across multiple pods” and you think: easy, I’ll just make the channels bigger. That thought has caused more production incidents than I care to remember — including one where we lost about 3,000 job records because a pod restarted mid-processing and nobody had thought about what “at-least-once delivery” actually means.

Lesson 22: Rate Limiting and Load Shedding — Say no before you fall over

Wed, 10 Dec 2025 00:00:00 +0000

The most honest thing a server can do is say “no.” Not crash, not time out after 30 seconds, not queue work indefinitely until memory explodes — just return a clean 503 and let the caller decide what to do. I spent a long time thinking rate limiting was about protecting users from themselves. Then I got paged at midnight because a misbehaving client hammered an endpoint, the service queued everything politely, RAM climbed to the ceiling, and the process died. The fix wasn’t more memory. It was teaching the service to refuse work it couldn’t handle.

Lesson 21: Supervisor Patterns — Let it crash, then restart it

Tue, 25 Nov 2025 00:00:00 +0000

Erlang got famous for the “let it crash” philosophy. The idea is that trying to handle every possible error in every possible place produces fragile, complicated code. It’s often better to let a process crash cleanly and have a supervisor restart it. The supervisor knows how to bring the process back to a known-good state. The process itself just needs to do its job and fail fast when something’s wrong.

Lesson 20: Profiling Contention — pprof knows where your goroutines sleep

Thu, 13 Nov 2025 00:00:00 +0000

You’ve added goroutines, you’ve got worker pools, you’ve written careful concurrent code — and the service is still slower than expected. Maybe goroutine count is climbing in your metrics dashboard. Maybe p99 latency has a long tail you can’t explain. Maybe a throughput test plateaus at 40% of what you thought the hardware should support.

This is where guessing stops and profiling starts. Go ships world-class concurrency profiling tools in the standard library — mutex profiles, block profiles, goroutine dumps, and the execution tracer. Most engineers know about the CPU and memory profiler. Far fewer use the concurrency-specific profiles, which is a shame because they find the exact thing that’s wrong in minutes instead of days.

Lesson 19: Ordering vs Throughput — You can have fast or ordered, pick one

Fri, 31 Oct 2025 00:00:00 +0000

Here’s a tension that comes up in almost every real concurrent system: you want to process things fast, which means doing them in parallel, but the output needs to come out in the same order the input arrived. These two goals are in direct conflict. Parallel execution means things finish in unpredictable order. Ordered output means you have to wait for the slowest thing in each batch.

The engineers who understand this tension build systems that make the trade-off explicitly. The engineers who don’t notice it build systems that either throttle themselves to single-goroutine throughput “to preserve order” or silently return results in the wrong order and wonder why tests fail non-deterministically.

Lesson 18: Backpressure Design — Slow down or blow up

Sun, 19 Oct 2025 00:00:00 +0000

Every system has a throughput ceiling. The question isn’t whether your service can be overwhelmed — it can. The question is what happens when it is. Does it slow down gracefully, maintaining correctness and giving the caller a clear signal? Or does it blow up — OOM-killed, goroutines piling up, latency spiking to infinity while requests pile up in an unbounded queue?

Backpressure is the mechanism that answers that question. It’s how a component communicates to its upstream: “I’m full — slow down.” Without it, fast producers eat slow consumers alive. In Go, backpressure is naturally expressed through blocking channel sends — and understanding when to block versus when to drop is one of the more interesting design decisions in concurrent systems.

Lesson 17: Go Scheduler Behavior — M:N scheduling is not magic

Wed, 08 Oct 2025 00:00:00 +0000

Most Go developers write concurrent code for years without thinking about the scheduler. That’s by design — the scheduler is supposed to be invisible. But eventually you’ll hit a situation where goroutines aren’t running when you expect them to, CPU cores are idle while goroutines pile up, or a CPU-bound workload is somehow slower with more goroutines. At that point you need a mental model of what’s actually happening under the hood.

Lesson 29: sync.Cond — The coordination primitive nobody teaches

Wed, 01 Oct 2025 00:00:00 +0000

Every Go concurrency course covers goroutines, channels, mutexes, WaitGroups, and maybe semaphores. Very few touch sync.Cond. It’s either treated as advanced or dismissed as unnecessary because “just use channels.” But there’s a whole class of coordination problem where channels are the wrong tool and sync.Cond is exactly right. Once you see the pattern, you’ll recognize it everywhere — and you’ll stop reaching for time.Sleep polling loops when you shouldn’t.

This is a bonus lesson. Not because the topic is minor, but because it builds on mutexes (Lesson 6) and you really need to understand lock ownership before sync.Cond clicks.

Lesson 16: Atomic Operations — Lock-free when you can, mutex when you must

Mon, 29 Sep 2025 00:00:00 +0000

The first time I looked at sync/atomic, it felt like a niche tool for systems programmers writing lock-free data structures. Turns out it’s one of the most practically useful packages in the standard library — and most Go developers reach for it way too late, after they’ve already built something with a mutex and then profiled it into submission.

Atomic operations are CPU-level instructions that read or modify memory as a single, indivisible unit. There’s no scheduler window between the read and the write. That means multiple goroutines can operate on the same memory location without a lock — and without any blocking. When your shared state is a counter, a flag, or a single configuration value, atomics are almost always the right tool.

Lesson 15: Semaphores for Concurrency Limits — A channel with a size is a semaphore

Thu, 18 Sep 2025 00:00:00 +0000

There’s a class of production bugs I see over and over — and the cause is almost always the same: nothing is telling the program to slow down. A spike in traffic arrives, every goroutine blasts outbound, and suddenly you’ve got five hundred simultaneous connections against a Postgres instance that’s configured for a hundred. The database starts rejecting connections. The application throws errors. Everyone’s paged at 2am.

The fix isn’t complicated. It’s a semaphore — a primitive that limits how many concurrent operations are in flight at once. Go doesn’t ship a dedicated semaphore type, but it doesn’t need to. A buffered channel of the right size is a semaphore, and that insight unlocks a whole class of resource-limiting patterns.

Lesson 14: errgroup for Structured Concurrency — All succeed or all cancel

Tue, 02 Sep 2025 00:00:00 +0000

There’s a pattern that comes up constantly in backend services: make N concurrent calls, collect all their results, and if any one of them fails, cancel the rest and return the error. This is the “all succeed or all cancel” pattern — and before errgroup, implementing it correctly required a non-trivial amount of boilerplate involving WaitGroup, error channels, and manual context cancellation. People got it wrong often enough that errgroup was created specifically to handle it.

Lesson 13: Timeouts Everywhere — Unbounded waits are production bugs

Tue, 26 Aug 2025 00:00:00 +0000

I have a strongly held opinion about timeouts: if you’re making a network call, a database query, or waiting on any external resource without a timeout, you’ve written a production bug. It just hasn’t fired yet. The network will eventually hang. The database will eventually have a slow query. The external API will eventually stop responding. And when it does, your goroutine will wait. And wait. And wait — holding a connection, a file descriptor, a slot in your worker pool — until the process runs out of resources or someone restarts it.

Lesson 12: Leak Prevention — Every goroutine must have an exit

Fri, 15 Aug 2025 00:00:00 +0000

Goroutine leaks are the memory leaks of concurrent Go. They’re slow, invisible, and tend to surface only under load — or worse, only after days of continuous running when the service has accumulated tens of thousands of stuck goroutines. I’ve debugged two production incidents that traced back to leaks in code that had been in production for months, completely unnoticed during normal load.

The root cause is almost always the same: someone started a goroutine and didn’t give it a way to exit. Not a way to exit eventually — a way to exit in every possible code path.

Lesson 11: Graceful Shutdown — Stop accepting, finish what you started

Sun, 03 Aug 2025 00:00:00 +0000

Most Go services handle startup carefully and shutdown carelessly. The startup code has retries, health checks, dependency validation. The shutdown code is os.Exit(0) or — if the developer was feeling generous — nothing at all, just letting the process get killed. That’s how you get dropped HTTP connections, half-written database records, uncommitted Kafka offsets, and on-call alerts at 3am.

Graceful shutdown has one principle: stop accepting new work, finish what you already started. That’s it. But implementing it correctly requires knowing the shutdown order, wiring up OS signals properly, and draining workers before pulling the plug.

Lesson 10: Pipelines and Stage Isolation — Each stage owns its output channel

Thu, 17 Jul 2025 00:00:00 +0000

A pipeline is how you turn a stream of work into a sequence of transformations without writing a single monolithic function that does everything. Each stage reads from one channel, does its thing, and writes to another. Stages compose. Stages are independently testable. And because each stage runs concurrently with every other stage, the whole pipeline processes multiple items simultaneously — like an assembly line rather than a factory worker doing each product start to finish.

Lesson 9: Worker Pools — Bounded concurrency or bust

Thu, 10 Jul 2025 00:00:00 +0000

Production Go codebases that run fine in staging will sometimes crater under real traffic. More often than not, the cause is unbounded goroutine creation. Some background job processor spins up a goroutine per message. The queue backs up. Suddenly there are 50,000 goroutines fighting for CPU, exhausting file descriptors, allocating gigabytes of stack space. The service falls over. The incident post-mortem says “we didn’t expect this load.” The real cause: no worker pool.

Lesson 8: Fan-Out / Fan-In — Distribute, collect, don''t leak

Tue, 24 Jun 2025 00:00:00 +0000

There’s a moment in every Go developer’s journey where the serial loop stops being acceptable. You’ve got 200 URLs to fetch, 500 records to transform, 50 API calls to make — and you’re doing them one at a time. The fix feels obvious: spin up goroutines. But spin them up without a plan and you’ve traded one problem for three.

Fan-out / fan-in is the pattern that makes parallel work manageable. Fan-out means distributing a stream of work across multiple goroutines. Fan-in means collecting their results into a single channel. Simple in concept, surprisingly easy to get wrong in practice.

Lesson 7: Race Conditions and the Go Memory Model — The bug you can't reproduce

Thu, 12 Jun 2025 00:00:00 +0000

Race conditions are the category of bug that makes senior engineers paranoid and junior engineers dismissive. They don’t reproduce consistently. Your tests pass. Your staging environment looks fine. Then, in production, under a specific load pattern at a specific time, two goroutines read and write the same memory address with no synchronization, and you get corrupted data — or a panic — or silently wrong results that sit in your database for three weeks before someone notices. Understanding why they happen at the language level is what separates engineers who prevent them from engineers who just get lucky.

Lesson 6: Mutexes Done Right — The boring tool that actually works

Sat, 31 May 2025 00:00:00 +0000

Channels get the spotlight in Go talks and blog posts — they’re the shiny, idiomatic, philosophically interesting tool. Mutexes feel like the C and Java baggage everyone was supposed to leave behind. But here’s what I’ve learned after several years of writing concurrent Go: mutexes are often the right tool, especially when you’re protecting shared state, and the problems you see in production are almost never “we should’ve used channels” — they’re “we copied the mutex” or “we forgot to narrow the critical section.” Learn to use mutexes properly and stop apologizing for them.

Lesson 5: sync.WaitGroup — Wait for everyone, then move on

Wed, 21 May 2025 00:00:00 +0000

sync.WaitGroup is one of the first concurrency primitives you reach for in Go, and also one of the first you misuse. The API looks deceptively simple — three methods, twenty minutes of reading, you think you’ve got it. Then you hit a negative counter panic in production, or you find out your Wait() returned while goroutines were still running, and you spend an afternoon learning the rules you thought you already knew.

Lesson 4: Buffered vs Unbuffered Channels — Buffering hides bugs

Mon, 12 May 2025 00:00:00 +0000

The first time someone tells you that buffered channels are faster, you believe them — and you spend the next year throwing make(chan T, 100) at every performance complaint, wondering why your system still deadlocks sometimes, and why those deadlocks disappeared when you bumped the buffer to 200. The buffer wasn’t fixing anything. It was delaying the problem. Understanding why requires actually thinking about what buffering changes — not at the performance level, but at the coordination level.

Lesson 3: Channel Ownership Rules — Who closes the channel?

Fri, 25 Apr 2025 00:00:00 +0000

At some point, you’re going to close a channel twice and the runtime is going to panic with close of closed channel. Or you’re going to close a channel from the wrong goroutine and send a value into it right after, and the runtime is going to panic with send on closed channel. These aren’t subtle race conditions that only show up under load — they’re logic errors that exist because nobody in the codebase agreed on who owns the channel. That agreement is the whole game.

Lesson 2: Cancellation with context.Context — Every goroutine needs a kill switch

Sun, 13 Apr 2025 00:00:00 +0000

Without context.Context, you have goroutines running long after the user who triggered them gave up, HTTP connections burning server resources for requests nobody’s waiting for, and database queries executing on behalf of clients that disconnected ten seconds ago. Context is the mechanism Go gives you to propagate “I changed my mind” — and most engineers don’t use it until something in production embarrasses them into learning it properly.

The Problem

Let’s start with the most common mistake — blocking on a slow operation with no timeout:

Lesson 1: Goroutine Lifecycle Management — Who owns this goroutine?

Sun, 06 Apr 2025 00:00:00 +0000

Nobody tells you that the first goroutine you “fire and forget” in production is the one that eventually takes down your service at 3 AM. You start it, it runs, and everything looks fine — until requests pile up, memory climbs, and your dashboards turn red because five hundred goroutines are blocked waiting on a channel that’ll never receive another value. The problem isn’t that goroutines are dangerous. It’s that nobody taught you to think about who owns them.

Lesson 25: Production Concurrency Architecture — Putting it all together

Sun, 22 Dec 2024 10:00:00 +0000

After 24 lessons of building blocks, let’s talk about how they compose in real systems. I’ve shipped concurrent Rust services handling millions of requests per day, and the architecture patterns that survive production are surprisingly consistent. Not because they’re clever — because they’re boring. Boring is good when your pager is involved.

This lesson is the blueprint I wish I’d had when I started building concurrent Rust systems for real.

Lesson 24: Testing Concurrent Code — Loom and beyond

Sat, 21 Dec 2024 09:15:00 +0000

I spent a full week writing tests for a lock-free queue. Ran them a thousand times — all green. Shipped it. Two days later, a production crash. A race condition that occurred roughly once every 50,000 operations under specific timing. My tests never hit it because standard testing can’t explore all possible thread interleavings. That’s when I found Loom.

Testing concurrent code is fundamentally different from testing sequential code. A test that passes doesn’t mean the code is correct — it means the code was correct for that particular thread scheduling. Run the same test with different timing and you might get a different result.

Lesson 23: GPU Computing from Rust — wgpu and compute shaders

Thu, 19 Dec 2024 11:35:00 +0000

The first time I ran a matrix multiplication on a GPU, my jaw dropped. A computation that took 8 seconds on an 8-core CPU finished in 40 milliseconds on a mid-range GPU. That’s a 200x speedup. GPUs have thousands of cores — small, simple cores designed for massively parallel uniform computation.

Rust’s GPU story has gotten remarkably good. wgpu gives you cross-platform GPU compute that works on Vulkan, Metal, DX12, and even in the browser via WebGPU. No CUDA lock-in.

Lesson 22: SIMD — Explicit vectorization

Tue, 17 Dec 2024 08:20:00 +0000

I had an image processing pipeline that took 4.2 seconds per frame on a single core. After rewriting the hot loop with SIMD intrinsics, it dropped to 0.9 seconds. Same core, same algorithm, 4.7x faster. SIMD doesn’t add more cores — it makes each core do more work per clock cycle.

SIMD (Single Instruction, Multiple Data) is the other kind of parallelism. While threads run code on different cores, SIMD runs the same operation on multiple data elements simultaneously within a single core.

Lesson 21: CSP-Style Concurrency — Go channels in Rust

Sun, 15 Dec 2024 10:40:00 +0000

Before I wrote Rust full-time, I spent two years writing Go. The goroutine-plus-channel model gets into your brain. You start thinking about problems as independent processes connected by typed pipes. When I switched to Rust, the first thing I looked for was the equivalent of Go’s select statement. Crossbeam has it — and in some ways it’s even better.

CSP (Communicating Sequential Processes) is the formal model behind Go’s concurrency. The idea: concurrent processes interact only by passing messages through channels. No shared memory. Each process is sequential internally. Concurrency comes from composition.

Lesson 20: The Actor Model with Rust — Message-passing architectures

Fri, 13 Dec 2024 14:20:00 +0000

The first time I saw Erlang’s actor model, I thought it was over-engineered. Every piece of state behind a process, every interaction a message. Then I worked on a system with 40 mutexes, 12 deadlock-prone code paths, and a debugging story that involved printf-ing thread IDs into a file and diffing them. I became an actor model convert that week.

Rust doesn’t have a built-in actor system like Erlang or Akka. But the building blocks — channels, threads, ownership transfer — make building one surprisingly natural.

Lesson 19: Thread-Local Storage — Per-thread state

Wed, 11 Dec 2024 09:55:00 +0000

I was optimizing a JSON serializer that allocated a buffer for every call. Under profiling, those allocations were 30% of the cost. The fix? A thread-local buffer that gets reused across calls on the same thread. No synchronization needed. No contention. Each thread has its own buffer. Throughput doubled.

Thread-local storage is the ultimate escape hatch from synchronization overhead. If each thread has its own copy, there’s nothing to synchronize.

Lesson 18: Debugging Deadlocks and Data Races — Tools and techniques

Mon, 09 Dec 2024 11:30:00 +0000

The worst deadlock I ever encountered wasn’t between two mutexes. It was between a mutex and a channel. Thread A held a lock and tried to send on a full bounded channel. Thread B was the consumer for that channel but was waiting to acquire the same lock before it could receive. Clean deadlock. Took me four hours to find because I was looking at mutex ordering and the real problem was a channel.

Lesson 17: Lock-Free Data Structures in Rust — Beyond mutexes

Sat, 07 Dec 2024 08:50:00 +0000

A few years ago I was profiling a metrics collection service and found that 60% of the CPU time was spent on mutex contention. Thirty-two threads, one mutex protecting a counter map. The actual “work” — incrementing counters — took nanoseconds. The locking overhead was three orders of magnitude more expensive than the operation it protected.

That’s when lock-free data structures become worth the complexity. When the lock is the bottleneck, remove the lock.

Lesson 16: Barriers and Once — Synchronization primitives

Thu, 05 Dec 2024 10:15:00 +0000

I was building a benchmark suite once — eight threads, each measuring throughput of a different operation. The problem was that threads started at different times depending on OS scheduling. Thread 0 might start 50ms before thread 7, skewing the results. I needed all threads to start their measurement at exactly the same time.

That’s what barriers do. Everyone waits at the barrier until the last thread arrives, then they all proceed together.

Lesson 15: Condvar — Waiting for conditions

Tue, 03 Dec 2024 14:40:00 +0000

Early in my career, I wrote a producer-consumer queue using a mutex and a busy-wait loop. The consumer would lock the mutex, check if there’s data, unlock, sleep for 10 milliseconds, and repeat. It worked, but it burned CPU doing nothing and had up to 10ms of latency on every message. My tech lead pointed me to condition variables, and the latency dropped to microseconds while CPU usage went to near zero.

Lesson 14: parking_lot — Faster mutexes

Sun, 01 Dec 2024 09:25:00 +0000

I switched a high-contention service from std::sync::Mutex to parking_lot::Mutex and saw lock acquisition time drop by 30% under load. The API is nearly identical — it was a find-and-replace job. That’s the kind of optimization I like: massive payoff, zero complexity cost.

parking_lot is one of those crates that probably should have been in the standard library. It provides the same synchronization primitives as std, but faster, smaller, and with more features.

Lesson 13: Fan-Out Fan-In in Rust — Parallel pipelines

Fri, 29 Nov 2024 12:50:00 +0000

One of the most satisfying architectures I’ve built was a real-time analytics pipeline. Events came in through a single ingestion point, fanned out to eight processing workers, then fanned back in to a single aggregator that wrote results to the database. Throughput went from 2,000 events/second to 14,000 with zero data loss.

Fan-out/fan-in is one of those patterns that looks simple on a whiteboard but has real subtlety in implementation. Getting the channel lifecycle right, handling errors, managing backpressure — that’s where things get interesting.

Lesson 12: Worker Pool Patterns — Bounded concurrency

Wed, 27 Nov 2024 08:35:00 +0000

I blew up a production database once by spawning unlimited concurrent connections. A batch job that normally processed 100 items suddenly got 50,000. Each item opened a database connection. The connection pool maxed out, then the OS ran out of file descriptors. The database server stopped accepting connections from any service. All because I wrote for item in items { thread::spawn(...) } without thinking about bounds.

Worker pools solve this. Fixed number of threads, bounded queue, backpressure when the system is overloaded. After that incident, I never write unbounded concurrency again.

Lesson 11: Crossbeam — Scoped threads and lock-free structures

Mon, 25 Nov 2024 11:05:00 +0000

Before Rust 1.63 added thread::scope to the standard library, crossbeam was the only ergonomic way to spawn threads that could borrow local data. Even now that std has scoped threads, crossbeam remains essential. Its channels are faster than std::sync::mpsc, it provides lock-free data structures, and its utilities fill gaps the standard library doesn’t cover.

I reach for crossbeam in nearly every concurrent Rust project. Here’s why.

Crossbeam Channels

The biggest win: crossbeam’s channels are multi-producer, multi-consumer (MPMC) and significantly faster than std::sync::mpsc.

Lesson 10: Rayon — Data parallelism made easy

Sat, 23 Nov 2024 15:20:00 +0000

I had a batch processing job — reading 50,000 JSON files, parsing them, running validation, writing results. Single-threaded, it took 12 minutes. I added Rayon, changed .iter() to .par_iter(), and it dropped to 90 seconds. Five characters added to my code. That’s it.

Rayon is probably the most impressive crate in the Rust ecosystem for the effort-to-impact ratio. If you have CPU-bound work that operates on collections, Rayon makes parallelism trivial.

Lesson 9: Send and Sync — The traits behind thread safety

Thu, 21 Nov 2024 09:40:00 +0000

There’s a moment in every Rust developer’s life when they try to send an Rc<RefCell<Vec<String>>> to another thread and get a wall of compiler errors. They Google the error, see something about Send and Sync, patch the type to Arc<Mutex<Vec<String>>>, and move on without understanding why.

I did exactly that for six months. Then I needed to write a custom type that crossed thread boundaries, and I had to actually learn what these traits mean. Turns out they’re beautifully simple once you see the design.

Lesson 8: Memory Ordering — Relaxed, Acquire, Release, SeqCst

Tue, 19 Nov 2024 13:15:00 +0000

Memory ordering is the thing that separates people who use concurrent code from people who write concurrent primitives. I avoided understanding it for years, using SeqCst everywhere like a safety blanket. It worked. But it left performance on the table and, more importantly, left me unable to read half the lock-free code I encountered.

So here’s the actual explanation — no hand-waving.

Why Ordering Matters

Modern CPUs and compilers reorder instructions for performance. Your code says “write A, then write B,” but the CPU might execute B first if there’s no dependency between them. On a single thread, this is invisible — the final result is the same.

Lesson 7: Atomics — Lock-free primitives

Sun, 17 Nov 2024 07:45:00 +0000

I once replaced a Mutex<u64> counter in a hot path with an AtomicU64 and saw throughput jump 40%. Not because mutexes are slow — they’re fast. But for a single integer being incremented by 32 threads, the overhead of acquiring and releasing a lock millions of times per second adds up to real time.

Atomics are the foundation of lock-free programming. They let you do thread-safe operations on primitive values without any lock at all.

Lesson 6: Arc> — The shared mutable state pattern

Fri, 15 Nov 2024 10:30:00 +0000

I remember staring at Arc<Mutex<HashMap<String, Vec<u8>>>> in a codebase and thinking “this is the ugliest type I’ve ever seen.” Three months later, after debugging a similar system in Go that had zero type safety around its concurrent map access, I came crawling back to Rust’s ugly-but-correct approach.

The Arc<Mutex<T>> pattern is everywhere in Rust concurrent code. Understanding why it exists — not just how to type it — is the key.

Lesson 5: Shared State — Mutex, RwLock, and poisoning

Wed, 13 Nov 2024 16:55:00 +0000

The nastiest production bug I ever tracked down involved a Java ConcurrentHashMap that was “thread-safe” in the API sense but not in the logic sense. Two threads would read a key, both see it’s absent, both insert with computed values, and one would silently overwrite the other. The map itself was fine — the access pattern was broken.

Rust’s Mutex won’t save you from logic bugs either. But it will absolutely prevent you from forgetting the lock in the first place.

Lesson 4: Channels — mpsc and beyond

Mon, 11 Nov 2024 09:10:00 +0000

The first concurrent system I built that actually worked well was a log aggregation pipeline. Multiple producers writing log lines, one consumer batching and flushing to disk. No shared state, no locks, no races. Just messages flowing through a pipe.

That experience sold me on message passing. And Rust’s channel implementation makes it surprisingly ergonomic.

The Problem: Shared State Is Hard

You can share state between threads with mutexes. But every shared mutable variable is a coordination point, a potential bottleneck, and a source of bugs. The more threads touching the same data, the harder the code is to reason about.

Lesson 3: move Closures — Sending data to threads

Sat, 09 Nov 2024 14:20:00 +0000

When I first started writing threaded Rust code, I hit the same compiler error about forty times in one afternoon. Something about closures borrowing values that might be dropped. I kept slapping move on closures until things compiled, without really understanding what was happening.

That’s a terrible way to learn. So here’s the actual explanation I wish I’d had.

The Problem: Closures and Thread Lifetimes

When you pass a closure to thread::spawn, the new thread might run for an arbitrary amount of time. It could outlive the function that spawned it. It could outlive the variables it references.

Lesson 2: std::thread — Spawning and joining

Thu, 07 Nov 2024 11:45:00 +0000

A few years back I was reviewing a Go service that spawned goroutines like confetti at a parade — hundreds of them, no tracking, no lifecycle management. When the service shut down, half those goroutines just vanished mid-work. Orphaned database connections everywhere. The Go runtime made it so easy to fire off concurrent work that nobody stopped to think about cleanup.

Rust’s threading model forces you to think about it. Every thread gives you a handle. Every handle demands acknowledgment. You can ignore it — but you have to do so explicitly.

Lesson 1: Why Rust Concurrency Is "Fearless" — The compiler has your back

Tue, 05 Nov 2024 08:30:00 +0000

I once spent three days chasing a race condition in a Java service that only manifested under production load. The bug? Two threads updating a shared HashMap — no synchronization, no errors at compile time, no warnings. Just silent data corruption that showed up as incorrect billing amounts. Three days of my life, gone, because the language didn’t care.

Then I tried to write the same bug in Rust. The compiler said no.