Security on

Lesson 8: Security Fuzzing — Finding vulnerabilities before attackers do

Sun, 18 May 2025 13:29:00 +0000

I found a panic in a production parser by accident last year. A user in Japan sent a request with a multi-byte UTF-8 character right at a boundary where our code was slicing a string by byte index. It worked fine for ASCII. It worked fine for most Unicode. But this particular combination of character and position triggered an index-out-of-bounds panic that crashed the request handler.

I fixed the bug in ten minutes. What bothered me was that we’d had unit tests, integration tests, and even some property-based tests — and none of them caught it. The input space was too large. The edge case was too specific. A human writing test cases would never think to put a 3-byte UTF-8 character at exactly that offset.

Lesson 7: Sandboxing and Privilege Dropping — Least privilege

Thu, 15 May 2025 07:41:00 +0000

Here’s a pattern I’ve seen too many times: a Rust web service runs as root in a Docker container because “it needs to bind port 443.” The service handles user uploads, parses JSON, processes images, and talks to a database — all with root privileges. If any part of that pipeline has a vulnerability, the attacker gets root on the container. And if the container isn’t properly isolated, they might get the host too.

Lesson 6: Supply Chain Security — Lockfiles, vendoring, and trust

Mon, 12 May 2025 10:08:00 +0000

The xz backdoor was a wake-up call for the entire industry, but honestly, supply chain attacks had been happening for years before that — just more quietly. Typosquatting on npm, malicious PyPI packages, compromised maintainer accounts. The question isn’t whether Rust’s ecosystem is vulnerable to supply chain attacks. It is. The question is what you’re doing about it.

I spent a week last year hardening our build pipeline after we realized that a cargo build on our CI server was pulling fresh crate downloads from the internet with no verification beyond what Cargo does by default. If crates.io got compromised, or if our DNS got hijacked, we’d be compiling and shipping attacker code with zero friction.

Lesson 5: Dependency Auditing — cargo-audit and cargo-deny

Fri, 09 May 2025 14:55:00 +0000

You know what keeps me up at night? Not my own code — I can review that. It’s the 200+ transitive dependencies in my Cargo.lock that I’ve never read a single line of. Every one of those crates runs with the same permissions as my code. If any of them has a vulnerability, it’s my vulnerability.

This isn’t theoretical. In 2024, the xz backdoor showed that even core infrastructure maintained by a single person can be compromised. Rust’s ecosystem isn’t immune. We’ve had actual advisories for real crates — buffer overflows in parsing libraries, unsound unsafe code in popular crates, logic bugs in crypto implementations.

Lesson 4: Secret Management — zeroize and secure memory

Wed, 07 May 2025 08:23:00 +0000

A while back I was debugging a crash in production and pulled a core dump from the server. Sitting right there in the heap, in plain text, was a database connection string with credentials. The service had loaded the secret from Vault on startup, stored it in a regular String, and that String stayed in memory for the entire process lifetime. When it crashed, the secret got written to disk in the core dump.

Lesson 3: Cryptography — ring, RustCrypto, and sodiumoxide

Mon, 05 May 2025 11:47:00 +0000

I’m going to say something controversial: most developers should never write cryptographic code. Not because they’re not smart enough — because the field is absurdly hostile to even tiny mistakes. A single branch in your constant-time comparison function leaks timing information. A reused nonce in AES-GCM completely destroys confidentiality. An ECDSA implementation with a biased random number generator leaks your private key after enough signatures.

But you still need to use cryptography. Every production system needs hashing, encryption, signatures, or key derivation at some point. The trick is picking the right library, using it correctly, and understanding just enough of the theory to avoid the common footguns.

Lesson 9: Dependency Scanning — govulncheck before you deploy

Mon, 05 May 2025 00:00:00 +0000

The security posture of your Go application is not just about the code you write — it is also about the code you import. A typical Go microservice will have dozens of direct and transitive dependencies. Any one of them might have a known vulnerability in the specific version you are using. Unlike the bugs in your own code, these vulnerabilities are publicly catalogued, exploits are often published, and attackers scan for them at scale.

Lesson 2: Input Validation and Sanitization — Trust nothing

Sat, 03 May 2025 16:12:00 +0000

A few months ago, I was reviewing a PR where someone had written a REST API handler that took a user-supplied filename, appended it to a base path, and opened the file. The code compiled perfectly. Clippy was happy. Tests passed. And it was a textbook path traversal vulnerability — ../../etc/passwd would work just fine.

Rust’s type system protects you from memory corruption. It doesn’t protect you from trusting user input. That’s still on you. And honestly? It’s where most production vulnerabilities in Rust code are going to come from.

Lesson 1: Memory Safety — What Rust gives you for free

Thu, 01 May 2025 09:34:00 +0000

Last year I inherited a C++ service that had been “battle-tested” in production for three years. Within a week of digging through crash dumps, I found two use-after-free bugs, a buffer overread that leaked heap data into API responses, and a data race in the connection pool that only triggered under load. Three years. Battle-tested. Right.

That experience is what finally pushed me from “Rust is interesting” to “Rust is non-negotiable for anything touching the network.” The memory safety guarantees aren’t just a nice-to-have — they’re the single biggest security win you get by choosing Rust.

Lesson 8: Secure HTTP Defaults — Your production server needs these headers

Mon, 10 Mar 2025 00:00:00 +0000

When I run the Go HTTP server in the standard library with no configuration, it does a lot of things right — it is fast, it handles HTTP/2, it is well-tested. But it also does a handful of things that are wrong for production: no timeouts, no response headers that protect against common browser-based attacks, and server identification information in responses. These are not bugs in the standard library; they are defaults appropriate for development that need to be changed before you deploy.

Lesson 7: JWT Caveats — JWTs are not sessions

Sat, 18 Jan 2025 00:00:00 +0000

JWT has become the default answer to “how should I handle authentication tokens?” in the Go community. I have shipped JWTs in production and I have also shipped systems that would have been much simpler and more secure with server-side sessions. The point of this lesson is not that JWTs are bad — it is that they carry specific security risks that are easy to overlook, and they solve a specific problem (stateless authentication across services) that not every application actually has.

Lesson 6: Password Hashing — bcrypt or argon2, nothing else

Mon, 02 Dec 2024 00:00:00 +0000

Password hashing is one of those topics where the correct answer is short and clear, the wrong answers are numerous and subtle, and developers who confidently implement one of the wrong answers often do not know they did anything wrong until a database is leaked and journalists start writing about it. I have sat in a post-mortem where the words “we were using MD5” were spoken in a conference room full of very quiet people. That was not my code, but I understood how it happened — MD5 was the “hash function” the developer knew, and they did not realize it was entirely unsuitable for passwords.

Lesson 5: Auth Middleware — Authentication is not authorization

Tue, 22 Oct 2024 00:00:00 +0000

A few years ago I audited a Go API where every endpoint was protected by an authentication middleware. The middleware checked for a valid JWT, extracted the user ID, and set it in the request context. The developer was proud of it — every route was secured. The problem was that the product had a concept of “organizations” — users belonged to organizations — and the API let you fetch any organization’s data as long as you were authenticated. The authentication was solid. The authorization was completely absent.

Lesson 4: SSRF and Injection — The URL your user gave you might be localhost

Wed, 18 Sep 2024 00:00:00 +0000

I reviewed a Go service that let users provide a “webhook URL” — we would call that URL when their account had a notification. The service fetched a preview of the URL to display in the UI. The developer who built it figured that since it only fetched the URL and never executed what was returned, it was safe. They had not considered that http://169.254.169.254/latest/meta-data/iam/security-credentials/ is also a URL.

SSRF — Server-Side Request Forgery — is the class of vulnerability where an attacker tricks your server into making an HTTP request to a target of their choosing, typically to access internal services that are not exposed to the internet. AWS instance metadata, Redis, internal Kubernetes API servers, admin dashboards, other services in your VPC — all are reachable from a server that blindly follows user-supplied URLs.

Lesson 3: TLS Configuration — Your default HTTP server is unencrypted

Mon, 12 Aug 2024 00:00:00 +0000

When I started writing Go HTTP servers, I thought TLS was someone else’s problem. A load balancer in front of my service handled HTTPS termination, so my service only ever spoke plain HTTP on the internal network. That is a common and often defensible architecture. But it means that if anyone gains access to your internal network — a compromised service, a misconfigured cloud security group, a rogue container — all traffic between your services is plaintext. I learned this lesson not from a breach but from a penetration test report that listed it as a finding with a crisp paragraph explaining exactly why it mattered.

Lesson 2: Secret Handling — Env vars are not a vault

Mon, 08 Jul 2024 00:00:00 +0000

I once found database credentials committed directly in a Go source file — not in a toy project, in a staging environment of a real product. The developer who wrote it knew it was wrong, but they were “just testing” and forgot to revert it. That file lived in the repository for eight months before anyone noticed. By then, the credentials had been rotated, but the history was permanent.

Secret handling in Go is not primarily a coding problem. It is a discipline problem. The code patterns are simple. The failures happen when you treat secrets as a detail to clean up later, or when you assume that environment variables are inherently safe because they are not in source code.

Lesson 1: Input Validation — Trust nothing from the wire

Wed, 05 Jun 2024 00:00:00 +0000

The first production security incident I dealt with involved a simple integer field in a JSON body. The client sent a negative number where we expected a positive quantity. We hadn’t validated it. The result was a negative charge on a purchase — we were effectively paying customers money. That taught me more about input validation than any security course.

Every byte that arrives over the wire is adversarial by default. Not because every user is malicious, but because your assumptions about the data are not enforced by anything except your own code. HTTP gives you a byte stream. JSON decoding gives you Go values. Neither of those steps checks whether the values make sense for your business logic. That job is yours.