Compilers on

Lesson 4: Code Generation — From AST to bytecode or machine code

Fri, 20 Dec 2024 00:00:00 +0000

The tree-walking interpreter in Lesson 3 works, but it has a ceiling. Every time you evaluate an expression, you traverse the AST from scratch. No caching, no precomputed form, no optimization. For a REPL or a small scripting language this is fine. For a production language runtime — something that runs for hours, executes millions of operations — you want a tighter inner loop. That tighter loop is a virtual machine executing bytecode. And the step that produces bytecode from the AST is code generation.

Lesson 3: AST and Evaluation — Walking the tree to compute results

Thu, 03 Oct 2024 00:00:00 +0000

After the lexer and parser, we have a tree. A beautiful, hierarchical, unambiguous representation of the program. Now we need to make it do something. The simplest way to execute a program from its AST is to walk the tree recursively and compute results as you go. No intermediate representation, no bytecode, no machine code — just a recursive function that pattern-matches on node types and returns values.

This is a tree-walking interpreter. It is not the fastest approach (we cover bytecode and code generation in Lesson 4), but it is the most direct. Many production language implementations started here — and some, like early Ruby and Python, stayed here for a long time.

Lesson 2: Parsing — Tokens to AST, recursive descent

Thu, 25 Jul 2024 00:00:00 +0000

The lexer gave us tokens. The tokens are still flat — a sequence with no hierarchy. The expression 1 + 2 * 3 produces six tokens, but it does not yet say that 2 * 3 should be computed before adding 1. That grouping, that hierarchy, is what the parser builds. By the time the parser is done, we have a tree. A specific kind of tree: an Abstract Syntax Tree, or AST. Everything after the parser works on this tree: evaluation, type checking, code generation, optimization. The tree is the program.

Lesson 1: Lexing — Turning text into tokens

Mon, 06 May 2024 00:00:00 +0000

I wrote my first lexer as a side project after reading the first chapter of “Writing an Interpreter in Go.” I expected it to be hard. It was not. A lexer is conceptually one of the simpler pieces of a compiler: read characters, group them into meaningful chunks, throw away whitespace. What surprised me was how much clarity it brought to everything downstream. Once I had tokens instead of characters, every subsequent step became easier to reason about. The text became structured. And the structure was entirely my design.