Llm on

Lesson 8: ML Data Pipelines — polars and processing at speed

Sat, 30 Aug 2025 14:15:00 +0000

Last quarter I inherited a Python data pipeline that prepared training data for our recommendation model. It processed 50 million rows. Took 3 hours. Used 64GB of RAM. Everyone accepted this as normal — “big data is slow.” I rewrote it in Rust with polars. Same data. 4 minutes. 6GB of RAM. My teammates thought I was lying until they ran it themselves.

Polars isn’t just “pandas but faster.” It’s a fundamentally different approach to DataFrame operations — lazy evaluation, query optimization, true parallelism, and a Rust-native API that makes data pipeline code genuinely pleasant to write.

Lesson 7: On-Device Inference — ONNX Runtime and candle

Tue, 26 Aug 2025 07:49:00 +0000

I run a sentiment analysis model on every support ticket that comes in. At first I used the OpenAI API — about 2 cents per ticket. Sounds cheap until you do the math: 10,000 tickets a day, $200/day, $6,000/month. For a model that classifies text into “positive,” “negative,” and “neutral.”

Switched to a local ONNX model running on a $50/month VM. Same accuracy. Latency dropped from 300ms to 8ms. Cost dropped to roughly zero. Not every task needs GPT-4 — and Rust is arguably the best language for running models locally because you get C++ performance without the C++ pain.

Lesson 6: Building MCP Servers in Rust — Model Context Protocol

Fri, 22 Aug 2025 10:23:00 +0000

The first time I heard about MCP, I dismissed it as yet another protocol nobody would adopt. Then Claude Desktop shipped with MCP support, then Cursor, then Windsurf, then half the AI tools I use daily. Turns out when Anthropic publishes a spec and immediately supports it in their flagship products, adoption happens fast.

MCP — Model Context Protocol — is a standardized way for AI models to discover and use tools, access data sources, and interact with external systems. Think of it as USB for AI: a universal interface so models don’t need custom integrations for every data source. And Rust is a fantastic language for building MCP servers because they need to be fast, reliable, and run for a long time without leaking memory.

Lesson 5: Agent Architectures in Rust — ReAct, planning, and loops

Wed, 20 Aug 2025 13:08:00 +0000

I built my first “AI agent” by stuffing a system prompt into a while loop and hoping for the best. It worked — sometimes. Other times it’d get stuck in infinite loops, burn through $50 of API credits hallucinating tool calls that didn’t exist, or confidently produce completely wrong answers after three rounds of “reasoning.”

The problem wasn’t the LLM. The problem was me treating agent design as an afterthought. Good agents need structure — clear state machines, well-defined stopping conditions, and guardrails that prevent runaway behavior. This is where Rust’s type system pays massive dividends, because you can encode these constraints at the type level.

Lesson 4: Embeddings and Vector Search — Semantic search in Rust

Mon, 18 Aug 2025 08:55:00 +0000

I spent a week building a keyword search system for internal documentation. Regex patterns, stemming, tf-idf scoring — the whole nine yards. Then someone searched “how do I deploy” and got zero results because every doc said “deployment process” instead of “deploy.” That’s when I switched to embeddings.

Embeddings map text into high-dimensional vectors where semantically similar content lives close together. “Deploy” and “deployment process” end up near each other in vector space even though they share almost no characters. It’s a fundamentally different approach to search, and once you’ve used it, keyword search feels like the dark ages.

Lesson 3: Tool Calling / Function Calling Patterns — Agents need tools

Sat, 16 Aug 2025 16:42:00 +0000

Here’s something that took me embarrassingly long to internalize: LLMs don’t do things. They generate text that describes doing things. The tool calling protocol is just the model saying “hey, I’d like you to call this function with these arguments” — and then your code actually does it.

This distinction matters because the entire tool calling system is essentially a serialization contract. The model generates JSON conforming to a schema you provided, you execute the function, and you send the result back. Get the schema wrong, and the model hallucinates arguments. Get the execution wrong, and you’ve got a broken agent. Get the result format wrong, and the model can’t make sense of what happened.

Lesson 2: Streaming LLM Responses — SSE and WebSockets

Thu, 14 Aug 2025 11:37:00 +0000

The first time I demoed an LLM-powered feature to stakeholders, I made the rookie mistake of using non-streaming responses. The CEO asked a question, hit enter, and stared at a blank screen for eight seconds. “Is it broken?” No — it was thinking. But by the time the response appeared, she’d already mentally moved on to the next agenda item.

Streaming changes everything. Users see tokens appearing in real-time, which feels responsive even when the total generation time is identical. But implementing streaming in Rust? It’s one of those things that’s surprisingly nuanced once you get past the happy path.

Lesson 1: Building LLM API Clients in Rust — Type-safe AI calls

Tue, 12 Aug 2025 09:14:00 +0000

Last month I watched a coworker’s Python script silently swallow a malformed response from the OpenAI API. The choices field came back empty, the code plowed ahead with choices[0], and the whole pipeline crashed at 2 AM. Nobody got paged because the error handler was also broken. Classic.

That’s the moment I decided to rebuild our LLM integration layer in Rust. Not because I’m some Rust evangelist who thinks Python is evil — I use Python daily. But when you’re making API calls that cost real money and feed into production systems, maybe you want a type system that actually catches things before runtime.

Lesson 6: Agent Architectures — Building autonomous agents in Go

Thu, 10 Jul 2025 00:00:00 +0000

An agent is a loop: observe, think, act, repeat. The LLM is the “think” step — it decides what to do next given the current state. Your Go code handles “observe” (gathering context), “act” (executing tool calls), and the loop control that keeps everything running. I’ve built agents that write and execute code, agents that browse the web, and agents that orchestrate multi-step data pipelines. The underlying architecture is always the same few patterns, and Go’s concurrency makes the execution layer clean and fast.

Lesson 5: Embedding and Vector Search — Semantic search in Go without Python

Sun, 18 May 2025 00:00:00 +0000

For a long time, embedding-based semantic search felt like Python territory. The tutorials all pointed to LangChain, FAISS, and numpy. But the actual operations — generate an embedding vector, store it in a database, query for nearest neighbors — map directly onto Go’s strengths: clean HTTP client code for the embedding API, pgx for PostgreSQL with pgvector, and fast concurrent query pipelines. I’ve built production semantic search systems entirely in Go and they’re fast, maintainable, and don’t require a Python sidecar.

Lesson 4: Tool Calling Patterns — Letting the LLM invoke your Go functions

Wed, 12 Mar 2025 00:00:00 +0000

Tool calling is where LLM integrations get genuinely powerful. Without tool calling, you’re limited to asking the model to generate text. With tool calling, you can build a conversational interface where the model decides which of your functions to call, calls them, incorporates the results into its reasoning, and decides whether to call more tools or return a final answer. I’ve used this to build support assistants that query databases, coding assistants that run test suites, and research tools that fetch live web content — all driven by the model’s judgment about which tools to invoke.

Lesson 3: Streaming Responses — Token-by-token output without buffering the whole response

Wed, 18 Dec 2024 00:00:00 +0000

If you’ve ever used a non-streaming LLM endpoint in a user-facing feature, you know the experience it creates: the user submits a question, watches a spinner for 8 seconds, then suddenly gets a wall of text. Streaming changes this completely — the user sees output appearing word by word, which feels responsive and alive even when the total time-to-complete is identical. In Go, streaming LLM responses means reading Server-Sent Events (SSE) from the API and piping them to the client in real time. It’s genuinely one of the nicer concurrency patterns I’ve implemented.

Lesson 2: LLM API Clients — Calling Claude, GPT, and Groq from Go

Sun, 06 Oct 2024 00:00:00 +0000

Most Go developers approach LLM APIs the same way they approach any REST API — write an HTTP client, handle errors, parse JSON. That instinct is correct, but LLM APIs have a few characteristics that require specific handling: they’re slow (seconds, not milliseconds), they have complex nested response structures, they support streaming, and the model selection and token management have real cost implications. This lesson is about building Go clients that handle all of this properly.

Lesson 1: Building MCP Servers in Go — Give AI agents tools with the Model Context Protocol

Tue, 30 Jul 2024 00:00:00 +0000

When Claude or another AI assistant needs to look up a database record, call an internal API, or read a file from your filesystem, it can’t do that on its own — it needs tools. The Model Context Protocol (MCP) is Anthropic’s open standard for giving AI agents exactly those tools. An MCP server is a small program you write that exposes tools via a JSON-RPC protocol; the AI client calls your server to invoke them. I find this genuinely exciting as a Go developer: Go’s concurrency model and fast startup time make it a natural fit for MCP servers.