Rust and AI/LLM Integration on Atharva Pandey

Lesson 8: ML Data Pipelines — polars and processing at speed

Sat, 30 Aug 2025 14:15:00 +0000

Last quarter I inherited a Python data pipeline that prepared training data for our recommendation model. It processed 50 million rows. Took 3 hours. Used 64GB of RAM. Everyone accepted this as normal — “big data is slow.” I rewrote it in Rust with polars. Same data. 4 minutes. 6GB of RAM. My teammates thought I was lying until they ran it themselves.

Polars isn’t just “pandas but faster.” It’s a fundamentally different approach to DataFrame operations — lazy evaluation, query optimization, true parallelism, and a Rust-native API that makes data pipeline code genuinely pleasant to write.

Lesson 7: On-Device Inference — ONNX Runtime and candle

Tue, 26 Aug 2025 07:49:00 +0000

I run a sentiment analysis model on every support ticket that comes in. At first I used the OpenAI API — about 2 cents per ticket. Sounds cheap until you do the math: 10,000 tickets a day, $200/day, $6,000/month. For a model that classifies text into “positive,” “negative,” and “neutral.”

Switched to a local ONNX model running on a $50/month VM. Same accuracy. Latency dropped from 300ms to 8ms. Cost dropped to roughly zero. Not every task needs GPT-4 — and Rust is arguably the best language for running models locally because you get C++ performance without the C++ pain.

Lesson 6: Building MCP Servers in Rust — Model Context Protocol

Fri, 22 Aug 2025 10:23:00 +0000

The first time I heard about MCP, I dismissed it as yet another protocol nobody would adopt. Then Claude Desktop shipped with MCP support, then Cursor, then Windsurf, then half the AI tools I use daily. Turns out when Anthropic publishes a spec and immediately supports it in their flagship products, adoption happens fast.

MCP — Model Context Protocol — is a standardized way for AI models to discover and use tools, access data sources, and interact with external systems. Think of it as USB for AI: a universal interface so models don’t need custom integrations for every data source. And Rust is a fantastic language for building MCP servers because they need to be fast, reliable, and run for a long time without leaking memory.

Lesson 5: Agent Architectures in Rust — ReAct, planning, and loops

Wed, 20 Aug 2025 13:08:00 +0000

I built my first “AI agent” by stuffing a system prompt into a while loop and hoping for the best. It worked — sometimes. Other times it’d get stuck in infinite loops, burn through $50 of API credits hallucinating tool calls that didn’t exist, or confidently produce completely wrong answers after three rounds of “reasoning.”

The problem wasn’t the LLM. The problem was me treating agent design as an afterthought. Good agents need structure — clear state machines, well-defined stopping conditions, and guardrails that prevent runaway behavior. This is where Rust’s type system pays massive dividends, because you can encode these constraints at the type level.

Lesson 4: Embeddings and Vector Search — Semantic search in Rust

Mon, 18 Aug 2025 08:55:00 +0000

I spent a week building a keyword search system for internal documentation. Regex patterns, stemming, tf-idf scoring — the whole nine yards. Then someone searched “how do I deploy” and got zero results because every doc said “deployment process” instead of “deploy.” That’s when I switched to embeddings.

Embeddings map text into high-dimensional vectors where semantically similar content lives close together. “Deploy” and “deployment process” end up near each other in vector space even though they share almost no characters. It’s a fundamentally different approach to search, and once you’ve used it, keyword search feels like the dark ages.

Lesson 3: Tool Calling / Function Calling Patterns — Agents need tools

Sat, 16 Aug 2025 16:42:00 +0000

Here’s something that took me embarrassingly long to internalize: LLMs don’t do things. They generate text that describes doing things. The tool calling protocol is just the model saying “hey, I’d like you to call this function with these arguments” — and then your code actually does it.

This distinction matters because the entire tool calling system is essentially a serialization contract. The model generates JSON conforming to a schema you provided, you execute the function, and you send the result back. Get the schema wrong, and the model hallucinates arguments. Get the execution wrong, and you’ve got a broken agent. Get the result format wrong, and the model can’t make sense of what happened.

Lesson 2: Streaming LLM Responses — SSE and WebSockets

Thu, 14 Aug 2025 11:37:00 +0000

The first time I demoed an LLM-powered feature to stakeholders, I made the rookie mistake of using non-streaming responses. The CEO asked a question, hit enter, and stared at a blank screen for eight seconds. “Is it broken?” No — it was thinking. But by the time the response appeared, she’d already mentally moved on to the next agenda item.

Streaming changes everything. Users see tokens appearing in real-time, which feels responsive even when the total generation time is identical. But implementing streaming in Rust? It’s one of those things that’s surprisingly nuanced once you get past the happy path.

Lesson 1: Building LLM API Clients in Rust — Type-safe AI calls

Tue, 12 Aug 2025 09:14:00 +0000

Last month I watched a coworker’s Python script silently swallow a malformed response from the OpenAI API. The choices field came back empty, the code plowed ahead with choices[0], and the whole pipeline crashed at 2 AM. Nobody got paged because the error handler was also broken. Classic.

That’s the moment I decided to rebuild our LLM integration layer in Rust. Not because I’m some Rust evangelist who thinks Python is evil — I use Python daily. But when you’re making API calls that cost real money and feed into production systems, maybe you want a type system that actually catches things before runtime.