Why I built my own model-agnostic LLM harness (and why you should too)

Nightloom is a model-agnostic LLM harness in Rust: one chat core, a CLI REPL and a Tauri desktop app sitting on top of it, and six provider adapters (Anthropic, OpenAI, Gemini, Groq, OpenRouter, plus anything that speaks chat/completions) that all normalize onto the same event stream.

Two disclosures before we go any further. First, this is not my first attempt. Nightloom is just the latest iteration of an application I keep renaming and rebuilding, every time a new model comes out, to sort of gauge AI progress with my “own personal benchmark” which hopefully is sufficiently out of distribution to measure true model intelligence. The other application that I am building in this way is GG Engine, a Rust Vulkan game engine, with modern features and toolsets.

I didn’t write most of it. Fable did. That’s the actual experiment here: point a frontier model at a real systems project (a harness for talking to models, built by a model) and find out where it shines and where it confidently walks into walls. The results are scattered through this post.

This post is the why. The how is going to be a whole 4-part series, A to Z, because it turns out there is a lot of how.

You don’t own your conversations

Every harness I tried couples three things that have no business being coupled: the vendor’s wire format, the conversation state, and the UI. Start a session with one provider and you’ve basically married it. The transcript lives in that vendor’s shapes, the reasoning artifacts only replay against that vendor’s API, and switching means starting over. There is no prenup.

It gets worse with context management. Almost everything treats context operations as destructive. Compact a conversation and the original is gone. Hit the window limit and something silently falls off the end. It’s a filing cabinet that shreds the oldest folder every time you add a new one, and it won’t tell you which folder it was.

That bothered me enough to do something about it. Nightloom is built on one rule, held stubbornly: the session log is the source of truth. It’s append-only. What the provider receives, what the transcript renders, what the context gauge reads are all projections of that one log. Compaction, rewind, and dropping a bloated tool result are markers in the log, never deletions. So /rewind can take you back to any user message, /context drop can evict a 40k-token tool result from the next request, and both are reversible, because nothing was ever actually thrown away.

”Model-agnostic” is mostly a lie

Here’s what I didn’t appreciate before this iteration: the chat part of being model-agnostic is easy. Text in, text out, everyone streams SSE, you write a small adapter per vendor and move on.

Reasoning is where it falls apart. The obvious agnostic move is to drop reasoning artifacts when you replay history to the model. That turns out to be wrong for three of the four wire dialects I support:

• Anthropic requires signed thinking blocks back inside a tool loop, and rejects what it can’t verify.
• Gemini 3 hard-requires its thoughtSignature echoed on function calls. Omit it and round two of every tool loop 400s. (Gemini 2.5 never validates any of this, which makes it a fun thing to discover on upgrade.)
• OpenAI’s Responses API wants the reasoning item replayed by id, in stream order. The summaries it streams at you are not the replayable artifact, a sentence I could only have written after meeting the 400 error personally.
• Plain chat/completions hosts require nothing at all, which sounds relaxing until you notice it’s the exception.

So in Nightloom, reasoning replay tokens are separate types per vendor. A handle issued by Anthropic can’t be sent to Gemini, and the compiler enforces what the APIs would otherwise punish at runtime with a 400 mid-conversation. This is the single biggest reason the “one log, many providers” design works: you can switch vendors mid-session and keep the conversation, because the projection layer knows exactly what each dialect needs back and what it must never see.

You find real things

The best argument for building your own harness is that you stop treating the API as magic, and you start finding things. Sometimes the thing you find is in the API. Sometimes it’s in your model.

My favorite: I asked Fable to split the system prompt so a small stable preamble came first and the per-turn status info came later, specifically to protect the prompt cache. Fable did it cleanly: good structure, sensible breakpoints, exactly what I asked for. Then we measured a four-turn session and every turn reported cache_read_input_tokens: 0. The cacheable prefix was about 300 tokens, below Anthropic’s 1024-token minimum, so the API had been silently ignoring the breakpoint the entire time. We had carefully engineered protection for a cache that never existed. That’s the experiment in miniature: the model does the plausible thing brilliantly, and it does not stop to ask whether the plausible thing is real. After fixing the breakpoint placement, turn 4 read back 1,032 cached tokens and wrote only a 548-token delta: $0.0025 for the turn instead of $0.0041.

Pricing tables hold similar traps. Gemini quotes a “cache write” price that is actually explicit-cache storage, per million tokens per hour, which belongs nowhere near a per-token column. You only learn this stuff when you’re watching the numbers being wired into a cost display right in front of you, and the dashboards refuse to line up.

And when you own the harness, you get to pick the defaults. Every Nightloom tool declares an effect (read-only, session, or mutating), and the default is mutating, meaning it asks before running. A tool that never answered the question, like one arriving over MCP, is something nobody vouched for. A safe-looking default would let every future tool skip the approval gate by omission. I am, after all, letting a model edit my files. Some paranoia is load-bearing.

The series

That’s the pitch: your harness is the room you live in, and you should own the walls, even if a model is doing most of the drywall. The series will build the whole thing end to end. Canonical types and the provider trait, the adapters and their per-vendor quirks, the tool loop and approval gate, MCP, session logs and replay, the desktop shell, and evals that grade by inspecting the disk instead of asking a model for its opinion. Along the way I’ll keep score on what Fable got right the first time and where I had to grab the wheel.

The repo is public if you’d rather read ahead.