Part 1 covered the basic idea: keep your sources immutable, let an LLM own a cross-linked Markdown wiki on top of them, and drive the whole thing through an agent that reads a plain-text schema. That much is essentially Andrej Karpathy’s original sketch, built out.
It works. But a knowledge base is one of those things that’s easy at a hundred pages and hard at a few thousand, and most of what I’ve built since is a response to the specific ways the simple version started to strain. Each of these began as a problem I actually hit — not a feature I set out to add.
The conventions drift, so make them lintable
The schema starts as prose: “every page should have a title, a summary, a type, and its sources.” For a while the model follows it. Then a page slips through without a summary, or with a type that doesn’t exist, and nobody notices until it breaks something downstream.
The fix was to make the conventions machine-checkable. Pages carry structured metadata, and a lint command walks the whole vault and reports what’s wrong — pages missing required fields, links that point nowhere, orphans nothing links to. The schema stopped being a suggestion and became a contract you can check. This is the single biggest departure from the original gist, and everything else depends on it.
Pages go stale silently, so track provenance and flag rot
A wiki page is a summary of its sources at a moment in time. Then a source gets updated, or a new one lands, and the page that summarized the old version is quietly wrong. In a growing collection this happens constantly and invisibly.
So every page declares which sources it’s built from, and the linter compares timestamps: if a page is older than something it depends on, it gets flagged as stale. The wiki tells me what’s rotting instead of waiting for me to trip over it. It’s a small idea that changes how much you can trust the thing.
Separate analyses disagree, so keep one source of truth
Say you keep several analyses that all lean on the same handful of assumptions. You update one assumption in one place, forget the others, and now your own wiki gives two different answers to the same question. I did exactly this and it’s corrosive — the whole point is to have something you can trust.
The fix is boring and effective: shared assumptions live on one canonical page with a change log, and every analysis that uses them references it. Change an assumption once, and every dependent page gets flagged for a refresh. It’s the database idea of normalization, applied to prose.
Sources contradict each other, so surface it instead of hiding it
The original pattern assumes sources basically agree. In real life they don’t — two articles cite different numbers, a person’s title changes across documents, today’s source disagrees with one you filed last year. The tempting move is to have the model pick one and smooth it over. That’s the wrong move, because the disagreement is often the most useful thing in the file.
So contradiction handling became its own small toolkit: a dedicated section when two sources genuinely disagree, dated corrections when I override something in conversation, superseded-but-preserved blocks that keep the old claim visible with a line through it. The rule is always to show the conflict, never to quietly resolve it.
Prose isn’t enough, so add typed pages
Karpathy’s wiki is prose summaries, and prose is right for ideas and events. But a lot of what you want to remember is structured — the things you own, the accounts you hold, the books and films you’ve been through. Cramming those into free text throws away everything you’d want to filter or sort by later.
So some page types grew real structure: a richer, consistent set of fields for each category. The wiki stopped being only a pile of essays and became something closer to a personal database with prose attached — you can still read a page, but you can also ask structured questions across a whole category.
The model knows the facts but acts wrong, so split off memory
This one took me a while to see. The wiki is facts about the world. But there’s a second kind of thing you want the assistant to retain: how it should behave — your preferences, the corrections you’ve made, the traps to avoid. Those aren’t facts about the world; they’re instructions about the work.
Mixing them was a mistake. A fact (“this person’s role is X”) belongs in the wiki; a behavior (“don’t re-explain the basics to me”) belongs in a separate memory layer the assistant reads before it acts. Splitting the two killed a lot of duplication and, more importantly, made the assistant act right, not just know right.
You can’t find the page you need, so measure retrieval
Here’s the failure mode nobody warns you about: the perfect page exists, and the model can’t find it. A wiki you can’t retrieve from is a write-only diary. And “the LLM will just search” hides how bad naive search can be — plain substring matching whiffs on a normal question that doesn’t happen to contain the exact words.
So I started measuring it: a small test harness of real questions with known-good answers, scored on whether the model actually opens the right pages. When the numbers were bad, I fixed the search — a proper tokenizing/BM25 index instead of substring matching — and the recall jumped. If you take one thing from this post, take that one: a knowledge base is only as good as your ability to find things in it, and that’s a number you can and should measure.
The theme
Karpathy’s gist gives you the shape, and the shape is genuinely most of the value. But making it something you trust at scale turns out to be editorial engineering: provenance, staleness, consistency, contradiction, structure, retrieval. None of it is glamorous. All of it is the difference between a knowledge base and a very tidy pile of guesses.
The nice part is that the LLM does the labor and you do the design. You decide the conventions; it lives inside them. And the conventions themselves — that plain-text schema from Part 1 — quietly became the thing I value most in the whole system. The pages are the output. The schema is the asset.
There’s a Part 3, but I’m leaving it open on purpose. The most interesting parts are still ahead — more than one vault, more than one person, running it somewhere other than your own machine — and this field moves fast enough that a good deal of what belongs there doesn’t exist yet. Call it a placeholder for the future: the things we’ll build as the tools keep changing under us.