Smorster

Karpathy's LLM Wiki: What It Means & How to Build One

Tonbi's AI Garage · 16m 55s · Watch on YouTube · 15 sources

Decision Card

Effort: Afternoon project (~2–4 hours) — write a CLAUDE.md schema file, drop 5–10 sources into a raw/ folder, have Claude Code ingest them, and open the resulting markdown folder in Obsidian.

Honest take: The demo auto-backfills web-search results into the wiki as “permanent pages” with no human review, which quietly contradicts the pattern’s own division of labor (human curates sources, LLM only maintains) — unvetted search content becomes ground truth. The video also never demonstrates the “lint” maintenance pass it lists as one of the three core operations, so the claim that the wiki “stays healthy as it grows” is asserted, not shown; and it misexpands RAG as “retrieval automated generation” (it’s retrieval-augmented generation).

Concrete next steps:

  • Read Karpathy’s original llm-wiki gist — it’s the actual spec the video paraphrases, including the schema conventions (~10 min).
  • Prototype: create raw/, wiki/, and a CLAUDE.md schema, ingest 3 sources with Claude Code, and verify the cross-links render in Obsidian (~2–3 h).
  • Compare against the prebuilt version: Hermes Agent’s bundled llm-wiki skill implements the same pattern off the shelf (~30 min).
  • Skip if your document Q&A is one-shot — if you query a corpus a handful of times and move on, plain RAG or NotebookLM is cheaper than paying tokens to pre-compile a wiki you’ll never revisit.

TL;DR

The video explains Andrej Karpathy’s “LLM wiki” pattern — instead of RAG re-retrieving from raw documents on every query, an LLM agent incrementally builds and maintains a persistent, interlinked markdown wiki (raw sources → wiki → schema file) so knowledge compounds across sessions. The presenter then builds one live in Claude Code with Opus 4.6 for trading-strategy research, showing ingestion of eight transcripts, wiki-link navigation in Obsidian, and automatic backfilling of the wiki from web searches when a query exceeds what’s stored.

Key Points

  • Karpathy’s recent post on “LLM knowledge bases” proposed using LLMs to build personal, persistent knowledge bases for research topics, and this video reconstructs the pattern hands-on. 00:11
  • The core problem with RAG: nothing accumulates — the LLM re-pieces fragments from scratch on every query, with no memory or cross-references. 01:15
  • The wiki pattern inverts this: synthesis happens at ingest time, not query time, so cross-references and contradiction flags already exist when you ask. 01:35
  • Architecture has three layers: immutable raw sources, an LLM-owned markdown wiki, and a schema/config file (like CLAUDE.md) defining structure and workflows. 02:34
  • Three core operations: ingest (process a new source), query (answer from the wiki, optionally filing good answers back in), and lint (health-check for contradictions, orphans, and gaps). 03:52
  • Ingest is a seven-step pipeline (read → extract → summary page → update entity/concept pages → flag contradictions → update index → append log), and a single source can touch 10–15 wiki pages. 04:48
  • The economics argument: humans abandon wikis because maintenance outgrows value, but LLM maintenance cost is near zero, so the wiki actually stays maintained. 06:05
  • Four design principles: explicit (no hidden embeddings), yours (local files), file-over-app (universal markdown), and bring-your-own-AI (any model, even fine-tuned on the wiki). 06:34
  • The live demo builds a trading-strategy wiki in Claude Code with Opus 4.6, ingesting eight video transcripts via parallel agents and browsing the result in Obsidian. 08:46
  • When a question exceeds the wiki’s contents, the agent web-searches and automatically backfills the new findings as permanent pages — this is how the knowledge base “gets smarter on its own.” 13:57
  • A basic version is claimed to be buildable in about an hour in Claude Code. 16:30

Notable Quotes

“The LM incrementally builds and maintains a persistent wiki, structured interlinked markdown files sitting between you and your raw sources.” 02:01

“The wiki is a code base, and then Obsidian is the IDE, and the LM is the programmer, and the schema is the style guide.” 03:35

“Humans abandon wikis because the maintenance burden grows faster than the value.” 06:07

Verified Claims

  • Claim: Karpathy recently proposed the “LLM knowledge bases” pattern in a viral post with follow-ups. 00:02 — Sources: Karpathy’s original post on X, llm-wiki gist. Verdict: Confirmed (posted April 2026 with a follow-up “idea file” gist).

  • Claim: Karpathy’s formulation is that the LLM “incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources.” 02:01 — Source: llm-wiki gist. Verdict: Confirmed (near-verbatim from the gist).

  • Claim: RAG stands for “retrieval automated generation.” 00:54 — Source: Lewis et al. 2020, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (arXiv:2005.11401). Verdict: Disputed — it is retrieval-augmented generation; a verbal slip, but repeated.

  • Claim: The pattern’s three layers (immutable raw sources / LLM-owned wiki / schema config) and three operations (ingest, query, lint), with a single ingest touching 10–15 pages, come from Karpathy’s design. 02:34 — Source: llm-wiki gist. Verdict: Confirmed (the gist specifies all three layers, all three operations, and the 10–15-page figure).

  • Claim: “Humans abandon wikis because the maintenance burden grows faster than the value” is Karpathy’s argument for why LLM maintenance changes the economics. 06:05 — Source: llm-wiki gist. Verdict: Confirmed (gist: “maintenance burden grows faster than value”).

  • Claim: The demo runs in Claude Code with Claude Opus 4.6. 08:46 — Sources: Anthropic: Claude Opus 4.6, Anthropic: Claude Opus 4.8. Verdict: Confirmed — Opus 4.6 shipped February 5, 2026, though it has since been superseded (Opus 4.8 and the Claude 5 family are newer).

  • Claim: The same wiki can be driven by other agents, e.g. “open claw” or “Hermes agent.” 15:20 — Sources: Milvus: What Is OpenClaw?, Hermes Agent bundled llm-wiki skill. Verdict: Confirmed — OpenClaw (ex-Clawdbot/Moltbot) is a real open-source agent, and Nous Research’s Hermes Agent ships Karpathy’s llm-wiki as a built-in skill.

  • Claim: “File over app” — markdown-based storage keeps the wiki interoperable with any tool. 07:05 — Source: Steph Ango, “File over app”. Verdict: Confirmed — the phrase is Obsidian CEO Steph Ango’s philosophy, accurately applied here.

Tools, Papers & Standards Mentioned

Follow-up Questions

  1. How well does the pattern hold up at scale — after hundreds of ingested sources, does the “lint” pass actually catch drift, duplicated concepts, and contradictions, or does wiki quality degrade the way RAG chunking does? (One month-in field report exists: R&D World’s evaluation.)
  2. What review gate should sit between “agent web-searched an answer” and “answer filed as a permanent wiki page,” so the raw-sources-are-ground-truth guarantee isn’t silently broken?
  3. Karpathy suggests fine-tuning a model on your wiki as the next step — at what wiki size does fine-tuning (or a periodic distillation pass) actually beat just loading pages into a long context window?

Sources