Smorster

We Just Hit the Local LLM Tipping Point (Colibri)

Devsplainers · 8m 01s · Watch on YouTube · 17 sources

Decision Card

Effort: Weekend project — clone github.com/JustVugg/colibri, free up ~400 GB on a fast NVMe drive, download the int4 GLM-5.2 expert file (~370 GB, most of a day on home broadband), build the single C file, and run your first overnight prompt.

Honest take: The video’s own numbers undercut its “8,000 tokens a day” pitch — that rate requires the SSD running flat-out 24 hours straight, which the video itself admits risks thermal throttling on cheap drives; and “sits above Claude Opus” holds only on LMArena’s Code Arena Frontend board, while Opus 4.8 still leads the harder agentic benchmarks (SWE-bench Pro 69.2 vs 62.1) per BenchLM’s comparison.

Concrete next steps:

  • Read the Colibri README benchmark table before committing disk space (~15 min) — real cold-start speed is 0.05–0.1 tok/s, lower than the video’s headline number.
  • On a 128 GB Apple Silicon Mac, try antirez’s ds4 instead — 27 tok/s interactive with DeepSeek V4 Flash beats Colibri’s sub-1 tok/s for the same “local frontier model” goal (~1 hr setup).
  • Subscribe to llama.cpp issue #20757 and vLLM RFC #38256 to catch expert-cache support landing in mainstream runtimes (~5 min).
  • Skip if you need interactive chat speed, or you don’t have ~400 GB of free NVMe — at 0.1 tok/s this is strictly a batch/overnight tool.

TL;DR

Colibri is a ~1,300-line pure-C inference engine that runs GLM-5.2 — a 744B-parameter mixture-of-experts model — on a 25 GB-RAM laptop by pinning the ~10 GB dense core in memory and streaming the ~370 GB of routed experts from SSD on demand, exploiting the fact that only ~5% of parameters activate per token. The catch is speed (0.1–1 token/second), so it’s useful only for overnight or air-gapped jobs, but it signals a broader shift: antirez’s ds4, llama.cpp, and vLLM are all starting to treat the SSD as a memory tier.

Key Points

  • GLM-5.2’s 4-bit weights occupy 370 GB — normally requiring ~$40k of GPUs or a 256 GB Mac Studio — yet Colibri runs it in 25 GB of RAM. 00:54
  • Naive SSD streaming fails for dense models: every parameter is needed per token, meaning ~30 seconds of disk reads per word even on the fastest consumer SSD. 01:22
  • GLM-5.2 is a mixture-of-experts model: each layer has 256 experts, a router picks the 8 best per token, and the other 248 stay idle. 01:51
  • Experts are not human-legible specialists (“a math guy, a code guy”) — they’re unlabelled weight blobs with overlapping jobs, so you can’t prune to a “coding subset.” 02:23
  • Only ~40B of 744B parameters (~5%) do work per token — the rest is dead weight that doesn’t need to be in RAM. 02:44
  • Colibri pins the dense core (attention, embeddings, one shared expert — ~10 GB at 4-bit) in RAM and leaves ~21.5k experts of 19 MB each on SSD, loading them into an LRU cache on demand. 03:01
  • Unlike llama.cpp’s mmap (blind, blocking page faults), Colibri knows which 8 experts are needed the moment the router fires, fetches them asynchronously, and even runs the next layer’s router early on partial information. 04:27
  • A usage log persists across launches, pinning your personal hot experts into spare RAM — the engine measurably gets faster with use. 04:08
  • The speed ladder: 0.1 tok/s on a budget SSD, 0.28 on top PCIe 5 drives, 0.37 on a 128 GB Framework laptop, ~1 tok/s on an M5 Max — the current ceiling. 05:18
  • The trend is bigger than one repo: antirez shipped ds4 (DeepSeek experts streamed off a MacBook SSD), llama.cpp has open expert-cache proposals, and vLLM is building its own version. 06:21
  • The key unlock is routing predictability: Colibri already guesses next-layer experts at 72% accuracy; if that climbs, the disk penalty largely evaporates. 06:50

Notable Quotes

“A page fault can’t see the future. A router can, a little.” 05:01

“Serious runtimes are starting to treat the SSD as a memory tier instead of a filing cabinet.” 06:43

“For the price of one consumer SSD, the floor for touching a frontier model dropped out of the data center and onto a desk.” 07:37

Verified Claims

  1. GLM-5.2 is a 744B-parameter MoE model with ~40B active per token, routing through 8 of 256 experts per layer. 01:47 Sources: eesel — What is GLM-5.2?, claudefa.st GLM-5.2 specs Verdict: Confirmed — sources corroborate 744B total / ~40B active and “8 of 256 experts plus one shared expert,” though some coverage cites 753B total and 384 experts, so exact architecture numbers vary by source.

  2. GLM-5.2 ranks above Claude Opus on the latest LMArena benchmarks. 00:14 Sources: BenchLM — Opus 4.8 vs GLM-5.2, MindStudio comparison Verdict: Confirmed, with a caveat — GLM-5.2 (Max) is #2 on LMArena’s Code Arena Frontend (Elo 1,595, above Opus 4.8 Thinking at 1,561), but it’s behind Claude Fable 5, and Opus 4.8 still leads SWE-bench Pro, NL2Repo, and SWE-Marathon.

  3. Colibri is a ~1,300-line pure-C engine that pins a ~10 GB dense core in RAM and streams ~370 GB of experts from SSD with LRU caching, async I/O, and hot-expert pinning. 00:29 Sources: Colibri GitHub repo, GIGAZINE coverage Verdict: Confirmed — repo documents ~9.9 GB int4 resident dense core, on-disk expert library, adaptive cache pinning, and bounded async read pool; the repo counts 19,456 routed experts vs the video’s “21 and a half thousand.”

  4. Real-world speed runs ~0.1 tok/s on the author’s laptop, up to ~1 tok/s at the top of the consumer ladder. 05:09 Sources: Colibri GitHub repo, Developers Digest write-up Verdict: Confirmed — repo lists 0.05–0.1 tok/s cold on 25 GB baseline systems and 1.07 tok/s on a single RTX 5070 Ti; the video’s 1 tok/s ceiling matches the top of the streaming configurations.

  5. Colibri predicts the next layer’s experts with 72% accuracy. 06:53 Source: Colibri GitHub repo Verdict: Confirmed — repo states routing is “71.6% predictable one layer ahead,” which the video rounds to 72%.

  6. antirez (Redis creator) shipped ds4, a Metal/C engine streaming DeepSeek experts off a MacBook’s SSD. 06:24 Sources: ds4 GitHub repo, knightli.com analysis Verdict: Confirmed, with nuance — ds4 exists and targets DeepSeek V4 Flash on Apple Silicon, but its main tricks are asymmetric 2-bit MoE quantization and an SSD-backed KV cache (reaching 27 tok/s on a 128 GB MacBook Pro), a different balance than Colibri’s expert streaming; a glm5.2 branch also exists.

  7. llama.cpp has open proposals for expert caching, and vLLM is building its own version. 06:33 Sources: llama.cpp issue #20757 — two-tier expert cache, vLLM RFC #38256 — incremental MoE expert offloading Verdict: Confirmed — both projects have active proposals for exactly this expert-cache/offload pattern.

  8. Streaming won’t kill your SSD because reads don’t wear out flash — writes do. 05:40 Sources: ATP — SSD endurance specs (TBW/DWPD), Read Disturb Errors in MLC NAND Flash (arXiv) Verdict: Confirmed, minor caveat — endurance ratings measure writes only and reads don’t wear cells, though heavy repeated reads cause “read disturb” errors that controllers must manage; the video’s thermal-throttling warning is the more practical concern.

Tools, Papers & Standards Mentioned

Follow-up Questions

  1. How far can cross-layer expert prediction accuracy be pushed (from today’s ~72%) with a small learned predictor, and does routing predictability differ systematically between coding, math, and prose workloads?
  2. Does ds4’s asymmetric quantization approach (2-bit experts, higher-precision attention/routing) compose with Colibri-style SSD streaming to shrink the on-disk expert file — and what quality is lost at 2-bit vs 4-bit on GLM-5.2?
  3. When llama.cpp’s #20757 or vLLM’s #38256 land, does the SSD-as-memory-tier pattern generalize across MoE families (DeepSeek, Qwen, Kimi), or does it depend on GLM-5.2’s specific expert granularity (many small ~19 MB experts)?

Sources