Smorster

AirLLM Tutorial: Run 70B Models with Low Memory Usage

Seb Intel · 38s · Watch on YouTube · 4 sources

Decision Card

Effort: Weekend experiment — pip install airllm, download a 70B model (~140GB disk), run the quickstart example on a machine with 4GB+ VRAM or Apple Silicon; expect a few hours mostly waiting on model download and reshard.

Honest take: This 38-second promo omits the deal-breaking trade-off: layer-by-layer loading turns a memory bottleneck into a disk I/O bottleneck, yielding roughly 0.5–2 tokens/second (vs. 10–20 on an A100), and AirLLM’s own author says it is “not very suitable for interactive scenarios like chatbots.” Also, flash attention is a general LLM optimization the video presents as an AirLLM feature, and the “runs on your MacBook” framing skips the ~140GB+ of disk space and the resharding step required first.

Concrete next steps:

  • Read the AirLLM GitHub README for the supported-model list, MacOS/MLX install path, and the 4GB-VRAM claims — 15 min.
  • Read the author’s Hugging Face deep-dive post explaining the layered-inference and flash-attention math before believing benchmark claims — 20 min.
  • If curious, benchmark tokens/sec on your own hardware with an 8B model first (fast to download) before committing 140GB of disk to a 70B — 1–2 hours.
  • Skip if you need interactive/chat-speed inference — at 0.5–2 tokens/sec this is only viable for offline batch work (RAG indexing, document analysis); use a quantized smaller model via Ollama/llama.cpp instead.

TL;DR

AirLLM is an open-source (Apache-2.0) Python library that runs 70B-parameter models on ~4GB of VRAM by loading one transformer layer at a time from disk instead of holding the whole model in memory. The video’s claims are essentially accurate but omit the critical cost: inference is roughly 10× slower than conventional serving, making it suitable for offline batch tasks rather than chat.

Key Points

  • Core pitch: you no longer need a supercomputer to run a 70B-parameter model. 00:01
  • The enabling tool is an open-source library called AirLLM. 00:06
  • Mechanism: instead of loading the entire model at once, it loads it layer by layer from the hard drive. 00:08
  • Analogy given: reading a book page by page rather than memorizing the whole thing first. 00:14
  • Flash attention is cited as keeping memory usage nearly flat even with long inputs. 00:17
  • Claims models like Llama 3.3 70B can run directly on a MacBook or gaming computer. 00:23
  • Selling points: fully private, zero API costs. 00:27
  • Target audience named as students, researchers, and indie developers. 00:30
  • Engagement-bait close: comment “Air” to receive the link (no link given in the video itself). 00:34

Notable Quotes

“You no longer need a supercomputer to run a 70 billion parameter AI model.” 00:01

“Instead of loading the entire model at once, it loads it layer by layer from your hard drive.” 00:08

“Models like Llama 3.3 70B can run directly on your MacBook or gaming computer. Fully private with zero API costs.” 00:23

Verified Claims

  1. AirLLM is an open-source library. 00:06 Sources: AirLLM GitHub repo (Apache-2.0 license). Verdict: Confirmed.

  2. It runs 70B models on low-memory hardware — no supercomputer needed. 00:01 Sources: AirLLM GitHub repo (headline claim: “70B inference with single 4GB GPU”), author’s Hugging Face post (measured <4GB GPU memory during inference). Verdict: Confirmed — memory-wise; speed is the unstated cost.

  3. The trick is loading the model layer by layer from disk. 00:08 Sources: AirLLM GitHub repo (“only one layer on the GPU at a time”), Hugging Face post (~1.6GB per layer for a 70B model, 1/80 of the full model). Verdict: Confirmed.

  4. Flash attention keeps memory usage nearly flat with long inputs. 00:17 Sources: Hugging Face post (flash attention reduces attention memory complexity from O(n²) to O(n)). Verdict: Confirmed — though it is a general LLM optimization used by AirLLM, not an AirLLM invention as the video implies.

  5. Llama 3.3 70B can run directly on a MacBook. 00:23 Sources: AirLLM GitHub repo (MacOS/Apple Silicon supported via MLX + PyTorch; Llama 3 70B explicitly supported). Verdict: Confirmed for Llama 3-family 70B on Apple Silicon; Llama 3.3 specifically is not called out in the README, and ~140GB+ of disk is required.

  6. Fully private with zero API costs. 00:27 Sources: AirLLM GitHub repo (fully local inference of open-weight models). Verdict: Confirmed — inherent to local inference; you still pay in disk space and time.

  7. Implied claim: this is a practical way to use a 70B model day-to-day. 00:32 Sources: Hugging Face post (author: low-end GPUs “quite slow… not very suitable for interactive scenarios like chatbots”), BrightCoding layer-wise inference guide (~0.5–2 tokens/sec vs 10–20 on an A100), skeptical analysis on Medium. Verdict: Disputed — works, but only for offline/batch use at current speeds.

Tools, Papers & Standards Mentioned

Follow-up Questions

  1. What real tokens/sec does AirLLM achieve on Apple Silicon (M-series, unified memory + MLX) versus a 4GB discrete GPU — does fast unified memory meaningfully close the disk-I/O gap?
  2. At what model size does AirLLM’s layer-swapping beat a straightforward 4-bit quantized model in llama.cpp/Ollama on the same hardware, in both quality and speed?
  3. Does AirLLM’s block-wise quantization (claimed up to 3× speedup) measurably degrade output quality on reasoning benchmarks compared to unquantized layer-wise inference?

Sources