Saltar al contenido principal

Colibrì: Run a 744B MoE From Your SSD (and What It Really Costs You)

Avanzado
What you'll learn
  • Understand the three-tier trick — dense weights in RAM, routed experts on disk, optional VRAM tier — and why MoE makes it possible at all
  • Read the benchmark table honestly: which hardware gets 0.1 tok/s, which gets ~2, which gets ~6, and why NVMe bandwidth, not CPU, is the ceiling
  • Know the four gotchas that cost people days: int8 MTP heads, O_DIRECT on the wrong drive, speculative decoding that loses, and SSD wear
  • Get from git clone to a first streamed token on GLM-5.2 or DeepSeek V4.1 Flash
  • Decide when Colibrì beats a GGUF-plus-Ollama offload, when to rent a GPU instead, and when a hosted API is simply the right answer

On July 1, 2026 a developer publishing as JustVugg pushed a repository with a one-line promise: run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. The Show HN thread hit 453 points nine days later. By mid-September the project is at 30.8k stars and version 1.11.0, released September 13, added a ninth model family (DeepSeek V4.1 Flash, 552B) that reads the published checkpoint natively with no conversion step. It is the top AI repository on GitHub's daily trending list as this page is written.

The pitch sounds like a trick, and in a sense it is one. This page explains the trick precisely enough that you can predict what it will do on your machine before you download 370 GB.

The one-sentence version​

A mixture-of-experts model only touches a small slice of its weights per token, so Colibrì keeps the always-used part (attention, embeddings, shared experts) resident in RAM and reads the rarely-used part (the routed experts) from NVMe on demand — trading tokens per second for the ability to run a 744-billion-parameter model on a laptop.

Why MoE makes this possible​

Take GLM-5.2 as the reference case, since it is the model the project was built around:

QuantityValue
Total parameters744B
Active per token~40B
Dense (always-on) parameters~17B → ~9.9 GB resident as int4
Routed expertsroughly twenty thousand across all layers → ~370 GB on disk as int4
Expert weight that changes token to token (cold)~11 GB
Minimum RAM / comfortable RAM16 GB / 24 GB

Every token needs the dense 9.9 GB plus the handful of experts the router picks in each layer. The dense part fits in RAM. The experts do not — but you only need a few of them at a time, and the next token often reuses many of the same ones. So the engine treats VRAM, RAM and NVMe as one memory hierarchy: a per-layer LRU cache of recently used experts in RAM, a learned "pinned hot-store" of experts that are used most often, an optional VRAM tier if a GPU is present, and the disk as backing store for everything else.

The design choice that makes it a single C file per model rather than a framework: no BLAS, no Python at runtime, no GPU required. The GLM engine started at about 1,300 lines of C. Zero dependencies also means the OS page cache becomes a free second cache layer for expert reads.

Four mechanisms worth understanding​

1. Router lookahead: routing is 71.6% predictable one layer ahead​

The expensive part of streaming is waiting for disk. Colibrì runs a lookahead thread (enabled with PILOT=1) that predicts which experts the next layer will need and starts the reads early. The README's measured figure is that routing decisions are 71.6% predictable one layer ahead, which is enough to hide a large share of the I/O latency behind compute. This is why "tok/s after warmup" is a very different number from "cold tok/s".

2. Dual-SSD streaming: two drives, one model, summed bandwidth​

Put a full copy of the model on a second NVMe and the engine streams from both at once. Each expert hashes deterministically to one drive, weighted by each drive's measured bandwidth, so there is no duplicate caching and the effective read bandwidth is roughly the sum of the two. The DeepSeek V4 Flash benchmark below (RTX 5080 plus two NVMe drives, ~1.6 tok/s) is a dual-drive configuration.

3. Compressed KV state that survives a restart​

Models with multi-head latent attention (MLA) let Colibrì store a compressed KV state — 576 floats per token instead of 32,768, a 57× reduction. The engine persists that state to a .coli_kv file, so reopening a conversation after a restart does zero re-prefill and is byte-identical to a session that never stopped. On a machine where prefill runs at streaming speed, not re-prefilling a long context is the difference between usable and not.

4. Semantics are guaranteed, speed is not​

The project's stated contract: insufficient fast memory only makes it slower; precision and router behavior never silently change. The forward pass is validated token-exact against a transformers oracle. This is the opposite trade from most "fits on a laptop" quantization stories, where you get speed by accepting a different model.

The benchmark table, read honestly​

Numbers from the project's README (GLM-5.2 unless stated), warm cache where noted:

HardwareTok/sWhat it means
6× RTX 5090, full expert residency in VRAM5.8–6.8 (TTFT ~13 s)The ceiling: no disk I/O at all. This is a GPU rig, not "hardware you already own".
128 GB RAM desktop, CPU only, warm cache~1.8The realistic best case for a fat workstation without GPUs.
Single RTX 5070 Ti (~24 GB VRAM tier)1.07A laptop-class GPU roughly doubles the cold-disk floor.
RTX 5080 + 32 GB RAM + 2 NVMe, DeepSeek V4 Flash~1.6 at 3k context, after warmupThe dual-SSD path on a smaller model.
25 GB RAM laptop, slow NVMe, cold0.05–0.1The headline "744B on 25 GB" configuration. About one token every 10–20 seconds.

The variable that decides your number is random-read bandwidth from the drive, not CPU. A cold token needs on the order of 11 GB of expert reads; a PCIe 5.0 NVMe at 13–15 GB/s therefore caps a cold, unprefetched setup near 1 tok/s, and lookahead plus cache hits are what get you above that. Commenters in the launch thread reported cache hit rates around 23% on a 128 GB Apple M5 Max and 0.091 tok/s on an old dual-Xeon with 192 GB of DDR3 — "viable for overnight jobs".

So the honest use cases are: overnight or asynchronous batches, private offline experiments with a frontier-class model, research into how MoE inference actually behaves, and warm multi-turn chat on a machine with 100 GB+ of RAM. Interactive pair-programming on a 32 GB laptop is not on the list.

Nine model families, and what each one costs in disk​

ModelTotal / activeDiskRAM minNotes
OLMoE7B / 1B~7 GB8 GBThe "does my build work" model
Qwen3.635B / 3B~20 GB24 GB (full RAM residency)Optional GPU; 7× speedup measured on dual 8 GB cards
Qwen3.8-Flash-Next125B + 51B n-gram / 6B~185.5 GB16 GBCPU only
DeepSeek V4 Flash284B / 13B~167 GB16 GBNative fp4 experts; GPU optional (Pascal or newer)
GLM-5.3-Flash321B / 40B~195 GB25 GBVision-capable
DeepSeek V4.1 Flash552B / 16B~203 GB16 GBNew in 1.11.0; reads the released checkpoint with no conversion
GLM-5.2 / 5.3744B / 40B~372 GB16 GBThe reference model; MIT-licensed weights
Inkling975B / 41B~469 GB25 GBbf16 dense is 49.4 GB resident by default; an int4 tool brings it to 25 GB hosts
Kimi K32.8T / 104B~1.6 TB32 GB+Native MXFP4; requires the raw checkpoint, no conversion available

Quantization is per family, not one global setting: int4 group-scaled for GLM and Inkling, int8 or block-FP8 for OLMoE and Qwen3.8, native MXFP4 for Kimi K3, native fp4 experts with fp8-e4m3 dense for the DeepSeek V4 line. If you already have Kimi K3 or DeepSeek V4.1 Flash weights from a vLLM deployment, Colibrì reads those same files.

Four gotchas that cost people days​

  1. MTP heads must be int8. Multi-token-prediction draft heads quantized to int4 give 0–4% draft acceptance — speculative decoding silently does nothing (issue #8). The pre-converted GLM-5.2 container on Hugging Face is int4 for the model and int8 for MTP for exactly this reason.
  2. O_DIRECT helps on some drives and hurts on others. Bypassing the page cache measured +34% on NVMe drives with a DRAM cache and bandwidth headroom, and neutral-to-negative on QLC, DRAM-less and virtualized disks. Benchmark both on your drive; do not copy someone else's flag.
  3. Speculative decoding is often a net loss on real chat. On multi-turn conversations the MTP and grammar drafters accepted roughly 1–10 of every 24 tokens, and replaying the engine's recurrent attention state for rejected suffixes cost more than the accepted drafts saved. For DeepSeek V4 the project measured 495 seconds of rejected-suffix replay and ships with both drafters off by default (V4_DRAFT=0, V4_MTP=0). Turn them on only if you measure a win.
  4. SSD wear is real. Heavy expert reads drive page-cache churn; the README warns about wear on consumer drives, and the launch thread's advice was to disable swap and consider a read-only mount for the model directory. A 370 GB model streamed for hours per day is a workload consumer NVMe was not sold for.

One more, less a gotcha than a limitation: the learned expert pinning can overfit to your routing history. It helps repeated workloads; the maintainers say it still needs held-out, cross-session A/B tests before you should trust the gains on new prompts.

From clone to first token​

Guided walkthrough1 of 5
  1. git clone https://github.com/JustVugg/colibri && cd colibri/c && ./setup.sh — then make -C c glm for GLM-5.2, or make -C c inkling / make -C c kimi_k3 for the others. Start with OLMoE (7 GB) to prove the build before committing hundreds of gigabytes of downloads.

A minimal first run on GLM-5.2 (Linux, one NVMe)

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh
make -C c glm
./coli convert --model /nvme/glm52_i4        # one-time download + convert
COLI_MODEL=/nvme/glm52_i4 ./coli plan          # see what lands where
COLI_MODEL=/nvme/glm52_i4 PILOT=1 ./coli chat  # first streamed tokens

Colibrì vs. the alternatives you already know​

PathWhat it is good atWhere Colibrì differs
llama.cpp / Ollama with GGUF offload (our Ollama guide)Huge model catalog, GPU offload, mature tooling; the GLM-5.2 page covers the 239 GB dynamic 2-bit GGUF routeGGUF offload needs the whole quantized model in RAM plus VRAM (256 GB for GLM-5.2 at 2-bit). Colibrì needs 16–24 GB of RAM and a big disk, at the same precision the lab released, not 2-bit.
vLLM / SGLang on rented GPUs (Kimi K3 deploy guide)Dozens of tokens per second, batching, multiple usersFast because everything is resident in HBM. Colibrì exists for the case where that hardware is not available or not allowed.
Hosted API (DeepSeek, Z.ai, Moonshot)Cheapest per token by far, no hardwareThe only reason to prefer Colibrì is that the weights and the prompts must never leave the machine, or you want to study MoE inference itself.

The right way to think about Colibrì is not "a faster llama.cpp". It is the tool that makes a frontier-sized open model available on a machine that could not otherwise load it, at a speed you schedule around rather than wait on. For a privacy-first stack that mixes a small fast local model for interactive work with a large slow one for batch jobs, see A private local AI stack.

Pulsa Intro o Espacio para girar la tarjeta. Usa las flechas izquierda y derecha para moverte entre las tarjetas.Término mostrado.
1 / 7

Check yourself

0/5
  1. Why can a 744B model run on a 16–25 GB machine with Colibrì at all?
  2. What is the binding constraint on tokens per second when experts are streamed cold?
  3. Why does the project ship DeepSeek V4 speculative decoding turned off by default?
  4. You quantized GLM-5.2 yourself, all tensors to int4, and speculative decoding shows ~2% draft acceptance. What happened?
  5. Which workload is Colibrì a good fit for on a 32 GB laptop with one NVMe?

Sources & further reading​

Next​