跳到主要内容

Strata: a 125B MoE at 94 tok/s on a 12 GB Card — and the Exact Bill

高级
What you'll learn
  • Understand the three architecture facts that make a 125B model interactive on a 12 GB card — and why none of them is 'aggressive quantization'
  • Read Strata's own benchmark table honestly: the headline 94 tok/s is the *lowest* quant on the ladder, and the top quant costs 43% of it
  • Price the quality bill precisely: 91.3% of SWE-bench Verified next to 98.7% of LiveCodeBench, and what that gap tells you about where pruning actually hurts
  • Know the four gotchas the README headline does not carry — including a default that passes needle-in-haystack while quietly losing long-document accuracy
  • Decide between Strata, plain llama.cpp with --n-cpu-moe, and a hosted API for your specific job

On 24 September 2026 a repository called Strata appeared with a one-click installer and a claim that reads like a typo: run a 125-billion-parameter multimodal model on a gaming PC, at speeds you can hold a conversation with. Ten days later it has 10,622 stars, 935 forks and 265 open issues, and its Hacker News thread is at 473 points with 245 comments. One commenter's summary of the moment: "LLM threads the world over are spammed with Strata links."

The claim is true. The interesting part is what it costs, because almost nobody is writing that part down — the project's own headline numbers contain no accuracy column at all. This page supplies it.

The one-sentence version​

Qwen3.8-Flash-Next activates only about 6B of its 125B parameters per token, so Strata keeps the per-token path (attention, routers, shared expert, output head) in VRAM, pins all 24,576 experts in system RAM, and streams a 28.8 GB lookup table from SSD — buying interactive speed instead of the batch speeds that expert-streaming engines settle for.

If you have met Colibrì, this is the opposite end of the same hierarchy. Colibrì puts the experts on disk to reach 744B-parameter models at 0.05–6.8 tok/s. Strata puts the experts in RAM to reach 44–94 tok/s on a model a fifth the size. Different tier, different use case: one is for overnight batch, this one is for a coding agent you actually wait on.

Three architecture facts, none of them "quantization"​

People assume the trick is brutal compression. Compression is involved, but three properties of the model do most of the work, and knowing them lets you predict behaviour on your own machine.

1. The sparsity ratio is roughly 20 to 1​

QuantityValue
Total transformer parameters125B
Activated per token~6B
MoE experts per layer512 (10 routed + 1 shared active)
Layers48
Total experts in the model48 × 512 = 24,576
Hidden size2,560

Ten routed experts out of 512 means roughly 2% of the expert bank is touched for any given token. Expert weights are therefore large but cold — exactly the profile you want to push down the memory hierarchy. The per-token path that stays hot is small enough to sit on a 12 GB card.

2. The model is bigger than "125B" and smaller in RAM than its file size​

This is the fact that explains the surprisingly modest 32 GB RAM floor, and it is almost always reported wrong.

Qwen3.8-Flash-Next is 176.9B parameters in total: a 125B transformer, a 51B n-gram embedding table, and a 4B multi-token-prediction module. The n-gram table is a fixed lookup structure used for prompt lookup — it is not matrix-multiplied, so it does not need to be resident. Strata reads it unbuffered from SSD.

Take ISTA-DASLab's Coder release as the worked example:

ComponentFile sizeResident in RAM?Stored at
Transformer weights (256/512 experts kept)29.6 GBYes1.89 bpw effective over the original 125B
N-gram table28.8 GBNo — read from SSDfixed 4.5 bpw
Total on disk58.4 GB29.6 GB residentvs 354 GB at BF16

So a 58 GB download has a 29.6 GB memory budget. That is the whole reason a 32 GB machine is on the compatibility list, and it is why putting this model on a mechanical drive is a configuration error rather than a slow choice — the table is on the hot path for every prompt.

3. Only 12 of the 48 layers grow a KV cache​

Flash-Next is a hybrid-attention model. Three of every four layers use Gated DeltaNet (GDN), a linear-attention mechanism that compresses history into a fixed-size recurrent state. The remaining layer uses Qwen Sparse Attention (QSA) for precise retrieval across the full context. The split is 36 GDN layers to 12 QSA layers.

The consequence is the one local-inference guides keep missing: on a conventional 48-layer transformer, context cost scales with all 48 layers. Here it scales with 12. That — not quantization — is why a 262,144-token native window (extensible to 1M) is affordable on consumer hardware, and why Strata can offer a --kv-resident flag that keeps the context cache in RAM with only attention's working set in VRAM.

The memory map​

TierWhat lives there
VRAMAttention and DeltaNet mixers, routers, shared experts, output head, MTP draft layer, part of the KV cache above 64K — then an expert cache that fills whatever VRAM remains with the most-used experts
RAMAll 24,576 experts, pinned. The CPU computes non-cached experts in place with AVX-512/AVX2 kernels, concurrently with GPU work
SSDThe 28.8 GB n-gram table, read unbuffered (--ple-io direct; use --ple-io ram on a rotational disk)

Two mechanisms make the expert cache work better than a static split. The shipped expert profile ranks all 24,576 experts by how often they fired on training data, so the cache starts warm. And from 0.1.31 there is router lookahead: while the CPU works on one layer, a background thread runs the next layer's router to predict which experts are about to be needed and prewarms the OS file-cache pages for them, overlapping I/O with compute. An adaptive tier system (--adapt-every) then reshapes the cache around your actual conversation.

Reading the benchmark table honestly​

These are the project's own measurements. Note what the quantization column does to them.

RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAM

QuantizationDecode (tok/s)Prefill (tok/s)
Q2_0942,650
IQ2_XS792,090
IQ3_XXS621,750
IQ3_S531,620
Coder (pruned)552,180

RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM

QuantizationDecode (tok/s)Prefill (tok/s)
Q2_0601,160
IQ2_XS521,110
Coder (pruned)441,420

The headline "94 tok/s" is Q2_0 — the bottom rung. IQ3_S, which the project labels highest quality, runs at 53 tok/s: you surrender 43% of the advertised speed to climb the ladder. Community reports fill in the rest of the spread, and they are wide: 124 tok/s on a 4090 with 128 GB of DDR5 and a 7950X3D; about 30 tok/s on a 3080 (10 GB) with a 3600X; 33 tok/s on an RTX 2060 at Q2_0 but 21 at IQ3_XXS; around 25 tok/s on an M1 Max at 3-bit; and 7 tok/s decode with 60 tok/s prefill on a GPU-less 8845HS laptop with 96 GB.

Read that list as a rule rather than a leaderboard: decode speed tracks RAM bandwidth and quantization level, not GPU class. The 2060 beating the 3080 at Q2_0 is not a measurement error. The card holds the hot path; the system memory streams the experts.

The quality bill​

Here is what the headline numbers leave out. There are two separate compressions in play, and they fail in different ways.

GSQ — the quantizer​

Gumbel-Softmax Quantization is post-training scalar quantization that learns grid assignments and group scales jointly, which narrows the usual gap to vector quantization at low bit-widths while keeping files loadable as standard GGUF. Retained experts land at 3.5 bits per weight.

RCO — the pruner​

Riemannian Constraint Optimization chooses which experts to delete. It treats the feasible set as a smooth manifold so per-layer budgets are enforced exactly, with no penalty coefficient to tune, and supports a different budget per layer simultaneously. The Coder release keeps 256 of each layer's 512 experts — still 10 active per token — calibrated on code, agentic and vision data.

What the pruning costs, measured​

BenchmarkBF16 baseGSQ-RCO CoderRetention
SWE-bench Verified82.8075.6091.3%
LiveCodeBench v687.4386.2898.7%

Do not read those as "roughly the same." The gap between them is the single most useful number on this page.

LiveCodeBench is short-horizon: read a problem, emit a solution. It barely notices that half the expert bank is gone — 1.3% off. SWE-bench Verified runs long agentic trajectories against real repositories, and there the same model drops 7.2 points. Same weights, same quantization, six times the relative damage.

Expert pruning costs you horizon, not knowledge. The model still knows how to write the function. What degrades is holding a plan across forty tool calls — exactly the capability an agentic coding loop is made of. One commenter's word for it was "lobotomizes," and the benchmark pair is the receipt.

The authors are explicit about the matching scope limit: calibration covered code, agentic and vision data, and "degradation outside this set is an accepted cost." If you point the Coder variant at prose, translation, or non-code reasoning, you are off-distribution by design, with no published number for how far.

Four gotchas the headline does not carry​

Watch out
  • The speed mode called ESP is documented by the project as removing refusal directions. That is a safety-behaviour change shipped inside a performance feature — treat it as a different model, not a faster one.

1. ESP trades alignment and output distribution for speed. The project's own documentation records that its speed-projection path degrades safety by removing refusal directions, raises perplexity on code by 15%, and drops top-1 token agreement with the stock model to about 93%. A 7% disagreement rate on next-token choice compounds hard across a long generation. If you enable it, you are no longer running the model you benchmarked.

2. 4-bit KV passes needle tests while losing long-document accuracy. The shipped default is 8-bit KV. --kv q4_0 applies a Hadamard rotation before quantizing, halves cache memory and is about 4% faster at 128K — at the cost of 8–12% higher perplexity on long documents. The documentation notes needle tests still pass. That is the trap: needle-in-haystack retrieval surviving is not evidence that long-context reasoning survived. The installer offers you 4-bit above 4K context, so this is a decision you will be asked to make before you have any reason to understand it. There is also a --kv k8v4 hybrid — 8-bit keys, rotated 4-bit values — for 23% less KV memory with measurably slower prompts.

3. Identical weights, different engine, different answers. One HN commenter ran a 50-image coordinate-localization benchmark through both engines using the same GGUF and the same vision adapter: Strata produced a median error of 154.8 pixels (mean 168.8); llama.cpp produced 46.5 (mean 81.4). Their assessment was that the gap "is as large as the jump from a 9B model to a 35B model." The project's documentation supplies a plausible mechanism in the same family of problems — its IQ-pack multi-token kernels round differently from the single-token CPU paths, with an opt-in reproducibility flag (STRATA_IQ_MT_MIN=1) added in 0.1.30. The lesson generalises past this project: when a new engine claims a large speedup on weights you already have, run your own eval on your own task before you trust the output, because "same file" does not mean "same answers."

4. Past 262K you are in untested territory. Beyond the native window, yarn RoPE scaling applies, and the documentation describes it as experimental with quality measured on a single machine. Community reports of tool-calling failures and spelling errors at 150K+ are consistent with that caveat.

Getting a first token​

Guided walkthrough1 of 5
  1. Minimum is 12 GB VRAM on an NVIDIA RTX 20/30/40/50 series or a supported AMD Radeon (RX 7700 XT and up, RX 9060 XT, RX 9070/9070 XT, Radeon AI PRO R9700, RX 6800/6900), 32 GB system RAM, and about 80 GB of free space. Make the storage an SSD: the 28.8 GB n-gram table is read on every prompt, so a rotational disk is not a slow configuration but a broken one.

Expose the server on your LAN with an API key

START-HERE.bat --setup --host 0.0.0.0 --api-key <your-secret>

The flags worth knowing, and what each one trades

--ple-io direct      # unbuffered n-gram table reads (use 'ram' on a rotational disk)
--kv-resident 32768  # keep the context cache in RAM, only attention's working set in VRAM
--prefill auto       # chunk prompts up to 8,192 tokens for GPU streaming
--pcie-frac N        # share of missing experts fetched over PCIe vs computed on CPU
--pool-workers N     # CPU threads computing non-cached experts
--adapt-every N      # how often the expert cache re-ranks around your conversation
--kv q4_0            # halve KV memory: ~4% faster at 128K, 8-12% worse perplexity on long docs
--kv k8v4            # 8-bit keys, rotated 4-bit values: 23% less KV memory, slower prompts
--spec-min-p         # draft-acceptance confidence threshold for speculative decoding
What you'll learn
  • Settings are split between CLI arguments, the per-model strata-<model>.json args block, and STRATA_* environment variables, and the documentation does not consistently say which lives where. Useful env vars include STRATA_PF_FUSED (fused prompt kernels), STRATA_DECODE_TIMING=1 (profile decode windows), STRATA_RESIDENT_PIN=0 (disable page-locking in low-RAM resident mode) and STRATA_IQ_MT_MIN=1 (single-token-path rounding for reproducibility).

Which rung do you actually want​

Your jobPickWhy
Agentic coding, long tool-call trajectoriesFull model at IQ3_S or Unsloth UD-IQ4_XS (~80 GB RAM)SWE-bench is where pruning costs 7.2 points. Do not pay that on your main workload
One-shot code generation, snippets, reviewCoder (pruned, 32 GB RAM)98.7% of LiveCodeBench for half the expert bank. This is the rung the pruning was designed for
Vision and coordinate workStock llama.cpp with ggml-org/Qwen3.8-Flash-Next-GGUFThe one head-to-head measurement on identical weights favours llama.cpp by 3x on localization error
Maximum tok/s for throwaway draftingQ2_094 tok/s, and roughly 80% of the model's "memories" at 2 bits by one commenter's information-theoretic estimate. Fine for drafts, not for facts
Long documents, 128K+ contextFull model, 8-bit KV, stay inside 262K4-bit KV and yarn scaling are both documented as lossy or experimental
Reliability over speedStock llama.cpp with --n-cpu-moeSlower on the same hardware, but it is the reference implementation your eval should be measured against
Anything where being wrong is expensiveA hosted APIThe HN consensus from people who own this hardware: they still default to cloud for demanding work

The honest baseline: llama.cpp with --n-cpu-moe​

Strata is not a from-scratch engine. It is built on llama.cpp / ggml (MIT), transcribes i-quant GPU kernels from upstream, uses llama.cpp's quantized MMQ dot-product kernels, and takes its vision encoder from llama.cpp's mtmd library. The packaging and the Flash-Next-specific scheduling are the contribution. That matters practically: the baseline is one flag away.

--n-cpu-moe N moves the MoE feed-forward weights of the first N layers to CPU. The reasoning is the same sparsity argument this page opened with: expert stacks are large and rarely touched, so they buy the most VRAM for the least speed penalty, while attention, embeddings and the KV cache are small and touched every token and belong on the GPU. More VRAM means a lower N and a higher ceiling before the CPU side becomes the bottleneck. If the model already fits in VRAM, raising N degrades throughput steadily — the flag is a way to fit, not a way to go faster.

Reported community numbers put llama.cpp around 21 tok/s where Strata reaches 100+ on comparable hardware. That is a real and large gap. It is also the gap you are buying with a ten-day-old codebase, 265 open issues, documented kernel-rounding divergence, and no accuracy column. Decide deliberately rather than by headline.

What you'll learn
  • The GGUF ecosystem for this model is unusually crowded: ISTA-DASLab's GSQ-RCO releases (1.89M and 351k downloads), unsloth (1.4M), the official ggml-org build, bartowski, and several abliterated and 'uncensored' community variants. Download count is popularity, not quality. The two with published benchmark tables are ISTA-DASLab's; the reference build for comparison is ggml-org's.

What this run of the hype cycle is actually teaching​

Strip out the specific project and three durable lessons remain, and they will apply to the next engine that trends too:

  1. Sparsity, not compression, is what makes big models local. A 125B MoE with 6B active can be more comfortable on consumer hardware than a 32B dense model, because the constraint it puts you under is RAM capacity and bandwidth rather than VRAM capacity. Check the active-parameter count before the total.
  2. Compression has a shape, not a magnitude. "91% retained" and "99% retained" came from the same artifact on the same day. Which number you get depends entirely on whether your task is short-horizon or long-horizon. Ask which benchmark the retention figure was measured on.
  3. An engine is part of the model. Identical weights produced a 3x difference in localization error between two engines. Kernels round, caches quantize, speed modes project. Your eval has to cover the engine, not just the checkpoint.

Check yourself

0/6
  1. Why can a 125B-parameter model run on a 12 GB GPU here, when a 32B dense model would struggle?
  2. The Coder variant retains 98.7% of LiveCodeBench v6 but only 91.3% of SWE-bench Verified. What does that gap tell you?
  3. You enable 4-bit KV cache and your needle-in-haystack test still passes at 128K. What have you established?
  4. A 58.4 GB download is listed as needing only 32 GB of system RAM. Why is that not a contradiction?
  5. The same GGUF and vision adapter gave a median localization error of 154.8 px on one engine and 46.5 px on another. What is the right conclusion?
  6. What does llama.cpp's --n-cpu-moe flag do, and when does raising it hurt?
按 Enter 或空格键翻转卡片。使用左右方向键在卡片之间切换。已显示术语。
1 / 11

Sources & further reading​

Next​