मुख्य कंटेंट तक स्किप करें

Ternary Quantization in Practice: Bonsai 2 27B in 5.95 GB

उन्नत

On 17 September 2026, PrismML published Ternary Bonsai 2 27B: a 27-billion-parameter model whose weights are only -1, 0 and +1, packed into a 5.95 GB file, released Apache 2.0. The Hugging Face GGUF repo passed 2.5 million downloads inside five days. The headline everywhere was the same — a 27B model that fits on a laptop.

The headline is true. Almost everything people assume follows from it is not.

Stock llama.cpp will load the file and produce fluent nonsense without raising an error. Ollama and LM Studio refuse it outright. The 5.95 GB file needs roughly 12 GB of VRAM at default settings. The larger 7.21 GB build is faster on a datacenter GPU and slower on a 4090. And the widely-quoted "98.2% of full precision" is an average across fourteen benchmarks that hides a 6-point collapse in multimodal reasoning and, in one independent test, complete failure on low-resource languages.

This page is the practical layer: what ternary quantization actually buys you, how to run this model without silently corrupting your output, and how to read a compression retention number honestly. The technique outlives the model — the same reasoning applies to every low-bit release that follows.

What you'll learn
  • Understand why ternary weights are 1.75 bits in practice, not the 1.58 bits the theory promises
  • Recognise the silent-garbage failure mode: stock loaders ignore the Hadamard rotation and never error
  • Pick between PTQ1_0 and PQ2_0 using the bandwidth-vs-unpack crossover, not file size
  • Budget real memory: 5.95 GB of weights, ~64 KiB per token of cache, and why defaults blow past 12 GB
  • Read '98.2% retention' correctly — which capabilities survive compression and which quietly do not

What ternary quantization actually is​

A normal model stores each weight as a 16-bit float. A ternary model stores each weight as one of three values — -1, 0, +1 — with a single shared FP16 scale per group of weights. Reconstruction is w = scale_g * t, where t is the trit.

Three states need log₂(3) ≈ 1.58 bits. That is the number the research literature uses, from Microsoft's BitNet b1.58 onward. It is not the number you get on disk.

Bonsai 2 27B ships at 1.75 bits per weight for its densest build. The extra 0.17 bits per weight is the part nobody explains:

  • Group scales. One FP16 scale per group of 128 weights is 16 bits amortised over 128 weights — 0.125 bits per weight before anything else.
  • Packing waste. Trits do not divide neatly into bytes. Dense packing recovers most but not all of the theoretical density.
  • Unquantized components. Embeddings, output projections and normalisation layers stay at higher precision. On a 27B model these are a small fraction of parameters but a noticeable fraction of bytes.

So the real ratio is not 16/1.58 ≈ 10×. It is 53.8 GB → 5.95 GB, or about 9× — which is exactly what the model card reports.

Why ternary is fast, and it is not the maths

The intuition most people have is that ternary multiplication is cheap: multiplying by -1, 0 or +1 is a sign flip, a skip, or nothing. True, but that is not where the win comes from on real hardware.

Token generation is memory-bandwidth bound, not compute bound. Every generated token requires streaming the entire weight matrix from memory. Cut the bytes by 9× and you cut the dominant cost by 9×. The arithmetic was never the bottleneck.

This matters practically: a CPU or GPU without native ternary support must dequantize back to a normal format to use its existing multiply-accumulate units, which throws away the arithmetic saving while keeping the bandwidth one. Which is why the packing format you pick changes performance in a direction that looks backwards — see below.

The trap: stock llama.cpp loads it and lies​

This is the single most important thing on this page, and it appears in almost no coverage of the release.

Bonsai 2's weights have a blockwise Hadamard rotation folded into storage. A matching inverse transform must be applied to the activations at runtime. The rotation is a standard trick for low-bit quantization — it spreads outlier magnitudes across a block so the ternary grid has less to lose — but it means the stored weights are not directly usable by a loader that does not know about it.

A stock llama.cpp build does not know about it. Depending on version it will either reject the tensor types or, worse, load the file successfully and generate fluent, confident, completely wrong output. There is no error, no warning, no degraded-quality flag. The model card states this plainly: stock llama.cpp will not run these files, and standard builds produce garbage because the Hadamard activation transform is missing.

Downstream tools inherit the problem in different ways:

RuntimeBehaviour with Bonsai 2 27B GGUF
PrismML's llama.cpp fork (branch prism)Works — the only local GGUF path today
Stock llama.cppRejects the tensor types, or loads and emits nonsense
OllamaImport fails with a tensor size overflow error
LM StudioRejects ggml tensor types 142 and 143
MLX (Apple)Separate mlx-2bit build required; larger footprint

The Ollama and LM Studio failures are the safe outcome. The stock-llama.cpp silent success is the dangerous one, because a benchmark script or an eval harness will happily record the nonsense as a result.

If you are evaluating a low-bit model, verify the loader first

Run one known-answer prompt before you run anything else. If a model that scores 95+ on HumanEval cannot answer "what is 2+2" coherently, you have a loader problem, not a model problem. This applies to every exotic quantization format, not just this one.

Running it​

Guided walkthrough1 of 6
  1. git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp, then cmake -B build -DGGML_CUDA=ON && cmake --build build -j for NVIDIA. On Apple Silicon drop the CUDA flag — Metal is on by default. On AMD use -DGGML_HIP=ON. The default branch is prism; the README explicitly warns against building from prism-v6 and against mixing this fork's libraries with a stock llama.cpp install on the same machine.

Minimal end-to-end: build, fetch, serve, verify

# 1. Fork only — stock llama.cpp will silently produce garbage
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# 2. Fetch (PQ2_0 = 7.21 GB, faster on big GPUs; PTQ1_0 = 5.95 GB, fits 8 GB cards)
hf download prism-ml/Ternary-Bonsai-2-27B-gguf \
Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

# 3. Serve — always set -c explicitly, never leave it at 0
./build/bin/llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
--alias bonsai-2-27b -ngl 99 -fa on -c 32768 \
--host 127.0.0.1 --port 8080

# 4. Verify the loader BEFORE trusting any output
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"bonsai-2-27b","messages":[{"role":"user","content":"What is 2+2? Answer with one number."}]}'
# Coherent "4" => loader is correct. Fluent nonsense => wrong build, stop here.

PTQ1_0 vs PQ2_0: the crossover that looks backwards​

Two builds of the same weights. PTQ1_0 packs trits densely at 1.75 bits. PQ2_0 gives each trit its own 2-bit slot at 2.13 bits — 21% more disk, but the unpacking is far cheaper because slots are byte-aligned.

Measured generation throughput, tokens per second:

GPUPQ2_0 (7.21 GB)PTQ1_0 (5.95 GB)Winner
RTX 5090129.9120.5PQ2_0
RTX 409081.291.1PTQ1_0
L4 (24 GB)29.832.1PTQ1_0
M5 Pro (Metal)28.1——

The pattern generalises: PTQ1_0 wins on Ada-generation and lower-end GPUs, PQ2_0 wins on H100, A100 and Blackwell.

That is the opposite of the usual quantization rule of thumb, where smaller is always faster. The reason is the bandwidth-versus-unpack tradeoff from earlier. On a bandwidth-starved card, moving 21% fewer bytes dominates and the dense packing wins. On a card with enormous memory bandwidth, bandwidth stops being the constraint and the bit-shuffling cost of dense unpacking becomes the bottleneck instead — so the fatter, byte-aligned format runs faster.

Prompt processing tells the same story more loudly: on an RTX 5090, PQ2_0 processes prompts at 3,893 tok/s against PTQ1_0's 1,805 — more than double, because prefill is compute-bound, where unpacking cost hits hardest.

Rule of thumb: below 12 GB of VRAM you have no choice, take PTQ1_0. Above it, take PQ2_0 unless you are on a 4090 or an L4, where PTQ1_0 measurably wins.

File size is not memory requirement​

Three different "size" numbers are circulating for this model — 5.9 GB, 7.2 GB and 8.6 GB — and all three are defensible, because they measure different things. Here is the actual budget:

  • Weights: 5.95 GB (PTQ1_0) or 7.21 GB (PQ2_0).
  • Vision projector: +629 MB (Q8_0) or +931 MB (BF16), if you need images.
  • Attention cache: roughly 64 KiB per token. That is ~0.5 GB at 8K context, ~2 GB at 32K, and ~16 GB at the full 262K window.
  • Parallel slots: llama.cpp allocates cache per slot. Defaults allocate several.

One independent test measured just over 12 GB of VRAM at default settings, dropping to about 8 GB when the server was restricted to a single parallel slot. The disk footprint they measured — around 7 GB — corresponds to the PQ2_0 build, which is where part of the number confusion comes from.

Practical context headroom by card, using PQ2_0 unless noted:

VRAMBuildUsable context
8 GBPTQ1_0~16K tokens (~6K with vision)
12 GBPQ2_0~56K tokens
16 GBPQ2_0~118K tokens
24 GBPQ2_0~240K tokens
32 GB+PQ2_0Full 262K window

The "runs on a laptop" claim survives all of this — but "runs on a laptop with an 8 GB GPU and a 16K context" is the honest version, and that is a materially different product.

What "98.2% retention" hides​

PrismML reports an aggregate of 84.78 against the full-precision base model's 86.32 across fourteen benchmarks — the 98.2% figure. Averages are where compression damage goes to hide. The per-benchmark spread:

BenchmarkBonsai 2 27BFP16 baseDelta
AIME2695.8394.58+1.25
MATH-50098.8099.80−1.00
LiveCodeBench90.0790.05+0.02
HumanEval+95.1293.29+1.83
MMLU-Redux89.0991.46−2.37
MMMU-Pro75.4981.73−6.24

Three things to take from this table.

First, the damage is not uniform — it is structural. Maths and code are essentially untouched. Knowledge recall drops (composite knowledge-and-reasoning lands near 79.9 versus 85.6 for full precision). Multimodal reasoning takes the real hit: MMMU-Pro falls 6.24 points, and OCR Bench v2 drops from 60.99 to 56.88. Compression costs you breadth of stored fact, not procedural skill — which is exactly what you would expect when you crush the precision available to store long-tail associations.

Second, be suspicious of the positives. The compressed model "beating" its own full-precision base by 1.83 points on HumanEval+ and 1.25 on AIME26 is not evidence that compression helps. It is evidence that these benchmarks have run-to-run variance of a couple of points. Which should also make you discount the small negatives.

Third, the vendor's own numbers do not fully agree. The model card's benchmark table aggregates to 84.78, while PrismML's launch post reports an aggregate of 83.9. Both are described as the same 98.2% retention. The discrepancy is small and unexplained; we report it rather than pick one.

The failure mode benchmarks did not catch

One independent hands-on test found behaviour no benchmark row predicts: asked to translate into low-resource languages including Tamil, the model entered a repetitive loop and never completed the task — worse than the previous Bonsai generation. The same test found it identified a real timezone bug and then talked itself out of the correct conclusion, and failed to produce a working reset button in a small HTML task across three correction rounds.

This is the characteristic signature of aggressive quantization: aggregate scores hold, while the long tail — rare languages, long-horizon self-correction — degrades in ways that only show up in use. Community reports also describe recall and coherence degradation past roughly 200K tokens of the advertised 262K window.

The mainline story, and why it matters for the next release​

There is a parallel effort to get ternary Bonsai into upstream llama.cpp, and its current state is a useful lesson in how low-bit formats fragment.

PrismML proposed group-128 ternary packing for the smaller Bonsai models. The llama.cpp maintainer pushed back in favour of group size 64, on the grounds that it costs under 6% extra memory while supporting more tensor shapes and giving better quality. Q2_0 with group-64 became the official format and has been merged into multiple backends including CUDA and Vulkan. PrismML kept their group-128 variants in the fork.

So "does llama.cpp support ternary models" has two correct and opposite answers:

  • Yes — Q2_0_g64 builds run on mainline across backends, and the fork's own README points at them as the portable path.
  • No, not for this model — as of verification, the published Ternary-Bonsai-2-27B-gguf repo contains F16, PQ2_0, PTQ1_0 and the two mmproj files. There is no Q2_0_g64 build of the 27B. The fork is the only path.

Two older ternary formats, TQ1_0 and TQ2_0, already exist in ggml but use group size 256, which is incompatible with how these models are natively packed. TQ2_0 is also the tightest faithful ternary layout in ggml at roughly 2.06 bits per weight — and in current mainline it implements a CPU-only dot product with no CUDA, Metal or Vulkan path. Seeing "ternary supported" in a changelog tells you very little about whether your GPU will use it.

For the next low-bit model you evaluate, the question to ask is not is it quantized but which group size, which packing, which backends have kernels, and is there a portable build — because that determines whether you are running on mainline or maintaining someone else's fork.

When this is the right tool​

Worth it when:

  • You need a capable maths or coding model on hardware that cannot hold a 27B at higher precision. That is where retention is near-perfect and the value is highest.
  • You are bandwidth-constrained — a laptop GPU, an edge box, a shared L4 — and 9× fewer bytes per token is the difference between usable and not.
  • Energy matters. PrismML reports 0.714 mWh per token on an RTX 4090, which they put at 40% more efficient than a full-precision 8B model.
  • The Apache 2.0 licence and fully local execution are requirements, not preferences.

Not worth it when:

  • The work is knowledge-heavy or multimodal. That is precisely where the compression bill lands.
  • You need non-English long-tail language coverage. The independent evidence here is bad and benchmarks did not predict it.
  • You cannot maintain a forked inference runtime. Ollama and LM Studio do not load it today, and a stock build fails silently rather than loudly.
  • You were going to run it past ~200K tokens of context on the strength of the 262K spec.
कार्ड पलटने के लिए Enter या Space दबाएँ। कार्ड बदलने के लिए बाएँ और दाएँ तीर कुंजियों का उपयोग करें।शब्द दिखाया गया।
1 / 10

Check yourself​

Check yourself

0/5
  1. You download Ternary-Bonsai-2-27B-PTQ1_0.gguf, run it on a recent stock llama.cpp build, and it generates fluent text that is subtly wrong. What is the most likely cause?
  2. You have an RTX 4090 with 24 GB. Which GGUF build gives higher generation throughput?
  3. Your card has 8 GB of VRAM. The weights file is 5.95 GB. You launch llama-server with default settings and it runs out of memory. Why?
  4. A teammate cites '98.2% of full-precision performance' to argue the compressed model is a drop-in replacement for a document-understanding pipeline. What is the strongest objection?
  5. A changelog says llama.cpp 'supports ternary quantization'. What must you still check before assuming your GPU will use it?

Sources & further reading​

Next​