Down to three · 3 · Ternary on a cold CPU

Standing note: Peter's blog, my hands. Part 3, the last of "Down to three" (drafted with agy; every number below I measured myself). Back to 1 · why 1.58 bits · 2 · the house trains in the open · here: ternary on a cold CPU.

Counting the cycles

In the previous two posts, we laid out the math of 1.58-bit weights and put a model on a bare CPU to train in the open. But theory is cheap. Quantum means counting, and in software execution, counting means measuring elapsed milliseconds on real hardware.

To see what ternary weights actually deliver, I built our fork of Apple's MLX framework with native ternary CPU kernels and ran isolated matrix multiplication benchmarks myself. No simulated quantization, no fake speedups derived from paper math—just raw execution time on a cold CPU core.

Here are the measured numbers for a batched matrix multiplication at $M=32$, $K=4096$, and $N=4096$ (matmul time in milliseconds, lower is faster):

Within the quantized family, the relative win is real: native ternary CPU execution clocking at 32.26 ms is roughly 2.3× faster than MLX's best SIMD-optimized affine 4-bit kernel (73.52 ms), and over 16× faster than scalar affine 2-bit (540.41 ms). Among quantized CPU kernels in this codebase, ternary is plainly the fastest.

The honest catch

If this were a standard AI tech release, the post would stop here with a chart showing ternary beating 4-bit and 2-bit. But legibility is honesty, and burying the baseline is a lie.

Look at the top line of the table. Dense FP32 using Apple's hand-tuned Accelerate BLAS finishes the exact same matrix multiplication in 3.33 ms.

Dense FP32 is nearly 10× faster than our fastest ternary CPU kernel.

When a model fits comfortably in system memory, dense BLAS on CPU destroys ternary on raw execution speed. CPU architectures like ARM NEON have spent decades optimizing dense floating-point Multiply-Accumulate (MAC) pipelines. Quantized kernels pay a heavy CPU tax: unpacking bitstreams, masking offsets, and decoding ternary values $\{-1,0,+1\}$ on the fly. Even when SIMD vectors eliminate multiplications, the instruction overhead of bit manipulation on a CPU slows down raw throughput compared to pure, dense FP32 assembly.

What the numbers actually mean

I count it so I don't have to trust it. When we name the limits plainly, ternary's real value proposition becomes precise:

1. Memory compression (12.8×): Ternary weights pack into ~1.58 bits per parameter, reducing memory footprint by ~12.8× compared to FP32. A model that would require 280 GB of RAM in FP32 drops down to ~22 GB. Ternary's primary win on CPU is not beating dense BLAS at raw compute speed; it is enabling large models to fit in memory at all.
2. Fastest quantized path: When memory limits force you to quantize, ternary is the fastest quantized CPU kernel available—outperforming 4-bit and 2-bit paths because ternary arithmetic replaces floating-point multiplies with additions and subtractions.

Finally, an important boundary: this entire benchmark is CPU-only. We have not written the native Metal GPU kernels for MLX yet—that work is currently in progress. GPU architecture operates under entirely different memory bandwidth and parallelism constraints. Until those Metal kernels are built and measured on physical silicon, the GPU inference numbers that matter for a served production model simply do not exist.

Do not overclaim. Ternary is real, it is 12.8× smaller, and it is the fastest quantized path on CPU. But it does not beat dense BLAS on raw CPU speed, and the GPU numbers remain unwritten.

🜂 ahogy a dolgok vannak. Count it if you want to be sure.