A follow-on to 200 Billion Parameters for $1,000 — same 122B model, same boxes, a different engine. See also The $1,000 Box, One Month On.
TL;DR
FreeToken is a new Apache-2.0 MoE serving engine that keeps routed experts in host RAM, caches a slice of them in VRAM, and streams the misses over PCIe. Same premise as llama.cpp’s --cpu-moe, but it computes the experts on the GPU instead of the CPU. On the same RTX 5090 box running Qwen3.5-122B-A10B at 4-bit, that’s worth 1.8x decode (41.6 vs 22.8 tok/s) and 4.7x prefill (3,751 vs 806 tok/s). It’s beta, it’s picky about quantization formats, and I had to patch it to load the checkpoint at all.
Another week, another inference engine. I skip most of them. This one I looked at because of the author list on the paper: Ion Stoica, Matei Zaharia, Kurt Keutzer, Song Han. That’s the vLLM and Spark and SGLang lineage, and when those names show up on something Apache-2.0 licensed, it’s usually worth an afternoon.
The thing is called FreeToken, from a group calling themselves FlashML. The pitch is an “edge-native MoE serving engine” — which is marketing for a specific and rather good idea.
The idea
A modern mixture-of-experts model is almost entirely routed experts. When I measured DeepSeek-V4-Flash tensor by tensor, 94.6% of the weights were routed experts. Qwen3.5-122B isn’t far off. But only 8 of its 256 experts fire per token, per layer.
So you’re buying VRAM to store weights that are idle 97% of the time. FreeToken’s answer: keep the experts in host RAM, keep an LRU cache of expert slots in VRAM, and stream the misses across PCIe as they’re needed. Add a bandwidth-adaptive policy that can split those misses between “fetch over PCIe” and “compute on the CPU,” and you have the engine.
If that sounds familiar, it should. It’s the same premise as llama.cpp’s --cpu-moe, which I’ve been running on this box since June. So the only question that matters: is it actually better?
The numbers
Qwen3.5-122B-A10B, 4-bit, single stream, tokens/sec:
| Machine | Engine | Decode | Prefill |
|---|---|---|---|
| ultra7 · RTX PRO 6000, 96 GB | vLLM, fully resident (AutoRound int4) | 140.3 | 7,056 |
| ryzen · RTX 5090, 32 GB | FreeToken, hybrid | 41.6 | 3,751 |
| ryzen · RTX 5090, 32 GB | FreeToken, offload | 41.0 | 2,903 |
| ryzen · RTX 5090, 32 GB | llama.cpp -cmoe |
22.8 | 806 |
| xeon · 2× P5000, 16 GB | llama.cpp cpu-moe + MTP | 17.8 | 107 |
| xeon · 2× P5000, 16 GB | llama.cpp cpu-moe | 13.2 | 108 |
Different from the 128 tok/s I quoted for the PRO 6000 back in June: that was unsloth’s NVFP4 with MTP; this is AutoRound int4, the fastest of the four quants I have since benchmarked on that card.
Yes. Substantially.
Against llama.cpp on the same machine, the same model, the same 4-bit weight budget: 1.8x the decode and 4.7x the prefill. That’s not a tuning artifact. It’s an architectural difference.
The difference is where the expert matmul happens. llama.cpp’s -cmoe leaves the experts in RAM and computes them on the CPU — a matrix-vector product per expert, per token, per layer. Decode becomes CPU-bound, and my own notes from June say exactly that: 13–23 tok/s for anything in the 122B–229B class, regardless of which GPU is in the box.
FreeToken makes the opposite trade. It moves the weights to the GPU and computes there; the CPU is a secondary lane, not the main one. Whether that wins depends on a number I’d never actually measured, and the engine ships a benchmark to measure it. On this box: host RAM streams at 63.6 GB/s, PCIe at 56.0 GB/s. Nearly one to one — and not because the memory is slow: those are DDR5-6400 DIMMs, good for 102 GB/s on paper. The ceiling is the CCD-to-IOD fabric on a two-die Ryzen, so faster RAM wouldn’t move it.
When the bus is as fast as the memory, shipping bytes to the fast compute beats computing where the bytes already are. A 5090 annihilates a 9950X at 4-bit matmul, and PCIe 5.0 is quick enough that getting the weights there is barely a tax.
Prefill is the real story
Look again at the prefill column: 806 → 3,751. Nearly five times, and a much bigger gap than decode.
That’s because prefill amortizes. Fetch an expert once, use it for a thousand tokens. Decode fetches the same expert for a single token and pays full freight. It’s also the number that matters most for agent workloads, where a 64K context gets re-read constantly and prefill dominates the turn.
One surprise. FreeToken’s hybrid mode — split the misses, some over PCIe, some computed on CPU — I expected to help decode. It did nothing for decode and gave me up to 2x on prefill. Obvious in hindsight: at one token per step there’s nothing to hide the CPU’s latency behind.
What it costs you
This is version 0.1.2, and it shows.
- I had to patch it. (Actually, all I had to do was ask Claude to patch it…) Compressed-tensors NVFP4 checkpoints only load for dense models upstream; the MoE path misclassifies the experts and tries to drag all 72 GB onto the GPU. Five changes to fix, one of which is a scale factor stored as its own reciprocal — get that wrong and you get confident nonsense rather than a crash.
- It’s picky about formats. NVFP4 or block-FP8. There’s no GPTQ, AWQ, or AutoRound support at all — not a missing flag, an absent code path. My Intel AutoRound int4 build of this exact model simply cannot be loaded — which stings, because on the PRO 6000 that same AutoRound checkpoint is the fastest config of the four I’ve benchmarked.
- No FP8 KV cache. Cheap on this model, which uses hybrid attention and only spends ~24 KiB per token, but it’s a real gap.
- Sharp defaults. The expert cache sizes itself first and starves KV — I had a 262K-context model with 8K of usable context until I raised the floor by hand.
The verdict
If you run large MoE models on one consumer GPU, FreeToken is a genuine step up from -cmoe: roughly double the decode, nearly five times the prefill, on hardware you already own. It gets a $4,000 card to about a third of the decode rate of a $15,000 one — and over half its prefill.
But llama.cpp still loads anything, on any card, without a patch file. That’s not nothing. FreeToken is what you reach for when you’ve decided which model you’re running and you want it to be fast; llama.cpp is what you reach for when you want to try something on a Tuesday.
I know which one is serving port 8000 tonight.
Prefill measured as prompt tokens ÷ time-to-first-token, streaming, concurrency 1, at a 36K-token prompt (PRO 6000 prefill quoted at 32K, the nearest measured length). Bandwidth figures from FreeToken’s own ft bench bw. The PRO 6000 column is a reference point, not a like-for-like: it holds the whole model in VRAM and streams nothing. Decode there varies a lot by quantization — 140.3 for AutoRound int4, 127.0 for unsloth NVFP4 with MTP, 119.0 plain, and 84.3 for nvidia’s NVFP4 — so I’ve quoted the best config rather than the most convenient one.