The $1,000 Box, One Month On: A Model Zoo and Multi-Token Prediction

A follow-on to 200 Billion Parameters for $1,000: Running 4-Bit Quants on eBay Hardware.

TL;DR

When we first wired up the $1,000 Xeon box — dual 2016-vintage E5-2698 v4s, 128 GB of DDR4, and a pair of $250 Quadro P5000s — it ran three big MoE models at faster-than-you-read speeds. Nothing in that hardware has changed. Today the same ~$1000 box serves 44 models, including four 200B-plus giants the original post never mentioned: DeepSeek-V4-Flash (284B), Tencent’s Hunyuan Hy3 (295B), MiniMax-M2.7 (230B), and Step3.7-Flash (198B). The bigger story is multi-token prediction (MTP) — letting a model draft its own next few tokens and verify them in a single pass. On our flagship Qwen3.5-122B it lifted decode from 11 to 17 tok/s (+52%); on dense 27B models it nearly doubled throughput, and a coder model hit 58 tok/s at 96–99% draft acceptance. The catch: MTP only pays when acceptance is high. On a cheap MoE it’s a net slowdown. Measure acceptance, not architecture.

The zoo grew — same box, no new hardware

The original piece made a narrow claim: for about $1,000 of eBay parts you can run a handful of 4-bit, 200B-class MoE models locally, trading concurrency and prefill speed for access. A month of nights-and-weekends later, that same chassis is a router in front of 44 distinct model presets — dense chat models, OCR pipelines, embedding and rerank endpoints, and a shelf of frontier-scale MoEs loaded on demand.

The heavyweights are the headline. All of these decode on the same 40 cores, with their expert tensors parked in system RAM (--cpu-moe) while attention and the KV cache ride the two 16 GB Pascal cards:

Note the pattern the original post predicted and this table confirms: active parameters, not total, set decode speed. Hy3’s 21B-active experts make it the slowest thing in the rack despite not being the largest. DeepSeek-V4-Flash’s 13B-active MLA gets it to 7.4 — and a single upstream patch (fused hyper-connection ops, llama.cpp #25585) bumped it there from 5.2, a free +42% for the cost of a rebuild.

MTP: the model drafts, then checks its own work

The interesting lever since the last post is multi-token prediction, a flavor of speculative decoding. Instead of a separate draft model, the newer Qwen3.6, Hy3, and Qwen3.5-122B GGUFs bundle a tiny “NextN” head inside the file. That head proposes the next few tokens; the full model verifies them in one forward pass. Every accepted guess is a token you got almost for free. Because verification is batched and our cpu-moe models are memory-bandwidth-bound, that extra check costs almost nothing — the win is close to pure.

The numbers on our slow box:

• Qwen3.5-122B, the flagship: 11.1 → 16.9 tok/s (+52%), with 90–96% draft acceptance. This is the single biggest quality-of-life change since launch — the original “about 10 tok/s, faster than you read” is now comfortably past 15.
• Dense 27B models: qwen3.6:27b went 12.7 → 24.0 (~1.9×) at 95.7% accept; the coder variant, 13.2 → 24.6.
• A 3B-active coder MoE: qwopus3.6:35b-coder reaches 58 tok/s with MTP on, riding 96–99% acceptance on actual code.

The catch: acceptance decides, not architecture

MTP is not free money. It wins only when the draft head is usually right — high acceptance amortizes the verification pass. When acceptance is low, you pay for guesses you throw away.

We learned this twice. First, turning MTP on for Gemma4-26B — a cheap 3B-active MoE — was a net slowdown, 45 → 33 tok/s: the model already decodes cheaply, so draft overhead swamped the gain. Second, Hy3 shipped at ~44% acceptance and sat exactly at break-even (4.96 tok/s, tied with the non-MTP control). The fix wasn’t more drafting — it was less. Setting spec-draft-p-min = 0.75, so the head only drafts tokens it’s confident about, pushed acceptance to 83–87% and nudged decode to 5.12. The coder’s 96–99% is why a low-active MoE that “should” lose with MTP wins big instead: on code, the next token is nearly always predictable.

So the rule from a month of benchmarking is blunt: measure acceptance for your workload before trusting MTP. Coding and QA love it; creative writing can actually regress. Architecture is a weak predictor.

Prefill is still the tax

None of this fixes the original post’s honest caveat: prefill on old CPUs is slow, and cpu-moe makes it slower. The 200B-class models still ingest context at 40–80 tok/s, so a long document is a coffee break. MTP helps decode, not prefill — the draft head is tiny and adds nothing to the prompt pass. If your use is single-user, conversational, and decode-bound, the box has quietly gotten 50–90% faster on the same silicon. If you feed it 40K-token prompts all day, the answer is the same as it was: buy newer cores, or wait.

The thesis holds, and got cheaper per token: frontier-scale local inference isn’t a hardware problem. It’s a patience-and-tuning problem — and the tuning keeps paying off.