A follow-on to The $1,000 Box, One Month On: A Model Zoo and Multi-Token Prediction, which followed 200 Billion Parameters for $1,000: Running 4-Bit Quants on eBay Hardware.
TL;DR
Cleaning out the junk drawer, I found 64 GB of DDR4 I’d paid $50 for before the rampacolypse. The sensible move was listing it on Facebook Marketplace for some astronomical number like $600. Instead I plugged it into the eBay Xeon box from the last two posts, taking it from 128 GB to 192 GB — the first hardware change in the entire series.
Then I handed the box to Claude Opus and told it to go find the wins: pick the quants, run the benchmarks, keep the ones that pay. I stayed out of the way except for a handful of calls where I overrode it. That division of labor turns out to matter to the story, because four of the five quant upgrades made models slower, one by 23%, and the only outright win was an upgrade Opus recommended skipping and I ran anyway. Along the way a piece of folklore in my own notes — that i-quants dequantize slower than K-quants on CPU — failed head-on twice and had to be thrown out.
The useful result is two rules that actually predict what a quant change costs on a CPU-offloaded box, and they are not the same rule:
- Prefill tracks bytes added. +1.1 GB is free. +25 to +48 GB costs 15–20%.
- Decode tracks expert bits-per-weight, not quant format. Fewer bits per weight, less RAM traffic per token, faster decode — whatever the letters in the filename say.
Higher quant ≠ better. On this box it’s usually quality bought with speed, and you should know the exchange rate before you spend 40 GB of download on it.
And the other thing more memory appears to buy — huge context windows — turns out to be mostly theoretical at these prefill rates. Filling DeepSeek’s context to 128K means 35 minutes before the first token appears. Point a coding agent at it with a 64K context and a 20-turn loop takes 6h 14m, against under two minutes for a smaller model on a consumer Blackwell card. I raised zero context windows this round, and the section near the end does that arithmetic in full.
The junk drawer
The box in this series has never had a hardware change. That was sort of the point — two posts about squeezing 200B-class models out of $1,000 of 2016-vintage eBay parts, and all the gains came from software: --cpu-moe, layer offload tuning, split modes, and multi-token prediction. Same dual E5-2698 v4s, same pair of $250 Quadro P5000s, same 128 GB of DDR4.
Then I found four 16 GB sticks in a drawer. Fifty dollars, bought well before DRAM prices went vertical. Current Marketplace comps for the same kit are genuinely absurd — I could have flipped it for something in the neighborhood of $600 and felt clever.
I put it in the server instead.
192 GB total, 188 usable. In practice that lifted the ceiling on how much model I can pin in RAM from about 105 GB to about 165 GB, because these models are mlock‘d — pinned into anonymous memory so the kernel can’t page an expert out to swap mid-generation. That pin is the whole trick from the first post, and its size is the constraint on how good a quant you can run.
So: new headroom, five models sitting at compromise quants chosen when the ceiling was 105 GB. Obvious move, right? Upgrade them all.
How this actually got run
Worth being explicit about the working arrangement, because it shaped the result.
I didn’t do this optimization pass by hand. I gave Claude Opus the box — the repo, the router, shell access, the benchmark scripts — and a standing instruction: figure out which models can move to better quants now that there’s headroom, measure everything, keep what pays and revert what doesn’t. It picked the tiers, probed the files, ran the A/Bs, and wrote the results into the repo’s notes as it went.
My interventions were few and mostly consisted of pushing for more than it wanted to do:
- It proposed a middle tier for DeepSeek; I asked why we weren’t going straight to the top 4-bit tier. (That one blew up spectacularly. Read on.)
- I extended the same top-tier question to two more models.
- It wanted to delete a rejected 144 GB quant; I told it to archive it to a spare array instead.
- It recommended skipping the Nemotron upgrade as pointless. I told it to run the benchmark anyway.
- On hy3, it laid out a speed-versus-quality trade and asked which I wanted. I took the quality side, eyes open, and told it to skip the second perplexity run to save half an hour.
That last category is the one I’d point at for anyone doing this. The model was rigorous about measuring — arguably more rigorous than I would have been, since it caught its own bad reasoning twice — but it was consistently conservative about what was worth measuring. The single biggest throughput gain of the exercise came from overruling a well-argued “don’t bother.”
The obvious move was wrong four times out of five
Here’s what actually happened. Every one of these is a same-build A/B — old quant and new quant measured back to back on the same binary, on a settled box, with the old files kept on disk until the comparison was done.
| Model | Swap | Δ size | Decode | Prefill | Verdict |
|---|---|---|---|---|---|
| Nemotron-3-Super-120B | Q4_K_M → Q4_K_XL | +1.1 GB | +7.8% | −1.4% | Kept — the one win |
| MiniMax-M2.7-230B | IQ4_XS → Q4_K_XL | +30 GB | −1.2% | −16% | Kept for quality |
| Step3.7-Flash-198B | IQ4_XS → Q4_K_XL | +25 GB | −6.1% | −14.4% | Kept for quality |
| DeepSeek-V4-Flash-284B | IQ3_XXS → Q4_K_XL | +48 GB | −23% | −17% | Reverted |
| Hunyuan Hy3-295B | Q2_K → IQ3_XXS | +8 GB | −28% | −3.7% | Rejected |
| Hunyuan Hy3-295B | Q2_K → Q3_K_M | +32 GB | −29% | −20% | Kept — deliberate quality call |
Four of six swaps made the model slower. One made it dramatically slower. The single upgrade that improved throughput was the one Opus had recommended skipping on the grounds that it barely changed anything.
Let me take the interesting failures in order, because each one broke a different assumption.
Failure 1: the MXFP4 trap (−23%)
DeepSeek-V4-Flash was running at UD-IQ3_XXS — a 3-bit quant — purely because the 4-bit version wouldn’t fit in the old 128 GB pin ceiling. This was the upgrade the RAM was for.
Opus initially queued a middle tier. I asked why we weren’t just going to UD-Q4_K_XL, the top 4-bit tier, since 144 GB now fit comfortably. No good reason surfaced, so we killed the first download and pulled the big one.
Decode fell from 6.94 to 5.33 tok/s. Minus 23%. Prefill fell 17%. The memory pin verified clean, swap was idle, the box wasn’t contended. It was just slower.
The reason is a genuinely counterintuitive property of this particular model, and it’s worth internalizing because more models are shipping this way. DeepSeek-V4-Flash is natively MXFP4 — a 4-bit floating-point format the model was quantization-aware-trained in. So the tiers don’t work the way you assume. A higher tier doesn’t replace MXFP4 with something better; it preserves more of it:
| Tier | MXFP4 share of expert tensors |
|---|---|
| UD-Q4_K_XL | 100% |
| UD-IQ4_XS / Q3_K_M | ~35% |
| UD-IQ3_XXS | 0% |
Pascal has no FP4 tensor cores. Every MXFP4 expert matmul on this box is a software dequantization loop. So the “best” quant of this model is the one that hands my nine-year-old GPUs the largest possible pile of arithmetic they have no silicon for. The shipped 3-bit quant is the only tier in the entire lineup with zero MXFP4 in it — which makes it not a compromise but very possibly the optimal choice for this hardware, and I’d been treating it as a placeholder for a year.
On a Blackwell card the ranking inverts completely. Same file, same flag, opposite answer.
Opus wanted to delete the 144 GB after the revert. I told it to archive the files to a spare array instead — if a future llama.cpp build lands a fast Pascal MXFP4 path, the test is a cp away rather than another overnight download.
The cheap lesson, and it was cheap for a reason: the load log printed ftype: MXFP4 MoE while the model was coming up. Opus read that line and quoted it back to me as confirmation the right file had loaded — correct, and completely missing what it implied about throughput on a GPU with no FP4 units. The information that would have cancelled the download arrived before the download finished, and we both walked past it.
Failure 2: folklore that survived because nobody tested it
Both MiniMax and Step3.7 were on IQ4_XS. Having seen the DeepSeek result, I asked whether the same top tier made sense for these two. Opus said yes — and expected a speed win on top of the quality win, citing a rule sitting in my own repo notes: i-quants dequantize slower than K-quants on CPU. These were textbook i-quant → K-quant swaps. Should be free speed.
MiniMax: decode flat (−1.2%), prefill −16%. Step3.7: decode −6.1%, prefill −14.4%.
Tested head-on, twice, the rule lost twice. And when Opus went back to find where it came from, it turned out to rest on two incidental observations from unrelated work — never once a controlled comparison. It had been sitting in my notes for months looking like knowledge because nothing had contradicted it, mostly because nothing had asked it a direct question. I’d written it; the model inherited it and dutifully repeated it right up until the bench said no.
Both models stayed on the new quant anyway — I told it to keep them and delete the old files. That’s a deliberate trade: buying output quality with prefill throughput, with the price now known. The point isn’t that the upgrade was wrong, it’s that we’d have made the same call for the wrong reason and filed a fictitious speed win next to it.
Failure 3: a rule invented, written into the docs, and retracted inside a day
Hy3 is the 295B monster, and the slowest thing in the rack — 21B active parameters per token means the heaviest expert matmul here by a wide margin. It was running Q2_K, a 2-bit quant, and my question was reasonable enough: with 64 GB more RAM, can we do better than 2-bit?
I also asked Opus to measure perplexity on the current quant first, so any quality claim about the replacement had a real baseline instead of vibes.
Step one, Q2_K → IQ3_XXS (+8 GB): decode fell 28%. An 8 GB increase should be nearly free by the prefill rule — and prefill was indeed nearly free, −3.7%. But decode cratered.
Opus had an explanation ready. This was a K-quant → i-quant swap, the reverse direction from MiniMax and Step3.7, and it lost badly. So the folklore wasn’t dead, just directional: i-quant dequantization is genuinely slower on CPU, and this test had isolated it. That went into the repo notes as the finding.
Step two, Q3_K_M — pure K-quant experts, no i-quant kernel involved anywhere: 4.12 tok/s.
IQ3_XXS had measured 4.16.
A dead heat. Whatever slowed hy3 down had nothing to do with quant format, because the format changed completely and the number didn’t move. What both quants share is more bits per expert weight than Q2_K had.
That’s the real mechanism, and it should have been obvious from the first post’s own framing: cpu-moe decode is memory-bandwidth-bound. Every token streams the active experts out of DDR4. Fewer bits per weight means fewer bytes crossing the bus per token means more tokens per second. Format determines quality at a given size; bit-width determines speed. Q2_K is fast here because it is small, full stop.
To its credit, Opus caught this itself and retracted the rule in the repo notes without being prompted — it flagged the contradiction the moment the Q3_K_M number came in. But that’s a save, not a clean play. The rule shouldn’t have been written after one test.
What makes it worth writing up rather than quietly fixing: two different explanations predicted the identical result for the first test. “i-quants are slow” and “more bits are slow” both forecast a decode loss for Q2_K → IQ3_XXS. One datapoint could not tell them apart. The wrong one got picked and recorded as established. It took a second test the two hypotheses disagreed about — a pure K-quant at higher bit-width — to separate them.
When two mechanisms fit your one datapoint, you don’t have a finding. You have a coin flip you haven’t noticed you’re calling.
Hy3 stayed on Q3_K_M, and that one’s on me. Opus laid the trade out straight — quality up, −29% decode and −20% prefill — and asked which I wanted. I took quality on a model I use for hard single-shot questions rather than volume, and told it to skip the second perplexity run to save half an hour of box time. It’s down to about 4.1 tok/s, below reading speed and a genuinely different experience from the 5.8 it used to do. The Q2_K files went to the spare array, restorable in a cp if I decide that was a bad call.
The one that won, which the model told me not to do
Nemotron-3-Super was on Q4_K_M. The Q4_K_XL was only 1.1 GB bigger, and Opus probed the tensor tables before downloading anything: of 80 expert tensors, exactly three differed. Two moved Q8_1 → Q8_0, one moved Q4_K → Q5_K.
Its recommendation was to skip it, and the argument was good. Three tensors out of eighty, essentially no added precision, some prefill cost for the extra bytes — nothing to gain on either axis. On the evidence presented, that’s the right call.
I told it to run the benchmark anyway. Partly because 1.1 GB is a cheap experiment, mostly because by that point three confident predictions had already been wrong and I’d stopped trusting the reasoning ahead of the measurement.
Decode: 8.07 → 8.70 tok/s. +7.8%. Prefill flat. The only throughput win of the entire exercise, from the swap that had been argued down to nothing.
The best read is that the two Q8_1 → Q8_0 tensors hit a better-optimized CPU dequantization path — a kernel-quality difference, not a precision difference. Which means the analysis was fine as far as it went and simply answered a different question than the one that mattered: a tensor-count delta tells you about quality, not about speed. Three tensors can’t add meaningful precision. Three tensors can absolutely land on a faster code path.
A +7.8% gain from three changed tensors is exactly the shape of a measurement artifact, so it got the full treatment before going in the table: re-run the new quant (8.69, 8.70 — reproduced), then put the old quant back and re-benchmark it under the same conditions (8.07, matching its 8.12 from before the swap). That’s the single best reason to keep the old files until the comparison is finished — without them the honest control doesn’t exist and you’re comparing against a number from a different week. Opus did this unprompted, which is the flip side of the conservatism: rigorous once measuring, just reluctant to start.
There’s also a mundane failure mode worth flagging if you automate this. Midway through the Nemotron run, another process on the network hit the router and evicted the model — the box serves one model at a time, so anything else using it can wipe out your benchmark. I spotted the number looked off and told it to retest. A fully unattended loop would have recorded that as data.
The two rules, and why they’re separate
Six measurements, one coherent picture. The costs are independent and you have to price them separately:
Prefill ≈ bytes added. Prefill streams the entire weight set through the CPU to digest your prompt, so it pays the size increase in full and linearly.
| Δ size | Prefill cost |
|---|---|
| +1.1 GB | free |
| +8 GB | −3.7% |
| +25 to +48 GB | −15% to −20% |
Decode ≈ expert bits-per-weight. Decode touches only the active experts per token and is bandwidth-bound, so it tracks bits, not gigabytes and not format. A big quant jump that keeps bits-per-weight similar can leave decode roughly flat (MiniMax, −1.2%) while still costing 16% of prefill. A jump that raises bits-per-weight hits decode hard (hy3, −29%).
Practically, for anyone running MoE models with experts in system RAM:
- A large quant jump is quality bought with prefill. Budget ~15% and decide if you want it. Nobody’s stopping you — just don’t expect it to be free.
- A near-same-size tier bump is cheap and might win outright. Nemotron gained 7.8% for 1.1 GB. These are the ones worth trying.
- Read the expert tensor types, never the tier name. “Q4_K_XL” told us nothing useful about any of these five models. What mattered was inside the file.
- Keep the old quant until the A/B is finished. Non-negotiable. It’s the only honest control you’ll get.
The trick that would have saved us a 144 GB download
Rule 3 sounds like it requires downloading the thing you’re evaluating, which defeats the purpose. It doesn’t.
GGUF files put their tensor table near the front, and Hugging Face serves HTTP range requests. So you can pull the first ~40 MB of part 2 of a split model, parse the tensor metadata out of it, and read exactly which quantization type every expert tensor uses — in seconds, without moving the other 144 GB.
That’s how the MXFP4 percentages in the DeepSeek table got measured. Retroactively, unfortunately — Opus wrote the probe after the disappointing benchmark, to explain the result. Run first, it would have said “100% MXFP4 experts” and the whole 144 GB detour wouldn’t have happened.
It became standard procedure immediately after, and has already paid for itself twice: it’s what produced the tensor-level detail for every model in this post, and when a promising new 152 GB model came up at the end of the session, the same probe found its architecture isn’t in any released llama.cpp build yet (the upstream PR is still an open draft). That download never started.
What 192 GB actually bought
Not what I expected. Two of five models are genuinely better — Nemotron is faster and slightly higher precision, and MiniMax and Step3.7 are running quants I couldn’t fit before, at a prefill cost I can now quote exactly. DeepSeek is back on the quant it started with, but for the first time I know why that’s the right answer rather than the affordable one. The context windows, as covered above, stayed exactly where they were.
The real return was on the measurement discipline. Every table cell in my notes that we’d been treating as a baseline had quietly drifted with upstream builds — deepseek 7.12 → 6.94, minimax 8.05 → 7.46, step3.7 8.54 → 8.36, nemotron 8.60 → 8.12. Not one was a valid control anymore. Meanwhile a perplexity measurement from a build a month older reproduced to within noise (6.6996 → 6.7031), because perplexity measures weights and speed measures kernels, and only one of those changes when you rebuild.
Box state alone swung one decode number 14% — same model, same config, measured while a previous 137 GB memory pin was still unwinding, then again ninety seconds later. Both readings internally consistent across three runs. “Three tight runs” is not evidence of a clean measurement; it’s evidence of a stable wrong answer.
There was also a stretch spent chasing a load average of 1.3 during a perplexity run, which looked absurd on a 40-core box — I flagged it as obviously broken. It wasn’t: 45 of 46 threads were correctly asleep because that workload is GPU-bound. The suspicious number was the right number. Worth remembering that “this looks wrong” is a hypothesis too, and mine was the one that failed that time.
The context length you can hold but cannot use
There’s a second thing more RAM buys, and it’s the one I’d been most excited about: context. Freeing memory means you can allocate a bigger KV cache, and several of these models advertise 256K or even 1M token windows. The temptation is to crank every ctx-size to its trained maximum because you finally can.
Don’t. Run the arithmetic first.
Prefill — digesting the prompt before a single token comes out — is the slow phase on this box, and unlike decode, multi-token prediction does nothing for it. DeepSeek-V4-Flash prefills at a measured 62.4 tok/s. So:
131,072 tokens ÷ 62.4 tok/s = 2,101 seconds = 35 minutes
Thirty-five minutes of a dual-Xeon box at full tilt before the model emits its first character. The answer itself is the fast part: 500 tokens at 6.94 tok/s is another 72 seconds. Fill DeepSeek’s context to 128K and you wait ~36 minutes for a reply.
Here’s the full table at the prefill rates measured in this post:
| Model | Prefill | 32K prompt | 128K prompt | 256K prompt |
|---|---|---|---|---|
| gpt-oss-120B | 191.6 tok/s | 2m 51s | 11m 24s | 22m 48s |
| Nemotron-3-Super-120B | 71.1 tok/s | 7m 40s | 30m 43s | 1h 01m |
| Step3.7-Flash-198B | 65.6 tok/s | 8m 19s | 33m 18s | 1h 06m |
| DeepSeek-V4-Flash-284B | 62.4 tok/s | 8m 45s | 35m 00s | 1h 10m |
| MiniMax-M2.7-230B | 51.0 tok/s | 10m 42s | 42m 50s | 1h 25m |
| Hy3-295B | 34.8 tok/s | 15m 41s | 1h 02m | 2h 05m |
Hy3 at its full 256K window is a two-hour wait for the first token. And note the cruel interaction with everything above: the Q3_K_M upgrade cost 20% of prefill, which on a 256K prompt is roughly 25 minutes of pure penalty added to a job that was already unreasonable.
The question that actually settles it
Let me put this in the terms I care about, because I don’t run these models in a vacuum — I run agents. So: just how useful is the increased intelligence I’d get from switching to DeepSeek-V4-Flash for my OpenClaw or Hermes agent with a 64K context, if the turn time became 18½ minutes instead of the 5.7 seconds I get from Qwen3.6-35B-A3B on a single RTX 5090 — 17,500 tok/s prefill and 250 tok/s generation?
That comparison is deliberately not a datacenter card. It’s a 35B MoE with 3B active parameters on one consumer RTX 5090 — fully resident in 32 GB of VRAM, no CPU offload anywhere in the picture. Same 4-bit quantization philosophy as everything else in this series, just small enough to fit.
It is not cheap, mind you. That’s the ~$3,200 card in the ~$5,000 Ryzen box from the original post — five times what the Xeon cost. It’s been sitting there the whole series as the fast-but-expensive option. Hold that thought, it’s the actual decision at the end of this.
The arithmetic:
DeepSeek-V4-Flash on the Xeon box: 65,536 ÷ 62.4 tok/s = 1,050 s = 17m 30s Qwen3.6-35B-A3B on one RTX 5090: 65,536 ÷ 17,500 tok/s = 3.7 s
That’s a 280× gap on prefill alone. Now add a 500-token response. Qwen generates at 250 tok/s on the Blackwell card — 2 seconds. DeepSeek does 6.94 tok/s — 72 seconds, and that’s with every decode optimization from the last two posts already applied.
| Prefill (64K) | Decode (500 tok) | Full turn | |
|---|---|---|---|
| Qwen3.6-35B-A3B, one RTX 5090 | 3.7 s | 2.0 s | 5.7 s |
| DeepSeek-V4-Flash, Xeon box | 17m 30s | 1m 12s | 18m 42s |
| Ratio | 280× | 36× | 195× |
Note what the decode column does to the argument. Decode is the axis I’ve spent two posts optimizing — MTP, layer offload, split modes, quant selection — and it’s only a 36× gap. Prefill is 280×. The thing I can tune is not the thing that dominates, and on a 64K agent context prefill is 94% of the turn.
Now run an agent, which is the whole point. Agents don’t take one turn; they take dozens, each re-reading an accumulating context of tool output, file contents, and prior reasoning. A modest 20-turn loop at 64K context:
| 20-turn agent run | |
|---|---|
| Qwen3.6-35B-A3B, one RTX 5090 | 1m 54s |
| DeepSeek-V4-Flash, Xeon box | 6h 14m |
Under two minutes against six and a quarter hours. That is not a slower agent, it’s a different category of object — one you supervise interactively, and one you launch before bed and read about in the morning.
So is DeepSeek-V4-Flash smarter than Qwen3.6-35B-A3B? On a lot of tasks, honestly, yes — 284B total against 35B, 13B active against 3B, and you can feel it on hard reasoning. It does not matter. No plausible intelligence gain survives a 195× turn-time penalty in an agentic loop, because agents convert latency into wall-clock multiplicatively. A slightly worse answer in 6 seconds beats a better answer in 19 minutes every time you’re going to ask 20 more questions after it. The only regime where the big model wins is the one-shot hard question you’re willing to wait on — and notice that’s precisely not an agent workload.
And the mechanism behind that 195× is the same trick this whole series is built on. --cpu-moe is what makes 284B parameters possible on cheap hardware, and it is exactly what makes them unusable in a loop. Experts streaming out of DDR4 is a brilliant answer to “can I run this at all” and a terrible answer to “can I run this forty times in a row.” The 5090 wins here for one reason: it never touches system RAM.
Which brings back the price tag. At ~$5,000 for the Ryzen build against ~$1,000 for the Xeon, this is emphatically not “cheap hardware loses to expensive hardware.” It’s two genuinely different purchases that happen to share a name:
- $1,000 buys capability. 200B-class models you otherwise could not run at all, at conversational speed, one question at a time. Nothing that fits in the 5090’s 32 GB touches DeepSeek-V4-Flash on a hard problem — that model doesn’t fit on that card at any quantization, so this isn’t a speed tradeoff, it’s the difference between having the option and not.
- $5,000 buys throughput. A smaller model that answers in seconds, over and over, without complaint. That is what an agent actually consumes.
5× the money for a model with 1/8th the parameters, and for agentic work it is the correct buy. That sentence should feel wrong and doesn’t, which is the whole lesson: you are not buying intelligence per dollar, you are buying the shape of the latency curve your workload rides on.
Neither box is a mistake. They answer different questions, and the trap is assuming money spent on one buys you the other. I spent $50 expecting to improve my agent stack and improved something else entirely — a fine outcome, just not the one I’d budgeted for.
Model quality is what you shop for when latency is roughly comparable. When latency differs by two orders of magnitude, latency is the only spec that matters — and prefill speed, not decode speed and not parameter count, is what sets it for anything that reads before it writes.
So what is a big context window for, then?
It’s a ceiling, not a workload. The value is that a long conversation or a document-plus-history session doesn’t hit a wall and truncate. You get there incrementally — a few thousand tokens per turn, each turn’s prefill measured in seconds — and llama.cpp caches the prefix, so turn 40 only re-processes what turn 40 added. That prefix cache is what makes local long-context usable at all, and it’s also exactly what an agent defeats: agents rewrite their context, so the cache misses and you pay full prefill again. Accumulating to 100K over an afternoon of chat is completely fine. Arriving at 100K in one paste, or churning it every turn, is not.
The practical rule I’ve landed on: on this box, prompts should be under ~10K tokens for interactive work (about 2½ minutes of prefill on the slow models, tolerable) and long context should be reached by conversation, not by upload. Anything where you genuinely need to dump 100K tokens of fresh material and get an answer — full-repo analysis, a long PDF, bulk document extraction — is the exact workload this hardware is worst at, and it’s worth being honest that a cloud API or a Blackwell card will do it in seconds. Agentic workloads sit squarely in the bad category and belong on the fast GPU regardless of how much smarter the big model is.
This is also why I declined to raise any context windows during this session, despite now having the memory to do it. The KV cache would fit. The wait wouldn’t. More RAM bought better weights, not usable context — and it took doing the division to see that those are separate purchases.
What I’d tell someone handing a box to a model
The autonomy worked, with a shape to it. Opus was better than I would have been at the parts that are tedious and unglamorous: capturing fresh controls instead of reusing stale ones, re-running surprising results, reloading an old quant to get an honest A/B, retracting its own rule when a later test contradicted it, writing all of it down as it went. That’s most of the actual work, and it’s exactly the part humans skip at 11pm.
What it needed from me was appetite. Left alone it converged on the defensible choice — the middle tier, the skipped benchmark, the well-argued “nothing to gain here.” Three of my five interventions were just run it anyway, and one of those produced the only speed win of the entire exercise. The other two were about optionality: archive the rejected quant instead of deleting it, keep the old files until the comparison is done.
Which sounds like a knock and isn’t, quite. A model that only proposes well-justified experiments is a good property when it’s driving your production box unattended. It just means the human’s job shifts from doing the work to deciding which unjustified experiments are cheap enough to run anyway. On a $1,000 box with a spare 1.5 TB array, that bar is low.
The thesis from the first post still holds and got a little sharper. Frontier-scale local inference isn’t a hardware problem, it’s a patience-and-tuning problem — and this round the tuning mostly consisted of discovering that four confident predictions, several of them from rules in my own notes, were wrong. The box serves 48 models today on the same $1,000 of eBay parts, plus $50 of DDR4 I’d forgotten I owned.
Which, at current Marketplace prices, means the upgrade carried an opportunity cost of about $600. I regret nothing.