The Nightly From Hell: Bisecting vLLM to Find a Working NVFP4 Build

Two 5070 Tis, one 35B model — Part 2 of 3: bisecting vLLM and diagnosing SM120

TL;DR

Native FP4 (NVFP4) is the format Blackwell was built for, and Qwen3.6-35B ships an NVFP4 checkpoint. Getting it to run on two consumer RTX 5070 Ti cards took two separate battles. First, a crash: on SM120 the FP4 GEMM throws an “illegal instruction” unless you feed FlashInfer three specific environment variables that select the 120f code path, undocumented tribal knowledge without which the model won’t load. Second, a correctness regression: the :nightly vLLM image produced pure garbage (empty completions and <|box_end|> spam) under our sharded config. We bisected nine daily nightly builds to pin the regression to a 49-commit window, ruled out the obvious suspects, and froze production on the last good pin. Along the way we root-caused a recurring ~17-second engine freeze to GDN kernel JIT recompilation. This is the debugging story.

Why bother with NVFP4 at all

The production model (Part 1) is Intel AutoRound INT4. It works, it’s stable, it stays on the latest nightly. So why chase a second, fussier build?

Because NVFP4 is native to the hardware. Blackwell’s tensor cores have first-class FP4 support; nvidia/Qwen3.6-35B-A3B-NVFP4 is a mixed-precision checkpoint (W4A16 NVFP4 experts + FP8 attention/KV) designed to hit the flashinfer_b12x MoE kernel, a native-FP4 path that on paper should be the fastest way to run this model on this silicon. If it lived up to that, it’d be the better production build. Finding out whether it did meant getting it stable first. That’s where the trouble started.

Battle 1: the SM120 “illegal instruction”

The very first boot didn’t produce wrong answers. It crashed, hard, with an illegal-instruction fault inside the FP4 GEMM. This is the kind of failure that eats an afternoon because it looks like a broken build when it’s actually a missing incantation.

Consumer Blackwell (SM120) comes in two ISA flavors that FlashInfer distinguishes as 120a and 120f. The 120a variant lacks the TMA warp-specialized grouped GEMM that the FP4 path wants; route the FP4 GEMM through it and you get an illegal instruction. The fix is to force FlashInfer onto the 120f code path with three environment variables, all required together:

- FLASHINFER_CUDA_ARCH_LIST=12.0f
- FLASHINFER_FORCE_SM=120f
- FLASHINFER_DISABLE_VERSION_CHECK=1

That last one matters because the pinned FlashInfer build’s version string doesn’t match what vLLM expects, and the check aborts otherwise. None of this is in a getting-started guide; it’s assembled from issue threads and one working reference deployment. Miss any of the three and the model either crashes or silently refuses the native path.

And selecting the native path is itself a choice you have to make explicitly:

- --moe-backend=flashinfer_b12x

Leaving it on auto lets vLLM pick the flashinfer_cutlass FP4 MoE path, which is broken on SM120. Two more traps sit right next to it: do not set VLLM_USE_FLASHINFER_MOE_FP4=1 (routes to the broken cutlass MoE), and do not force --attention-backend flashinfer (breaks SM120 hybrid models, vllm#40677; leave attention on auto, which resolves to FlashInfer anyway). The single most important post-boot check is one grep:

docker logs qwen36-moe-nvfp4-b12x 2>&1 | grep -Ei "FLASHINFER_B12X|NvFp4 MoE"
# WANT: "Using 'FLASHINFER_B12X' NvFp4 MoE backend"

There’s a benign red herring here: a marlin.py warning that says “GPU does not have native support for FP4.” That’s only the dense linear layers falling back to Marlin; the experts, which are the bulk of the compute, still run on b12x. If you see the b12x line, you’re good; if you don’t, don’t ship it.

With the three env vars and the forced backend, the crash was gone and the model loaded. Then it started lying.

Battle 2: the garbage-output regression

The vllm/vllm-openai:nightly image at the time (build 0.23.1rc1.dev672, dated 2026-07-01) loaded cleanly, engaged b12x, passed its health check, and produced garbage: empty completions, or streams of <|box_end|> special-token spam. Same checkpoint, same config, incoherent output.

The catch: this only happened under sharding. The reference single-card deployment on a 32 GB RTX 5090 ran the same model fine. Something in the interaction between a recent vLLM change and our multi-GPU split was corrupting the model’s output.

When you have a build that works and a build that doesn’t and a pile of daily images in between, you stop theorizing and start bisecting.

The bisect

vLLM publishes a nightly image essentially every day. We pulled the run of dailies spanning 2026-06-22 through 2026-07-01, nine builds, and ran the identical sharded NVFP4 config against each, checking a fixed prompt for coherent output (“What is the capital of France?” → must say Paris; a 17×23 arithmetic check; special-token sanity).

The window collapsed fast:

The regression landed between dev552 and dev601: 49 commits. Everything from dev601 onward, including the :nightly tag everyone pulls by default, was broken for our config.

Ruling out the obvious suspects

Forty-nine commits is a lot, and the tempting move is to blame the biggest, scariest change in the range. Two candidates stood out:

• The FlashInfer 0.6.13 bump. Given that all our SM120 pain lives in FlashInfer, a version bump inside the bad window looked like an obvious culprit. We tested around it. It wasn’t the regression.
• A mamba/GDN align-prefix-cache change. Qwen3.6 is a hybrid model with mamba-style GDN state; a change touching prefix-cache alignment for exactly those layers is a plausible way to corrupt sharded output. Also not it. Ruled out.

We could not pin the regression to a single commit in the window before the practical answer took over: we don’t need the exact commit, we need a working pin. dev552 (2026-06-29) is coherent, sharded, and, importantly, newer than the dev245 build the reference 5090 deployment runs. So dev552 became the production pin for NVFP4: the newest build that’s proven-good under our config.

image: vllm/vllm-openai@sha256:7feb2a09304e3b2d38e224a100316e84fe3205faa7605060609e2c02179cbca6
# 0.23.1rc1.dev552 — newest nightly confirmed clean under sharded NVFP4/b12x

Pinning by digest, not by the :nightly tag, is the whole lesson. The upstream regression (tracked as #47365) remains unfixed, which means NVFP4 is frozen on this pin: it can’t take newer nightlies until upstream fixes it. That freeze is a real cost, and it’s one of the reasons the other build (AutoRound, which happily rides the latest nightly) stayed in production. More on that trade-off in Part 3.

The fp8 KV detour

A smaller battle inside the big one: what dtype to use for the KV cache. NVFP4 is FP4 for the weights, but the KV cache is a separate decision, and the checkpoint is calibrated for fp8 KV. The obvious move on Blackwell is to reach for nvfp4 KV to save even more memory, but that path is hard-gated to SM100/B200 in vLLM and raises a requires sm100f error on our SM120 consumer cards. It simply doesn’t exist for us.

We briefly suspected fp8 KV itself as the source of the garbage output and tried disabling it, a dead end. The reference 5090 deployment runs fp8 KV correctly with the same b12x backend, which is proof it’s not the bug. So --kv-cache-dtype=fp8 stayed, matching the working reference and the checkpoint’s calibration. The lesson recurs throughout this project: when a known-good reference deployment exists, match it first and change one thing at a time. Most of the “obvious” culprits are things the reference already proves innocent.

Cudagraphs, enforce-eager, and capture sizing

There was also a performance knob we nearly left money on the table over. Early debugging runs used --enforce-eager to take the compiler and cudagraph capture out of the picture while chasing the garbage-output bug — a reasonable move when you’re isolating a correctness problem. But eager mode is slow, and once the dev552 pin fixed the correctness issue, eager had to go: compilation and cudagraph capture were restored, matching how the reference deployment runs.

Cudagraph capture on SM120 has its own footgun. The reference 5090 deployment sets --max-cudagraph-capture-size=64 to dodge an SM120 cudagraph-replay crash at larger capture sizes. We took a different route: a PIECEWISE capture set explicitly bounded to the sequence sizes we actually run —

- '--compilation-config={"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4,8]}'

Because max-num-seqs is 8, capturing only sizes [1,2,4,8] keeps every captured graph far under the size that triggers the replay crash, so the standalone max-cudagraph-capture-size flag isn’t needed (and would in fact conflict — vLLM requires it to equal the max of the capture sizes). Bounding the capture set to the real concurrency ceiling is both safer and lighter on capture time than a heavy long-context sweep.

The 17-second freeze

One more gremlin surfaced repeatedly during benchmarking, and it’s worth documenting because it will fool your measurements. Intermittently, the engine would simply freeze for 8 to 17 seconds mid-serving, then resume as if nothing happened. No error, no OOM.

The trigger is a request-shape transition: a large prefill followed by a small decode-shaped request (or vice versa). Qwen3.6’s GDN linear-attention layers use a CuteDSL kernel that is JIT-compiled per problem shape. When the shape changes sharply, the kernel recompiles — and that recompilation blocks the engine (vllm#47458). On the NVFP4 build the effect was pathological: decode throughput measured cold, right after a big prefill, read ~113 tok/s; the same decode measured warm, after the kernel was cached, read ~171 tok/s.

The practical consequences:

1. Benchmarks must be warm. The first small-shape request after big prefills eats the stall. We switched to best-of-N warm timings to get numbers that reflect steady state rather than a compile.
2. Decode feels erratic to a user hitting the server with mixed request sizes, especially on NVFP4. It’s not a throughput problem, it’s a latency spike from a compiler running on the critical path.

There’s no clean fix short of pre-warming every shape you expect or an upstream change to cache across shapes. Knowing the cause at least turns a terrifying “the engine randomly hangs” into a bounded, explainable “the GDN kernel is recompiling.”

Where NVFP4 landed

After both battles, NVFP4/b12x runs correctly on the two 5070 Ti cards: native FP4 experts, FP8 attention and KV, coherent output, images working. Its prefill under TP came in a touch faster than the INT4 build’s TP (~7,700 vs ~6,800 tok/s), and it earned a genuine first in this project — working image requests under tensor parallelism — but only after we solved the vision-tower problem in Part 3.

The catch is everything around the pin: the flaky TP boot from Part 1, the frozen dev552 nightly, an undocumented three-variable crash fix, and a thin memory margin. NVFP4 is a validated capability, not the shipped default. Understanding exactly why it stayed on the bench — and how moving the vision encoder to the CPU changed what’s even possible — is Part 3.

Next — Part 3, “Evicting the Vision Encoder to the CPU”: how a FastAPI sidecar freed VRAM, unlocked NVFP4 under tensor parallelism, and taught us why “more KV cache” nearly killed the engine.