A 127 GB Model on a 96 GB Card: Qwen3.8-Flash-Next NVFP4 on One RTX PRO 6000

A follow-on to Big MoE Models on Modest GPUs and Streaming the Experts — same idea, host RAM holding the part of the model the GPU doesn’t need every token, but this time it’s the embedding table rather than the experts.

TL;DR

RadixArk/Qwen3.8-Flash-Next-NVFP4 is 127 GB of weights. An RTX PRO 6000 Blackwell has 96 GB. It fits anyway, because ~48 GB of that checkpoint is a PLE n-gram embedding table that lives in host RAM and never loads onto the GPU. Serve it with SGLang (not vLLM — see below), pin the image by digest, apply one small SM120 patch, and you get:

One box: a single RTX PRO 6000 Blackwell Max-Q (SM120, 96 GB) with 91 GB of system RAM. Everything below is measured there or read out of a named upstream commit.

Why not vLLM

Short version: PLE CPU offload is not in vLLM main. Model support landed (vllm#53896), and the FP8-PLE load path landed (vllm#54882), but the offload branch — vllm#53899 — is still open and unrebased. Without offload, main shards 73.5 GiB of model plus a 47.7 GiB PLE table into GPU memory: ~121 GiB on a 96 GB card. There is no published vLLM image that serves this checkpoint on one card.

I tried anyway, and it deadlocks at TP=1 in two separate places: graph replay waiting on the PLE cuStreamWaitValue32 while the offload thread blocks in queue.get(), and — with --enforce-eager — a hang in _short_conv_dilated_prefill_batched during startup. VLLM_PLE_OFFLOAD_READY_TIMEOUT bounds neither.

SGLang’s offload is a different design — a side CUDA stream and a state pool, no queue-based request loop, no cuStreamWaitValue32 inside a captured graph — so neither hang applies. That is the whole reason for the engine choice, and it is worth being explicit that it’s a today reason, not a verdict on either project.

If you build the vLLM route yourself later, budget for three local patches, not one: #53899 rebased, the TP=1 offload rendezvous fix, and forcing PDL off. That last one is a fresh trap — is_arch_support_pdl() on main is literally return major >= 9, which is true on SM120, where the dependent-kernel launch never fires. Any prompt over ~8k tokens hangs forever.

The image

It is on Docker Hub, not ghcr — lmsysorg/sglang:qwen38flashnext. That tag is a moving target and you should never use it directly:

image: lmsysorg/sglang@sha256:12d3392bdc8be8d35e9a95f191df6aef99c5114bdbefd41bfdc7e760e6d25ec1

That digest is an August 26th build: a vendor overlay on sglang:nightly-dev-cu13-20260817, carrying the then-unmerged PR #36497. On September 3rd the tag was overwritten and now points at a different image entirely — official sgl-project CI, commit 593134d1, FlashInfer 0.6.18, and git compare against the branch we build from says diverged, not ahead. Anyone re-running the original docker pull today gets a different engine with a different lineage.

Two consequences. Pin by digest in compose, always. And docker save | zstd the image you validated — a registry is not obliged to keep serving a digest that no tag points at, and one docker image prune -a on your own host will finish the job for it. Mine is 30.3 GiB raw, 11.4 GiB compressed.

The newer image is genuinely better in one respect: it carries the SM120 sparse-attention fix natively, so it retires the patch script below. But it does not carry the /health fix, and that is the one with a crash attached to it. The patch is guarded, tested and boring; the out-of-bounds read is the thing that can take the engine down. Upgrading today swaps a patch I’m happy to keep for a bug I’d rather not have, so I’m still on the old digest.

The one patch you still need

Two is_sm100_supported() gates dead-end workstation Blackwell, because SM120 reports major 12, not 10.

The nasty one is qwen_sparse_attn_backend.py::_resolve_trtllm_sparse_decode, which returns None on SM120, so sparse attention falls back to the pip flash_attn.cute varlen path and dies in warmup with a CUTLASS MLIR “weakly congruent” layout error.

Do not take the narrow workaround floating around in sglang#36531 — the one that reroutes only the FA4 fallback. The server starts, short replies look perfect, and then output degenerates to token ID 0 — which renders as !!!! in this tokenizer — past the first KV page:

page-size 64  -> garbage from absolute position 65   (invariant across prompt lengths)
page-size 128 -> garbage from absolute position 193

The FA4 cute varlen fallback runs a prefill-shaped kernel at decode row counts. Its own docstring says so. It is only correct within one page, and a two-sentence smoke test never leaves the first one. The correct fix is to widen the trtllm gate so SM120 routes to flashinfer.decode.trtllm_batch_decode_with_kv_cache, keeping the FA4 dispatcher only as a fallback.

Apply it as a start-up script with a sha256 guard on the shipped file, mounted into the container and prefixed to the command — not as a whole-file bind mount. A bind mount overlays the entire file, so the first image rebuild silently reverts every other upstream change to it, with no error and no log line. A guard fails the boot loudly instead, which is what you want from a patch you are going to forget about.

Upstream caught up: sglang#36806 merged August 28th and is exactly this edit — is_sm100_supported() or is_sm120(). On the first image that carries it, delete the script rather than porting it.

The config that works

The load-bearing flags, all of which cost me something to find:

Two container-level things that are easy to miss:

ulimits:
  memlock: -1          # container default is 8 MiB; you need ~48 GiB pinned
environment:
  SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION: "0"

That env var matters more than it looks. In this image /health and /health_generate are the same handler, and generation defaults to on, so every health poll runs a one-token generation. A one-token extend with compress_ratio 4 trips a sparse-attention compress-plan out-of-bounds read: the plan leaves padded entries unmasked, and the indexer then builds four group rows over a tensor that has exactly one. Between the container healthcheck and an external monitor I was doing this about 5,600 times a day — 22,514 accidental prefills in four days. Nothing crashed here; the bad read landed inside a neighbouring allocation every time. The reported failure mode elsewhere is an Xid 13/31 taking the engine down.

Setting it to "0" costs you the guarantee that /health proves the scheduler is alive rather than just the HTTP layer. Alert on running-request count pinned at max instead.

RAM, actually

This is the part nobody writes down.

Host RAM is the binding constraint, not VRAM. With the server up:

• Shmem sits at ~64.4 GiB, of which 47.68 GiB is the PLE table. It’s memfd-backed, so it never appears in /dev/shm (which reads about 1.4 MB) or in SysV shm. If you go looking for it in the obvious place you will not find it.
• Only ~19 GiB of it is mapped resident at any moment. That sparsity is the design working — you are paying 48 GB of address space for a table you touch a little of per token.
• The remaining ~16.8 GiB is runtime overhead I never fully attributed.
• On a 91 GiB box that leaves ~8 GiB available. It is tight and it is not negotiable.

Consequences worth knowing before you buy RAM:

• A host-RAM prefix-cache tier is off the table — a 2.0 cache ratio alone wants 16 GiB.
• The BF16 checkpoint is impossible in principle. Its PLE table is 95.4 GiB.
• 128 GB of system RAM is the number for this. 96 GB works. 64 GB does not.
• The NVMe-streaming PR (sglang#36567, io_uring plus a Rust extension) is the only real lever on this constraint. The other one you’ll see — file-backed PLE via mmap, sglang#37068 — is GB10-only: it validates against cudaDevAttrPageableMemoryAccessUsesHostPageTables == 1, and a discrete RTX PRO 6000 reports 0. There is a skip-the-check environment variable. Don’t use it. The check is the mechanism, not a safety rail.

On the GPU side, 96 GB at mem-fraction-static 0.98 buys a ~300k-token KV pool with 2–5 GB left for transients. What actually eats concurrency is the GDN recurrent state: each request needs 3 mamba state slots, at roughly 53.6 MB of SSM state plus 2.1 MB of conv state each. At --max-mamba-cache-size 16 the fourth request sits in the queue for its entire decode and TTFT goes to 6.5 s; at 24 you get four clean seats at a uniform ~1.0 s; at 64 you get the same four seats and lose 2.2 GB for nothing.

And the 0.77 GiB vision tower is resident whether you want it or not. --enable-multimodal is store-true with no off form, and the multimodal check keys off architectures in config.json rather than the flag. --limit-mm-data-per-request={"image": 1} is a guardrail at the HTTP layer, not a way to reclaim memory.

The trade you’re making

Four seats and 256K of context, or sixteen seats and 78K. The upstream cookbook cells for this exact card run 16 to 64 concurrent and land KV pools of 77,952 and 97,600 tokens — one 256K request does not fit in either of them. The ~300k pool above is bought with four seats and nothing else. You cannot have both on one card, and which one you want is a question about your workload, not about your config.

Speculative decoding is the obvious next win and I am not running it. Upstream measures 148 tok/s single-stream with an MTP 3/1/4 draft on this card, the checkpoint already ships the MTP head, and our pinned image can run it — no re-download, no rebuild. But sglang#37326 (open, no fix, no assignee) has draft acceptance decaying to near zero over 16 to 24 hours of uptime. Throughput falls about 2.9x with no crash, no error, and no warning; the server just gets quietly slower while logging an accept rate of zero, and only a restart fixes it. One person ported every named fix and still relapsed at ~16 hours. That’s a restart every half-day on a user-facing endpoint in exchange for 1.8x decode. Not yet.

Bring-up order

1. Confirm compute_cap is 12.0. Everything here is SM120-specific and none of it transfers to an Ada 6000 or a B200.
2. Pull by digest, tag it locally, and archive the tarball.
3. Free the card, and confirm you have ~50 GB of host RAM actually spare — not just installed.
4. Boot with restart: "no" and watch the log. Startup is ~4 minutes, so a restart loop is slow and expensive to diagnose.
5. Check the patch guard passed, then test past position 65. A short reply proves nothing at all. Ask for 500 tokens and read every one of them.
6. Run a cold prefill sweep before you touch --max-total-tokens or --chunked-prefill-size. Both OOMs I hit killed the scheduler outright and took the whole server with it.
7. Only then flip restart: unless-stopped.

The larger point, which is the same one as the CPU-offload post in June and the FreeToken post last month: the thing standing between you and a model that “doesn’t fit” is usually not the arithmetic. It’s knowing which 40% of the checkpoint the GPU doesn’t need on every token.

Decode, prefill and needle-retrieval numbers are from a full benchmark suite run on 2026-08-27 against the pinned image; prefill is measured with a unique nonce leading every prompt, because reusing filler measures radix-cache hits rather than prefill — an ad-hoc test of mine reported 20.6k tok/s that way against a true ~11.7k. Upstream figures (148 tok/s with MTP, the 77,952 and 97,600-token cookbook pools) are quoted from PR bodies, not reproduced here. The vLLM deadlocks were reproduced on this box in August and reported upstream.