Follows Twice as Fast on the Same Card, the Penny Royal post.
TL;DR
Qwen3.6-35B-A3B NVFP4 serves our chat, vision and tool-calling traffic from an RTX 5090. On
vLLM it decoded 267 tok/s for one request. Inspired by Penny Royal, John Pezzulli’s SGLang
fork for the RTX PRO 6000, I built one for this card and model in a day. The result is
Elsa: 626 tok/s for one request and 1,956 across eight, with 256K context, vision and
tools, now in production on two RTX 5090s. About half the gain is speculative decoding (MTP). The
other half is a dozen smaller fixes, and most of them were about memory, not speed. Stock
SGLang’s MTP fits one request and a 181K-token KV pool on 32 GB. Elsa fits eight requests
and 358K tokens. Two side findings: the CUDA version inside the container was worth 7%, and
a 575 W card on a narrow PCIe link performed within 1–3% of a 600 W one. The code is public:
jhsmith409/sglang-elsa.
Why it’s called Elsa
The grandkids were over, Frozen was on, and everyone was singing along. When I asked what to call the new server, the answer came back in unison: Elsa. The name has no technical meaning. After a day of watching numbers melt the old ones, it fits anyway.
The card and the model
The RTX 5090 is a consumer Blackwell card: SM120, the same architecture as the RTX PRO 6000,
but with 32 GB instead of 96. Memory bandwidth is about 1.8 TB/s. The model is
nvidia/Qwen3.6-35B-A3B-NVFP4: a 35B-parameter mixture-of-experts model with 3B active per
token, a hybrid of linear attention (Gated DeltaNet) and ten full-attention layers, plus a vision
tower and a built-in MTP head for speculative decoding. NVIDIA’s checkpoint stores the experts
in 4-bit NVFP4, the attention projections in FP8 and the MTP head in BF16. Loaded, it’s about
21 GB, which leaves roughly 11 GB for everything else, including a 256K-token context.
That last part is the whole story. On a 96 GB card you tune for speed. On a 32 GB card you tune for memory first, because every feature you turn on competes for the same few gigabytes.
Where we started
Production was a pinned vLLM build with FlashInfer’s b12x MoE kernels, 256K context, vision and prefix caching:
| vLLM (production) | Stock SGLang 0.5.22 | Stock SGLang + MTP | |
|---|---|---|---|
| decode, 1 request | 267 tok/s | 310 | 528 |
| decode, 8 requests (aggregate) | 1,145 | 1,201 | (1 at a time) |
| KV pool | — | 345K tokens | 181K tokens |
| requests that fit | 16 configured | 9 | 1 |
Stock SGLang didn’t even start on this card at first. Its automatic MoE backend choice picked a kernel built only for datacenter Blackwell (SM100), which crashed during CUDA-graph capture. Selecting the CUTLASS backend by hand fixed that. Turning on MTP nearly doubled single-request speed, but it ran one request at a time with a pool too small for a single 256K conversation. Speed wasn’t the hard part. Fitting it was.
Credit where it’s due: Penny Royal
Penny Royal is John Pezzulli’s open-source SGLang fork for Qwen3.8 on the RTX PRO 6000. My previous post covered running it in production. Penny Royal is the reason Elsa exists. It showed what a runtime tuned for exactly one model on one GPU family can gain, and it supplied most of the ideas I started from: SM120-specific small-batch kernels, FR-Spec draft vocabularies, launch settings chosen per card instead of per framework, and qualification habits that publish exactly what was measured. When I asked how Penny handles memory at startup, reading its code led straight to the biggest memory fix below. Elsa doesn’t share Penny’s code, because the model and card differ, but it shares its approach. Like Penny, most of the engineering was done by an AI: Claude Opus 5.5 wrote the patches, ran the benchmarks and the A/B tests, and I decided what to try and what to believe.
Buying back memory
Speculative decoding costs memory; most of the work went to buying it back. Elsa runs MTP with a larger pool than stock SGLang without it.
The draft was keeping its own copy of the vocabulary. The MTP draft shares the target model’s embedding and output head, but SGLang only connected them after it had measured free memory and sized the KV pool. The draft’s private copies, about 2 GB, were allocated, counted as used, and then freed too late to be used. Sharing them right after the draft loads took the pool from 181K to 244K tokens and from one request to three. It’s a small diff for the largest single gain in the project.
Mamba state. Each running request in a hybrid model needs linear-attention state as well as KV. Storing that state in BF16 instead of FP32 and using SGLang’s ReplaySSM verification instead of keeping a full state snapshot per draft token freed another gigabyte. Pinning the state slots explicitly stopped the automatic sizing from spending the savings on slots I didn’t need. Result: 303K tokens and eight concurrent requests.
The web server had a GPU it never used. The HTTP and tokenizer process held a 630 MB CUDA
context. FlashInfer and sgl_kernel asked for the GPU’s compute capability at import time, and
image decoding ran on the GPU. Elsa answers those questions from NVML, decodes images on the CPU,
and that process now never touches the GPU. That memory went to headroom rather than pool, which
mattered later: the first stress tests ran out of memory twice before it.
An 8-bit draft. The MTP head ships in BF16 because it isn’t part of the main quantization. Converting it to FP8 at load time changed nothing about acceptance, the share of drafted tokens the model keeps, and returned the pool to 358K tokens. The final pool fits a full 256K conversation plus about 100K tokens of other work, with MTP on.
Making decode faster
Cumulative changes, one request. The dips are steps that bought memory, not speed.
Skinny matrix multiplies. In decode, most matrix multiplies are skinny: four to sixteen rows of activations times a large weight matrix. For one 2048-to-64 projection in the linear-attention layers, cuBLAS picked tiles that left most of the GPU idle, taking 8.7 µs for 256 KB of weights. A split-K Triton kernel does it in 2.8 µs. The same kernel serves the shared experts, which the checkpoint stores as 4-bit weights used with 16-bit activations. Expanding those few layers to BF16 at load time is numerically identical and avoids a kernel that was spending its time on setup.
An FP8 fast path that never ran. SGLang already had a CUTLASS kernel for small-batch FP8 on SM12x cards. It never engaged for this checkpoint, because the weight scale was expanded per channel before the code checked for a single per-tensor scale. Fixing the order and enabling the kernel for this model’s shapes moved the attention projections from about 0.85 to 1.4 TB/s, with output bit-identical to cuBLAS.
FR-Spec, and a silent bug. The draft head scores all 248K vocabulary tokens every draft step. FR-Spec limits the draft to the most likely tokens; I built a 32K list from the model’s own output. Upstream’s version sliced rows out of the output head, but this checkpoint’s head is 4-bit and repacked for the Marlin kernel, so slicing it produces garbage. Nothing errors; the draft just guesses badly. Elsa builds a 4-bit hot-vocabulary head from the checkpoint instead: 37 MB per draft step instead of 290 MB. With drafting that cheap, five draft steps beat three.
One environment variable. FlashInfer’s fused MoE finalize step is off by default in SGLang. On here, it added 2%.
Those steps took single-request decode from 528 to 626 tok/s with MTP, and eight-request throughput to 1,956.
What didn’t work
Fusing the shared expert into the main MoE produced wrong output and was slower anyway. Triton decode attention avoided a per-step host synchronization in FlashInfer but slowed long contexts. SGLang’s adaptive draft length needed about 0.55 GB of CUDA graphs per draft length and ran out of memory. Splitting the decode graph into segments removed a launch gap in profiles and changed nothing in an A/B test. That one taught me to trust A/B tests over profiles: the profiler’s own overhead had created most of the gap it showed. Raising the static memory fraction from 0.88 to 0.90 passed benchmarks and failed a stress test with a 200K-token prompt.
The prefill that wasn’t slower
Early long-context tests made SGLang look slower than vLLM at reading prompts, by up to 20% at 30K tokens. Repeated runs with unique prompts showed it wasn’t: stock SGLang and vLLM tie at 11.0–11.2K tok/s at 120K tokens. My first measurement had been a cold server’s first request.
But Elsa was 8% slower than stock at long prompts. A profile of a 120K-token prefill explained it: 85% of the time went to the attention kernel, which ran 11 times per chunk instead of 10. The eleventh was the MTP draft layer, which attends over the whole prompt too. Giving the draft a 4K-token window only while it reads the prompt recovered the 8%, and drafting quality didn’t change, because the draft keeps full attention when it generates. The 225K-token time to first token went from 37.2 to 34.4 seconds, the same as stock.
| Prompt | Elsa prefill | Stock prefill | vLLM prefill | Elsa decode | vLLM decode |
|---|---|---|---|---|---|
| 30K | 25.8K tok/s | 26.4K | 24.0K | 381 | 252 |
| 120K | 11.1K | 11.2K | 11.0K | 297 | 223 |
| 225K | 6.53K | 6.56K | 6.65K | 256 | 196 |
CUDA versions matter, but only for some kernels
Production runs in Docker, like the vLLM container it replaces. The first Elsa image was 6% slower than the same code running natively. Most kernels can’t explain that, because PyTorch, cuBLAS and SGLang’s prebuilt kernels ship inside the Python wheels and are identical everywhere. But FlashInfer and SGLang also compile kernels at startup, using whatever CUDA compiler is installed, and the container’s was older than the host’s. Same code, three base images:
| Container base | 1 request (aggregate) | 4 requests |
|---|---|---|
| CUDA 13.0.2 | 520 tok/s | 1,300 |
| CUDA 13.3.1 | 559 | 1,329 |
| CUDA 13.4.2 | 542 | 1,310 |
13.3.1 won. Its compiler is the same CUDA 13.3 build Penny Royal is validated on. A practical detail: those compiled kernels are cached by FlashInfer version and GPU type, not by compiler version, so changing the base image needs a fresh cache. Otherwise the new compiler never runs.
Two 5090s: 600 W on PCIe 5.0 x16 vs 575 W on PCIe 4.0 x4
Elsa now runs on two RTX 5090s: one capped at 600 W in a PCIe 5.0 x16 slot, and one at 575 W in a PCIe 4.0 x4 slot, a quarter of the lanes at half the speed per lane. Same image, same tests:
| Elsa | 600 W, PCIe 5.0 x16 | 575 W, PCIe 4.0 x4 |
|---|---|---|
| 1 / 4 / 8 requests, thinking off | 626 / 1,343 / 1,956 | 630 / 1,320 / 1,955 |
| 1 / 4 / 8 requests, thinking on | 511 / 1,252 / 1,771 | 494 / 1,212 / 1,768 |
| prefill at 120K / 225K | 11.0K / 6.51K | 10.9K / 6.45K |
They’re within 1–3% of each other. Once the weights are loaded, inference barely uses the PCIe link, so a narrow slot costs almost nothing. The small differences that remain are in prefill and with thinking on, where the GPU does the most compute per token, which is consistent with the 4% lower power cap. The old vLLM numbers on the two cards differed by 4–14%. The 575 W card was measured through the gateway while serving other traffic, which probably explains most of that.
Same RTX 5090, model and context throughout. Stock MTP fits only one request at a time.
Production
Both cards now run the same Elsa image from Docker Compose, with the GPU pinned by UUID, a health
check and restart: unless-stopped, exactly like the vLLM container it replaced. The gateway
didn’t need a single routing change: same port, model name, context length and thinking-off
default, and reasoning text arrives in the same field. The old vLLM container is stopped, not
deleted, in case I need to roll back. Every acceptance test passed on both: vision, tool calls
(streaming and non-streaming, thinking on and off), a needle hidden in 225K-token prompts,
GSM8K at 94–97.5% (vLLM: 94.5%), and two stress loads with seven long sampled generations next to
a 200K-token prompt.
What I’d tell you before you start
- On a small card, tune for memory first. Speculative decoding is the biggest speed win, and it’s worthless if it limits you to one request.
- Check what stock picks. The default MoE backend crashed; the FP8 fast path silently never ran; FR-Spec silently guessed with garbage. None of them logged an error.
- Trust A/B tests over profiles. A profiler adds its own overhead to the gaps you’re chasing.
- Benchmark warm, with unique prompts. A cold first request and a prefix-cache hit are both wrong numbers.
- Pin the compiler, not just the image. Kernels compiled at startup make your CUDA toolkit part of your performance.
From 267 to 626 tok/s for one request, from one request to eight with MTP on, and a KV pool that grew while doing it, on cards I already owned. The approach is John Pezzulli’s and Penny’s; the code is at jhsmith409/sglang-elsa.
Methodology: decode figures are per-request medians of 8 requests × ~1,000 tokens (C=1) and aggregates over 16 or 32 requests (C=4, C=8), from unique prompts, thinking off and temperature 0 unless noted. “Thinking on” means temperature 0.7 with model-default top-p/top-k. Long-context rows are medians of three runs with distinct prompts so no prefix cache could help. All measurements are mine, on these two cards, on October 23, 2026.