Evicting the Vision Encoder to the CPU (and Why We Shipped None of It)

Two 5070 Tis, one 35B model — Part 3 of 3: the CPU vision sidecar, the OOM that freed memory, and the production decision

TL;DR

Qwen3.6 is multimodal: it carries a BF16 vision transformer (ViT) that, under tensor parallelism, gets replicated on both 16 GB cards and OOMs image requests. Our fix was to evict the ViT from the GPU entirely: a CPU vision sidecar. A small FastAPI proxy intercepts image requests, runs the vision tower on the CPU with plain transformers, and forwards precomputed image_embeds to vLLM, which we tell to skip loading its own ViT via --enable-mm-embeds. It passed a 7/7 exact-parity check (including three identical misreads, proving the CPU tower is functionally the same model), and it did something nothing else could: it made NVFP4 run under tensor parallelism with working images for the first time. It also taught us a nasty lesson: freeing VRAM can crash a hybrid model, because the profiler hands the freed memory to the KV cache and starves the GDN activation scratch. And after all that, we chose not to ship it. Here’s why all of that is true at once.

The problem the ViT creates

Recap from Part 1: under tensor parallelism the vision tower is replicated per card, so an image request inflates memory symmetrically on both GPUs and OOMs them together. Under pipeline parallelism the ViT lives on rank 0 only, which is why PP can serve images and TP can’t. On 16 GB cards already 85%+ full of weights, the resident ViT is the margin between working and dead.

But notice what that implies: the ViT is in the way. It occupies VRAM on the critical card, it blocks TP, and, for a MoE language model, it’s a comparatively small, embarrassingly CPU-friendly piece of compute that runs once per image, not once per token. What if it just… weren’t on the GPU?

The sidecar architecture

vLLM has a feature built for exactly this: --enable-mm-embeds. It lets a client send precomputed multimodal embeddings instead of raw pixels. Pair it with --limit-mm-per-prompt '{"image":0,"video":0}' and vLLM stubs out its vision tower entirely: the ViT weights are never loaded. The GPU gets that memory back.

Something still has to turn pixels into embeddings. That’s the sidecar:

client ──image_url──▶  vision-sidecar (:8006)  ──image_embeds──▶  vLLM (:8005)
                        │  FastAPI proxy
                        │  transformers Qwen3_5MoeVisionModel
                        │  CPU, 16 threads, BF16
                        └─ rewrites the chat request in flight

The proxy is a drop-in shim in front of vLLM. It watches chat completions for image_url content parts; when it sees one, it downloads the image, runs the CPU vision tower, serializes the resulting embedding tensor, and rewrites that content part into an image_embeds part before passing the request along. Text-only requests pass straight through untouched. To the client it’s just an OpenAI-compatible endpoint; to vLLM it’s a text-only model receiving embeddings.

The sidecar loads only the vision weights: it reads the model’s safetensors index and pulls just the model.visual.* shards into a transformers Qwen3_5MoeVisionModel. One subtlety worth calling out: the visual weights live on different shards in different checkpoints (shard 9/10 in the AutoRound build, shard 1/3 in NVFP4), so the loader auto-detects them from the weight index rather than hardcoding a shard number. That one change is what let the same sidecar serve both builds.

The wire-format gotchas

Getting the embeddings into vLLM correctly took two specific fixes that are easy to get wrong and produce opaque errors.

The embedding payload. Each rewritten content part looks like:

{"type": "image_embeds",
 "image_embeds": {"image_embeds": "<b64 torch.save of [n_tokens, 2048] bf16>",
                  "image_grid_thw": "<b64 torch.save of the grid shape>"}}

The grid_thw shape trap. The grid-shape tensor must be a per-item 1-D tensor of shape [3], not [1, 3]. vLLM’s chat_utils._merge_embeds stacks the parts and adds the batch dimension itself; hand it [1, 3] and you get a 400 error (size_per_item should be a 1-D tensor) with nothing pointing you at the extra dimension. Squeeze it to [3] and it works. Hours can disappear into that one.

“Does the embedding ever go back to the GPU?”

A fair question, and the answer is yes, and it clarifies exactly what the sidecar does and doesn’t move off the GPU.

Only the pixel → embedding computation happens on the CPU. The resulting embedding tensor is sent to vLLM, and vLLM copies it onto the GPU (pe_tensor.to(self.device) in the model runner) so it can be spliced into the token stream and attended over by the language model, which is entirely on the GPU. The sidecar doesn’t move the language model’s work to the CPU, which would be catastrophic. It moves only the one-shot vision encode, whose output is a modest [n_tokens, 2048] tensor that rides back to the GPU cheaply. The heavy, per-token transformer stays where it belongs.

Proving it’s the same model

Offloading a model component to a different framework (transformers on CPU vs vLLM’s own ViT on GPU) invites the fear that you’ve subtly changed the model’s behavior. We ran a same-boot A/B parity probe: identical images through the native GPU ViT and through the CPU sidecar, comparing the model’s actual text output.

7/7 exact match. The most convincing evidence wasn’t the correct reads. It was the incorrect ones. On three hard 120-pixel words the model misread the text, and the CPU sidecar produced the exact same three misreads (PERFECT/TPR/CORE) as the native GPU tower. Two independent implementations agreeing on the right answer is reassuring; two implementations making the same mistakes is proof they’re computing the same function. Needle-in-a-haystack retrieval held at 8/8 up to 192K context with concurrent image traffic. The CPU tower is the GPU tower.

The cost: CPU encoding runs about 3.2 seconds per megapixel, capped at ~1.1 MP (MAX_PIXELS) to bound latency. Sixteen encoder threads was the sweet spot on this hybrid P/E-core CPU; 24 threads was slower, because the extra threads land on efficiency cores and drag the batch.

The OOM that freeing memory caused

This is where it got strange. We unloaded the ViT expecting free VRAM and a happier engine. Instead the first 32K-token prefill OOM’d in a GDN kernel (chunk_fwd_o) and killed the engine.

The cause is a subtlety of hybrid models plus how vLLM sizes its cache. When the ViT stopped loading, ~1.5 GiB freed up on the critical card. vLLM’s memory profiler saw that free space and grew the KV cache pool to absorb it — from 196,608 tokens all the way to 481,689. But the GDN linear-attention layers need a chunk of activation scratch at prefill time (chunk_gated_delta_rule / chunk_fwd_o) that the profiler’s steady-state snapshot doesn’t fully account for. The ballooned KV pool ate the headroom that scratch needed, and the first real long prefill ran the card out of memory.

The instinct, “just cap the KV cache with --kv-cache-memory-bytes,” makes it worse on this hybrid model. That flag budgets only the paged KV; the mamba/GDN state cache lands on top of the budget, and we watched a card drop to 21 MiB free and OOM during cudagraph warmup. The only knob that behaves correctly is --gpu-memory-utilization, because it caps total usage and leaves the GDN scratch its implicit slack. Dropping util from 0.92 to 0.85 fixed it: a 267,386-token pool (still 1.36× the original) with ~2.7 GiB genuinely free for GDN scratch.

The same trap, worse, hit NVFP4 under TP: util 0.92 with the ViT unloaded ballooned the pool to 699,347 tokens and left 31 MiB free → cudagraph-capture OOM. Util 0.85 fixed it there too, though the margin stayed thin (176 MiB free on a near-ceiling prefill), which is why 0.82–0.83 would be the safer production number for that build.

Lesson: on a GDN/mamba hybrid, freeing VRAM is not automatically safe. The profiler will spend it on KV cache and starve the linear-attention scratch. Util is the only headroom knob that respects the hybrid.

The payoff: NVFP4 under tensor parallelism

The best result the sidecar bought us: with the ViT off the GPU, we ran the NVFP4/b12x model under TP=2 with working images — something that had never worked before, because the replicated ViT always OOM’d it (Part 1). On the dev552 pin (Part 2), text stayed coherent under sharding, images worked, b12x stayed engaged, and the KV pool hit 489,739 tokens (2.72×) with prefill ~7,700 tok/s and warm decode ~166 tok/s. The vision-tower blocker that made TP a non-starter for the multimodal model was simply gone.

Why we shipped none of it

And then production went back to AutoRound INT4, PP=2, native in-vLLM ViT — the sidecar switched off. That’s not a walk-back; it’s the workload talking.

• The production client sends large images — ~16K vision tokens each. The CPU sidecar caps at ~1.1 MP to keep latency sane, which would downscale the client’s images; uncapped, a single large image takes ~50 s to encode on CPU. For this client, native GPU encoding is both faster and lossless.
• PP runs the ViT natively on rank 0 with no OOM anyway. The problem the sidecar solves — ViT replication — is a TP problem. Production is PP. There’s nothing to fix.
• The workload is prefill-heavy, and PP’s prefill is 1.9× TP’s (Part 1). The sidecar’s whole reason to exist is to unlock TP’s bigger KV cache, which this workload doesn’t need.
• AutoRound rides the latest nightly; NVFP4 is frozen on dev552 until the upstream sharding regression is fixed (Part 2). Staying on AutoRound keeps us current.

So the sidecar, the NVFP4-under-TP result, and the 256K-context experiments it enabled are all validated capabilities on the shelf, not the running config. They’re the answer if the workload changes — text-heavy, high-concurrency, very-long-context, or the day #47365 gets fixed. Building them taught us more about this model’s memory behavior than any amount of reading would have. But the production box runs the boring, proven thing: INT4, pipeline-parallel, vision tower on the GPU where the client’s big images want it, 196,608 tokens of context, both cards busy, clean boots.

The best engineering outcome of a three-week investigation was knowing precisely why to keep doing what we were already doing — and having a validated escape hatch built for the day that stops being true.

A closing note

I should say who actually wrote this thing. The series opened with four developers and a toddler all running out of tokens on the same afternoon. I wasn’t one of them; I still had room left on my Fable plan, the same shiny new model that had drained everyone else’s quota by lunchtime.

So Fable wrote the sidecar. It read through the vLLM internals, wrote the encoder, hit the [1,3]-vs-[3] wall and got past it, and worked out why unloading the ViT was blowing up the GDN memory profiler. I aimed it at the problem, argued with it, and made the calls about what to ship. But the code is the model’s.

Which is a little funny, given the point of the whole project. The reason you build your own inference box is so you’re not at the mercy of someone else’s quota, and the only reason this particular box got built is that, that afternoon, I hadn’t hit mine. The escape hatch got built with the exact thing everyone else had just run out of. My granddaughter, for the record, was still at Zelky’s.

This concludes the 3-part series. Part 1 covered pipeline vs tensor parallelism on no-P2P consumer Blackwell; Part 2 covered bisecting vLLM nightlies and the SM120 FP4 crash fix; Part 3 covered the CPU vision sidecar and the production decision.