A vLLM patch + hand‑written SM120 SASS kernels: 2‑bit MoE experts + an FP4 "delta" cache that recovers precision — matching the official (NV)FP4 checkpoint's quality on consumer Blackwell cards.

vLLM-Moet: frontier MoE models on consumer/workstation Blackwell (SM120)

vLLM-Moet serves frontier Mixture-of-Experts models on RTX PRO 6000 and RTX 5090 (sm_120), where the official vLLM images do not run them or the official checkpoints do not fit. Two lines of work:

Official checkpoints on official vLLM images: the sm_120 gaps closed and the decode step rebuilt around small tensor-core kernels. DeepSeek-V4.1-Flash, served this way on 4× RTX PRO 6000 (the active line), gets 1.5× the decode and 2.5× the GPU KV tokens of the official image with only the fixes it needs to start; Qwen3.8-Flash-Next-FP8 decodes 1.5× faster.
2-bit experts on vLLM v0.24.0: GLM-5.2 (753B), DeepSeek-V4-Flash (159B) and Kimi-K2.7-Code (1T) on one to four cards that cannot hold their checkpoints.

DeepSeek-V4.1-Flash on 4× RTX PRO 6000

The official checkpoint (deepseek-ai/DeepSeek-V4.1-Flash: 552B plus a 196B Engram table, MXFP8 dense, MXFP4 experts, 510 GB) on four RTX PRO 6000 (96 GB, PCIe): TP4, 512K context, DSpark speculative decoding (k = 5), vision, thinking, tool calling, KV offload to host RAM and disk. Served image: vllm-moet-sm120:dsv41-nightly-49cb2bbf-r1, i.e. vLLM nightly 49cb2bbf (FlashInfer 0.7.2, DeepGEMM a7a92773) plus the patches in tools/dsv41_sm120/.

	measured on the served image, 2026-10-16
decode, 1 stream	prose 190–224 tok/s, code 436–466 tok/s (88–90 steps/s)
decode, 8 streams	prose 596–615 tok/s, code 1,494–1,597 tok/s aggregate
fresh prefill	163K tokens in 8.5 s, 391K in 21.4 s
KV cache	3.69M tokens on the GPUs (7× concurrency at 512K), 76.7M more in 64 GiB of host RAM, plus a disk tier
quality	GSM8K-200 193/200, needle 6/6 to 367K tokens, tool calls / JSON / vision pass

What we changed against official vLLM

	official vLLM image on sm_120	vLLM-Moet image
indexer pages	DeepGEMM's SM120 paged MQA logits admit 32/64-row pages, V4.1's ratio-1 layers use 128: the indexer asserts	128-row pages admitted, bit-exact to 64-row pages
KV formats	fp8 compressed record (584 B/state) and FP8 indexer cache (132 B/key); the checkpoint's FP4 record and the MXFP4 indexer are SM100-only	FP4 record (288 B) and MXFP4 indexer (68 B): 2× the GPU KV tokens (1.49M → 3.06M on the same image)
KV offload	one host copy per TP rank, 3.6 KB per token	one copy, 896 B per token
dense decode GEMMs	CUTLASS's 128-row tile on a 6-row batch; `wo_a` as BF16 weights through cuBLAS bmm	tensor-core MXFP8 GEMV: 1.8–3.4× faster, `wo_a` 2.9×
MoE at decode	every routed expert padded to 128 rows (grouped GEMM 2× slower); 7 launches from router to FC1; FC2's padding rows written back to HBM	64-row tiles, one fused launch, the padding dropped from L2
mHC boundary	on the model stream (the side-stream version is SM100-only)	off the critical path: +3–5 % steps/s
launch gaps	the indexer's `weights_proj` on 2 cuBLAS CTAs (the main stream waits ~26 µs per layer); in graphs each NCCL kernel starts 2.8 µs after its predecessor	one launch with cuBLAS's bits; 0.4 µs; programmatic dependent launch in our kernels
prefill	the indexer's TP row sharding starts at 65,536 rows, which a 4096-token chunk never reaches; Engram's `wkv` all-gathers on every chunk (+1.8 ms per layer)	sharding from 2048 rows (−4 % at 391K); `wkv` split over the ranks at decode only
`reasoning_effort`	`minimal` / `medium` answer HTTP 400	accepted (25 / 62)

Net on the same four cards, against the recipe's official image with only the fixes it needed to start (2026-09-17): single-stream decode 60 → 88–90 steps/s (1.5×), GPU KV 1.49M → 3.69M tokens (2.5×), quality unchanged. Most of the decode gain and 2× of the KV come from the changes above; newer vLLM bases added the rest, and most of the prefill speed-up (163K tokens: ~15.5 → 8.5 s). The changes are bit-identical to the paths they replace, except the dense GEMVs (within one bf16 rounding of the fp32 reference) and the two KV formats, which passed a quality gate (GSM8K-200, needle retrieval, greedy agreement). Most have a runtime switch (VLLM_MOET_*=0, docs/sm120-deploy.md).

Build and serve

Needs 4× RTX PRO 6000 (an RTX 5090 does not fit the model at TP4), Docker with the NVIDIA container toolkit, GPU peer-to-peer over PCIe, and ~330 GB of host shared memory per server (the loader's segments plus the 64 GiB offload region), so one server per host.

git clone https://github.com/kacper-daftcode/vLLM-Moet && cd vLLM-Moet
hf download deepseek-ai/DeepSeek-V4.1-Flash --local-dir /srv/models/DeepSeek-V4.1-Flash
export IMAGE=vllm-moet-sm120:dsv41-nightly-49cb2bbf-r1 \
  MODEL_DIR=/srv/models/DeepSeek-V4.1-Flash CACHE_DIR=/srv/cache/ds41
DOCKER_BUILDKIT=1 docker build -f Dockerfile.sm120-dsv41-nightly -t $IMAGE .   # ~10 min

# first start, ~22 min: FlashInfer autotunes into CACHE_DIR (on an empty cache the sweep OOMs at 0.95)
GPU_MEM_UTIL=0.92 docker/sm120/run-dsv41.sh
# later starts, ~20 min, as served: 0.94, KV offload to RAM + disk, indexer sharding from 2048 rows
docker stop -t 60 ds41-flash
KV_OFFLOAD_GIB=64 KV_OFFLOAD_FS_DIR=/srv/kvcache KV_OFFLOAD_PROMPT_ONLY=0 INDEXER_TP_SHARD_MIN_ROWS=2048 \
  docker/sm120/run-dsv41.sh

The OpenAI-compatible API listens on 127.0.0.1:8001 (BIND= publishes it; put an authenticating proxy in front). Every knob is an environment variable documented in the header of docker/sm120/run-dsv41.sh. vLLM never deletes disk-tier files, so run docker/sm120/kvcache-ttl.sh from cron; docker/sm120/ds41-watchdog.sh restarts a crashed server. Validate with tools/sm120_perf/spec_matrix.py (88–90 steps/s single stream) and tools/sm120_perf/needle_any.py.

Known limits: with the disk tier, each request on the 49cb2bbf base pays ~20 ms more before its first token (cause open); TP8 with a 1M window on eight cards was validated at the bring-up, not on this image.

The port with every gap and measurement: docs/dsv41-sm120-port.md. Build, serve, validate and every image tag: docs/sm120-deploy.md. Op-level tests: tools/dsv41_sm120/README.md. Head-to-head with an SGLang recipe on the same host (2026-09-19): docs/dsv41-sm120-vs-0xsero.md.

Qwen3.8-Flash-Next-FP8 on 4× RTX PRO 6000

The official vllm/vllm-openai nightly ran this 177B-A6B model on sm_120 at 65 decode steps/s as first deployed. Dockerfile.sm120-qwen38 keeps the model's numerics and changes where the time goes: the 51 GB n-gram PLE table stays on the GPUs instead of the CPU offload worker (whose handoff stalled every decode graph by ~1.2 ms), vLLM's CuTe-DSL skinny GEMM (gated to B300 upstream) runs the ~480 BF16 decode GEMMs on sm_120 with plans tuned for it, and the routed experts get a tuned Triton config plus an FP8 MoE GEMV with silu·up and the quantization fused in, bit-identical to Triton. TP4, MTP k = 3: 98.5 steps/s, prose 243 / code 346 tok/s single stream, 1.5× the first deployment's 65 steps/s and 137–160 / 228 tok/s (2026-09-19). Launcher docker/sm120/run-qwen38.sh; write-up: tools/qwen38_sm120/README.md.

2-bit experts on vLLM v0.24.0

A generated patch on official vLLM v0.24.0 (patch/, image Dockerfile.sm120-v024) serves checkpoints the hardware cannot hold: GLM-5.2 (753B) on two RTX PRO 6000 at 28–32 tok/s (128K window) and on four at 105 tok/s, DeepSeek-V4-Flash (159B) on one RTX 5090 at ~31 tok/s (32K) and on one RTX PRO 6000 at 161 tok/s, Kimi-K2.7-Code (1T) on four RTX PRO 6000 at 51 tok/s (256K).

2-bit experts at FP4 quality. The routed experts go to a sign-symmetric 2-bit codebook on hand-written SASS kernels (naive 2-bit fails through sign asymmetry, not error size); a delta cache and a confidence gate bring FP4 back where it matters.
Experts beyond VRAM. The 2-bit base can live in host RAM or NVMe pack files with the GPU as an expert cache; a miss is fetched and the step's graph replayed, bit-identical to a resident forward.
Against official v0.24.0. The release cannot serve these models on SM120 as shipped: the patch moves the DeepGEMM and FlashInfer pins, gates the cluster-launch top-k off, fixes the o_proj scale layout and captures CUDA graphs in thread-local error mode, and adds MTP under pipeline parallelism and an NVFP4 KV cache. With the VLLM_MOE_W2* knobs off, the SM120 fixes are the only change.

Hooks, knobs and measurements: docs/v024-port.md; quality method: docs/quality.md; kernels: kernels/MANIFEST.md. The recipes image serves each benchmarked configuration (bench/README.md):

DOCKER_BUILDKIT=1 docker build -f Dockerfile.sm120-v024 -t vllm-moet-sm120:v024 .
DOCKER_BUILDKIT=1 docker build -f Dockerfile.recipes -t vllm-moet-recipes:v024 .
docker run --rm --gpus all --network host --ipc host --shm-size 64g -v /srv/models:/models \
  -e HF_TOKEN=... vllm-moet-recipes:v024 glm-5.2-nvfp4/pro6000x4-tp4-mtp   # --list: all recipes

Benchmark results

Release v2026.08.28-x4-prefill — one row per supported recipe (bench/recipes/), measured by bench/runner/bench.py; full report: docs/benchmarks/v2026.08.28-x4-prefill.md. Single-stream decode and prefill are medians; batch is aggregate tok/s at the noted concurrency.

model	hardware	config	ctx	decode tok/s	batch	prefill 8K	needle	notes
deepseek-v4-flash	1x RTX 5090 (32 GB)	host-resident 2-bit base, GPU as expert cache	8K	38	—	—	—	acc 2.83 †
deepseek-v4-flash	4x RTX 5090 TP4	consumer-card throughput	16K	214.4	1 560 @32	6 101	—	acc 2.6 †
deepseek-v4-flash	2x RTX PRO 6000 TP2	throughput	24K	209.6	380 @3	5 791	—	acc 2.6 †
glm-5.2-exl3	4x RTX PRO 6000 TP4	EXL3 trellis base (2,2,1) resident via EP, MTP k=2, decode-wave M8+M16 dual-set	8K	71.9	196.7 @32	2 106	PASS ≤6K tok	acc 2.62
glm-5.2-nvfp4	2x RTX PRO 6000 TP2	host-resident base, 44 GiB/rank expert cache	32K	33	—	—	PASS ≤27K tok	acc 3 †
glm-5.2-nvfp4	4x RTX PRO 6000 TP4	2-bit base + MTP k=2, 128K window	128K	105	—	2 500	PASS ≤276K tok	†
glm-5.2-nvfp4	4x RTX PRO 6000 TP4	+ FP4 delta (auto) + confidence gate tau=0.60	128K	84	—	—	—	†
kimi-k2.7-code-nvfp4	2x RTX PRO 6000 TP2	host-resident base, 52 GiB/rank cache (~39% coverage)	16K	14.4	—	—	PASS ≤8K tok	†
kimi-k2.7-code-nvfp4	4x RTX PRO 6000 TP4	GPU-resident 2-bit + FP4 delta, 256K window	256K	51	222 @8	2 448	PASS ≤248K tok	†
kimi-k2.7-code-nvfp4	4x RTX PRO 6000 TP4	+ Eagle3 drafter (k=3, drafter TP4)	256K	57 (±44%)	—	—	—	acc 3.5 †

† imported from pre-harness measurements (README/docs history) — re-measured on the next release.

Quality vs native

The row below is retained as explicitly imported historical evidence, not as a fresh release gate. A correctness audit found that the W2 path had omitted the checkpoint's SwiGLU clamp and that the attached THINK artifact did not encode the request/baseline metadata claimed by the compact result. It must be re-measured through the schema-v2 harness before being certified again.

Quality release v2026.07.30-quality — dataset evals vs the committed native baselines (bench/baselines/, same tool/checkpoint/hardware, stock serving path). Cells: accuracy (Δpp vs native, completion-token inflation vs native). Full report: docs/benchmarks/v2026.07.30-quality.md; process: bench/README.md.

model	hardware	config	GSM8K	GPQA	GPQA think	needle	notes
deepseek-v4-flash	2x RTX PRO 6000 TP2	max-quality candidate (revalidation required)	95.5% (-1.5pp, tok +1.3%)	73.2% (-1.0pp, tok +10.6%)	—	—	†

† imported from the measurement campaign logs — re-measured by the harness on the next quality release.

Kernels and tools

blackwell-isa (a machine-readable SM120 SASS ISA database, searchable reference) and cubit (an SM120 assembler / disassembler built on it) produce the 2-bit line's SASS kernels in kernels/: stock CUDA compiles the block-scaled MMA, but not the scheduling and register shaping these kernels need.
The DeepSeek-V4.1 decode kernels are CUDA C++ compiled into the image, each with op-level tests, in tools/dsv41_sm120/ (sm120_gemv/, moe_quant_scatter/, mhc_overlap/, bf16_gemm/, moe_l2/), Qwen3.8's MoE GEMV in tools/qwen38_sm120/moe_gemv/; the profiling and benchmark scripts behind the numbers are in tools/sm120_perf/.

Repository layout

path	what
`Dockerfile.sm120-dsv41-nightly`, `tools/dsv41_sm120/`, `docker/sm120/run-dsv41.sh`	DeepSeek-V4.1-Flash: the served image, its patchers, kernels and tests, the launcher (`docker/sm120/` also holds its cron scripts)
`Dockerfile.sm120-dsv41`	the same on the recipe's `deepseekv41-flash-0909` image (served until 2026-09-23)
`Dockerfile.sm120-qwen38`, `tools/qwen38_sm120/`, `docker/sm120/run-qwen38.sh`	Qwen3.8-Flash-Next-FP8
`tools/sm120_perf/`	profiling and benchmark scripts
`patch/`, `Dockerfile.sm120-v024`, `Dockerfile.recipes`, `docker/serve_recipe.py`	the 2-bit line: the generated v0.24.0 patch (never edited by hand, see AGENTS.md), its image, the recipes image
`bench/`	recipes, the runner and committed results; the tables above are rendered from it and checked in CI
`kernels/`, `tools/*.py`	SASS sources, prebuilt SM120 cubins and generators; the 2-bit line's tests
`docs/`	port notes, the deploy guide, the quality method, benchmark reports

License

Apache License 2.0 (LICENSE); third-party components in NOTICE.