Recipes/Nanbeige4.3-33B · 1× RTX PRO 6000 Max-Q
Recipe 001 · Dense 33B · One Max-Q card · 325 W
94 tokens a second from a dense 33B, on one 325-watt card.
Nanbeige4.3-33B reads all of its weights for every token it writes, so one card is limited by memory bandwidth. Smaller weights and free drafts take it from 42.9 to 94.1 tok/s on code and 92.6 on reasoning, one request at a time. Every number here is a Verified run on LocalMaxxing, and one command starts the same server.
Why it is fast
The card is reading weights, so make it read less.
A dense model touches every weight for every token. The time to stream them out of memory sets a ceiling on decode speed, and drafts are the only way past it.
Stock FP8 reaches 79% of its ceiling
The FP8 checkpoint is 35.5 GB. Leave out the 2.4 GB embedding table, of which each token reads one row, and 33.1 GB moves per token. At 1.79 TB/s that allows about 54 tok/s. Stock vLLM measures 42.9.
NVFP4 lifts the ceiling to about 91
primitive-ai's NVFP4 checkpoint stores the weights in 4 bits, so 19.7 GB moves per token. vLLM measures 67.7 with no drafts. The same checkpoint on SGLang 0.5.21 measures 49.5; we have not profiled why.
Drafts get more than one token per read
vLLM looks up the last few tokens in the prompt and the text so far, proposes 3 that might follow, and checks them all in one pass over the weights. A pass costs about what a plain step does, so every accepted draft is close to free.
DFlash2 is strongest on code
ProCreations' DFlash2 is a 0.98B, 3-layer drafter that proposes a whole block of tokens from the model's own hidden states. Code is predictable: 49.5 to 105.3. Reasoning text is less so: 49.8 to 71.5. That is why the default profile uses n-gram drafts.
Why not Nanbeige4.2-3B as the drafter? Its vocabulary is the same, but its embedding table is padded to 166,144 rows against the 33B's 166,912, so vLLM refuses the pair, and its architecture would run on the slow Transformers fallback.
Eight setups, one card
Pick a profile by what you serve.
./start.sh serves the default. Gains are confirmed against the same engine's stock run over interleaved fresh boots, with a 95% confidence interval.
| Setup | Start with | Code tok/s | Reasoning tok/s | Confirmed gain |
|---|---|---|---|---|
| NVFP4 + n-gram drafts vLLM, Triton attention |
./start.sh fast |
94.1 | 92.6 | x1.37 [1.36-1.40] |
| NVFP4 + DFlash2 SGLang, 6 draft tokens |
./start.sh code |
105.3 | 71.5 | x1.75 [1.73-1.76] |
| FP8 + DFlash2 SGLang, 8 draft tokens |
./start.sh fp8-dflash |
93.4 | 66.4 | x1.79 [1.75-1.82] |
| NVFP4 stock vLLM |
./start.sh stock-nvfp4 |
67.7 | 67.7 | baseline |
| FP8 + n-gram drafts vLLM, Triton attention |
./start.sh fp8 |
60.0 | 58.1 | x1.37 [1.36-1.39] |
| NVFP4 stock SGLang |
not a profile | 49.5 | 49.8 | baseline |
| FP8 stock SGLang |
not a profile | 43.5 | 43.6 | baseline |
| FP8 stock vLLM |
./start.sh stock-fp8 |
42.9 | 42.9 | baseline |
Swipe the table for more →
Run it
One command downloads, starts and checks it.
Clone and start
git clone https://github.com/puretensor/Nanbeige4.3-33B-1x-RTX-PRO-6000-MaxQ
cd Nanbeige4.3-33B-1x-RTX-PRO-6000-MaxQ
./start.sh fast --print # show the exact docker command, start nothing
./start.sh # 21 GB of NVFP4 weights, then vLLM, then a smoke testThe first start compiles kernels and CUDA graphs, which takes a few minutes. It ends by printing the server's address.
Talk to it
curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "nanbeige43-33b",
"messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],
"max_tokens": 1024,
"chat_template_kwargs": {"enable_thinking": false}
}'Any OpenAI client works with base_url = "http://127.0.0.1:8000/v1". Set GPU, PORT or MODELS_DIR to change the card, the port or where weights go.
Measure your card
python3 bench.py # LocalMaxxing's prompts, temperature 0, median of 5It prints decode speed, time to first token and, on vLLM, tokens per step and milliseconds per step, so you can see which of the two differs from ours.
One RTX PRO 6000
96 GB, with about 88 GB free. These numbers are for the 325 W Max-Q.
Linux and Docker
With the NVIDIA Container Toolkit. Tested on driver 615.71.10.
Disk
21 GB for NVFP4, 34 GB for FP8, 1.9 GB for the DFlash2 drafter.
The hf CLI
pip install -U huggingface_hub. Weights download at pinned commits.
Receipts
Sixteen runs, all Verified, all linked.
Each LocalMaxxing run page holds the prompt, the model's output, the engine's own timings and the exact launch command.
How each number was measured
LocalMaxxing's canonical code-v1 and reasoning-v1 prompts, one request at a time, temperature 0, thinking off, 1,024 output tokens, 2 warmups discarded and the median of 5 kept. Decode is (output tokens − 1) divided by the time from the first token to the last.
How each setup was chosen
Our tuning harness changes one setting at a time, gates every candidate on output quality and degeneration, and keeps a change only if interleaved fresh boots (stock, tuned, stock, tuned, stock, tuned) confirm a gain of at least 1%. The CPU ran in a performance window.
Reproduced from a fresh clone
On 2 November we cloned the recipe, let it download its own weights and ran bench.py on the same card with CPU boost off. The default profile measured 95.2 and 92.8 tok/s, at 1.36 tokens per step and 14.3 ms per step, within 1.2% of the Verified runs. The DFlash2 profile measured 99.5 and 69.0, a few percent lower: SGLang does more work on the CPU between steps, so it misses the boost that vLLM's CUDA graphs do not need.
Credits
Nanbeige for the model and its FP8 checkpoint, primitive-ai for NVFP4, ProCreations for DFlash2, vLLM, SGLang, and LocalMaxxing for the prompts and verification.
Licenses
The recipe's scripts are MIT. The weights are Apache-2.0 and download from Hugging Face; nothing is redistributed.