Experiment · Local inference

A 188 GB model, a 72 GB GPU and a very fast SSD

8 November 2026 · By Jan Janssens · Read time: 5 min · ← All experiments

Qwen3.8-Flash-Next at Q8_0 is 188 gigabytes. Our machine has 72 GB of GPU memory and 96 GB of RAM, so a third of the model simply lives on the SSD and gets read while the model is talking. We ran Strata's new standard benchmark on exactly that setup: 66 to 71 generated tokens per second with thinking on, 2,400 tokens per second prefill at 16K, sanity checks 12 of 12. The run is now a pull request upstream and, as far as we can tell, the first RTX PRO 5000 72GB Blackwell entry in Strata's community benchmark list.

The result

The machine

• GPU. NVIDIA RTX PRO 5000 72GB Blackwell, driver 595.99.02, CUDA 13.2.
• CPU and RAM. AMD Ryzen 7 9800X3D, 8 cores; 96 GB of RAM.
• Storage. Samsung 9100 PRO 4TB NVMe, which is where the model files live.
• Engine. Strata terminal 0.2.12, engine 0.1.58 release build, commit 9780a8c2.
• Model. unsloth/Qwen3.8-Flash-Next-GGUF, Q8_0, six GGUF shards, 188 GB. Served with 2 slots, a 262,144-token context, int8 key-value cache and the model's MTP drafts enabled (--spec 4).

The trick: 188 GB does not fit anywhere

Flash-Next is a mixture-of-experts model, and Strata splits it the way these setups usually are: the shared layers go to the GPU, the experts stay in system RAM, and the engine pulls the expert weights it needs per token. That works beautifully when the experts fit in RAM. Ours do not: Q8_0 has about 119 GiB of expert weights and the machine has 89.7 GiB of RAM total.

So we told the engine to keep 66 GiB of experts resident and read the remaining ~53 GiB from the NVMe drive while it answers. Every number in this post includes that SSD traffic. This is not an all-in-RAM configuration, and the honest comparison is with other low-RAM streaming builds, not with the RTX 5090 reports where everything fits. The Samsung 9100 PRO is doing a lot of quiet work here.

The standard numbers

Strata recently defined a formal benchmark protocol (strata-qwen38-thinking-v1-draft.1, still a draft): fixed prompts, thinking mode on, temperature 1.0, a fresh engine started by the runner, server on loopback, hashes of every model file recorded, and a client-side stopwatch. We ran it with the runner so the folder is machine-checkable. Status: complete, 21 of 21 required requests, nothing skipped, 2 minutes 33 seconds of measured time after the model loaded.

Decode sits around 70 tokens per second with reasoning enabled, which for a thinking model you talk to interactively is comfortable: text appears faster than you read it. The first-token times are where the streaming shows, and they grow with the prompt: 3.4 s at 4K, 6.9 s at 16K.

One detail we liked: prefill speed rises with prompt size, from 1,163 tok/s at 4K to 2,297 at 16K and 2,453 at 32K in the quick suite. Fixed costs (waking up, reading the first expert pages) spread over more tokens, and bigger batches read the SSD more sequentially. Small prompts are the worst case for a streamed model.

The quick suite, with thinking off

We also ran the older community script (tools/strata_bench.py --long) against the same configuration freshly loaded: short chat decode 54.8 tok/s median, 4K / 16K / 32K fresh prompts at 1,163 / 2,297 / 2,453 tok/s, and an agent session that replays a growing conversation. There the prompt cache did its thing: 69.8 percent of tokens reused across turns, and decode medians up to 97 tok/s as MTP drafts hit well on the shorter fresh suffix.

Correctness, because speed without it is trivia

The standard suite embeds twelve sanity requests: four instruction-following, four retrieval, four tool-call checks. All twelve passed. A separate needle-in-a-haystack run found the needle 6 times out of 6 at 32K and 128K prompt tokens, at depths of 10, 50 and 90 percent. The Q8_0 quantisation is behaving itself.

Concurrency, with the honest footnote

The optional concurrency stage releases four requests at once. Our server has two slots, so two requests decode and two wait their turn; the rounds landed at 63 to 74 tokens per second in aggregate wall time. That is a throughput number for a queueing two-slot server, not four-way parallel decoding, and we said so in the submission.

Upstream

Both result folders, the machine-checkable standard run and the quick community report, are in pull request #5568 on the Strata repository, results only, no engine changes. The protocol is still a draft, so this is explicitly pilot data, and the caveats (low-RAM streaming, two-slot queueing, revision provenance) are written into the folders themselves rather than left to memory. If you have the same GPU, the profile and the run script are in there to compare against.