AMD local inference has come a long way recently, thanks to projects like radiance. I’ve been successfully running Qwen3.8-Flash-Next (FP8, with 4-bit IQ4 experts) on four Radeon RX 9070 XTs and 256 GB of RAM for large NL-to-SQL workloads, at around 220 tokens/s for a single session and 500-600 with eight at once. For this task, model quality is critical. Inference engines like Strata fit big models on consumer hardware by quantizing them to 2-3 bits per weight and, for single requests, by default even skipping some of a token’s experts when they aren’t on the GPU; I wanted to see whether this machine could run a much bigger model as it was released: DeepSeek-V4.1-Flash, with its original MXFP4 experts.
At 416 GB it is far bigger than the GPUs’ 64 GB of VRAM, so most of it has to come from RAM or the SSD. With only a few requests, each expert copied to a GPU is used for just one or two tokens, and one request at a time gives 25-35 tokens/s. But its compressed KV cache needs only about 10 MB per GPU for a 1,000-token request, so hundreds of requests fit at once. The obvious experiment: keep raising the batch size until something breaks. With a few tweaks, it went further than expected: with all 500 questions of the BIRD text-to-SQL benchmark at once, the box generates 228 output tokens per second (about 360 at its peak, measured over 10 seconds) and finishes the job in 3.5 minutes. On BIRD’s 2026 revision, whose prompts ask for reasoning before the SQL, it generates 155 tokens/s for 39 minutes.
| Original BIRD, at once | Generating, tok/s | Whole job, tok/s | Wh per 1k tokens |
|---|---|---|---|
| 1 | 34 | 14 | 12.8 |
| 8 | 59 | 33 | 5.0 |
| 64 | 79 | 46 | 3.5 |
| 256 | 164 | 108 | 1.72 |
| 500 | 228 | 151 | 1.29 |
| BIRD 2026, at once | Generating, tok/s | Whole job, tok/s | Wh per 1k tokens |
|---|---|---|---|
| 1 | 25 | 21 | 8.6 |
| 8 | 37 | 35 | 4.1 |
| 64 | 53 | 50 | 2.9 |
| 256 | 101 | 94 | 1.83 |
| 500 | 155 | 137 | 1.34 |
| 1,534 (dev, continuous) | 197 | 154 | 1.20 |
“Whole job” includes prompt processing. The last row: BIRD’s larger dev set in the same prompt format, fed through 512 slots (chart). Power (GPUs + CPU) ranges from 520 to 700 W across the batch sizes, so 500 at once needs six to ten times less energy per token than one at a time.
The answers hold up against the API deployments of the same model. BIRD, the text-to-SQL benchmark, has two versions of its 500-question mini-dev set: the original from 2024 and a 2026 revision with corrected reference answers and official prompts:
| Who runs the model | Original accuracy | cost | BIRD 2026 accuracy | cost |
|---|---|---|---|---|
| This box, DeepSeek-V4.1-Flash | 61.8% | €0.012 | 50.6% | €0.13 |
| DeepSeek API | 59.8% | €0.031 | 50.8% | €0.22 |
| DeepInfra | 60.2% | €0.015 | 47.2% | €0.12 |
| Wafer | 60.2% | €0.021 | 49.0% | €0.16 |
| Other models | ||||
| Qwen3.8-Flash-Next, this box | 67.2% | €0.005 | 32.4% | €0.025 |
| Claude Haiku 5.5 | 62.0% | €0.10 | 53.4% | €0.28 |
| GPT-6 Luna | 55.8% | €0.063 | 43.0% | €0.17 |
This box’s cost is the electricity of its GPUs and CPU (from their own sensors; RAM, board and power supply losses come on top) at an assumed €0.30/kWh. API costs were billed through OpenRouter on 16 November 2026, at $1.12 per euro. All runs with thinking off and temperature 0, except Haiku and Luna, which ran at their default sampling because they don’t accept temperature 0.
The two BIRD versions tell different stories about Qwen. On the original benchmark it clearly beats DeepSeek-V4.1-Flash, while on the revision it falls from 67.2% to 32.4%. Every other model loses about 9 to 13 points on the revision; Qwen loses 35. One explanation could be that some models know the original set too well; the revision’s new prompts and answers may also play a part.
In the BIRD 2026 job, the machine only processes prompts (prefill) for the first four and a half minutes; once every prompt is in, all 500 requests generate together, at its fastest while all of them are running and slowing as replies finish. The replies vary a lot in length, so for most of the second half only a shrinking group of long answers is still generating. With a continuous queue the batch stays full (third chart): BIRD’s dev set, 1,534 questions (including 496 of the original mini-dev’s 500) in the same prompt format, fed through 512 slots and refilled 16 at a time, keeps about 500 requests running for 32 minutes at about 210 output tokens per second, new prompts included. Over the whole 52-minute job it averages 154, at 1.2 Wh per 1,000 tokens, the lowest of all runs.
The inference server
| Price | ||
|---|---|---|
| GPUs | 4x Radeon RX 9070 XT, 16 GB each (RDNA4), each on its own PCIe 4.0 x16 link | ~€750 each (now €900-1,000) |
| Board | ASRock Rack ROMED8-2T | ~€750 |
| CPU | AMD EPYC 7443, 24 cores | €320 used |
| RAM | 256 GB DDR4-2666, 8 channels (121 GB/s measured from the CPU) | ~€1,000 used |
| GPU links | MCIO cables and adapters for the three GPUs not in a board slot | ~€140 each |
| Storage | Samsung 990 PRO 2 TB (the model file is 416 GB) | ~€350 |
| PSUs | Corsair HX1500i and 2x RM1000x (more than this setup needs) | ~€250 + €150 each |
| Total | ~€6,400 |
The EPYC board with its 256 GB of RAM at the back, one 9070 XT directly in a board slot and three more in the open frame in front, and a lot of cables. The single Radeon R9700 was not used here.
The EPYC server’s eight memory channels and four (seven in total) x16 PCIe links matter a lot here: most of the model resides outside VRAM, so memory capacity and bandwidth are central to this setup.
Optimizations for large batches
The engine is adapted from radiance, an inference server for AMD GPUs, and baKon’s DeepSeek-V4.1 port for it. This fork builds on that work to improve large-batch performance on this setup.
Of the model file’s 416 GB, 289 GB are experts (each token uses only a few), 111 GB Engram lookup tables and 17 GB everything else; the measured runs read the release’s own FP8 Engram tables (203 GB) from the SSD instead. baKon’s port already ran it with the experts that don’t fit on the GPUs streamed from a small RAM pool and the SSD. To scale efficiently to batches of hundreds, a few changes to memory, scheduling and kernels were needed:
- Cache experts in RAM. The experts take 269 GiB, more than the machine’s 256 GiB of RAM, which also has to hold everything else. So a cache of about 204 GiB keeps the experts the job uses, favoring the busiest ones in the usage profile (see Expert placement), and the rest are read from the SSD when needed.
- Give the memory to the experts. The cache is sized from the RAM that is actually free, so memory reserved for other things and never used costs expert space. Capping the prefix cache’s snapshot store and sizing the KV cache to what the batch needs leave more of it to the experts, so fewer have to come from the SSD.
- Batched expert passes. In steps with hundreds of tokens, each expert a layer needs is copied to the GPUs at most once (unless it is already there or the CPU computes it) and applied to all the tokens that use it, instead of handling experts one by one as they are streamed in.
- Process prompts together. Requests that arrive together have their prompts processed in large shared chunks, and generation starts once the prompts are in, so each step generates a token for as many requests as possible. Prompts that share a long schema reuse the shared part through radiance’s prefix cache, which the fork made work with this batched processing.
- Use the CPU alongside the GPUs. Each GPU has its own PCIe 4.0 x16 link, about 27 GB/s on its own; copying experts from RAM, all four together reach about 88 GB/s. The CPU cores alone read RAM at about 121 GB/s, and CPU and GPUs together reach about 160 GB/s, close to the 170 GB/s that eight channels of DDR4-2666 can deliver. So the GPUs alone leave RAM bandwidth unused. The CPU therefore computes some of the experts, during generation and prompt processing, reading them from RAM without crossing PCIe. Its code reproduces the GPUs’ arithmetic exactly, so in the reproducible mode the replies stay the same.
- Faster GPU kernels. Register-tiled expert kernels for prompt processing, and larger tiles for the dense layers' matrix-unit kernel (adapted from radiance’s tiled GEMM), both with identical results.
Smaller changes helped too, such as starting to read the next layer’s lookup-table rows earlier and packing prompt processing more tightly.
Expert placement
Speed also depends a lot on the placement profile: counts of how often each expert is used. baKon’s port can record these counts and keep the busiest experts on the GPUs; the fork’s RAM cache also uses them, holding on to the busiest experts longer. Expert use is very uneven, which is what makes this work: averaged over the 40 layers, the busiest 9% of experts handle about two thirds of the expert work while generating, and 9% is about how many experts the GPUs have room to keep.
The same counts per layer, first in the order the experts are stored, then sorted from busiest to least used:
Expert use while generating, on the same 64 BIRD questions. Darker cells carry more of their layer’s tokens.
The model file carries no profile. Without one, the experts kept on the GPUs are an arbitrary fixed set; the RAM cache adapts either way, filling with the experts the job uses while it runs. The same 256 questions of the original BIRD set (an earlier run) without a profile and with a recorded one:
| Expert usage profile | Job time |
|---|---|
| None | 221 s |
| Recorded from 64 public BIRD training questions (different databases) | 146 s |
The profile is included (see Try it). It contains counts, not prompts. Recording a profile for your own workload takes one command; the time depends on the calibration set.
Energy and cost
Batch size decides the cost. On BIRD 2026, one request at a time costs about €2.58 of electricity per million output tokens at €0.30/kWh; at 500 at once it’s €0.40, and the whole job costs about the same as through the APIs:
At home the price per kWh can be lower than the tariff: surplus solar power in summer, and in winter the 600-700 W heat the room (though obviously only about 1/3-1/5 as efficient as a heat pump!). Still, the main reasons to run it locally are control over the data and knowing exactly which model runs and how it runs.
Try it
The BatchSeek repo contains the modified radiance engine, a batch client, placement profiles and the BIRD tools. The converted model and the release’s FP8 Engram shards are on Hugging Face.
./serve-batch.sh --model model.rad --silu-table silu_table_gfx1201.bin --streams 512 \
--profile profiles/bird-train.profile.json \
--engram-file engram-fp8/model-00047-of-00048.safetensors,engram-fp8/model-00048-of-00048.safetensors
python3 batch_client.py my-questions.jsonl answers.jsonl --concurrency 512
The server speaks the OpenAI API, so any client works as long as it sends many requests at once. batch_client.py is
a small helper that does that for a JSONL file and keeps the answers in order.
The KV pool should be no larger than the batch needs: it takes VRAM from the experts on the GPUs. The repo’s README lists the settings of the runs in this post, for both BIRD versions and the dev set.
The default mode adds some numbers up in a different order on the GPU matrix units, so a reply can change slightly
with the batch around it. In an earlier test it was 8% faster at similar accuracy. With --reproducible, rerunning the same questions gave
byte-identical replies, and each optimization above was checked to leave the replies unchanged.
The measured runs used the release’s own FP8 Engram tables (--engram-file). model.rad also bundles IQ4_NL
tables from ISTA-DASLab’s GGUF release, used
when --engram-file is not given: the same speed and similar accuracy in the earlier BIRD runs, and 203 GB less to
download.
To record a profile for your own prompts:
./calibrate.sh --model model.rad --silu-table silu_table_gfx1201.bin my-prompts.jsonl my.profile.json
Credits
- radiance by Deadcode and contributors: the server, the HIP kernels and everything around them, including tcclaviger’s and keptn42’s all-reduce kernels and slydlake’s and ggz14’s MXFP4 matrix kernels. Full list in radiance’s NOTICE.
- baKon: DeepSeek-V4.1 in radiance (pull request 266): the model across GPUs, the MXFP4 import, the Engram tables, and streaming experts from RAM and disk.
- Related work applying similar ideas: large batches to amortize offloaded weights in FlexGen, MoE-Lightning and MoE-Gen, and experts computed on the CPU in Fiddler and KTransformers.
- DeepSeek for the open weights, ISTA-DASLab for the IQ4_NL Engram tables in the converted model, and the BIRD team for their benchmark and databases.
- Claude Opus 5.5, Claude Fable 5.2, GPT-6.1 Sol and GPT-6 Astra for help with optimizing inference.