AMD local inference has come a long way recently, thanks to projects like radiance. I’ve been successfully running Qwen3.8-Flash-Next (FP8, with 4-bit IQ4 experts) on four Radeon RX 9070 XTs and 256 GB of RAM for large NL-to-SQL workloads, at around 220 tokens/s for a single session and 500-600 with eight at once. For this task, model quality is critical. Inference engines like Strata fit big models on consumer hardware by quantizing them to 2-3 bits per weight and, for single requests, by default even skipping some of a token’s experts when they aren’t on the GPU; I wanted to see whether this machine could run a much bigger model as it was released: DeepSeek-V4.1-Flash, with its original MXFP4 experts.

At 416 GB it is far bigger than the GPUs’ 64 GB of VRAM, so most of it has to come from RAM or the SSD. With only a few requests, each expert copied to a GPU is used for just one or two tokens, and one request at a time gives 25-35 tokens/s. But its compressed KV cache needs only about 10 MB per GPU for a 1,000-token request, so hundreds of requests fit at once. The obvious experiment: keep raising the batch size until something breaks. With a few tweaks, it went further than expected: with all 500 questions of the BIRD text-to-SQL benchmark at once, the box generates 228 output tokens per second (about 360 at its peak, measured over 10 seconds) and finishes the job in 3.5 minutes. On BIRD’s 2026 revision, whose prompts ask for reasoning before the SQL, it generates 155 tokens/s for 39 minutes.

Original BIRD, at once Generating, tok/s Whole job, tok/s Wh per 1k tokens
1 34 14 12.8
8 59 33 5.0
64 79 46 3.5
256 164 108 1.72
500 228 151 1.29
BIRD 2026, at once Generating, tok/s Whole job, tok/s Wh per 1k tokens
1 25 21 8.6
8 37 35 4.1
64 53 50 2.9
256 101 94 1.83
500 155 137 1.34
1,534 (dev, continuous) 197 154 1.20

“Whole job” includes prompt processing. The last row: BIRD’s larger dev set in the same prompt format, fed through 512 slots (chart). Power (GPUs + CPU) ranges from 520 to 700 W across the batch sizes, so 500 at once needs six to ten times less energy per token than one at a time.

The answers hold up against the API deployments of the same model. BIRD, the text-to-SQL benchmark, has two versions of its 500-question mini-dev set: the original from 2024 and a 2026 revision with corrected reference answers and official prompts:

Who runs the model Original accuracy cost BIRD 2026 accuracy cost
This box, DeepSeek-V4.1-Flash 61.8% €0.012 50.6% €0.13
DeepSeek API 59.8% €0.031 50.8% €0.22
DeepInfra 60.2% €0.015 47.2% €0.12
Wafer 60.2% €0.021 49.0% €0.16
Other models
Qwen3.8-Flash-Next, this box 67.2% €0.005 32.4% €0.025
Claude Haiku 5.5 62.0% €0.10 53.4% €0.28
GPT-6 Luna 55.8% €0.063 43.0% €0.17

This box’s cost is the electricity of its GPUs and CPU (from their own sensors; RAM, board and power supply losses come on top) at an assumed €0.30/kWh. API costs were billed through OpenRouter on 16 November 2026, at $1.12 per euro. All runs with thinking off and temperature 0, except Haiku and Luna, which ran at their default sampling because they don’t accept temperature 0.

The two BIRD versions tell different stories about Qwen. On the original benchmark it clearly beats DeepSeek-V4.1-Flash, while on the revision it falls from 67.2% to 32.4%. Every other model loses about 9 to 13 points on the revision; Qwen loses 35. One explanation could be that some models know the original set too well; the revision’s new prompts and answers may also play a part.

Original BIRD: 500 questions in one batch500 requests (all sent at t=0), 31,643 output tokens, request phase 209 s ending 2026-11-16 23:10:17 UTC. Decode batch peak 500 at 70 s, time-weighted mean 229 while decoding, 80 tokens per decode step. Decode-step mean 224 tok/s; whole-run average 151 tok/s; . GPU power avg 528 W over the 4 GPUs; CPU avg 164 W.Original BIRD: 500 questions in one batchprocessing promptsgeneratingRequests in flightprompt processed, waiting0200400generatingwaiting for promptOutput tokens/s while generating15 s meanfaint: 5 s cellsreconstructed from per-request timings0250500300 tok/s224 tok/s average while generatingmeasured: 356 tok/sKV cache memory (% of the pool)0%25%50%75%100%Power (W)4 GPUs (avg 528 W)CPU (avg 164 W)10 s mean over 1 s samples0500050100150200seconds since the run's first request
All 500 questions of the original BIRD set in one batch. Its prompts ask only for the SQL, so answers are short and the whole job takes 3.5 minutes. Hover or tap for the values at any moment.
500 BIRD questions in one batch500 requests (all sent at t=0), 316,824 output tokens, request phase 2314 s ending 2026-11-16 22:23:10 UTC. Decode batch peak 500 at 264 s, time-weighted mean 226 while decoding, 155 tokens per decode step. Decode-step mean 154 tok/s; whole-run average 137 tok/s; . GPU power avg 494 W over the 4 GPUs; CPU avg 168 W.500 BIRD questions in one batchgeneratingRequests in flightprompt processed, waitingwaiting for prompt0200400generatingOutput tokens/s while generating15 s meanfaint: 5 s cellsreconstructed from per-request timings0200400300 tok/s154 tok/s average while generatingmeasured: 342 tok/sKV cache memory (% of the pool)0%25%50%75%100%Power (W)4 GPUs (avg 494 W)CPU (avg 168 W)10 s mean over 1 s samples05001,00005001,0001,5002,000seconds since the run's first request
All 500 BIRD 2026 questions in one batch, from the first request to the last reply. The official prompts ask the model to reason step by step before the SQL, so answers are long (about 630 tokens).
BIRD dev: 1,534 questions through 512 slots1534 requests (pool refill), 479,868 output tokens, request phase 3118 s ending 2026-11-17 01:51:41 UTC. Decode batch peak 512 at 123 s, time-weighted mean 344 while decoding, 187 tokens per decode step. Decode-step mean 197 tok/s; whole-run average 154 tok/s; . GPU power avg 500 W over the 4 GPUs; CPU avg 163 W.BIRD dev: 1,534 questions through 512 slotsgeneratinggeneratingRequests in flightprompt processed, waitingwaiting for prompt0200400generatingOutput tokens/s while generating15 s meanfaint: 5 s cellsreconstructed from per-request timings0200400300 tok/s197 tok/s average while generatingKV cache memory (% of the pool)0%25%50%75%100%Power (W)4 GPUs (avg 500 W)CPU (avg 163 W)10 s mean over 1 s samples05001,00005001,0001,5002,0002,5003,000seconds since the run's first request
BIRD’s dev set, 1,534 questions in the 2026 prompt format, through 512 slots refilled 16 at a time: the batch stays near 500 for half an hour. In the middle, generation pauses repeatedly while groups of new prompts are processed.
512 SQL questions in one batch512 requests (all sent at t=0), 45,510 output tokens, request phase 566 s ending 2026-11-16 09:14:50 UTC. Decode batch peak 512 at 398 s, time-weighted mean 283 while decoding, 358 tokens per decode step. Decode-step mean 269 tok/s; whole-run average 80 tok/s; . GPU power avg 799 W over the 4 GPUs; CPU avg 161 W.512 SQL questions in one batchprocessing promptsgeneratingRequests in flight0200400generatingprompt processed, waitingwaiting for promptOutput tokens/s while generating15 s meanfaint: 5 s cellsreconstructed from per-request timings0250500300 tok/s269 tok/s average while generatingKV cache memory (% of the pool)0%25%50%75%100%Power (W)4 GPUs (avg 799 W)CPU (avg 161 W)10 s mean over 1 s samples05001,0000100200300400500seconds since the run's first request
512 natural-language-to-SQL questions from a custom set (not public) with short answers, mostly just the SQL (about 90 tokens each), all sent at once, in the reproducible mode.

In the BIRD 2026 job, the machine only processes prompts (prefill) for the first four and a half minutes; once every prompt is in, all 500 requests generate together, at its fastest while all of them are running and slowing as replies finish. The replies vary a lot in length, so for most of the second half only a shrinking group of long answers is still generating. With a continuous queue the batch stays full (third chart): BIRD’s dev set, 1,534 questions (including 496 of the original mini-dev’s 500) in the same prompt format, fed through 512 slots and refilled 16 at a time, keeps about 500 requests running for 32 minutes at about 210 output tokens per second, new prompts included. Over the whole 52-minute job it averages 154, at 1.2 Wh per 1,000 tokens, the lowest of all runs.

The inference server

Price
GPUs 4x Radeon RX 9070 XT, 16 GB each (RDNA4), each on its own PCIe 4.0 x16 link ~€750 each (now €900-1,000)
Board ASRock Rack ROMED8-2T ~€750
CPU AMD EPYC 7443, 24 cores €320 used
RAM 256 GB DDR4-2666, 8 channels (121 GB/s measured from the CPU) ~€1,000 used
GPU links MCIO cables and adapters for the three GPUs not in a board slot ~€140 each
Storage Samsung 990 PRO 2 TB (the model file is 416 GB) ~€350
PSUs Corsair HX1500i and 2x RM1000x (more than this setup needs) ~€250 + €150 each
Total ~€6,400
Four 9070 XT GPUs in an open frame

The EPYC board with its 256 GB of RAM at the back, one 9070 XT directly in a board slot and three more in the open frame in front, and a lot of cables. The single Radeon R9700 was not used here.

The EPYC server’s eight memory channels and four (seven in total) x16 PCIe links matter a lot here: most of the model resides outside VRAM, so memory capacity and bandwidth are central to this setup.

Optimizations for large batches

The engine is adapted from radiance, an inference server for AMD GPUs, and baKon’s DeepSeek-V4.1 port for it. This fork builds on that work to improve large-batch performance on this setup.

Where the model lives: model file, RAM and VRAMModel file 388 GiB: experts 269, Engram 103, other 16. RAM 256 GiB: expert cache 204, the rest 52. VRAM 64 GiB on four GPUs: experts 23, KV cache 21, weights 11, arena, driver and headroom 9. Copies of the busiest experts sit on the GPUs and in the RAM cache; the rest stay on the SSD.Where the model livesArea is size. Copies of the busiest experts go onto the GPUs and into RAM; the rest stays on the SSD.ExpertsEngram tablesOther weightsKV cacheOS, buffers, driver, freeOther weights (attention, shared experts, embeddings): 16 GiBEngram lookup tables: 103 GiB in this file, read from the SSD as needed (the measured runs read the release's FP8 tables, 189 GiB, instead)Engram tables 103 GiBExperts: 269 GiB; copies of the busiest are also on the GPUs and in the RAM cache, the rest only on the SSDExperts 269 GiBExpert cache: 204 GiB, copies of the experts the job uses mostExpert cache 204 GiBThe rest: 52 GiB for the OS, the engine's buffers and the file cache for the Engram readsThe busiest experts: 23 GiB, about 330 a GPU23KV cache: 21.4 GiB (720,896 tokens shared by up to 512 requests)21Each GPU's share of the other weights: 10.5 GiB in allActivation arena, driver and headroom: 8.7 GiBModel file (SSD)388 GiBRAM256 GiBVRAM, 4 GPUs64 GiBRAM and VRAM as measured in the BIRD 2026 run, which read the FP8 Engram tables (189 GiB) from the SSD.
Hover or tap a block for its size.

Of the model file’s 416 GB, 289 GB are experts (each token uses only a few), 111 GB Engram lookup tables and 17 GB everything else; the measured runs read the release’s own FP8 Engram tables (203 GB) from the SSD instead. baKon’s port already ran it with the experts that don’t fit on the GPUs streamed from a small RAM pool and the SSD. To scale efficiently to batches of hundreds, a few changes to memory, scheduling and kernels were needed:

  • Cache experts in RAM. The experts take 269 GiB, more than the machine’s 256 GiB of RAM, which also has to hold everything else. So a cache of about 204 GiB keeps the experts the job uses, favoring the busiest ones in the usage profile (see Expert placement), and the rest are read from the SSD when needed.
  • Give the memory to the experts. The cache is sized from the RAM that is actually free, so memory reserved for other things and never used costs expert space. Capping the prefix cache’s snapshot store and sizing the KV cache to what the batch needs leave more of it to the experts, so fewer have to come from the SSD.
  • Batched expert passes. In steps with hundreds of tokens, each expert a layer needs is copied to the GPUs at most once (unless it is already there or the CPU computes it) and applied to all the tokens that use it, instead of handling experts one by one as they are streamed in.
  • Process prompts together. Requests that arrive together have their prompts processed in large shared chunks, and generation starts once the prompts are in, so each step generates a token for as many requests as possible. Prompts that share a long schema reuse the shared part through radiance’s prefix cache, which the fork made work with this batched processing.
  • Use the CPU alongside the GPUs. Each GPU has its own PCIe 4.0 x16 link, about 27 GB/s on its own; copying experts from RAM, all four together reach about 88 GB/s. The CPU cores alone read RAM at about 121 GB/s, and CPU and GPUs together reach about 160 GB/s, close to the 170 GB/s that eight channels of DDR4-2666 can deliver. So the GPUs alone leave RAM bandwidth unused. The CPU therefore computes some of the experts, during generation and prompt processing, reading them from RAM without crossing PCIe. Its code reproduces the GPUs’ arithmetic exactly, so in the reproducible mode the replies stay the same.
  • Faster GPU kernels. Register-tiled expert kernels for prompt processing, and larger tiles for the dense layers' matrix-unit kernel (adapted from radiance’s tiled GEMM), both with identical results.

Smaller changes helped too, such as starting to read the next layer’s lookup-table rows earlier and packing prompt processing more tightly.

Expert placement

Speed also depends a lot on the placement profile: counts of how often each expert is used. baKon’s port can record these counts and keep the busiest experts on the GPUs; the fork’s RAM cache also uses them, holding on to the busiest experts longer. Expert use is very uneven, which is what makes this work: averaged over the 40 layers, the busiest 9% of experts handle about two thirds of the expert work while generating, and 9% is about how many experts the GPUs have room to keep.

Expert use concentrationBusiest 9% of each layer's experts (about what the GPUs keep resident): prefill 61% of uses, decode 67% of uses.A few experts do most of the workshare of expert uses covered by each layer's busiest experts, 64 BIRD training questionsdecodeprefill← room on the GPUs (≈9%)the rest: RAM cache, then SSD0%25%50%75%100%0%25%50%75%100%share of each layer's 384 experts, busiest first61%67%
How often each expert was used on 64 public BIRD training questions, averaged over the 40 layers. Hover or tap to read off any share.

The same counts per layer, first in the order the experts are stored, then sorted from busiest to least used:

Animated heat map, 40 layers by 384 experts: cells start in stored order, scattered, then slide into each layer's sorted order: in every layer a dark band of a few dozen busy experts on the left, most experts barely used

Expert use while generating, on the same 64 BIRD questions. Darker cells carry more of their layer’s tokens.

The model file carries no profile. Without one, the experts kept on the GPUs are an arbitrary fixed set; the RAM cache adapts either way, filling with the experts the job uses while it runs. The same 256 questions of the original BIRD set (an earlier run) without a profile and with a recorded one:

Expert usage profile Job time
None 221 s
Recorded from 64 public BIRD training questions (different databases) 146 s

The profile is included (see Try it). It contains counts, not prompts. Recording a profile for your own workload takes one command; the time depends on the calibration set.

Energy and cost

Batch size decides the cost. On BIRD 2026, one request at a time costs about €2.58 of electricity per million output tokens at €0.30/kWh; at 500 at once it’s €0.40, and the whole job costs about the same as through the APIs:

Electricity price sliderAt €0.30/kWh the job's electricity costs €0.127; APIs: DeepSeek API €0.220, Wafer €0.160, DeepInfra €0.120. At €0.05/kWh (solar surplus) it is €0.0212.Cost of the 500-question jobdrag to set the electricity price€0solar surplustypical tariff€0.50electricity€0.30/kWhThis box (electricity)€0.127DeepSeek API€0.220Wafer€0.160DeepInfra€0.120Electricity only: 0.425 kWh measured (GPUs and CPU), no hardware. API costs as billed on 16 Nov 2026.
Drag the slider: the job’s electricity cost at your price, against the APIs’ bills for the same job.

At home the price per kWh can be lower than the tariff: surplus solar power in summer, and in winter the 600-700 W heat the room (though obviously only about 1/3-1/5 as efficient as a heat pump!). Still, the main reasons to run it locally are control over the data and knowing exactly which model runs and how it runs.

Try it

The BatchSeek repo contains the modified radiance engine, a batch client, placement profiles and the BIRD tools. The converted model and the release’s FP8 Engram shards are on Hugging Face.

./serve-batch.sh --model model.rad --silu-table silu_table_gfx1201.bin --streams 512 \
  --profile profiles/bird-train.profile.json \
  --engram-file engram-fp8/model-00047-of-00048.safetensors,engram-fp8/model-00048-of-00048.safetensors
python3 batch_client.py my-questions.jsonl answers.jsonl --concurrency 512

The server speaks the OpenAI API, so any client works as long as it sends many requests at once. batch_client.py is a small helper that does that for a JSONL file and keeps the answers in order.

The KV pool should be no larger than the batch needs: it takes VRAM from the experts on the GPUs. The repo’s README lists the settings of the runs in this post, for both BIRD versions and the dev set.

The default mode adds some numbers up in a different order on the GPU matrix units, so a reply can change slightly with the batch around it. In an earlier test it was 8% faster at similar accuracy. With --reproducible, rerunning the same questions gave byte-identical replies, and each optimization above was checked to leave the replies unchanged.

The measured runs used the release’s own FP8 Engram tables (--engram-file). model.rad also bundles IQ4_NL tables from ISTA-DASLab’s GGUF release, used when --engram-file is not given: the same speed and similar accuracy in the earlier BIRD runs, and 203 GB less to download.

To record a profile for your own prompts:

./calibrate.sh --model model.rad --silu-table silu_table_gfx1201.bin my-prompts.jsonl my.profile.json

Credits

  • radiance by Deadcode and contributors: the server, the HIP kernels and everything around them, including tcclaviger’s and keptn42’s all-reduce kernels and slydlake’s and ggz14’s MXFP4 matrix kernels. Full list in radiance’s NOTICE.
  • baKon: DeepSeek-V4.1 in radiance (pull request 266): the model across GPUs, the MXFP4 import, the Engram tables, and streaming experts from RAM and disk.
  • Related work applying similar ideas: large batches to amortize offloaded weights in FlexGen, MoE-Lightning and MoE-Gen, and experts computed on the CPU in Fiddler and KTransformers.
  • DeepSeek for the open weights, ISTA-DASLab for the IQ4_NL Engram tables in the converted model, and the BIRD team for their benchmark and databases.
  • Claude Opus 5.5, Claude Fable 5.2, GPT-6.1 Sol and GPT-6 Astra for help with optimizing inference.