Fast LLM inference on consumer AMD GPUs.
We run large models on a single AMD Radeon AI PRO R9700 (RDNA4, 32 GB) and publish what it takes to make them fast: kernels, serving configs and ready-to-run quantizations — measured, not guessed.
A vLLM fork for RDNA4 (gfx1201). Hand-written W4A8 MXFP4 kernels on the fp8 matrix cores, fused norm/quant, tuned decode and drafter attention, DFlash2 speculative decoding and int4 heads — Qwen3.8-27B at ~130 tokens/s decode on one card, 262k context. One Docker image, every change gated on a benchmark and a quality check.
Quantizations built for hardware with native MXFP4 (RDNA4, AMD Instinct MI350), each with a model card that says what it costs in quality and speed — including when you are better off with another build.
Decode by step time and speculative acceptance (not single noisy runs), quality by deterministic prompt-logprob comparison plus GSM8K, all on the same card and power cap. Numbers and scripts live in the fork's docs.
Credit where it's due: this work stands on vLLM, StillDeadcode/vllm-radiance and ggz14's RDNA4 kernels. Model weights belong to their authors and keep their licenses.