DeepSeek V4 Flash Runs on a Single AMD MI300X: 304B Parameters, No Quantization
On August 4, developer ryanzhou open-sourced a collection of configs and patches for running DeepSeek-V4-Flash-0731 on a single AMD MI300X in production. The HN post hit 365 points the same day. The 304B-parameter checkpoint runs with no quantization and no weight offload, fitting entirely into one card's 192GB of HBM.
Fitting 304B on one card
The MI300X has 192GB of HBM3 and 5.3TB/s of memory bandwidth, 2.4x the HBM capacity of an H100 SXM5, at roughly half the list price. Model weights take 156.67 GiB of HBM, leaving room for a 20GB GPU KV pool and a 96GiB CPU tier for evicted prefix-cache entries. One card handles 2-8 typical concurrent streams and bursts of up to 64 streams without OOM.
Measured performance
The numbers come from a pinned stack (vLLM ROCm nightly 0.26.1rc1.dev229 + AITER 0.1.19):
- Single-stream decode: 168.6 tok/s median
- 8 concurrent streams: 542 tok/s aggregate, 90.3 tok/s median per stream
- 64-stream burst: 830 tok/s aggregate, no OOM, no engine errors
- Prefill with tuned kernels: ~7.9-8.5K tok/s
- Context: 256K validated; the architecture supports 1M
What the patches fix
The official vLLM recipe targets NVIDIA and newer AMD hardware. Running reliably on MI300X required fixes for several issues: the FP8 format difference (MI300X's CDNA3 implements the AMD/Graphcore fnuz variant of E4M3, while MI325X and newer use OCP-standard FP8, so a kernel written for OCP semantics can be off by a factor of two in the scale domain), MoE routing at high concurrency, HIP-graph hazards in sparse MLA decode, CPU-KV synchronization, and several untuned kernel shapes. The repo collects these as reference diffs against upstream and pins the versions used in production.
Prior work came from Fergus Finn's MI300X worklog and the accompanying Doubleword repository (120 points on HN), which first identified the FP8 incompatibility and missing AITER fast paths.
Background
DeepSeek-V4-Flash-0731 is the Flash-series checkpoint DeepSeek updated on July 31, with roughly 430K downloads on Hugging Face. Running a 304B model on a single 192GB card matters because it pulls inference cost down from multi-GPU clusters to a single GPU, a practical option for self-hosted inference and edge deployments.