Developers and researchers hunting for ways to run large language models on their own hardware once reached for NVIDIA cards without a second thought. CUDA support made the choice automatic. Yet that assumption no longer holds. A hands-on test published this week shows AMD hardware delivering competitive speeds on everyday models while costing far less.
MakeUseOf contributor Dipan Saha put an ASUS ROG Flow Z13 through its paces. The laptop packs an AMD Ryzen AI Max 390 processor and Radeon 8050S integrated graphics with 32 compute units. Running on Arch Linux with Ollama, Saha compared Vulkan and ROCm backends across several popular models including gemma2:9b, mistral:7b, phi4:14b, deepseek-r1:8b and 14b, plus llava variants.
Results surprised many. The AMD iGPU kept pace with a mobile RTX 4060 in token generation rates. On larger discrete cards such as the RX 9070 XT, inference hit 95 tokens per second on 7-billion-parameter models. Those numbers put the hardware in striking distance of far pricier NVIDIA options. Setup hurdles remain. ROCm still threw out-of-memory errors on some workloads. Vulkan proved more stable and delivered consistent output.
But the story runs deeper than one laptop review. Recent industry tests paint a similar picture of narrowing gaps. A detailed comparison published in March by Viperatech examined the RX 7900 XTX against current NVIDIA cards for local inference. The 24-gigabyte AMD board offered strong value, often matching or beating mid-range NVIDIA performance when price entered the equation. ViperaTech noted that NVIDIA still leads in absolute speed yet AMD wins on dollars per token in many consumer scenarios.
Power efficiency tells another part of the tale. YouTube creator Alex Ziskind ran identical systems side by side, one with an RTX 5090 and the other with an RX 7900 XTX. Both produced roughly the same output on the same models. The AMD card did so while drawing half the electricity. That difference matters for users running models around the clock or in home offices where electricity bills add up fast. Clips from the test show the NVIDIA machine finishing slightly quicker on some prompts. Yet the margin looks small once total cost of ownership comes into view.
Software progress explains much of the change. AMD rolled out ROCm 7 earlier this year with dramatic gains. Official benchmarks claim more than 3.5 times better inference performance compared with the previous generation. Support for lower-precision formats such as FP4 and FP6 helped squeeze extra speed from the same silicon. The stack now works more smoothly with frameworks including vLLM, SGLang and Ollama. And last week AMD announced expanded collaboration with Anthropic. The deal includes up to two gigawatts of MI450 GPUs plus a $5 billion equity investment. Joint engineering on ROCm and Claude models should accelerate fixes that matter to local users too.
Community benchmarks back the momentum. A Medium post from the Data Science Collective tested a 700-euro AMD card against an RTX 5090 on DeepSeek-R1 14B. The NVIDIA flagship reached 122 tokens per second. The AMD hardware managed 43 tokens per second. Thirty tokens per second feels responsive for most interactive work. When that speed arrives at half the price and lower power draw, many teams reconsider their default vendor. The same article pointed out that anything above 30 tokens per second counts as usable for daily tasks. Data Science Collective called the AMD option a clear win on affordability and availability.
Price gaps look even wider on the used market. Current listings show RX 7900 XTX boards trading between $450 and $550. Equivalent NVIDIA RTX 4090 cards still command $1,000 or more. Prompt Quorum ran the numbers and found AMD delivering 15 to 25 percent better compute per dollar across several models. The trade-off sits in software maturity. ONNX Runtime support on AMD still trails CUDA, and some quantization paths require extra tweaking. But ROCm 7.1 now recommends MIGraphX as the primary execution provider, closing one long-standing weakness. Prompt Quorum concluded that the RX 7900 XTX matches RTX 4090 speeds at roughly 60 percent of the price once setup friction is accepted.
Local AI enthusiasts on X have taken notice. Posts from the past 24 hours celebrate the latest ROCm Windows preview that brings day-zero support for consumer Radeon cards and a reported 4.6x inference boost. One account summed it up: “AMD just shattered NVIDIA’s desktop AI monopoly.” Others point to upcoming Ryzen AI MAX PRO 400 chips that will ship with improved ROCm integration. The chatter mixes excitement with lingering skepticism about driver stability. Yet the volume of positive test results keeps growing.
Enterprise interest follows the same trend. AMD’s partnership with Anthropic signals confidence that ROCm can scale to frontier workloads. If the stack improves enough to support Claude training and inference at cluster scale, consumer tools will inherit many of those gains. Developers already report smoother multi-GPU setups under ROCm 7 thanks to better communication libraries and distributed inference primitives co-developed with open-source teams.
Of course NVIDIA refuses to stand still. The company’s latest cards pack more VRAM and specialized Tensor Cores that still deliver top token rates on the largest models. A Dev.to benchmark from last year already showed RTX 4090 variants crossing 100 tokens per second on optimized setups. Yet those gains come at premium pricing that prices out many independent researchers, students, and small teams. When an AMD board delivers 70 tokens per second at half the cost, the conversation changes.
Hardware alone does not tell the full story. Quantization, model choice, and backend selection all influence real-world results. Saha’s tests used standard Ollama images and the llm-benchmark suite to keep conditions fair. Other testers experiment with exllama, llama.cpp, or custom kernels. Outcomes vary. But the pattern repeats: AMD cards that once lagged far behind now sit close enough that total ownership cost tips the scale.
Power consumption deserves extra attention. Data centers and home labs alike face rising electricity prices. A card that matches performance while sipping half the watts suddenly looks strategic. LocalArch.ai highlighted this advantage in a November analysis, showing AMD options delivering strong tokens-per-watt numbers across inference tasks. The gap narrows further when users factor in cooling requirements and noise levels inside typical PC cases.
Challenges persist. AMD’s software stack still demands more manual configuration than CUDA on many Linux distributions. Windows support, while improved, remains newer. Some models with exotic architectures run better on NVIDIA today. And driver updates occasionally break previously working setups. Those realities explain why many organizations keep NVIDIA as the safe default. But safe no longer means only.
Look at the broader market. Used AMD Instinct accelerators have become popular among budget-conscious experimenters because of their high VRAM. Community forums buzz with guides for MI50 and MI210 cards paired with ROCm. Consumer Radeon boards benefit from the same software investments. The result is a widening pool of affordable hardware that can run 7B, 13B, and even some 30B-class models at interactive speeds.
Future updates look promising. AMD promises continued ROCm refinements focused on usability and broader framework compatibility. The Anthropic collaboration should produce concrete improvements in kernel performance and model support. At the same time, open-source projects keep adding AMD paths to popular tools. The flywheel is turning.
So what should teams do today? Start with clear requirements. If absolute maximum speed on the largest models matters most and budget is unlimited, NVIDIA cards remain the straightforward pick. For everyone else, AMD hardware now merits serious evaluation. Run the same benchmarks Saha and others published. Measure power draw. Calculate total cost. The numbers often surprise.
The local AI field no longer belongs to a single vendor. Choice brings experimentation, lower prices, and faster iteration. And that benefits every developer who wants models running on their own machines without cloud latency or surprise bills. The data keeps coming in. AMD has earned its place at the table.
WebProNews is an iEntry Publication