What is the best GPU for running local LLMs?

For most people, a used NVIDIA RTX 3090 (24GB) is the best value — its 24GB of VRAM runs models up to ~34B, which no cheaper card can match. If you want a new card, the RTX 5090 (32GB) is the fastest consumer option, and the RTX 3060 (12GB) is the best budget pick.

How much VRAM do I need for local AI?

8GB is the practical minimum for the useful 7-8B models. 16GB comfortably runs 13-14B models, 24GB runs 30-34B models, and 48GB (two 24GB cards or a 48GB workstation card) is the entry point for 70B models.

Should I buy a used RTX 3090 for AI?

For local LLMs, yes — it's the standout value. You get 24GB of VRAM for far less than a new 24GB+ card, and for inference its older architecture barely matters. Buy from a reputable seller and test it on arrival.

Can I use an AMD GPU for local LLMs?

Yes, but expect more setup work. AMD's ROCm has improved and the 24GB Radeon RX 7900 XTX is good value, but CUDA-only tools and smoother support still make NVIDIA the safer choice for most users.

Do more CUDA cores make an LLM faster?

Up to a point. Once a model fits entirely in VRAM, token speed is driven mostly by memory bandwidth, not core count. If the model doesn't fit, no amount of compute saves you — it spills to system RAM and slows dramatically. Capacity first, then bandwidth.

Best GPU for Running Local LLMs in 2026: A VRAM-First Buying Guide

The best graphics card for local AI isn't the fastest — it's the one with enough VRAM to hold your model. Here's how to pick a GPU for local LLMs by budget, from a $250 used card to the RTX 5090.

The short version

Buy VRAM, not benchmarks. A graphics card’s gaming frame rate is almost irrelevant for AI. What matters is whether the model fits in the card’s memory.
Best value: a used RTX 3090 (24GB) — 24GB unlocks 30-34B models for the lowest cost per gigabyte.
Best budget card: a new RTX 3060 (12GB) — cheap, efficient, runs the everyday 7-8B models well.
Fastest consumer card: the RTX 5090 (32GB GDDR7).
Avoid: 8GB cards if you can help it, and high-end cards with low VRAM (a 16GB card beats a faster 12GB one for AI).

This guide is the GPU-specific companion to our broader best hardware for local LLMs guide, which also covers Macs and mini PCs.

Why VRAM is the only spec that really matters

A large language model is a big block of numbers (the weights) that has to live in memory while it runs. If the whole model fits in your GPU’s VRAM, it runs at full speed. If it doesn’t fit, the software offloads the overflow to system RAM — and performance falls off a cliff.

That’s why a modest card with lots of memory beats a powerful card with little: a 16GB RTX 4060 Ti will happily run a 13B model that an otherwise-faster 12GB card has to cripple. Once a model fits, the next thing that matters is memory bandwidth (how fast the GPU can stream those weights), which is why newer GDDR7 cards feel snappier. Core count and clock speeds come dead last.

See the model-to-memory table in our hardware guide for exactly how much VRAM each model size needs.

NVIDIA recommendations by budget

NVIDIA is the default because its CUDA platform is what nearly every local-AI tool targets. Picks by tier:

Budget — 12GB

RTX 3060 (12GB): the entry-level champion. Inexpensive, low power, and 12GB is enough for 7-8B models and lightly-quantized 13B models. The best “just try local AI” card.

Mid-range — 16GB

RTX 4060 Ti (16GB): low power draw and 16GB make it a great always-on inference card.
RTX 5070 Ti / 5080 (16GB): much faster bandwidth; choose these if you want speed and can live with a 16GB ceiling.

Sweet spot — 24GB

Used RTX 3090 (24GB): the value pick of the whole market. 24GB runs 30-34B models and is plenty for most home users.
RTX 4090 (24GB): same memory, far more speed — the best single-card experience short of the 5090.

Flagship — 32GB

RTX 5090 (32GB GDDR7): the most VRAM and bandwidth on a consumer card. Runs 34B models effortlessly and 70B models with aggressive quantization.

Going bigger

For 70B-class models, run two 24GB cards (dual RTX 3090 = 48GB; most tools split the model automatically) or step up to a unified-memory machine.

What about AMD and Intel?

AMD Radeon RX 7900 XTX (24GB): the best non-NVIDIA option — 24GB for a competitive price. AMD’s ROCm software has matured and tools like Ollama support it, but you’ll hit more rough edges than on CUDA.
Intel Arc (e.g. A770 16GB): cheap 16GB cards with improving support, best treated as a tinkerer’s option rather than a safe default.

If you value your time over saving a little money, buy NVIDIA. If you enjoy the tinkering, AMD’s 24GB card is a legitimate value play.

Common mistakes to avoid

Buying for gaming benchmarks. The “fastest” gaming GPU at a price is often not the best AI card if it has less VRAM than a cheaper rival.
Getting an 8GB card. It works for 3-7B models but you’ll outgrow it fast. 12GB is the sane floor; 16GB is comfortable.
Ignoring the used market. Inference doesn’t stress a card the way mining did, and the used RTX 3090 is the single best-value AI buy. Test on arrival.
Forgetting power and cooling. A 24GB card under sustained load needs a real PSU and airflow — budget for both.

For the full picture including Apple Silicon and 128GB AI desktops, read our best hardware to run LLMs locally, or start small with our Raspberry Pi local-LLM tutorial.