← All GPUs

RTX 4090

24GB VRAM ·  Ada Lovelace (CUDA)

Quick answer: The fastest listed local LLM on RTX 4090 is llama3.2-3b at 210 tok/s decode, with 12 Q4_K_M models marked as fitting this hardware.
ℹ️ VRAM estimates assume GGUF format and 2048-token context. Running at 8k+ context adds 2–4GB. Tokens/sec = generation (decode) speed only.11 rows currently link to community benchmark sources.
ModelQuantVRAMTok/sDataFits
llama3.2-3bQ4_K_M2GB210Community benchmark✓
gemma3-4bQ4_K_M3GB190Community benchmark✓
qwen2.5-7bQ4_K_M4.7GB155Community benchmark✓
mistral-7bQ4_K_M4.7GB152Community benchmark✓
llama3.1-8bQ4_K_M5GB148Community benchmark✓
deepseek-r1-7bQ4_K_M4.7GB145estimate✓
gemma3-12bQ4_K_M7.5GB95Community benchmark✓
qwen2.5-14bQ4_K_M9GB92Community benchmark✓
phi-4-14bQ4_K_M9GB88Community benchmark✓
mistral-24bQ4_K_M14.4GB58Community benchmark✓
gemma3-27bQ4_K_M16.5GB48Community benchmark✓
qwen2.5-32bQ4_K_M19.5GB44Community benchmarkTight
llama3.1-70bQ4_K_M42GB—estimate✗ No
nemotron-51bQ4_K_M30.6GB—estimate✗ No
qwen2.5-72bQ4_K_M43.5GB—estimate✗ No
Improve this GPU page
Have a RTX 4090 benchmark run?

Add model, quantization, context length, and decode tok/s so this page can answer more long-tail local LLM searches accurately.

Contribute RTX 4090 data

You might also compare

gemma3-4b vs llama3.2-3b on RTX 4090
GPU-specific speed and VRAM fit
llama3.2-3b vs qwen2.5-7b on RTX 4090
GPU-specific speed and VRAM fit
llama3.1-8b vs mistral-7b
Quality and model-level tradeoffs
phi-4-14b vs qwen2.5-14b
Quality and model-level tradeoffs

RTX 4090 local LLM FAQ

What LLMs can run on RTX 4090?

12 Q4_K_M models in this dataset are marked as fitting on RTX 4090. Start with the table rows marked as fitting, then compare VRAM and decode tok/s.

What is the fastest local LLM on RTX 4090?

llama3.2-3b is the fastest listed model on RTX 4090 at 210 decode tokens per second.

How much VRAM does RTX 4090 have for local LLMs?

RTX 4090 has 24GB VRAM. Models near 80% or more of that capacity are marked as tight fits.

Can RTX 4090 run 70B local LLMs?

RTX 4090 is not marked as fitting the listed 70B-class Q4_K_M models in this dataset. Check the fit column before trying larger context windows.

What quantization should I use on RTX 4090?

Use Q4_K_M as the baseline for this site. It is the comparison format used across the current benchmark table and keeps VRAM requirements predictable.

Last updated: 2026-06-11 ·Source ·Improve this data