NeuroGenesysWhat LLMs can I run on a 8 GB GPU?
Last updated • By Will Schulz
On a 8 GB GPU you can run 6 of the 24 open-weight models we track with an 8K context. The largest is Gemma 3 12B at 3-bit (Q3_K_M), which needs 8.16 GB, and none of them fits unquantized at 16-bit.
Models that fit a 8 GB GPU
| Model | Best fit | Memory | Longest context | |
|---|---|---|---|---|
| Gemma 3 12B | 3-bit (Q3_K_M) | 8.16 GB | 8K | calculator |
| Gemma 2 9B | 3-bit (Q3_K_M) | 7.87 GB | 8K | calculator |
| Qwen3 8B | Q4_K_M | 7.37 GB | 8K | calculator |
| Llama 3.1 8B | Q5_K_M | 7.97 GB | 8K | calculator |
| Qwen2.5 7B | Q6_K | 7.92 GB | 8K | calculator |
| Mistral 7B v0.3 | Q5_K_M | 7.39 GB | 8K | calculator |
Memory is weights + FP16 KV cache + runtime overhead for one request, as in the VRAM calculator, assuming the runtime allocating 95% of the card. Longest context is the largest of 4K–128K that fits at that quantization; it is only usable up to the model's own context window.
Models too large for a 8 GB GPU
Even at 2-bit (Q2_K) these need more memory: Llama 3.1 70B, Llama 3.3 70B, Llama 3.1 405B, Llama 4 Scout, Llama 4 Maverick, Qwen2.5 14B, Qwen2.5 32B, Qwen2.5 72B, Qwen3 32B, Qwen3 30B-A3B, Qwen3 235B-A22B, Mixtral 8x7B, Gemma 2 27B, Gemma 3 27B, Phi-4 (14B), gpt-oss-20b, gpt-oss-120b, DeepSeek-V3 / R1.
Questions
What is the largest LLM that runs on a 8 GB GPU?
Of the models we track, Gemma 3 12B (12.2B parameters) at 3-bit (Q3_K_M), which needs 8.16 GB with an 8K context. Larger models need more memory or several cards.
Can a 8 GB GPU run a model at full 16-bit precision?
None of the models we track fits at 16-bit. Quantized to Q4_K_M (about 4.9 bits per weight) the smaller ones do, with little quality loss.
How much memory does a 8 GB GPU leave for context?
The KV cache grows with every token of context, so the longest context that fits depends on the model and quantization; the table lists it for each model, trying 4K, 8K, 16K, 32K, 64K and 128K tokens. These figures assume the runtime allocating 95% of the card.
