NeuroGenesysCan I run this model?
An LLM needs memory for three things: its weights, the KV cache that grows with every token of context, and some working space for the runtime. Pick a model, a quantization and a context length to see the total, and which GPUs and Apple Silicon machines can hold it.
Free · runs in your browser · updated
1 · Model and settings
2 · Memory needed
3 · What it fits on
GPUs (model split across cards)
- RTX 3090 24 GB4 cards
- RTX 4090 24 GB4 cards
- RTX 5090 32 GB2 cards
- RTX A6000 48 GBfits on 1
- RTX 6000 Ada 48 GBfits on 1
- L4 24 GB4 cards
- L40S 48 GBfits on 1
- A100 40GB 40 GB2 cards
- A100 80GB 80 GBfits on 1
- H100 PCIe 80 GBfits on 1
- H100 SXM 80 GBfits on 1
- H200 141 GBfits on 1
- B200 180 GBfits on 1
- AMD MI300X 192 GBfits on 1
Apple Silicon (one machine)
- Apple M2 Max64 GB config
- Apple M2 Ultra64 GB config
- Apple M3 Max (40-core GPU)64 GB config
- Apple M3 Ultra96 GB config
- Apple M4 Pro64 GB config
- Apple M4 Max (40-core GPU)64 GB config
Multi-card assumes the model is split evenly (tensor or pipeline parallel), with each card paying the fixed overhead. On a Mac the GPU can use about three-quarters of unified memory by default; you can raise the limit with sudo sysctl iogpu.wired_limit_mb, at the expense of the rest of the system. Vendor memory sizes are binary (a 24 GB card holds 25.8 × 10⁹ bytes), and that is accounted for.
Questions
How much VRAM do I need to run Llama 70B?
About 141 GB for the weights in BF16, so two 80 GB cards. At Q4_K_M (about 4.9 bits per weight) the weights take about 43 GB, which fits on one 48 GB card or a Mac with 64 GB of unified memory, with room for a modest context. At 128k context the BF16 KV cache adds about 43 GB more.
How do I work out an LLM's memory from its parameter count?
Weights take parameters × bits per weight ÷ 8 bytes: 8 billion parameters at 16 bits is 16 GB, at 4 bits about 4 GB. Add the KV cache (2 × layers × KV heads × head dimension × context × bytes per value) and a few percent for the runtime.
Why does context length use so much memory?
Every token in the context keeps a key and a value vector in every layer. Grouped-query attention, sliding-window layers and KV-cache quantization all shrink this, which is why two models of the same size can need very different memory at 128k tokens.
Do mixture-of-experts models need less memory?
No. A mixture-of-experts model runs only a few experts per token, which makes it fast, but every expert has to be in memory because any token can be routed to any of them. Memory follows the total parameter count, speed the active count.
How much of a Mac's unified memory can the GPU use?
By default macOS lets the GPU wire roughly three-quarters of unified memory. You can raise the limit with sudo sysctl iogpu.wired_limit_mb, at the cost of memory for the rest of the system.
Related
- Free toolLLM inference speed estimator
- Free toolInference economics (Academy lesson)
- Free toolCan I run it? Memory guides for every open model
- Free toolLLM API Cost CalculatorCompare what a workload costs per call, per day and per month across Claude, GPT and Gemini models, with prompt caching and long-context pricing.
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
- Free toolAI Eval Sample Size CalculatorHow many eval items you need: the margin of error on one model's accuracy, and the sample size to detect a difference between two models, paired or not.
- Free toolLLM Training Compute CalculatorTraining FLOPs from parameters and tokens (6ND), GPU-hours at a realistic utilization, wall-clock time and cost — or the compute-optimal model for a budget.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolLLM Judge Agreement CalculatorCheck an LLM-as-judge against human labels: Cohen's kappa with a 95% interval, the confusion matrix, sensitivity and specificity, and how many human labels the judge is worth.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
