NeuroGenesysHow fast will it run?
Generating text is limited by memory bandwidth: every new token reads the whole model from memory. Reading the prompt is limited by compute. Pick a model and hardware to estimate tokens per second, time to first token, and how throughput grows when you serve several requests at once. Every efficiency assumption is editable.
Free · runs in your browser · updated
1 · Model and hardware
2 · Estimated speed
A roofline estimate, not a benchmark: real numbers depend on the runtime (llama.cpp, MLX, vLLM, TensorRT-LLM), kernels and drivers. It ignores speculative decoding and prefix caching, and assumes the model fits in memory — check that with the VRAM calculator.
Questions
How many tokens per second will I get?
For one request, roughly memory bandwidth × efficiency ÷ model size in bytes. Llama 3.1 8B at Q4_K_M is about 4.9 GB; an RTX 4090 reads about 1,000 GB/s, so at 70% efficiency that is about 140 tokens per second, or about 130 once a 4k-token KV cache is read too.
Why is a Mac with lots of memory slower than a GPU?
Unified memory lets a Mac hold large models, but its bandwidth (273 GB/s on an M4 Pro, 546 GB/s on an M4 Max) is lower than a high-end GPU's (1,008 GB/s on an RTX 4090, 3,350 GB/s on an H100). Speed per token follows bandwidth, so a 70B model fits on the Mac but runs at single-digit to low double-digit tokens per second.
What is time to first token?
The delay before output starts, mostly spent processing the prompt. That step does about 2 × parameters × prompt tokens operations, so it scales with compute (TFLOPS) and grows with prompt length.
Why does batching increase throughput?
One read of the weights can serve every request in the batch, so total tokens per second rises almost linearly at first. It levels off when the hardware runs out of compute, or when each request's KV-cache reads come to dominate at long contexts.
Why are mixture-of-experts models fast for their size?
They only read the experts each token is routed to. gpt-oss-120b has 117B parameters but uses about 5.1B per token, so on hardware that can hold it, it generates faster than a dense 14B model.
Related
- Free toolVRAM calculator
- Free toolLLM API cost calculator
- Free toolCan I run it? Memory guides for every open model
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
- Free toolAI Eval Sample Size CalculatorHow many eval items you need: the margin of error on one model's accuracy, and the sample size to detect a difference between two models, paired or not.
- Free toolLLM Training Compute CalculatorTraining FLOPs from parameters and tokens (6ND), GPU-hours at a realistic utilization, wall-clock time and cost — or the compute-optimal model for a budget.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolLLM Judge Agreement CalculatorCheck an LLM-as-judge against human labels: Cohen's kappa with a 95% interval, the confusion matrix, sensitivity and specificity, and how many human labels the judge is worth.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
