NeuroGenesysWhat does it cost to train this model?
The compute to train a dense transformer is close to six FLOPs per parameter per training token. Divide by what your GPUs actually sustain and you have GPU-hours, days and dollars. Or start from a budget and get the model size and token count that make best use of it under the Chinchilla rule.
Free · runs in your browser · updated
1 · The run
2 · Estimate
Compute only, for one dense transformer run: C ≈ 6 × parameters × tokens. Excludes failed runs, experiments, evaluation, data and staff — usually several times the final run. Mixture-of-experts models use active parameters. GPU prices are indicative medians (source); specialist clouds and reserved capacity are cheaper.
Questions
Why 6 × parameters × tokens?
Each token costs about 2 FLOPs per parameter in the forward pass and about 4 in the backward pass. It ignores attention over the context, which adds little at typical sequence lengths, and the overhead of activation recomputation, which you can fold into utilization.
What is MFU?
Model FLOPs utilization: the share of the GPUs' peak that goes into useful model arithmetic. Well-tuned large runs reach roughly 35–50%; small or poorly tuned ones are lower. It's the biggest uncertainty in the estimate.
What does compute-optimal mean?
For a fixed compute budget, the model size and token count that reach the lowest loss. The Chinchilla study found about 20 training tokens per parameter. Many current models train on far more tokens than that, accepting more training compute for a smaller model that is cheaper to serve.
Is the compute cost the whole cost of a model?
No. It covers one final run. Experiments, failed runs, evaluation, data acquisition, staff and inference after launch usually cost several times more.
Related
- Researchphotonics-budget (research tool)
- Free toolLLM API Cost CalculatorCompare what a workload costs per call, per day and per month across Claude, GPT and Gemini models, with prompt caching and long-context pricing.
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
- Free toolAI Eval Sample Size CalculatorHow many eval items you need: the margin of error on one model's accuracy, and the sample size to detect a difference between two models, paired or not.
- Free toolVRAM Calculator: Can I Run This LLM?GPU memory for an LLM at any quantization and context length — weights, KV cache and overhead — and which GPUs and Apple Silicon Macs can hold it.
- Free toolLLM Inference Speed Estimator (Tokens per Second)Tokens per second and time to first token for a model on a GPU or Mac, from memory bandwidth and compute, and how batching changes throughput.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolLLM Judge Agreement CalculatorCheck an LLM-as-judge against human labels: Cohen's kappa with a 95% interval, the confusion matrix, sensitivity and specificity, and how many human labels the judge is worth.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
