Bayes' rule and the base-rate trap

A detector that is right 95% of the time can still be wrong about most of the cases it flags. Bayes' rule says exactly when and by how much, and it is the single most useful formula for reading any classifier, test, verifier or evaluation result.

Turning the conditional around

The product rule from the previous lesson can be written two ways: $P(H \cap E) = P(H)\,P(E \mid H) = P(E)\,P(H \mid E)$. Dividing by $P(E)$ gives Bayes' rule:

The denominator is the law of total probability. Each term has a name:

• $P(H)$, the prior: how plausible the hypothesis was before the evidence.
• $P(E \mid H)$, the likelihood: how expected the evidence is if the hypothesis is true.
• $P(H \mid E)$, the posterior: what you should believe after seeing it.

The result is named after Thomas Bayes, whose essay was published posthumously in 1763 after Richard Price edited it and communicated it to the Royal Society.^[1] Bayes treated a specific problem, inferring the unknown chance of an event from observed successes and failures. Laplace, independently and in more general form, set out the principle of reasoning from observed effects to probable causes in a 1774 memoir, and he did most to make it a working tool.^[2]

The rule is trivial to derive. Its practical difficulty is that people, including experts, routinely confuse $P(E \mid H)$ with $P(H \mid E)$.

Worked example: a good test for a rare condition

A screening test has 99% sensitivity ($P(+ \mid \text{disease}) = 0.99$) and 95% specificity ($P(- \mid \text{healthy}) = 0.95$, so the false-positive rate is 0.05). The condition has 1% prevalence. Someone tests positive. How likely are they to have it?

About one in six. The arithmetic is clearer as natural frequencies. Take 10,000 people:

There are 594 positives, of which 99 are real. The test is accurate; the false positives win because they are drawn from a pool 99 times larger. This is the base-rate trap: ignoring $P(H)$ and reading the test's accuracy as if it were the answer.

The trap is well documented. In 1978 Casscells, Schoenberger and Graboys asked 60 physicians, residents and students at Harvard-affiliated hospitals about a test for a disease with prevalence 1/1000 and a 5% false-positive rate.^[3] Assuming perfect sensitivity, the answer is $\frac{0.001}{0.001 + 0.05 \times 0.999} \approx 2\%$. According to a 2014 re-analysis, only 11 of the 60 answered correctly and the most common answer was 95%; a replication with 61 respondents at a Boston-area hospital found 14 correct, and 95% was again the most common answer.^[4]

The odds form: evidence multiplies

Bayes' rule is easiest to use in odds, $O(H) = P(H)/P(\neg H)$. Dividing Bayes' rule for $H$ by Bayes' rule for $\neg H$ cancels the awkward denominator:

In the screening example the prior odds are $1:99$ and the likelihood ratio of a positive result is $0.99/0.05 = 19.8$, so the posterior odds are $19.8/99 = 0.2$, that is $1:5$, a probability of $1/6$. In log-odds the update is additive: each independent piece of evidence adds its log likelihood ratio. A second independent positive test multiplies the odds by 19.8 again, giving $3.96:1$, a probability of about 0.80.

The odds form explains a number of results about selection with imperfect judges. In the lab's Thurstone bounds Best-of-N, the condition under which a shared exploit defeats plurality voting is that its conditional rate exceed $p/(1-p)$, the prior odds that a candidate is correct.^[5] When correct answers are rare, the bar a false answer must clear is correspondingly low.

Where the base rate bites in AI

Detectors and filters. Any system that flags a rare class (fraudulent transactions, harmful prompts, machine-generated text, a reward hack among thousands of samples) faces the same arithmetic. A false-positive rate that looks small on a balanced test set can dominate the flagged set in deployment, where the target class is rare. When you read a detector's reported accuracy, the first question is: at what prevalence?

Verifiers. In Best-of-N sampling a verifier scores many candidates and the highest score wins. If the base rate of correct candidates is low, even a verifier that rarely scores a wrong answer highly will, across enough candidates, be shown more high-scoring wrong answers than right ones. The test-time compute course develops this.

Calibration. Bayes' rule takes probabilities as inputs, so it is only as good as those numbers. A classifier's softmax output is a posterior only if the model is calibrated: among cases where it says 80%, about 80% should be right. Guo and colleagues found that modern deep networks were often poorly calibrated, more so than older, smaller networks, and that a single-parameter fix, temperature scaling, worked well on most datasets they tested.^[6] An uncalibrated score fed into Bayes' rule gives a precise-looking but wrong posterior.

Updating more than once, honestly

Bayes' rule can be applied sequentially: today's posterior becomes tomorrow's prior. The odds form makes this look like a sum of log likelihood ratios, but that is valid only if the pieces of evidence are conditionally independent given the hypothesis. Two tests that share a failure mode, or two language-model judges that read the same flawed context, are not two independent pieces of evidence. Counting them as independent double-counts. The lab's correlated critique note measured a same-context critic that missed all 12 planted errors that fresh critics caught; agreement between such a critic and the generator is weak evidence, however confident both are.^[7]

Key takeaways

• Bayes' rule, $P(H \mid E) \propto P(E \mid H)\,P(H)$, turns "how likely is this evidence under the hypothesis" into "how likely is the hypothesis given the evidence". The two are not the same.
• A test's accuracy says little about the meaning of a positive result until you know the base rate; rare targets make false positives dominate.
• In odds form, each independent piece of evidence multiplies the odds by its likelihood ratio. Natural-frequency tables make the arithmetic hard to get wrong.
• Model outputs are usable as posteriors only if calibrated, and repeated evidence adds up only if it is conditionally independent.

Sources