""" Verifier-bottleneck falsification machine (Focus Area 06 — the verifier ceiling) ================================================================================= Brief 06's deepest finding: under REPEATED SAMPLING, *coverage* (any-sample-correct) scales log-linearly across 4+ orders of magnitude and never plateaus, but real ACCURACY plateaus because the **verifier** that selects among samples is the true ceiling [Large Language Monkeys, arXiv:2407.21787]. Brief 06 leaves one thread explicitly OPEN (line 51 of docs/focus-areas/research/06-inference-test-time-compute.md): "How far can verifier quality be pushed before reward-hacking dominates; 'good enough' thresholds are task-specific." This is a SUBSTRATE-FREE question (no waves, no photonics) and the repo had nothing on it. This file is the falsification machine for it. It is a faithful Monte-Carlo model of the Best-of-N (BoN) / reward-model selection process used in repeated-sampling inference. No LLM is run; we model the *selection statistics*, which is exactly where the ceiling lives. ------------------------------------------------------------------------------------------ THE MODEL (one knob per real-world quantity) ------------------------------------------------------------------------------------------ A generator emits N candidate solutions for a problem. Each is independently CORRECT with probability p (= pass@1, the generator's per-sample competence). A verifier assigns each candidate a real-valued score; selection picks the argmax (BoN). The verifier is imperfect, and we decompose "imperfect" into TWO orthogonal axes: * d' (discrimination) : correct candidates score ~ N(d', 1); honest-wrong ~ N(0, 1). d' is the verifier's signal-detection separability. AUC = Phi(d'/sqrt2). d'=inf is a perfect verifier. * eps (exploitability) : each WRONG candidate is, with prob eps, an EXPLOITER — a Goodhart/blind-spot answer that the verifier scores HIGH: ~ N(d' + hack_margin, 1). With hack_margin>0 an exploiter beats a typical correct answer. eps models "the verifier has holes a search will eventually find" [LLMs Gaming Verifiers, arXiv:2604.15149]. Baselines AT EQUAL BUDGET (same N samples): * oracle : correct iff ANY sample is correct -> equals COVERAGE (the ceiling). * random-pick : pick a uniformly random candidate -> equals p (the no-search floor). * majority : plurality final-answer vote (uses NO verifier). Wrong answers land on a dominant distractor with prob q (mass distractor_mass) or scatter; this is the "everyone makes the same mistake" failure mode. The brutal baseline. ------------------------------------------------------------------------------------------ PRE-REGISTERED PREDICTIONS (chosen BEFORE running; kill criteria are numeric) ------------------------------------------------------------------------------------------ P1 (divergence — reproduce the brief). Coverage(N) is monotone increasing toward 1. verifier-BoN accuracy does NOT track it. KILL P1 if BoN accuracy stays within 2% of coverage across the whole sweep for an imperfect verifier (then there is no ceiling). P2 (HONEST noise does NOT reward-hack — the counterintuitive claim). For eps=0 (a merely NOISY but unexploitable verifier), BoN accuracy is monotone NON-DECREASING in N and asymptotes to a d'-limited ceiling below coverage. More sampling never hurts. KILL P2 if, for any tested d'>0 at eps=0, accuracy drops by >2% from its running max. P3 (EXPLOITABLE verifier DOES reward-hack). For eps>0 with hack_margin>0, BoN accuracy is NON-MONOTONE: it rises to a peak at a finite N* then DECLINES toward 0 as search finds the holes. KILL P3 if accuracy stays within 2% of its peak at N_max (no decline). P4 (the onset N* is set by exploitability, not discrimination). N* (samples you can safely spend) decreases as eps increases. Raising d' raises the PEAK height but does not rescue you from the eventual collapse. KILL P4 if N* fails to decrease with eps, or if raising d' removes the collapse at fixed eps>0. Headline if all hold: the dangerous axis of verifier quality is EXPLOITABILITY (eps), not inaccuracy (d'). An accurate-but-exploitable verifier is *worse under scale* than an inaccurate-but-honest one. "How far can you push N" = N*(eps), reported as a table. Run: ../.venv/bin/python verifier_bottleneck.py (numpy only; ~10-20s; seed=0) """ import numpy as np SEED = 0 N_GRID = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192] def auc(dprime): """Area under ROC for the noisy verifier: P(correct scores > honest-wrong scores).""" from math import erf, sqrt return 0.5 * (1 + erf(dprime / 2.0)) # Phi(d'/sqrt2) def bar(x, w=22): n = int(round(max(0.0, min(1.0, x)) * w)) return "#" * n + "-" * (w - n) def simulate(N, p, dprime, eps, hack_margin, distractor_mass, n_trials, rng): """One (config, N) cell. Returns dict of accuracies for each selection method.""" u_correct = rng.random((n_trials, N), dtype=np.float32) correct = u_correct < p wrong = ~correct exploiter = wrong & (rng.random((n_trials, N), dtype=np.float32) < eps) # verifier scores scores = rng.standard_normal((n_trials, N)).astype(np.float32) scores += correct * np.float32(dprime) scores += exploiter * np.float32(dprime + hack_margin) rows = np.arange(n_trials) sel = scores.argmax(axis=1) bon = correct[rows, sel].mean() coverage = correct.any(axis=1).mean() # == oracle # random-pick: expected accuracy is exactly p, but measure for honesty randpick = correct[rows, rng.integers(0, N, size=n_trials)].mean() # majority vote (verifier-free). count of correct bucket vs dominant-distractor bucket; # all other wrong answers are unique (count 1 each -> never win a plurality for N>=2). count_c = correct.sum(axis=1) distractor = wrong & (rng.random((n_trials, N), dtype=np.float32) < distractor_mass) count_d = distractor.sum(axis=1) # correct wins the plurality iff strictly more votes than the dominant distractor and >=1 maj = ((count_c > count_d) & (count_c >= 1)).mean() return { "coverage": float(coverage), "bon": float(bon), "random": float(randpick), "majority": float(maj), } def sweep(p, dprime, eps, hack_margin=1.0, distractor_mass=0.5, rng=None): """Sweep N for one config. Adaptive trials keep memory bounded at large N.""" if rng is None: rng = np.random.default_rng(SEED) out = {k: [] for k in ("N", "coverage", "bon", "random", "majority")} for N in N_GRID: n_trials = int(min(20000, max(3000, 3_000_000 // N))) r = simulate(N, p, dprime, eps, hack_margin, distractor_mass, n_trials, rng) out["N"].append(N) for k in ("coverage", "bon", "random", "majority"): out[k].append(r[k]) return {k: np.array(v) for k, v in out.items()} def peak_and_decline(bon): """Return (N*-index, peak, value_at_Nmax, max_drop_from_running_max).""" bon = np.asarray(bon) istar = int(np.argmax(bon)) running_max = np.maximum.accumulate(bon) max_drop = float(np.max(running_max - bon)) return istar, float(bon[istar]), float(bon[-1]), max_drop # ============================================================================== # def main(): rng = np.random.default_rng(SEED) print("=" * 86) print("VERIFIER-BOTTLENECK FALSIFICATION MACHINE (Focus Area 06, substrate-free)") print(" model: N samples, pass@1=p, verifier d'(discrimination) + eps(exploitability)") print(f" seed={SEED} N sweep: {N_GRID[0]}..{N_GRID[-1]} ({len(N_GRID)} points, geometric)") print("=" * 86) # ---------------------------------------------------------------------- # # P1 — coverage vs accuracy divergence (reproduce the brief's core claim) # ---------------------------------------------------------------------- # print("\n" + "-" * 86) print("P1 COVERAGE SCALES FOREVER, SELECTED-ACCURACY DOES NOT") print(f" config: p=0.10 d'=2.0 (AUC={auc(2.0):.3f}) eps=0.01 hack_margin=1.0") print("-" * 86) s = sweep(p=0.10, dprime=2.0, eps=0.01, rng=rng) print(f" {'N':>6} | {'coverage':>9} {'(oracle)':>8} | {'BoN-acc':>8} | {'majority':>8} | {'random':>7}") for i, N in enumerate(s["N"]): print(f" {N:>6} | {s['coverage'][i]:>9.3f} {bar(s['coverage'][i],8):>8} |" f" {s['bon'][i]:>8.3f} | {s['majority'][i]:>8.3f} | {s['random'][i]:>7.3f}") gap = s["coverage"] - s["bon"] p1_kill = bool(np.all(gap < 0.02)) print(f"\n coverage(N_max)={s['coverage'][-1]:.3f} BoN(N_max)={s['bon'][-1]:.3f}" f" -> gap={gap[-1]:.3f}") print(f" P1 {'FALSIFIED (no ceiling)' if p1_kill else 'HOLDS'}: " f"selected-accuracy is gated below coverage by the verifier" f" (max gap {gap.max():.3f}).") # ---------------------------------------------------------------------- # # P2 — honest (noisy, eps=0) verifier does NOT reward-hack: monotone # ---------------------------------------------------------------------- # print("\n" + "-" * 86) print("P2 AN HONEST-BUT-NOISY VERIFIER (eps=0) IS MONOTONE — MORE SAMPLING NEVER HURTS") print(" pre-registered: no drop >0.02 from running max, for any d'>0 at eps=0") print("-" * 86) p2_ok = True print(f" {'dprime':>7} {'AUC':>6} | {'BoN@N=1':>8} {'BoN@peak':>9} {'BoN@Nmax':>9}" f" {'maxdrop':>8} | verdict") for dp in (1.0, 2.0, 3.0): s = sweep(p=0.10, dprime=dp, eps=0.0, rng=rng) _, peak, atmax, drop = peak_and_decline(s["bon"]) ok = drop <= 0.02 p2_ok &= ok print(f" {dp:>7.1f} {auc(dp):>6.3f} | {s['bon'][0]:>8.3f} {peak:>9.3f} {atmax:>9.3f}" f" {drop:>8.3f} | {'monotone OK' if ok else 'NON-MONOTONE -> P2 KILLED'}") print(f"\n P2 {'HOLDS' if p2_ok else 'FALSIFIED'}: honest verifier noise does " f"{'NOT' if p2_ok else 'DOES'} induce reward-hacking; accuracy is " f"{'coverage-limited, not search-degraded' if p2_ok else 'degraded by search'}.") # ---------------------------------------------------------------------- # # P3 — exploitable verifier (eps>0) DOES reward-hack: peak then collapse # ---------------------------------------------------------------------- # print("\n" + "-" * 86) print("P3 AN EXPLOITABLE VERIFIER (eps>0, hack_margin=1.0) PEAKS THEN COLLAPSES") print(" config: p=0.10 d'=2.0 sweep eps") print("-" * 86) print(f" {'eps':>6} | {'N*':>6} {'peak':>6} | {'BoN@Nmax':>9} | {'decline':>8} | verdict") p3_ok = True ncrit = {} for eps in (1e-3, 1e-2, 5e-2): s = sweep(p=0.10, dprime=2.0, eps=eps, rng=rng) istar, peak, atmax, _ = peak_and_decline(s["bon"]) Nstar = int(s["N"][istar]) ncrit[eps] = Nstar decline = peak - atmax ok = decline > 0.02 p3_ok &= ok print(f" {eps:>6.3f} | {Nstar:>6} {peak:>6.3f} | {atmax:>9.3f} | {decline:>8.3f} |" f" {'collapses OK' if ok else 'no decline -> P3 KILLED'}") print(f"\n P3 {'HOLDS' if p3_ok else 'FALSIFIED'}: with exploitable holes, BoN accuracy " f"is non-monotone — past N* more compute LOWERS accuracy (reward hacking dominates).") # ---------------------------------------------------------------------- # # P4 — N* set by exploitability eps, not discrimination d' # ---------------------------------------------------------------------- # print("\n" + "-" * 86) print("P4 THE SAFE BUDGET N* IS SET BY EXPLOITABILITY (eps), NOT DISCRIMINATION (d')") print(" 'how far can you push the verifier' = N*(eps). p=0.10, hack_margin=1.0") print("-" * 86) print(f" {'eps \\ dprime':>14} | " + " ".join(f"{dp:>10.1f}" for dp in (1.0, 2.0, 3.0))) print(" " + "-" * 60) table = {} for eps in (1e-3, 1e-2, 5e-2): row = [] for dp in (1.0, 2.0, 3.0): s = sweep(p=0.10, dprime=dp, eps=eps, rng=rng) istar, peak, atmax, _ = peak_and_decline(s["bon"]) table[(eps, dp)] = (int(s["N"][istar]), peak, atmax) row.append(f"{int(s['N'][istar]):>4}({peak:>4.2f})") print(f" {eps:>14.3f} | " + " ".join(f"{c:>10}" for c in row)) print(" cells: N*(peak-accuracy). read DOWN a column: N* falls as eps rises (P4a).") print(" read ACROSS a row: peak rises with d' (ceiling),") print(" but N* / collapse persists (d' does not rescue, P4b).") # P4 checks: N* monotone non-increasing in eps (per d'); collapse persists across d'. p4a = all( table[(5e-2, dp)][0] <= table[(1e-2, dp)][0] <= table[(1e-3, dp)][0] for dp in (1.0, 2.0, 3.0) ) # d' raises peak at fixed eps? p4b = all( table[(eps, 3.0)][1] >= table[(eps, 1.0)][1] - 0.02 for eps in (1e-3, 1e-2, 5e-2) ) # but does NOT remove collapse (peak still strictly above value-at-Nmax for high d', high eps) p4c = (table[(5e-2, 3.0)][1] - table[(5e-2, 3.0)][2]) > 0.02 print(f"\n P4a (N* falls as eps rises, every column): {'HOLDS' if p4a else 'FALSIFIED'}") print(f" P4b (higher d' raises the peak): {'HOLDS' if p4b else 'FALSIFIED'}") print(f" P4c (high d' still collapses at eps=0.05): {'HOLDS' if p4c else 'FALSIFIED'}" f" -> discrimination buys ceiling height, not immunity.") # ---------------------------------------------------------------------- # # Equal-budget bake-off: when does the verifier earn its keep? # ---------------------------------------------------------------------- # print("\n" + "-" * 86) print("EQUAL-BUDGET BAKE-OFF — does BoN beat the verifier-free baselines? (p=0.10)") print(" winner among {BoN, majority, random} at a large budget N=1024") print("-" * 86) Nfix_idx = N_GRID.index(1024) print(f" {'dprime':>7} {'eps':>6} | {'BoN':>6} {'majority':>8} {'random':>7} | winner") for dp in (1.0, 3.0): for eps in (0.0, 1e-2, 5e-2): s = sweep(p=0.10, dprime=dp, eps=eps, rng=rng) b, m, r = s["bon"][Nfix_idx], s["majority"][Nfix_idx], s["random"][Nfix_idx] vals = {"BoN": b, "majority": m, "random": r} win = max(vals, key=vals.get) print(f" {dp:>7.1f} {eps:>6.3f} | {b:>6.3f} {m:>8.3f} {r:>7.3f} | {win}" f"{' <- verifier loses to consensus' if win != 'BoN' else ''}") # ---------------------------------------------------------------------- # print("\n" + "=" * 86) print("SUMMARY") print("=" * 86) print(f" P1 coverage>>selected-accuracy divergence : {'HOLDS' if not p1_kill else 'FALSIFIED'}") print(f" P2 honest noise (eps=0) is monotone : {'HOLDS' if p2_ok else 'FALSIFIED'}") print(f" P3 exploitable (eps>0) peaks then collapses : {'HOLDS' if p3_ok else 'FALSIFIED'}") print(f" P4 N* governed by eps, not d' : " f"{'HOLDS' if (p4a and p4b and p4c) else 'FALSIFIED'}") print(f"\n N*(eps) at d'=2.0, p=0.10 (how far you can push BoN before it hurts):") for eps in (1e-3, 1e-2, 5e-2): print(f" eps={eps:<6} safe up to ~N={ncrit[eps]:>5} samples") print("\n Reframing of brief 06's 'verifier quality is the ceiling':") print(" quality has TWO axes. DISCRIMINATION (d') sets the ceiling HEIGHT; only") print(" EXPLOITABILITY (eps) decides whether more sampling HELPS or HURTS. The") print(" reward-hacking danger is exploitability, not inaccuracy — an accurate but") print(" exploitable verifier is *worse under scale* than a noisy but honest one.") if __name__ == "__main__": main()