NeuroGenesysWas the threshold measurable at all?
Pre-registration stops goalposts moving, but it can't stop a goalpost being placed where no result could reach it. Before registering a threshold, compute three things: the standard error of the exact statistic under the null, the ceiling of the instrument on ideal data, and — for range or band statistics — how wide the band is when the truth is perfectly flat. A threshold inside any of them is unmeasurable.
After the run, a miss gets one of three verdicts — malformed, underpowered or substantive — and never “the claim was right after all”. Background: Could the instrument have shown it?.
1 · The registered prediction
2 · Could it be measured?
null SE from the Mann–Whitney U variance; observed SE by Hanley–McNeil.
The rule
- Never register a one-sided threshold at a null value (“AUROC ≤ 0.5”, “≤ chance”). Register the equivalence band |A − 0.5| ≤ 2 SE.
- Measure the instrument first. If your reference decoder tops out at 0.86 on clean data, a 0.90 threshold tests the decoder, not the hypothesis.
- Band statistics are wide by construction. The range of k noisy points grows like the expected range of k normals — about 4.3 SE at k = 38.
- Underpowered is not refuted. It is unmeasured, and a reformulated claim is post-hoc until re-registered.
Run it locally (Python)
pip install numpy python threshold_audit.py # reruns the original audit of four near-misses # or import the calculators: from threshold_audit import auroc_null_se, auroc_se, prop_se, mean_se, verdict
- threshold_audit.pycalculators + the original audit · requires numpy
The browser version computes the expected range of k normals exactly; the Python audit used a rounded 3.7 SE for k = 38, where the exact value is 4.28. That makes E4's flat-truth band wider (0.150, not 0.130) — the original conclusion holds, with less room than it reported.
