Research • Findings ledger

Findings ledger

Last updated Nov 12, 2026 • By Will Schulz

A running ledger of every finding from the lab's continual-learning research programme, with its evidence level and where it is recorded. Entries are appended as results land; nothing is removed, and a correction is a new entry rather than an edit. Dates are when the result was obtained.

157 entries across 16studies. Research Desk posts and papers are linked; reports and research notes live in the lab's private research archive and are shown by path.

How is each finding graded?

Every entry carries one of four evidence levels, and a post-hoc flag when the comparison or explanation was chosen after seeing results. The level says how far the result has been taken, not how much we believe it.

L0 engineering: an observation about tooling or a pipeline, not a research result.
L1 toy: our 2-layer transformer on a synthetic knowledge base (or MLPs on MNIST / a point-mass simulator), 3–5 seeds, uncorrected tests.
L2 small LLM: Qwen2.5-1.5B-Instruct with LoRA on real facts, 3 seeds.
L3 literature: published result we have read but not reproduced; numbers are the authors' own.
post hoc: the comparison or explanation was chosen after seeing results.
Pending: a result promised by a pre-registration or plan that has not yet been obtained.

Jump to: 1. LLMs editing open-model weights · 2. Memory, world models and source trust · 3. Order of addition and companion knowledge · 4. Seven anti-forgetting methods, head to head · 5. Consequence-consistent updates, pre-registered study · 6. Long update streams · 7. Weight editing to prevent forgetting · 8. Genesys program: selection and verifier tests · 9. Stopping confident wrong expert votes · 10. Logical coherence of LLM world models · 11. Elastic weight consolidation and contradicting updates · 12. Cross-generational teaching and examination · 13. Model self-knowledge and confident errors · 14. Merging and splitting MoE experts · 15. Optimal router architecture · 16. Civilization as a superintelligence model · Pending · Caveats

1. LLMs editing open-model weights

Nov 09, 2026 • 5 entries

Findings 1.1–1.5
#	Finding	Level	Recorded
1.1	Automated weight modification is good at removing things (refusals, reasoning loops, specific facts) and poor at adding capability or knowledge. Nearly every result stops at ~8B parameters.	L3 synthesis	`reports/LLMs modifying open model weights.md`in the research archiveA model rewrote another model's weights. Read what it took away.Research Desk
1.2	MIT CSAIL's weight-space programming (2 Nov 2026): an LLM agent subtracted one direction from 131,072 weights (0.016%) of Qwen3.5-0.8B; MATH-500 7.9%→48.1%. Mechanism was removing a looping failure (token-limit hits 93%→33%), not adding skill. Unreviewed, small samples.	L3	same as 1.1
1.3	Knowledge-editing efficacy is ~99% under teacher-forced scoring but 38.5% vs 96.8% under free generation (Mirage of Model Editing); adversarial suffixes recover "erased" facts >85% of the time. Edits suppress, they do not overwrite.	L3	same as 1.1
1.4	PostTrainBench: best coding agent 23.2 vs 51.1 for official instruct models; agents trained on test data, downloaded instruct checkpoints, used found API keys.	L3	same as 1.1
1.5	Abliteration costs little on MMLU but up to 18.8 points on GSM8K (Yi-1.5-9B); no post-training safeguard has survived adaptive attack; pretraining data filtering (Deep Ignorance) is the most durable defence.	L3	same as 1.1

2. Memory, world models and source trust

Nov 09, 2026 • 6 entries

Findings 2.1–2.6
#	Finding	Level	Recorded
2.1	Split-MNIST class-incremental: naive 0.197 → replay (500 images) 0.852 → replay + sleep-style distillation 0.888 ± 0.006. Sleep adds a small real margin on top of the buffer.	L1	`reports/Memory world models and source trust.md`in the research archiveWhat a model lets into long-term memory matters more than how it learnsResearch Desk
2.2	Source tagging gave the best accuracy under mixed-reliability sources (0.944) and cut a targeted 7→1 label attack from 47% to 6%.	L1	same as 2.1
2.3	Weighting whole sources by estimated reliability backfired: attack rate rose 0.468 → 0.696. Per-example posterior weighting fixed it (0.935) — post hoc.	L1Post hoc	same as 2.1
2.4	Gating each example on agreement with trusted memory before consolidation held a poisoning attack to 0.037, where trusted replay alone left it at 0.936 — post hoc design.	L1Post hoc	same as 2.1
2.5	World models: exploration-coverage data gave 3.9× lower error than narrow policy data; multi-step objectives 3.2× lower 20-step rollout error; easy-first curriculum was a null result. Replay prevents forgetting of changed dynamics only with a dynamics-context signal.	L1	same as 2.1
2.6	Literature: sparse memory finetuning cut forgetting 89%→11% (1.3B); Google's "Sleep" credits most of its gain to self-generated practice data, not consolidation; ~250 poisoned documents backdoor models of every size tested; ClashEval models adopt wrong retrieved text >60% of the time.	L3	same as 2.1

3. Order of addition and companion knowledge

Nov 09, 2026 • 17 entries

Findings 3.1–3.17
#	Finding	Level	Recorded
3.1	Bridging knowledge added alongside new facts does not protect old knowledge. Toy: paraphrases −2.0 pp, context about at-risk entities −2.1 pp vs filler (null); entailed ripple facts −11.4 pp (harmful).	L1	`reports/Continual learning order and companion data.md`in the research archiveTo stop a model forgetting, rehearse what it knew. New context won't do it.Research Desk
3.2	Old facts re-expressed in never-trained phrasings protect nearly as well as verbatim replay: toy 94.2% vs 95.8% (47.6% with nothing).	L1	same as 3.1
3.3	Replicated at 1.5B: new facts only 33.2 ± 10.0% old-fact retention; + bridging 32.3 ± 11.2; + re-expressed old facts 88.9 ± 0.9; + verbatim replay 91.8 ± 1.8; + model's own answers 86.0 ± 9.4. 60 counterfactual edits + 15 verified 2024–26 real changes; updates learned 95–100%.	L2	`reports/Continual learning order and companion data.md`in the research archiveTo stop a model forgetting, rehearse what it knew. New context won't do it.Research Desk`research_notes/Continual learning order and companion data/experiments_llm.md`in the research archive
3.4	Rehearsing half of the old facts protected the never-rehearsed half too (82–84% vs 32%) at 1.5B. General-knowledge canary 83.8% new-only vs 95–96% with rehearsal.	L2	same as 3.1
3.5	Re-expressed rehearsal trails verbatim replay by 15.3 ± 9.6 pp on the trained wording (75.3 vs 90.6) at 1.5B — a cost the toy could not show.	L2	same as 3.1
3.6	94% of forgotten answers under new-only training were the just-trained counterfactual, not random errors — post hoc.	L2Post hoc	same as 3.1
3.7	Order: recency dominates without replay (stage-1 facts 22.8% vs stage-8 100%); hardest-first kept +26.8 pp old knowledge but cost −19.7 pp on updates; interleaving relation types beat random by ~8 pp; blocking by entity cluster worst (−19.3 new, −41 update); dependency order no effect. Literature: 22.8-point spread across orders for plain LLM fine-tuning; random order is the safe default.	L1L3	same as 3.1
3.8	Correction: targeted rehearsal does beat random when targeting by the model's low confidence (+2.9 pp full budget, +4.5 at quarter budget); gradient-cosine targeting is worse than random (−1.5/−2.6). The earlier verdict that targeting does not help was based on gradient cosine only.	L1Correction	`reports/Continual learning order and companion data.md`in the research archiveTo stop a model forgetting, rehearse what it knew. New context won't do it.Research Desk`experiments_followup.md`in the research archive
3.9	A dry run (apply update on a copy, rehearse what flipped) ties low-confidence targeting at full budget and costs 2.26× vs 1.26× compute; its extra benefit is mostly on probed facts (unprobed +1.5/+3.3 pp over random).	L1	same as 3.1
3.10	Rehearsal preserves outdated knowledge. Superseded facts at their natural 2.6% pool share cost 11 pp of update accuracy; forced into the pool, update accuracy 23.0% vs 73.5% pruned. Pruning fixes the direct answer but not dependents: 93% of updated persons still gave the old city. Adding new dependent facts fixes only covered persons (99%) with no spillover. Any rehearsal roughly doubled stale city answers vs none (45.5%→93.0%).	L1	same as 3.1
3.11	Forgetting predictors: low confidence AUROC 0.737; gradient/representation similarity 0.65–0.69; "shares an entity with a new fact" 0.403 (those facts were less likely to be forgotten).	L1	same as 3.1
3.12	Self-generated supplements came within 2.6 pp of KB-derived ones; making 30% of them wrong cut retention 19.8 pp overall and 31.5 pp on the targeted facts. No LLM generator is needed if the knowledge base is available; generated material must be verified.	L1	same as 3.1
3.13	Rehearsal source at 1.5B: token KL to the frozen base ties gold replay overall (92.0 vs 91.8% held-out) but protects never-rehearsed facts better (+5.4 ± 0.4 pp, t = 23.0, trained wording; +4.0 held-out, n.s.) at −2.6 pp on rehearsed ones. Replaying the base's own answers as hard targets is no better than gold (−1.2 pp).	L2	`research_notes/Continual learning order and companion data/experiments_llm_followup.md`in the research archive
3.14	Stale knowledge at 1.5B: superseded facts were 29–31% of a natural rehearsal pool; pruning them raised update accuracy +22.8 ± 7.7 pp (t = 5.1) to 98.9%. Also pruning stale 2-hop dependents cut stale dependent answers −37.2 ± 6.7 pp (t = −9.6), far more than the toy's −7.0, but 59.7% stayed stale.	L2	`research_notes/Continual learning order and companion data/experiments_llm_followup.md`in the research archive
3.15	Adding new dependent facts at 1.5B fixed the covered entities (+77.4 pp, to 96.3%) and none of the held-out ones (+1.9 pp, n.s.), as in the toy: consequences do not propagate from the updated fact.	L2	`research_notes/Continual learning order and companion data/experiments_llm_followup.md`in the research archive
3.16	Does not replicate 3.8 at 1.5B: rehearsing the lowest-confidence facts was worse than random — −6.7 ± 2.7 pp held-out (t = −4.2) at full budget, −8.0 (n.s.) at quarter budget. Low base log-prob predicted which facts an update flips with AUROC 0.57 (toy 0.737) — post hoc.	L2Post hoc	`research_notes/Continual learning order and companion data/experiments_llm_followup.md`in the research archive
3.17	A dry run at 1.5B helps only the facts it probed: at quarter budget +14.8 pp on probed vs −3.3 (n.s.) on unprobed; at full budget it ties random replay (+0.5 pp) for 2.1× the compute.	L2	`research_notes/Continual learning order and companion data/experiments_llm_followup.md`in the research archive

4. Seven anti-forgetting methods, head to head

Nov 09, 2026 • 10 entries

56 conditions × 5 seeds on the toy testbed; random replay = 95.8% retention, naive = 47.6%.

Findings 4.1–4.10
#	Finding	Level	Recorded
4.1	Sequential LoRA forgot far more than full fine-tuning at the learning rate needed to learn (14.2% retention). Extra compute without rehearsal did not help: naive 2× 49.0%, 4× 40.7%.	L1	`reports/Seven continual learning methods tested.md`in the research archive`research_notes/Seven continual learning methods tested/experiments_results.md`in the research archive
4.2	Sparse memory slots: 78–99% retention. The control Meta never ran — updating the same number of random slots — loses 53–70 pp, so the frequency ranking does the work, not sparsity. Learns less: 85% immediate new-fact accuracy vs 98%.	L1	same as 4.1
4.3	Multi-timescale blocks (Titans/Nested Learning style): no better than an equal-parameter single-rate network (+4.6 ± 7.6, n.s.) and slightly worse than freezing the slow blocks (−2.4, t = −3.6).	L1	same as 4.1
4.4	Sleep consolidation: consolidating into slow weights hurt in every arm (−33 to −40 pp). Self-generated practice helped only as pseudo-rehearsal (+16 vs repeating new facts) and was 32 pp below random replay at equal compute. Minimal implementation; the method was designed for models with context windows.	L1	same as 4.1
4.5	Output anchoring: KL to the frozen model on new-fact inputs blocks learning (5% immediate accuracy); on old queries (λ = 10) −3.6 retention but +11.2 new-fact accuracy vs replay. On-policy replay lost to gold replay at the same 1% buffer (−23.6) and its low-score ablation ran opposite to the paper.	L1	same as 4.1
4.6	Update-direction restriction: O-LoRA beat IncLoRA (+8.0) but all LoRA arms were 68–82 pp below replay; AlphaEdit's published threshold left ~1 free dimension per layer in the toy (3.4% new-fact accuracy).	L1	same as 4.1
4.7	Forecast-then-replay: low-confidence selection +2.8 vs random (≈74% of the oracle gain); the Jin & Ren forecaster −3.8, AUROC 0.573, worse than its prior term alone (0.655).	L1	same as 4.1
4.8	Stacking: replay + anchoring + projection was the best point (97.7% retention, 83.5% new-fact accuracy, 92.5% update accuracy, 1.55× compute), beating replay at 2× compute by +2.3/+15.0. Every pairwise interaction sub-additive (replay×anchoring −13.6, replay×projection −8.5, anchoring×projection −7.8); still sub-additive on a log-forgetting scale — post hoc scale.	L1Post hoc	same as 4.1
4.9	First measurement of "stiffening": real only for methods that consume subspace. O-LoRA λ = 5 needed +18 extra steps per stage by stages 7–8 (t = 19.2); history-aware projection +13.5 steps, free dimensions 516→370. Plain fine-tuning and replay got faster over the stream.	L1	same as 4.1
4.10	Replication record: Meta's sparse memory and all of Google's Titans/ATLAS/HOPE/Sleep have no official code; the one faithful ARC rebuild of Sleep scored 20% vs the paper's 80%; O-LoRA's released code uses a different loss from its paper; at 7–8B two 2026 re-runs put O-LoRA below plain sequential LoRA; the 82–93% on-policy result (OPR, Tsinghua) uses a 50-prompt buffer and ships code.	L3	`reports/Seven continual learning methods tested.md`in the research archive`research_notes/Seven continual learning methods tested/experiments_results.md`in the research archivethe four research notes in that folder

5. Consequence-consistent updates, pre-registered study

rebuilding Phase 0

Nov 09, 2026 • 5 entries

Pre-registration frozen at commit 700ba5a (SHA-256 21ee5bbf…a3e2a8) before any main run. A machine restart on 2026-11-10 erased Phase 0's working data, including the frozen Wikidata snapshot, before any test result. Phase 0 is being rebuilt from live Wikidata with the same code and seeds (deviation D7); its model steps start after the CIFAR-100 study finishes.

Findings 5.1–5.5
#	Finding	Level	Recorded
5.1	MEMIT's apparent instability on a second chunk (z-NLL 15.2) was caused by degenerate smoke-test covariance statistics; with real text statistics it fits normally (0.28). Fix: a 1e-6 ridge in the closed-form solve.	L0	`research_notes/Consequence consistent updates/LOG.md`in the research archive`preregistration.md §17`in the research archive
5.2	AlphaEdit's null space with real statistics at 1.5B is 3,635 of 4,864 directions at layer 4 — ample room to learn, unlike the toy (4.6). Will shrink with the full 400k-token statistics.	L0	same as 5.1
5.3	WikiBigEdit's `update` tag conflates object replacement with one-to-many addition and does not release the replaced object; stale-answer scoring therefore needs Wikidata revision history. TriviaQA's HF card carries no licence.	L0	`phase0/audits/licenses_and_checks.md`in the research archive
5.4	Six pre-run deviations logged (D1 generator deferred → KB-only arm in the CCU slot; D2 judge audit deferred; D3 probe caps; D4 second candidate batch; D5 Phase 1 before the human probe audit; D6 8 tuning configs per condition).	L0	`preregistration.md §17`in the research archive
5.5	Pending: Phase 1 gap check (does in-weights consequence consistency trail in-context editing by ≥5 points at 1.5B?). Stop rule applies.	Pending	to be appended

6. Long update streams

Nov 09, 2026 • 5 entries

Toy: 1,000 updates in 100 stages, 8 conditions, 3 seeds (seeds 3–4 stopped to free the machine). Qwen2.5-1.5B + LoRA: 300 real counterfactual updates in 30 stages, seed 0 only; the pipeline arm was stopped at stage 20 under swap pressure and its numbers were recovered from log lines, not saved JSON.

Findings 6.1–6.5
#	Finding	Level	Recorded
6.1	Protection that looked strong over 8 stages collapses over 100: output anchoring alone fell from 92.1% to 20.4% base-fact retention and projection alone from 85.9% to 11.2%, close to naive fine-tuning's 6.7%.	L1	`research_notes/Seven continual learning methods tested/experiments_longstream.md`in the research archive
6.2	Replay holds but erodes slowly on the toy (91.0% → 73.0% over 100 stages); the replay + anchoring + projection stack held 82.2%, significantly above replay (t = 7.2). On Qwen 1.5B, plain replay kept old facts flat at 90.8–93.0% across all 300 updates, while the general-knowledge canary drifted from 93.9% to 82.8%.	L1L2	same as 6.1
6.3	Only the expanding projection stiffened, and mildly: its free subspace shrank from 470 to 133 dimensions (−72%) and it was the only condition whose learning did not speed up over the stream (+10 steps vs replay in the last block, t = 7.9), though it still learned every stage fully.	L1	same as 6.1
6.4	Stale knowledge accumulates in two places. In an unpruned replay buffer, superseded facts grew from 0.23 to 6.0 per stage and stale answers rose to 29.3% (8.7% with pruning). On updates' 2-hop consequences, every replay variant left 60–88% stale answers; the pipeline that adds new consequences cut this to 20.5% (−61.7 pp vs replay, t = −32).	L1	same as 6.1
6.5	The consequence-aware pipeline beat replay on update quality on the toy (+16.3 pp update accuracy, −23.7 pp stale answers) but not on retention. On Qwen 1.5B it was worse than gold replay at every common checkpoint, with a 30.7% stale rate on the held-out template vs 0.0–4.0% for replay. Untested explanation: it rehearses earlier updates using the model's own answers, which are the old value where an update has not generalised; the diagnostic was lost in the truncation.	L1L2	same as 6.1

7. Weight editing to prevent forgetting

Nov 09, 2026 • 8 entries

Toy testbed: ~45 conditions × 5 seeds (W1–W4), baselines reproducing earlier studies exactly. Qwen2.5-1.5B + LoRA: post-hoc repair (W3) only, 3 seeds. Every edit was chosen by a fixed rule; no LLM agent was in the loop.

Findings 7.1–7.8
#	Finding	Level	Recorded
7.1	Retiring superseded facts: a rank-one (ROME/MEMIT-style) overwrite plus a pruned rehearsal pool with the new consequences cut stale 2-hop answers from 93.0% to 9.5% (−83.5 pp, t = −32.4) vs pruning alone, raised update accuracy on a never-trained arrangement from 71.0% to 99.0%, and lifted never-trained consequences from 2.2% to 51.8%. Edit site and strength tuned on seed 0 — post hoc.	L1Post hoc	`reports/Weight editing to prevent forgetting.md`in the research archive`research_notes/Weight editing to prevent forgetting/experiments_results.md`in the research archive
7.2	The overwrite survived the recovery probes: stale answers stayed ≤1.5% after 50 steps of unrelated fine-tuning and after relearning on 20 related or unrelated old facts, while without the edit relearning brought them back (41.8% → 62.5%). Costs: −3.8 pp unaffected retention (t = −5.3), 1.31× compute, edits of 48 ± 16% of the edited matrix's norm per stage, and more loss under later unrelated fine-tuning. An optimised adversarial prompt search and linear-probe decodability were not tested.	L1	same as 7.1
7.3	Directional suppression of a single fact (projecting the old-vs-new answer direction out of the writing weights, key-gated) and F-Learning subtraction were indistinguishable from a random-direction control. Global abliteration-style projection of 40 directions erased stale answers but cut retention to 60.2% (a toy-width effect, untested at LLM width).	L1	same as 7.1
7.4	Removing the interfering part of each update beat compute-matched replay only when probing every old fact (98.6% vs 95.8%, t = 6.9; +4.4 pp new-fact accuracy); with a probe the size of replay's pool (~7% of old facts) it fell to 80.8% (−15.0, t = −9.4). Both variants made contradicting updates stick worse (44–77% stale vs ~7% for fixed projection); neither stiffened late learning.	L1	same as 7.1
7.5	Post-hoc repair never matched replay at equal compute on the toy: interpolation, rollback and random half-reset lie on one curve trading new facts for old; spectral truncation and batched rank-one restores of flipped facts barely helped (the latter recovered 42% of flips).	L1	same as 7.1
7.6	At Qwen 1.5B, halving the LoRA update after training (α = 0.5, chosen on dev every seed) restored old facts from 33.5% to 83.0% and matched replay on never-rehearsed facts (85.1% vs 84.1%), but cost 24.4 pp of update accuracy on the evaluation-only template and let stale answers rise to 22%. Layer, component and direction rollback each recovered only 9–11 pp. Zeroing only the MLP part of the update restores old facts but deletes the update: new facts and the forgetting they cause share the same weights.	L2	same as 7.1
7.7	Adding the overwrite to the replay + anchoring + projection stack cut stale 2-hop answers from 62.5% to 27.5% (−35.0, t = −12.3) at −3.0 pp retention and −4.2 pp new-fact accuracy; adding directional suppression instead changed nothing (+0.2).	L1	same as 7.1
7.8	Engineering: on Qwen 1.5B, re-training the identical no-rehearsal adapter matched the earlier run on average (−1.2 pp, t = −0.3) but varied −8 to +6 pp per seed (MPS non-determinism or library drift), so single-seed differences under ~6 pp at this scale are noise.	L0	same as 7.1

8. Genesys program: selection and verifier tests

in progress

Nov 09, 2026 • 18 entries

The evidence behind the Genesys program's two claims (forgetting as selection; verifier exploitability bounds Best-of-N), set against a literature audit across continual learning, Best-of-N, agent tool selection and continual world models. Three pre-registered tests are running; see /programs/genesys.

Findings 8.1–8.18
#	Finding	Level	Recorded
8.1	Forced vs routed skills, Llama 3.1 8B, in-context library 4 → 64 (400 requests, 200 probes per size): mis-selection ε rose 0.000 → 0.030 (Δ CI +0.014 to +0.052) and forced accuracy on the original skills rose 0.910 → 0.975, so no forgetting; but ε explained only 6% of the 0.237 routed-accuracy drop. The rest was execution of harder skills.	L2 8B, in-context	`research_notes/Forced vs routed skill selection/README.md`in the research archive
8.2	The class-incremental factorization is Kim et al.'s 2022 theorem (CIL loss ≤ within-task + task-prediction loss); its OOD-to-task-prediction bound weakens as tasks grow. Selectors that beat replay on CIFAR-100 (HAT+CSI 63.3, TPL 76.53 vs DER++ 53.7 / 69.73) are trained on replay data or synthetic negatives.	L3	same as 8.1
8.3	Selection is often the weak link and sometimes unnecessary: HiDe-Prompt's task-identity inference reaches at most 67.74% while its final accuracy is above 91% (a shared head recovers from wrong routing); RanPAC, with no task-ID step, reaches 92.2% on CIFAR-100 vs 84.1–86.3% for prompt-selection methods.	L3	same as 8.1
8.4	Real verifiers: weak ones decline at large N (Cobbe 2021 beyond ~400 samples; OASST RM −5.61% below the base policy at N = 2¹³; log-prob selection 53.34 → 51.96 on MATH from N = 8 to 64), strong ones plateau (PRM 78.2% at N = 1,860; ArmoRM 40.50 → 41.41% on MATH). RewardBench 2 accuracy correlates r = 0.87 with Best-of-N gains at N = 16.	L3	same as 8.1
8.5	At 52 / 102 / 202 skills on frontier models, wrong-skill selection explains about 38% / 57% / 67% of the pass-rate drop (Song & Wei 2026), the opposite of 8.1's small-library result; skill routing degrades sharply below ~12B (Xu et al. 2026). The widely quoted tool-count figures in arXiv 2606.30317 v1 were withdrawn in v2.	L3	same as 8.1
8.6	Continual world models: mixture-of-dynamics experts with an inferred regime date to MOLe (2018); continual model-based RL that works runs on replay (Continual-Dreamer); no published oracle-vs-inferred-selector ablation names selection as the bottleneck.	L3	same as 8.1
8.7	Pending: real-judge Best-of-N (Llama 3.1 8B, 32 candidates on 100 GSM8K / MATH-500 / HumanEval+ problems; Skywork-Reward-V2-Qwen3-0.6B, Qwen2.5-Coder-7B judge, self log-prob). H1 non-monotone with cross-fitted N, H2 ε orders N, H3 ε* consensus rule. Frozen at commit `eb41a5c`.	Pending	`research_notes/Judge exploitability on real Best-of-N/preregistration.md`in the research archive
8.8	Pending: Split-CIFAR-100 confirmatory run, H1–H4 (selection dominates for frozen experts; explicit selection trails GDumb; ε grows with task count; between-method differences are mostly selection), 10 seeds.	Pending	`research_notes/Forgetting as selection at scale/preregistration.md`in the research archive
8.9	Pending: forced vs routed phase 1b, predictions P1–P4 (3B routes worse than 8B; 30B-A3B no worse; documented descriptions cut ε; mis-selection explains under half of the drop in every run).	Pending	`research_notes/Forced vs routed skill selection/README.md`in the research archive
8.10	Resolves 8.9 (P1, P2): at 64 in-context skills, mis-selection ε falls steeply with model size: Llama 3.2 3B 0.170, Llama 3.1 8B 0.030, Qwen3-VL 30B-A3B 0.003 (Δ vs 8B +0.140 [+0.100, +0.182] and −0.028 [−0.049, −0.010], Newcombe). Mis-selection explained under half the routed drop in every run (P4); kill criterion not triggered, though the 3B's forced accuracy on original skills fell 8.5 points [−16.0, −0.9].	L2	`research_notes/Forced vs routed skill selection/README.md`in the research archive
8.11	Resolves 8.9 (P3): richer tool documentation (instruction plus a worked example) did not lower ε for Llama 3.1 8B at K = 64: 12/400 both ways, Δ +0.000 [−0.025, +0.025]. The confusions moved to different skill pairs rather than disappearing.	L2	`research_notes/Forced vs routed skill selection/README.md`in the research archive
8.12	Not predicted: the 3B model still answered 42 of its 68 mis-selected requests correctly, ignoring the chosen skill's instruction; the 8B followed the wrong skill (3 of 12 right). For small models, ε overstates the cost of a wrong choice. Observed after the fact — post hoc.	L2Post hoc	`research_notes/Forced vs routed skill selection/README.md`in the research archive
8.13	Resolves 8.7 (H1, H2): no real judge made Best-of-N accuracy fall by N = 32. Across a reward model, an LLM judge and self log-probability on GSM8K, MATH-500 and HumanEval+, accuracy at a cross-fitted N never beat N = 32 (rm +0.000, llm −0.009 [−0.021, +0.004], logprob +0.000; Holm p = 1.00). Measured ε (0.02–0.22) did not order N (Spearman ρ = +0.26 [−0.37, +0.78]). Both kill criteria were triggered: no saturation claim and no ε-predicts-budget claim for real judges at N ≤ 32.	L2	`research_notes/Judge exploitability on real Best-of-N/README.md`in the research archive
8.14	Resolves 8.7 (H3): a plurality vote over the top-scored half of 32 candidates matched or beat taking the single top-scored candidate in all 6 math cells, by +0.03 to +0.17 (largest for self log-probability on MATH-500, 0.45 → 0.62). Every cell was below its ε, so only the below-threshold half of the ε rule was tested.	L2	`research_notes/Judge exploitability on real Best-of-N/README.md`in the research archive
8.15	Resolves 8.8 (H1, H4): on Split-CIFAR-100 at 10 tasks (10 seeds), frozen per-task experts were right 96% of the time within the task they were given but routed an input to the right task 36% of the time: selection error exceeded within-task error by +0.604 [+0.598, +0.610]. Across Naive, GDumb, DER++ and BiC, selection carried S = 0.91 [0.90, 0.92] of the between-method variance in log accuracy (≥ 0.5 predicted). Oracle-task accuracy 0.786 passes K1. Interim: final Holm over H1–H4 waits on the T = 5 / 20 runs, which can't change these two decisions (bootstrap p < 0.0001).	L1 CIFAR-100 ResNet, 10 seeds	`research_notes/Forgetting as selection at scale/phase1/analysis/interim-2026-11-11.md`in the research archive
8.16	Resolves 8.8 (H2), against the prediction: explicit selection over frozen experts did not trail GDumb; it beat it, 0.342 vs 0.157 (Δ +0.186 [+0.180, +0.192]). Avalanche's GDumb, retrained from scratch on a 2,000-item memory under the shared 50-epoch budget, is far weaker here than on Split-MNIST. The experts still trailed BiC (0.403) and Joint (0.533) and only narrowly beat DER++ (0.315).	L1 CIFAR-100 ResNet, 10 seeds	`research_notes/Forgetting as selection at scale/phase1/analysis/interim-2026-11-11.md`in the research archive
8.17	8.15's selection share is not carried by Naive alone (whose task accuracy is fixed at 0.100): over GDumb, DER++ and BiC only, S = 0.68 [0.67, 0.70]. Computed after seeing the confirmatory value — post hoc.	L1 CIFAR-100 ResNet, 10 seedsPost hoc	`research_notes/Forgetting as selection at scale/phase1/analysis/interim-2026-11-11.md`in the research archive
8.18	Resolves 8.8 (H3), and closes it: for frozen experts with max-softmax selection, ε grew with task count, 0.534 (T = 5) → 0.644 (T = 10) → 0.788 (T = 20); both steps +0.109 [+0.099, +0.119] and +0.144 [+0.136, +0.153]. Within-task accuracy rose (0.905 → 0.961 → 0.981) as tasks got smaller, while accuracy fell 0.422 → 0.342 → 0.208, so the whole decline is selection. Final Holm over H1–H4 on the full 80-run grid: H1, H3, H4 confirmed (adjusted p < 0.0001), H2 not (p = 1.00), as in 8.15–8.16.	L1 CIFAR-100 ResNet, 10 seeds	`research_notes/Forgetting as selection at scale/phase1/analysis/results.md`in the research archive

9. Stopping confident wrong expert votes

Nov 09, 2026 • 10 entries

Ten pre-registered experiments from a literature review on making untrained experts stop voting confidently for the wrong answer. Split-MNIST class-IL, frozen per-task MLP experts (K = 5), 5 seeds, run once; baselines reproduce the forgetting-as-selection paper (ACC 0.633 raw pixels, 0.786 streaming-PCA features). ε is scored on the K-expert consensus. Code and full results: continual-learning/EXPERIMENTS-selector-abstention in the research repo.

Research DeskPenalising wrong answers only works if someone else is grading
Related paperCatastrophic forgetting in class-incremental learning is a verifier-exploitability problem

Findings 9.1–9.10
#	Finding	Level	Recorded
9.1	A selector that never reads expert confidence, RanPAC (random ReLU projection + ridge) over the streaming-PCA features, reached class-IL ACC 0.926 ± 0.004 at ε 0.067, against 0.857 for nearest-class-mean, 0.892 for shrunk Mahalanobis and 0.786 for the expert-confidence selector. ε over 2 → 5 tasks rose 0.016 → 0.067 (expert confidence: 0.071 → 0.213). The PCA basis uses the whole unlabelled stream and the ridge λ was picked on a labelled calibration split.	L1	`reports/Stopping confident wrong expert votes.md`in the research archiveresearch_notes/Stopping confident wrong expert votes/
9.2	Experts were calibrated on their own task (ECE 0.011) but not on other tasks' inputs (ECE 0.697); only 34% of raw inputs they scored ≥ 0.95 were from their own task. A shared temperature left ε unchanged (0.367 → 0.368); a per-expert isotonic map to P(mine) cut it to 0.336.	L1	same as 9.1
9.3	A reject threshold at the selector worked only on an independent score: Kalai score (+1 / 0 / −t/(1−t)) at t = 0.9 was +0.545 at 71% coverage on a density selector's margin, against +0.114 at 21% coverage on expert confidence; area under the risk–coverage curve 0.025 vs 0.098.	L1	same as 9.1
9.4	A penalty for wrong answers without other-task data did not help: experts with a Deep Gamblers abstain output trained on their own task only changed ε by −2% at t = 0.5 and raised it 54% at t = 0.75, where the abstain output fired on 18% of own-task and 20% of other-task inputs.	L1	same as 9.1
9.5	With other-task data the same idea worked: a "not mine" output trained on a buffer of past tasks' images, then re-tuned on the final buffer, lifted raw-pixel ACC 0.633 → 0.733 / 0.847 / 0.871 at 10 / 50 / 100 images per task. Its pre-registered kill fired: ε (0.124 at 100) never fell below the buffer-free RanPAC selector's. A post-hoc head on frozen expert features did worse (ε 0.303).	L1	same as 9.1
9.6	Self-made negatives with no stored data, a third class of rotated and flipped own-task images, lifted raw-pixel ACC 0.633 → 0.760 (+12.7); on PCA features the gain was +3.0, a hair under the kill threshold. Generic outlier exposure (permuted pixels, noise) gave +2.3 / +1.1.	L1	same as 9.1
9.7	Per-expert affine calibration on a 200-per-task buffer raised raw max-logit and energy selection by 5.4 points but lowered max-softmax selection (0.633 → 0.585); buffer-free percentile calibration lowered ACC in every arm (−1 to −5 points).	L1	same as 9.1
9.8	A task-local reconstruction gate (per-task PCA subspace, k = 20) beat expert confidence: 0.866 raw with the pre-registered normalization, and 0.915 (ε 0.075) without it, using only each task's own data. The unnormalized arm was secondary.	L1	same as 9.1
9.9	Deciding the task over a batch of same-task inputs took a density selector's ε from 0.095 to 0.002 at 8 inputs and 0 at 16; expert confidence went 0.213 → 0.012 at 8.	L1	same as 9.1
9.10	ε grows sub-linearly with task count because wrong experts steal the same inputs: pairwise wrong-win rates were non-uniform (CV 0.69, max pair 0.39 vs mean 0.15), yet independence models over-predicted ε (0.47–0.49 vs 0.367 measured; synthetic 10 tasks 0.87 vs 0.55), and allowing per-pair rates barely helped. The overlap is inferred from that over-prediction, not measured directly.	L1	same as 9.1

10. Logical coherence of LLM world models

Nov 10, 2026 • 21 entries

A literature synthesis on whether LLMs hold a logically coherent model of reality, and two pre-registered experiments it proposed, run once on the section 3 toy testbed (5 seeds each): iterated self-closure ("chewing", E3, plus a replication on 5 fresh seeds) and gradient kernels as predictors of edit ripple (E2, plus a pythia-160m check). A third, belief repair (E1), measured MaxSAT repair of real LLM beliefs (llama3.2 3B and llama3.1 8B) on MQuAKE 2-hop chains.

Findings 10.1–10.21
#	Finding	Level	Recorded
10.1	Accuracy and coherence come apart: a frontier model was 98.9% accurate on single questions but 60.3% consistent across logically equivalent rewrites (CRTBench, an unreviewed 2026 preprint), and after a rank-1 weight edit accuracy on facts that should change fell 0.97 → 0.01 (Hase et al.). Consistency-repair methods report consistency gains several times their accuracy gains; none reports the coherently-wrong rate, the share of repaired conflicts resolved toward the false belief.	L3 synthesis	`reports/Logical coherence of LLM world models.md`in the research archiveresearch_notes/Logical coherence of LLM world models/
10.2	Iterated self-training converges to whatever its selector rewards: without an external anchor the documented end states are self-consistent errors (model collapse, majority-vote self-reward collapsing into one template answer). Anchors that work in the literature are real data accumulated rather than replaced, verifier or ground-truth filters, and early stopping on held-out data.	L3 synthesis	same as 10.1
10.3	E3, H1 supported: iterated self-closure converged in every arm. Unanchored, it changed 5.3% of 1,888 beliefs in round 1 and 0.06% in round 10 (Δ −5.3 [−5.8, −4.7]); every seed fell below 1% per round by round 2–3, and no endpoint moved more than 2.1 pp between rounds 10 and 30.	L1	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.4	E3, H1b not supported: the unanchored fixed point was coherent but not true (global consistency 76.3 → 94.8%; 44.0% of updated persons coherently wrong), but the pre-registered rise in coherently-wrong updated persons was +4.0 pp [−0.7, +8.7], because 40% were already coherently stale at round 0. It was +7.0 [+1.0, +13.0] in replication and +5.5 [+2.4, +8.6] pooled over 10 seeds (post hoc pooling). On half-forgotten new persons coherently-wrong rose 10.0 → 55.0% (+45.0 [+31.7, +58.3]).	L1	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.5	Self-labelling paraphrases with the model's majority belief reverted edits learned in one phrasing: update accuracy in that phrasing fell 73.5 → 55.5% under unanchored closure and → 53.5% under plain self-rehearsal, within one round (−20.0 [−25.2, −14.8] and −22.5 [−28.0, −17.0] pp over 10 seeds).	L1	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.6	E3, H2 supported: anchoring the loop to the 228 told facts left 0.0% of updated persons coherently wrong, against 44.0% unanchored (Δ −44.0 [−53.7, −34.3]; replication −51.0). Global consistency could not tell the two fixed points apart (94.7% vs 94.8%).	L1	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.7	E3, H3 not supported as pre-registered: at 16 anchor examples per 64, pruned and unpruned anchored closure both reached 100% (Δ 0.0, a ceiling). Pruning only made settling about 10× faster (+11.5 ± 2.9 pp update accuracy over rounds 1–5). In a post hoc anchor-weight sweep, unpruned closure reached 76.0% update accuracy at round 10 with 1 anchor per 64 and never moved with 0, while pruned closure was at 100% by round 3 at every weight.	L1Post hoc	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.8	E3, H4 supported: pruned self-closure gave 100% correct 2-hop answers for uncovered updated persons, against 0% for gold consequences on the covered half (Δ +100.0; replication +97.2 [+94.0, +100.4]), the first NeuroGenesys propagation beyond trained entities (cf. 3.15). The reach comes from the loop querying every entity: on never-queried probe persons 2-hop accuracy rose only +14.0 [+10.5, +17.5] pp unanchored and +8.5 [−2.6, +19.6] pruned (+5.5 / +4.0 in replication).	L1	`reports/Iterated self closure consolidation.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E3_iterative_closure/experiments_results.md`in the research archive
10.9	E2, H1a failed: in the update stream, low confidence predicted forgetting better than every gradient kernel, AUROC 0.737 [0.712, 0.762] against 0.663 for the best kernel variant (t = −8.1), 0.592 for unsigned \|cos\| and 0.632–0.684 for CLaRE-like activations. Both earlier stream numbers (0.737, 0.654) replicated exactly (cf. 3.11).	L1	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.10	For small isolated edits to a converged model the ranking flipped: confidence 0.540, signed gradient cosine 0.684 (within-edit 0.731). A structural rule, "same relation as the edit", beat every kernel at 0.831, and within relation strata signed cosine, Adam-sign and CLaRE tied at 0.70–0.71 — post hoc relation analysis.	L1Post hoc	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.11	Kernel coupling between an edit and its should-change 2-hop fact predicted how far that answer moved (Spearman +0.409 [+0.260, +0.557]; failure AUROC 0.646, Adam-sign 0.699), where the old answer's confidence did not (0.522; Spearman −0.03). Propagation itself was rare: 8.6% [4.3, 12.8] of dependents updated, against a 0.4% bias control.	L1	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.12	The gradient cosine's sign detects contradictions: superseded facts in other phrasings had cos −0.306 / −0.286 with the edit (97% negative; AUROC 0.969 / 0.960 vs neutral), while the same query carrying the edit's new label had +0.731, a perfect separation (AUROC 1.000) that activation similarity cannot make on identical inputs — post hoc control. Stale 2-hop dependents also stood out (0.818), against the pre-registered null.	L1Post hoc	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.13	Steering rehearsal by the kernel backfired: unsigned-kernel rehearsal lost 5.2 ± 3.5 pp retention to random (t = −3.3, replicating 3.8), and inside a relation-map pipeline kernel rehearsal cost 16.1 pp retention (t = −10.0) while consequence examples cut covered stale 2-hop answers 88.2 → 11.3%. With low-confidence rehearsal instead, at full budget: 97.5% retention, 78.0% update accuracy, 18.5% covered stale 2-hop, against 95.7 / 64.0 / 94.6 for random, at 1.29× compute. Uncovered persons stayed stale (88.7% vs 89.7%).	L1	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.14	pythia-160m check (55 typed facts, one counterfactual edit set, 36% flips): signed gradient cosine tied low confidence on flips (0.685 vs 0.681) and beat it on large log-prob drops (0.781 vs 0.575); CLaRE-like activations were best (0.775 / 0.805), but "same answer type" alone scored 0.701, and within an answer type confidence beat the kernel on flips (0.684 vs 0.632; post hoc split). Directional only.	L2 pythia-160m, one edit setPost hoc	`reports/Kernels predicting edit ripple.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E2_kernel_ripple/experiments_results.md`in the research archive
10.15	E1, H1 not supported: evidence priority did not win; entrenchment did. When LLM beliefs on MQuAKE 2-hop chains (400 chains 3B, 250 chains 8B) were repaired by weighted MaxSAT, weighting by degree in the global constraint graph gave 85.8% / 95.2% accuracy and a 49.8% / 15.6% coherently-wrong rate (3B / 8B), against 79.4% / 92.0% and 65.0% / 31.2% for confidence. Evidence priority (20% of atomic facts verified) cut coherently-wrong to 60.8% / 21.2% but lost to entrenchment (−6.6 [−8.0, −5.3] and −1.7 [−3.2, −0.4] pp accuracy).	L2 llama3.2 3B / llama3.1 8B, Ollama	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.16	Model confidence was barely better than random at choosing which belief gives way: +1.3 pp [0.3, 2.3] (3B) and +1.2 pp [−0.0, 2.5] (8B) accuracy, with no difference in coherently-wrong rate. After confidence repair every chain was consistent, yet 65.0% (3B) and 31.2% (8B) still held a false belief; 32% of the 3B solver's flips moved a correct belief to wrong.	L2 llama3.2 3B / llama3.1 8B, Ollama	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.17	Why entrenchment won: the models accepted true premises but rejected true 2-hop conclusions (Q-form accuracy 41.8% / 63.2% on true composed answers vs 80.7% / 95.4% on true atomic facts), and confidence repair dropped a premise to keep the confident "No". Entrenchment raised true-conclusion accuracy to 67.5% / 91.6%. Degree did not predict truth (AUROC 0.50 / 0.47); an "atomic premises first" rule recovered most of the gain (84.2% / 93.6%) — post hoc diagnosis.	L2 llama3.2 3B / llama3.1 8B, OllamaPost hoc	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.18	Most of the repair gain came from making each belief's three phrasings agree: form agreement alone gave 80.6% (3B) and 89.7% (8B) under confidence, from 70.9% and 78.9%. Adding gold composition and exclusion logic then changed accuracy by −1.2 pp [−2.1, −0.3] (3B) and +2.4 pp (8B) under confidence, and +3.5 / +4.5 pp under entrenchment — post hoc comparison.	L2 llama3.2 3B / llama3.1 8B, OllamaPost hoc	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.19	E1, H8 supported: LLM-judged constraints roughly halved the gain. Confidence repair gained +4.8 pp (3B) and +4.2 pp (8B) over no repair with judged relations, against +9.3 and +12.0 pp with gold logic on the same chains (−4.5 [−6.3, −2.8] and −7.8 [−10.4, −5.4]). Judge contradiction recall was 0.34 (3B) and 0.42 (8B), precision 0.71 and 0.44; coherently-wrong rose to 75.8% and 53.8%. Judged on 120 (3B) and 80 (8B) chains.	L2 llama3.2 3B / llama3.1 8B, Ollama	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.20	E1, H6 supported: LLM elicitation and relation checking took about 23 s (3B) and 38 s (8B) per chain, CPU-only on a shared machine, against a 0.1 ms RC2 solve; relation checking alone was 1.1×10⁵ / 2.3×10⁵ times the solve. The solver's one real cost is skipping decomposition: one merged 68,400-belief instance took 282.5 s, against about 1.2 s solved per connected component.	L2 llama3.2 3B / llama3.1 8B, Ollama	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive
10.21	E1, H7 not supported: digital Ising stand-ins were worse than exact RC2. Simulated annealing and tuned simulated bifurcation reached the optimum on 76/105 and 72/105 components (3B; 8B 71/90 and 74/90) and lowered downstream accuracy from 79.5% to 71.1% and 66.5% (3B) and from 91.3% to 82.5% and 75.4% (8B), while being 30–100× slower per component.	L2 llama3.2 3B / llama3.1 8B, Ollama	`reports/Belief repair rules and solver cost.md`in the research archive`research_notes/Logical coherence of LLM world models/experiments/E1_belief_repair/experiments_results.md`in the research archive

11. Elastic weight consolidation and contradicting updates

Nov 10, 2026 • 8 entries

A literature synthesis on EWC (theory, estimators, 1–8B results, coherence) and the pre-registered toy experiment it proposed (E4): an EWC λ sweep with an exact Fisher on the section 3 testbed's stream of contradicting updates, KL anchors matched on retention, and a per-fact Fisher ledger. Every decision was fixed on dev seeds 100–101; test seeds 0–4 were run once.

Findings 11.1–11.8
#	Finding	Level	Recorded
11.1	EWC is the local quadratic of a KL anchor on old inputs, so it protects what its Fisher data exercises: in 2026 translation fine-tuning of Llama 3.1 8B it held the general-benchmark drop to 1.7 points against 11.0 for SFT, but task-specific instruction following its Fisher set missed recovered only to 35.3 (base 71.1). The Fisher estimator shifts EWC's optimal λ by orders of magnitude, so published "EWC" rows judge one implementation.	L3 synthesis	`reports/Elastic weight consolidation deep dive.md`in the research archiveresearch_notes/Elastic weight consolidation deep dive/
11.2	Correction: the Fortifying-knowledge report quoted the SLoRA paper (ACL 2026) from search snippets as saying "regularization methods remain weak at this scale, often falling below Seq-LoRA." The paper reads: "On Llama3.1-8B and Llama3-8B, methods such as EWC and GEM often fall below the simple Seq-LoRA baseline." The quoted Qwen2.5-7B figures (EWC 44.22, O-LoRA 46.47) were right, but both fall below Seq-LoRA (51.54), and EWC below the untuned base (48.62).	L3Correction	`reports/Elastic weight consolidation deep dive.md`in the research archiveresearch_notes/Elastic weight consolidation deep dive/`reports/Fortifying knowledge in continual learning.md (corrected)`in the research archive
11.3	E4, H2 and H1b supported: at the dev-chosen λ = 1e5, unrelated retention rose 44.3 → 92.0% while edit success on contradicting updates fell 62.5 → 13.5%, stale paraphrases rose 10.8 → 81.7% and unpropagated 2-hop consequences 32.0 → 89.3%. Per decade of λ: retention +16.5 ± 3.9, stale paraphrase +28.4 ± 2.3, stale held 2-hop +21.8 ± 5.8 pp.	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive
11.4	E4, H1 gap rejected (kill rule K1): stiffness did not widen the gap between a direct edit and its consequences. The edit–paraphrase gap moved +1.2 ± 2.5 pp per decade of λ (threshold +3) and the edit–held-2-hop gap narrowed by 14.7 ± 6.6; edits, paraphrases and trained consequences fell together (−19.0, −20.2, −16.8 pp per decade).	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive
11.5	The new answer is written beside the old one at moderate stiffness: among successful direct edits, stale paraphrases rose 7.6 → 23.8 → 39.0% at λ = 0 / 1e4 / 3e4, and at λ = 1e4 the paraphrase log-prob gap went −2.53 → +0.78 nats with the old answer in the top 2 for 66.0% (vs 24.5%). At λ = 1e5 only 3–7 of 40 edits landed (K3), so blockage dominates there.	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive
11.6	EWC was far more intransigent than the KL anchor it approximates: at matched retention (92.0% vs 94.8%, n.s.), a KL anchor on old queries kept edit success 55.5 ± 8.0 pp higher (t = −19.2), and 47.5 ± 8.2 higher at 85% retention. Including the superseded facts' own queries in the KL anchor cost +18.5 ± 9.7 pp stale paraphrases and +14.6 ± 3.2 stale trained consequences at equal retention.	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive
11.7	E4, H4 rejected (kill rule K2): the per-fact Fisher ledger was a null. Releasing the superseded fact's Fisher and its 2-hop dependent's (4.7% of the mass) changed stale paraphrases by −1.0 ± 1.3 pp and stale held 2-hop by −0.7 ± 1.1, with edit success identical (13.5%); mass-matched random release and global decay did as well (within 0.7 pp). The superseded fact supplied only 0.41% of the stiffness its correction faced (0.51% with its dependent, 0.76% with every fact about the person).	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive
11.8	No targeted release helped (exploratory arms, declared before the test run): zeroing stiffness on the coordinates the update pushes hardest against matched random coordinates at the ledger's mass (edit −1.5 ± 2.8, stale paraphrase −1.2 ± 3.3); at 5× that mass (24% of the Fisher) stale rates fell 17–19 pp for −13.7 ± 6.1 pp unrelated retention, and targeting was worse than random on edit success (−5.5 ± 1.4).	L1	`reports/EWC stiffness and stale facts.md`in the research archive`research_notes/Elastic weight consolidation deep dive/experiments/E4_ewc_stiffness/experiments_results.md`in the research archive

12. Cross-generational teaching and examination

Nov 11, 2026 • 9 entries

A literature synthesis on older model generations as teachers, examiners and auditors of their successors in graded succession, and the first two of its eight proposed experiments, pre-registered and run once on the section 3 toy-KB testbed (10 seeds): T3, who supplies replay when the old data is gone, and T2, whether a parent's exam catches a successor's real capability loss.

Findings 12.1–12.9
#	Finding	Level	Recorded
12.1	No published study or prior ledger result shows that teaching and examination across generations preserve capabilities better than rehearsal; at best they match it, over short chains, with a check from outside the lineage. Agreement between generations cannot promote a claim (it cannot separate truth from coherent error, 10.6), and a parent grading its descendant is a lenient judge.	L3 synthesis	`reports/Cross generational teaching and examination.md`in the research archiveresearch_notes/Cross generational teaching and examination/
12.2	T3, H1 rejected: replay generated by the previous generation was half as stale on revised employers as an unpruned real buffer (0.138 vs 0.300, diff −0.163 [−0.198, −0.130]) and matched a pruned buffer (0.150; kill rule K2 fired), at 2.4 pp lower retention (0.951 vs 0.973; H3 supported). The teacher had already absorbed earlier stages' updates (labels 96.9% gold), so its replay carried revisions forward; a frozen parent several generations old is untested.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T23_toy_kb/experiments_results.md`in the research archive
12.3	T3: every filter that lowered staleness in generated replay cost retention. A teacher-confidence filter cut direct staleness 8.2 pp (0.138 → 0.055) for −9.0 pp retention (H4 rejected), KL soft targets 5.3 pp for −5.7 pp (H5 rejected); the record filter cut it 2.3 pp at no retention cost (n.s.). 2-hop consequences stayed 0.88–0.95 stale under every replay source.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T23_toy_kb/experiments_results.md`in the research archive
12.4	T2, H1 supported: across 320 candidate successors (121 with ≥ 5 pp real regression), an exam of the parent's 200 most confident facts keyed on its answers ranked regression at AUROC 0.897 vs 0.816 for a held-out test of new facts plus 5% old (diff +0.081 [+0.055, +0.109]). Gold keys reached 0.994. One-per-subject coverage items ranked best (0.968) but falsely failed 53.8% of non-regressed successors at a 95% pass mark, against 2.5% for confident items and 1.0% with superseded items removed (H2 as stated rejected: revisions were not the cause).	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T23_toy_kb/experiments_results.md`in the research archive
12.5	T2, H3 supported: on 40 never-trained 2-hop facts where the parent errs, the parent's answer key repeated 46.4% of its descendants' wrong answers against 12.4% for an independently pretrained examiner (per-seed diff 0.336 [0.288, 0.383]). A same-lineage examiner waves through about half the errors a successor inherited from it.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T23_toy_kb/experiments_results.md`in the research archive
12.6	T3b, H1–H4 supported: generated replay grows staler as the teacher ages. Direct stale rate 0.138 with a 1- or 2-stage-old teacher, 0.198 at 4 stages, 0.300 with the frozen pretrained parent (frozen minus lag 1: +0.163 [+0.130, +0.198]). The record filter brings the frozen parent back to 0.150, and the older teacher retains 2.0 pp more (0.971 vs 0.951). A frozen parent that labels its own training queries perfectly reproduces a stored buffer exactly, so H2 held by construction: generated replay's advantage in T3 came from the teacher having learned the revisions, not from generation.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T3b_teacher_lag/experiments_results.md`in the research archive
12.7	T5 (toy KB), H1a, H1b, H2 supported, H3 not: a newer generation contradicts 93.4% of its ancestor's outdated direct claims but only 17.5% of outdated 2-hop consequences. An older auditor's flags on its successor are 45.9% precise, and the supersession record raises that only to 50.0% (predicted ≥ 90%). Post hoc, most false flags are the auditor's own errors: of flags on unchanged facts, 50% were real regressions, 19% recoveries by the successor, 31% both wrong.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T5_reciprocal_audit/experiments_results.md`in the research archive
12.8	T4 Phase A (numpy), H1, H2 and H5 not supported; H3a, H3b and H6 supported. The best solver was the best teacher at every label-noise level (Spearman ρ 0.964, 0.929, 0.786 at 0, 20, 40% noise). Students beat overfitted teachers: at 40% noise a 40-epoch MLP solves 0.629 and teaches 0.719. Confidence filtering helped as a teaching filter (+1.4 pp, reversing the prediction). A 10% probe student ranks teachers at ρ 0.945 with the full one.	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T4_solve_vs_teach/experiments_results.md`in the research archive
12.9	T7 (toy KB), H1–H3 not supported and the kill criterion fired: choosing exam items by discrimination selected the same items whether discrimination was measured against agreement with the parent or against gold-graded ability (wrong-key share 0.030 both, vs 0.051 random). Forgetting, not revisions, dominated differences between successors. Discrimination-selected exams also ranked successors worse than random items (Kendall τ 0.65 vs 0.78).	L1 toy, 10 seeds	`reports/Cross generational teaching and examination.md`in the research archive`research_notes/Cross generational teaching and examination/experiments/T7_examiner_reward/experiments_results.md`in the research archive

13. Model self-knowledge and confident errors

experiments queued

Nov 11, 2026 • 7 entries

A literature synthesis on whether language models know what they don't know, and how confident errors arise and are caught. All entries are published results (L3); numbers are the authors' own, and several sources are unreviewed 2025–26 preprints. Six experiments were designed from it: SK1–SK4 are pre-registered and queued (SK1, SK2 and SK4 on local CPU, SK3 on local MPS) and SK5–SK6 are designed only. None has results yet.

Findings 13.1–13.7
#	Finding	Level	Recorded
13.1	Self-uncertainty signals measure instability, not error: semantic entropy flagged confabulations at average AUROC 0.790 over 30 task–model pairs (P(True) 0.698, token entropy 0.691), and exact-answer hidden-state probes reached AUC 0.83–0.95 on 7–8B models, but on TriviaQA errors the model repeats ("consistently incorrect") probe AUC fell to 0.59–0.67 (Farquhar et al. 2024; Orgad et al.).	L3 synthesis	`reports/Model self knowledge and confident errors.md`in the research archiveresearch_notes/Model self knowledge and confident errors/
13.2	The "do I know this?" gate keys on entity familiarity: in Claude 3.5 Haiku a default "can't answer" circuit is inhibited by known-entity features, and in Gemma 2 linear known/unknown-entity directions causally control refusal (Lindsey et al. 2025; Ferrando et al., ICLR 2025). Fine-tuning on unknown facts raised hallucination roughly linearly (β ≈ −8.3 vs +7.3 for known facts), and relabelling them "I don't know" lifted answered-question accuracy from 43.0% to 61.8% at 3 points less coverage (Gekhman et al. 2024).	L3 synthesis	same as 13.1
13.3	Training and grading reward guessing: 9 of 10 popular leaderboards Kalai et al. examined give no credit for abstaining, reasoning fine-tuning lowered abstention by 24% on average across 20 models (AbstentionBench), and RL on verifiable rewards improved in-domain calibration while worsening out-of-domain ECE (0.40 → 0.46 on Qwen2.5-7B).	L3 synthesis	same as 13.1
13.4	Proper-score rewards undo it: RLCR (correctness minus Brier of verbalized confidence) cut in-domain ECE from 0.37 to 0.03 at 62.1% vs 63.0% accuracy, and OOD ECE from 0.46 to 0.21; a discrete abstain action against a fixed penalty over-refused or was normalised away under GRPO. Calibration training lowers confident errors mostly by lowering confidence (a calibrated 4B model reached 5.8% SimpleQA accuracy).	L3 synthesis	same as 13.1
13.5	System guards inherit the model's blindness to confident errors: conformal methods certify population rates under exchangeability, not individual answers; in ClashEval six top LLMs adopted incorrect retrieved content over their own correct prior more than 60% of the time; and no cascade or retrieval-trigger paper reports confident-error rates before and after the guard. Only guards that add information (search-backed verification, an independent model family, structure, density for OOD inputs) catch stable wrong beliefs.	L3 synthesis	same as 13.1
13.6	Under conversational pressure confidence is nearly uninformative: a bare "Are you sure?" flipped 46% of answers (FlipFlop) and SycEval found sycophancy in 58.19% of rebuttal cases. A 2026 study of nine reasoning models reported verbalized-confidence ROC-AUC 0.57 for correctness, but that paper (arXiv 2602.13093) is internally inconsistent across its tables and versions, so only its direction is used.	L3 synthesis	same as 13.1
13.7	Against NeuroGenesys's own results, the literature confirms selector-abstention SA0, SA2, SA5 and SA9 and E1's finding that confidence is near random for repair (10.16), but partly contradicts the expert-votes rule "never use the voter's own confidence": sampling disagreement, probes and RL-trained confidence work on errors the model is unstable about, not on stable ones. E1's coherently-wrong rate (65.0% / 31.2%) has no external counterpart. Experiments SK1–SK4 are pre-registered and queued, with no results yet.	L3 synthesis	same as 13.1

14. Merging and splitting MoE experts

MS0 complete

Nov 11, 2026 • 12 entries

A literature synthesis on when to merge, split or spawn experts in continual learning with frozen or isolated experts, and the first of nine experiments pre-registered from it. Entry 14.1 is a re-reading of existing ledger results (8.18), not a new experiment; 14.2–14.7 are published results (L3), with numbers the authors' own. Entries 14.8–14.12 are MS0, a pre-registered zero-training reanalysis of the R0 Split-CIFAR-100 expert libraries (single consolidations, fusions, retirements and splits of a T = 10 library, scored under four routers, test seeds 0–9, run once). MS1–MS4, MS7 and MS8 are queued; MS5–MS6 are design freezes.

Findings 14.1–14.12
#	Finding	Level	Recorded
14.1	Re-reading of existing ledger results (8.18), not a new experiment: the Split-CIFAR-100 task-count sweep is a controlled uniform merge and split. The R0 cache's T = 5 tasks are the unions of adjacent T = 10 task pairs for all 10 seeds (5 of 5 pairs each; no other pairing matches; checked for MS0), so going 10 → 5 tasks merges adjacent expert pairs, each retrained from scratch on its pair's data. Under max-softmax selection that merge raised system accuracy 0.342 → 0.422 (selection accuracy +0.110, within-task accuracy −0.056); the matching uniform split, 10 → 20, lowered it to 0.208 (ε +0.144, within-task +0.020). In log terms the merge's selection gain (+0.27 in ln A_sel) is about 4.5 times its capacity loss (0.06). The merge-and-split reading was chosen after seeing 8.18 — post hoc.	L1 CIFAR-100 ResNet, 10 seedsPost hoc	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`reports/Merging and splitting MoE experiment plan.md`in the research archive`research_notes/Forgetting as selection at scale/phase1/analysis/results.md`in the research archive
14.2	Merging erases a choice the router was making: REAP puts the irreducible error of a summed-gate merge at about E[(g_i+g_j)²] · Var[r] · E‖f_i − f_j‖², while pruning error is bounded by the pruned expert's gate mass. At 50% expert reduction on Qwen3-30B-A3B, REAP pruning kept the coding average at 0.541 against a 0.558 baseline while merging fell to 0.364 (HC-SMoE) and 0.397 (M-SMoE); merging stayed competitive at 25% compression and on multiple choice (HC-SMoE 0.542 vs REAP 0.503 at 50%).	L3 synthesis	same as 14.1
14.3	Usage frequency is a poor merge or prune signal: on Mixtral, frequency-based pruning scored below random (60.77 vs 63.04 at six of eight experts; NAEE), and on Qwen3-Coder-480B at 50% it collapsed coding to 0.010 while REAP's gate-weighted activation norm held 0.619. Grouping signals moved from router logits (best in fine-tuned Switch-T5, COPA 68.00 vs 60.00; MC-SMoE) to expert-output similarity in retraining-free decoder LLMs (0.5459, router logits last at 0.3153; HC-SMoE). Calibration data matters: the same pruning rule kept Mixtral's GSM8K at 37.07 calibrated on MATH but 24.87 on C4.	L3 synthesis	same as 14.1
14.4	Weight-space merging works when experts share an origin: MagMax, with no exemplars, reached 84.16 on CIFAR-100/5 and 76.75 on CIFAR-100/50, and cut first-task forgetting from 25.7% to 9.8% relative to sequential fine-tuning, but its accuracy still fell monotonically with merged tasks (84.16 → 80.41 → 78.49 → 76.75 at T = 5/10/20/50). No continual-learning paper reports old-task accuracy immediately before and after a merge event.	L3 synthesis	same as 14.1
14.5	Splitting has one principled criterion: splitting steepest descent splits a unit whose splitting matrix has a negative minimum eigenvalue at a plateau, into exactly two equal-weight children along ±v_min, reducing loss by −ε²λ_min/2; a split-grown keyword-spotting net reached 95.36% with 282.6K parameters against DS-CNN's 94.85% with 413.7K (Liu et al. 2019). Nobody has computed it for MoE experts. Initialisation decides divergence: with a reused router, copies plus α = 0.01 noise suffice (Orthogonal Growth); with a new router, Drop-Upcycling's r = 0.5 re-initialisation reached 40.3 against 34.5 from scratch.	L3 synthesis	same as 14.1
14.6	Selection error, not forgetting, dominates outside the lab too: CN-DPM's classifiers barely forget (Split-CIFAR-100 per-task accuracy 55.42 at initialization, 55.24 at the end), but its generative gate picks the right expert only 48.18% of the time on 5-task Split-CIFAR-10 and 31.14% on 20-task Split-CIFAR-100, leaving end-to-end accuracy at 20.10 against 73.80 for iid-offline. Co-trained MoEs differ: in continual pre-training, penalty-balanced routers recovered within about 500 steps and the MoE forgot less than a dense model (FineWeb loss increase 0.037 vs 0.066 with 40% replay; Thérien et al.).	L3 synthesis	same as 14.1
14.7	Synthesis decision rules, not yet tested: merge a pair the router often confuses or barely distinguishes, keep a pair it reliably tells apart (merging it is forgetting), split only at a within-task plateau with loss-relevant multimodality and a router shown to separate the children, and spawn only from a buffered coherent novelty stream behind a held-out old-task ε gate. No published system implements these rules and none of their thresholds has been calibrated; the literature has measured neither merge-event forgetting, nor ε before and after a split, nor a head-to-head of grow, reuse and merge policies. Experiments MS0–MS8 are pre-registered; MS0 (14.8–14.12) tests the first two rules on cached libraries.	L3 synthesis	same as 14.1
14.8	MS0: one merge is a selection fix whose sign depends on the router. Replacing one adjacent pair of a T = 10 library by their union expert (retrained on both tasks' data) raised system accuracy under max-softmax routing by +0.0058 [+0.0033, +0.0084] (35 of 49 events positive; Holm p 0.004), gaining selection (+0.0084) and losing within-task accuracy (−0.0063). Under a confidence-free RanPAC router on ViT features (baseline accuracy 0.671, ε 0.178) the same merges cost −0.0068 [−0.0084, −0.0051]; the interaction is −0.0125 [−0.0145, −0.0106].	L1 CIFAR-100 ResNet, 10 seeds	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`research_notes/Merging and splitting MoE experts/experiments/MS0_single_merge_split_reanalysis/experiments_results.md`in the research archive
14.9	MS0: router confusion, not representation similarity, predicts which merge pays. The pair's validation confusion under max-softmax correlated with its test merge gain at Spearman ρ +0.645 [+0.518, +0.790] (pre-registered threshold 0.30). Feature CKA (+0.02), ViT prototype overlap (−0.22) and the weaker parent's accuracy (−0.21) did not. Under the RanPAC routers no predictor reached ρ 0.30.	L1 CIFAR-100 ResNet, 10 seeds	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`research_notes/Merging and splitting MoE experts/experiments/MS0_single_merge_split_reanalysis/experiments_results.md`in the research archive
14.10	MS0: fusion beat consolidation, falsifying the pre-registered prediction (H0f). Adding the union expert alongside its two parents raised max-softmax accuracy by +0.0115 [+0.0104, +0.0126] (every event positive), against +0.0058 for replacing the parents; its negative-flip rate was 0.007 against 0.036 and its worst single-class drop −0.001 against −0.434. Consolidation's mean gain hid class-level losses above the pre-registered 0.01 flip ceiling. Under class-level RanPAC routers fusion equals consolidation on routing by construction.	L1 CIFAR-100 ResNet, 10 seeds	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`research_notes/Merging and splitting MoE experts/experiments/MS0_single_merge_split_reanalysis/experiments_results.md`in the research archive
14.11	MS0: one split hurts under self-report routing and usually helps behind a strong router. Replacing one T = 10 expert by its two T = 20 halves cost −0.0171 [−0.0197, −0.0150] under max-softmax (99 of 100 events negative), with selection −0.0195 against within-task +0.0049. Under RanPAC on ViT features 79% of splits raised accuracy (1% under max-softmax), mean +0.0022 [+0.0014, +0.0029]. Of the candidate split triggers, only the expert's task-given error tracked the split's value (ρ +0.33 max-softmax, +0.26 RanPAC-ViT); two-means silhouette, an Anderson–Darling unimodality test, k-means agreement and load did not (\|ρ\| ≤ 0.09 under max-softmax; secondary analysis, MS2 is confirmatory).	L1 CIFAR-100 ResNet, 10 seeds	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`research_notes/Merging and splitting MoE experts/experiments/MS0_single_merge_split_reanalysis/experiments_results.md`in the research archive
14.12	MS0: no expert was a liability, and the pre-registered utility rule (λ 0.05, μ 1.0, lower confidence bound, negative-flip ≤ 0.01 and worst-class ≥ −0.20 gates) was too conservative. Leave-one-out marginal contributions were positive for every expert under every router (minimum +0.0025 max-softmax, +0.048 RanPAC-ViT). The rule accepted 0 of 50 consolidations under max-softmax (missing 2 that paid on held-out data) and 16 of 50 under first-session RanPAC (1 false accept, 21 missed gains, +0.0019 per event).	L1 CIFAR-100 ResNet, 10 seeds	`reports/Merging and splitting MoE experts.md`in the research archiveresearch_notes/Merging and splitting MoE experts/`research_notes/Merging and splitting MoE experts/experiments/MS0_single_merge_split_reanalysis/experiments_results.md`in the research archive

15. Optimal router architecture

Nov 12, 2026 • 6 entries

R0–R5 of the router study, pre-registered (v1.0) before any test data. One SlimResNet18 expert per task on Split-CIFAR-100 (90% of each task's data), T ∈ {5, 10, 20}, seeds 0–9, 30 runs cached once; routers fitted offline on 250 cached training images per class and scored once on the test split. Paired bootstrap over seeds, Holm over H1–H6. Sanity gate passed: within-run max-softmax ε at T = 10 was 0.654 (8.18: 0.644). The LLM skill-routing arm (R6) is queued.

Findings 15.1–15.6
#	Finding	Level	Recorded
15.1	H1 supported, small, against the pilot's prediction of a tie. A confidence-free RanPAC router over the first task's frozen backbone (stream-legal, no pretraining) had lower selection error than the experts' energy score at T = 10: ε 0.614 vs 0.627, Δ +0.014 [+0.003, +0.027], Holm p = 0.003. ε stayed above 0.6, so without pretraining the representation, not the rule, is the binding constraint.	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive
15.2	H2 supported: the same rule over ImageNet ViT-B/16 features cut ε at T = 10 from 0.627 (energy) to 0.178, Δ +0.450 [+0.440, +0.460]. ImageNet's classes overlap CIFAR-100's, so this arm shows what a router can do given a strong representation, not that one can be learned from the stream.	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive
15.3	H3 supported: from 5 to 20 tasks, ε grew by +0.030 for the ViT RanPAC router (0.159 → 0.189) against +0.257 for energy (0.514 → 0.771), Δ +0.227 [+0.213, +0.239]. The stream-legal router's growth (+0.235) was close to energy's.	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive
15.4	H4 not supported: abstaining on the router's top-1 minus top-2 margin ranked errors slightly worse than the chosen expert's own max-softmax at T = 10 (AURC 0.158 vs 0.153, Δ −0.006 [−0.012, −0.000]; 0.452 vs 0.429 for the stream-legal router, unadjusted). Inside one expert, confidence still predicts within-task correctness; the MNIST result (9.3) compared the margin with confidence across experts. At T = 20 the ViT margin was better (0.140 vs 0.157, unadjusted).	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive
15.5	H5 supported but negligible: top-2 soft mixing beat argmax routing with the ViT router by +0.003 [+0.002, +0.004] accuracy (0.674 vs 0.671); with the weaker stream-legal router the gain was +0.031 (unadjusted). Mixing pays only when routing is uncertain.	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive
15.6	H6 supported: with the ViT features, a 100-way linear readout and no experts at all scored 0.803 at T = 10, against 0.671 for the best routed system, Δ +0.132 [+0.127, +0.138]. Under the strong router, experts were right on only 76–85% of the inputs routed to them (89–98% under max-softmax, which routes them their easy inputs). At this scale, the frozen-expert design adds nothing over a readout of the shared features. Secondary: a replay-buffer logistic router was 0.15–0.17 worse in ε than RanPAC in both spaces (consistent with 9.5).	L1 CIFAR-100 ResNet experts, 10 seeds	`research_notes/Optimal router architecture/results/results.md`in the research archive

16. Civilization as a superintelligence model

in progress

Nov 12, 2026 • 10 entries

Experiments from the synthesis on human civilization as a model for superintelligent systems: societies of LLM agents, shared knowledge stores and network structure. CS3 (does a visible hub, measured-error weighting or lineage mixing change collective estimation?) was pre-registered (v1.0, frozen before the main run) with lineages A = Claude Haiku 5.5 and B = GPT-4.1-mini via the API; lineage C (open weights) is deferred until GPU access. CS5a and CS2a are CPU tiers: a toy shared-store test (non-confirmatory) and the network gate for CS2. CS1 was frozen as v2.0 on 2026-11-12 at a reduced four-cell scope and is running on the Claude plan path; CS2b's conformity gate failed (16.10) and its main run is deferred; CS4–CS6 await later versions.

Findings 16.1–16.10
#	Finding	Level	Recorded
16.1	CS3, H2 holds: a visible hub decides which way a society of LLM agents moves. Revising estimates after seeing a misleading hub's answer raised collective error by 15.2%, and a truthful hub cut it by 26.3% (difference 0.41, Holm p < 0.001). The swing is smaller than in the human study it replicates (0.62), mainly because a misleading hub hurts less (+15% vs +19%) and a truthful one helps less (−26% vs −43%).	L2 Claude Haiku 5.5 + GPT-4.1-mini agents	`reports/Civilization as a superintelligence model.md`in the research archive`research_notes/Civilization as a superintelligence model/experiments/experiments_results.md`in the research archive
16.2	CS3, H1 fails: without a hub, decentralised exchange did not lower collective error (ΔE +0.007 [−0.016, +0.031] on Wikidata birth years, pre-registered bar ≤ −0.05; Holm p = 1.0), although mean individual error fell 1.6%.	L2 Claude Haiku 5.5 + GPT-4.1-mini agents	same as 16.1
16.3	CS3, H3 fails: weighting agents by their measured error was no better than weighting by self-reported confidence (error ratio 1.001 [0.987, 1.016], bar ≤ 0.97; Holm p = 1.0).	L2 Claude Haiku 5.5 + GPT-4.1-mini agents	same as 16.1
16.4	CS3, H4 holds, the largest effect in the run: mixing Claude and GPT lineages cut the error correlation between agents from 0.864 to 0.763 (Wikidata) and 0.969 to 0.809 (MMLU-Pro), a mean drop of 0.131, and lowered median error from 44.0 to 35.2 years and from 0.242 to 0.169 (Holm p < 0.001).	L2 Claude Haiku 5.5 + GPT-4.1-mini agents	same as 16.1
16.5	CS3, H5 fails, as pre-registered: deliberation with measured-error revision did not beat sealed answers at equal budget. Against a 24-sample same-model median it scored 0.997 of its error (bar ≤ 0.97; p = 0.85), and against the best single lineage 1.28× (single-lineage society) and 0.98× (mixed, p = 0.63). The 0.997 cell was first NaN; an item whose baseline samples all failed the answer format was dropped from that comparison only, and all 168 other frozen numbers reproduced exactly (deviation D13). 277,912 calls, $9.73.	L2 Claude Haiku 5.5 + GPT-4.1-mini agents	same as 16.1
16.6	CS5a: in a shared store fed by 12 tiny transformers in 3 lineages with 30% planted false facts, the share of false claims promoted after 8 rounds ordered NONE 0.426 > SAME 0.346 > AGREE 0.314 > CROSS 0.258 > EXEC 0.072. Peer agreement removed only 26% of false claims (H1 holds), same-lineage review admitted 75% of the planted lineage's false claims (H2 holds), and an executable registry check beat cross-lineage re-derivation by 0.186 with 1.9× its throughput (H4 holds). Cross-lineage re-derivation cut errors only 25% relative to same-lineage review (H3, ≤ 0.35 predicted, fails), and false claims did not compound over rounds (H5 fails). Non-confirmatory tier, 10 seeds.	L1 toy KB, 10 seeds	same as 16.1
16.7	CS2a (gate for CS2): on NK landscapes (N = 20, 16 agents, 1,000 landscapes per K) sparse networks beat the fully connected one under copy-the-best (+0.011 at K = 5, +0.016 at K = 10), the Lazer–Friedman sign. The conformity half failed in v1 (rule defect) and passed only degenerately in v2: under the Barkoczi–Galesic copy-the-majority rule agents copied in ≤ 0.22% of agent-rounds, because identical 20-bit solutions almost never co-occur. Independent search beat every copying arm by about 5 points.	L1 NK-landscape simulation	same as 16.1
16.8	CS2a v2, exploratory E2 (stated before the run): at Barkoczi–Galesic's own parameters (N = 15, K = 7, 100 agents, 200 rounds, 200 landscapes) their reversal reproduces. The sparse lattice wins under copy-the-best, the full network wins under copy-the-majority (best agent +0.0015, p = 0.048; mean agent ≈ +0.014), and the interaction is D = +0.0116 [+0.0071, +0.0162], p < 0.001. The degenerate v2 pass was a property of the 20-bit, 16-agent setting, not the rule.	L1 NK-landscape simulation, exploratory	same as 16.1
16.9	Running Haiku 5.5 through the Claude plan (`claude -p`) instead of the API is not a transport change: Claude Code injects about 550 tokens of identity and user context, which lowered Wikidata birth-year accuracy from 23% to 16% (p = 4×10⁻⁷) and raised first-call code format failures from 6.3% to 25%, with replies asking the user for the rule. A no-clarification clause cut failures to 10.4% (all output-limit cutoffs), and with a 2,500-token limit the plan path reproduced the API's CS1 depth curve (VOTE@B 0.92, 0.50, 0.33, 0, 0 vs 0.94, 0.44, 0.25, 0, 0 at d = 1–5). CS1 onward uses the plan path as its own lineage-A condition with its own drift reference.	L0 plan vs API backend	same as 16.1
16.10	CS2b conformity gate fails: on graded ListDSL program-search tasks (partial credit by construction, 5 of 24 candidates passing a difficulty screen), 16 Haiku agents copied a Barkoczi–Galesic majority 0.25 times each over 24 rounds in a fully connected network and 0.22 in islands, against a pre-stated bar of ≥ 1. A majority existed in 88% of agent-rounds but was strictly better than the agent's own program in only about 1%. With CS2a (degenerate on 20-bit strings) and its replication (the reversal appears only at the original NK parameters), conformity as a network mechanism depends on the landscape: it is untestable on program search, and CS2b's conformity half is reported as not testable.	L2 Claude Haiku 5.5 agents, 4 tasks	same as 16.1

Which results are still pending?

Promised by the plans and pre-registrations above; each will be appended to its study when it lands.

Optimal router architecture R6 (queued behind the coherence session's SK runs): does embedding kNN routing beat Llama 3.1 8B in-context routing over 64 skills, including on held-out request templates?
Cross-generational teaching T1a and T1b (pre-registered and queued on CPU): does a false rule (GSM8K) or a false fact (MQuAKE) planted in a parent's lessons propagate down a four-generation LLM teaching chain, and does a checker or a 20% knowledge-base spot-check stop it where self-confidence and same-family judges do not?
Graded succession by earned authority, experiments S1–S5 (pre-registered and queued on CPU behind R6): does evidence-gated authority beat a hard switch and a plain average, and does a protected constitution hold under a dominant successor?
The rank-one retirement edit (7.1) at Qwen 1.5B, with an optimised adversarial prompt search and linear-probe decodability of the old fact.
Long streams at 1.5B with more seeds and the pipeline run to completion (the current run is seed 0 only, truncated at 200 updates).
Pre-registered study Phase 0 freeze and Phase 1 gap check.
Model self-knowledge experiments SK1–SK4 (section 13), pre-registered and queued.
Merging and splitting experts MS1–MS8 (section 14), pre-registered and queued; MS0 is complete (14.8–14.12).
Civilization as a superintelligence model CS1, CS2b and CS4–CS6 (section 16): v2.0 pre-registration after the plan-path re-baseline; CS2b's conformity half is untestable on program search (16.10). The CS1 main run (v2.0 frozen 2026-11-12 at four cells: VERB vs SUM, SOLO@B, VOTE@B, 3 seeds) is running on the Claude plan.

What are the standing caveats?

All L1 and L2 results come from small models, synthetic or counterfactual facts, 3–5 seeds and uncorrected multiple comparisons; several learning rates were chosen from single pilots or on an evaluation seed. Published (L3) numbers are self-reported by their authors and, for most 2025–26 methods, unreplicated. Nothing here has been shown above 1.5B by us or above ~14B by anyone.

Last updated Nov 12, 2026. The ledger is kept as data in the lab's repository and this page, the Markdown ledger and the site map are generated from it.

Findings ledger

How is each finding graded?

1. LLMs editing open-model weights

2. Memory, world models and source trust

3. Order of addition and companion knowledge

4. Seven anti-forgetting methods, head to head

5. Consequence-consistent updates, pre-registered study

6. Long update streams

7. Weight editing to prevent forgetting

8. Genesys program: selection and verifier tests

9. Stopping confident wrong expert votes

10. Logical coherence of LLM world models

11. Elastic weight consolidation and contradicting updates

12. Cross-generational teaching and examination

13. Model self-knowledge and confident errors

14. Merging and splitting MoE experts

15. Optimal router architecture

16. Civilization as a superintelligence model

Which results are still pending?

What are the standing caveats?

Leadership

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

Will Schulz