NeuroGenesysFix the selector, not the skills
When the skills still work but the router picks the wrong one, retraining the skills does not help. This tool fits three cheap repairs to the selector. First, it puts each candidate's scores on one scale, so a habitually confident expert stops winning picks it should lose. Second, it sets a margin below which a pick is deferred to a verifier, a larger model or a person. Third, it finds the smallest shortlist a verifier needs to see.
Every choice is fitted on one half of the log and reported on the other half. A filter must be estimated, never trained on the data it judges. Background: Catastrophic forgetting is a verifier-exploitability problem. To check whether a regression is mis-selection in the first place, use selection-reliability.
Tune · from a decision log with scores
One JSON object per decision with a targetand the selector's scores, either as scores ({candidate: score}) or as selection/v1 candidates ([{id, score}]). The scores must include the target. Nothing leaves your browser.
Run it locally (Python, standard library only)
pip install selrel # or the tarball on the selection-reliability page
selrel simulate --mode overconfident --out d.jsonl
selrel tune d.jsonl --target-eps 0.02 --out selector.json
# in the agent
from selrel import Selector
sel = Selector(**json.load(open("selector.json")))
choice, margin = sel.select(scores) # choice is None: defer- README.mdlog format and limits
- selrel/selector.py
