NeuroGenesysDid your agent forget, or pick wrong?
Agents get worse as they learn. New tools, skills and memories crowd the router and the retriever, and the right capability is still there but no longer chosen. Teams then retrain what was never broken. NeuroGenesys measures how often your selectors pick wrong, tells forgetting from mis-selection, and gates updates before they ship.
Four checks
Diagnose a regression
Split a drop into forgetting (the skill fails even when forced) and mis-selection (the skill works but the selector picked another). Each has a 95% interval, and forgetting is judged only on skills that existed before.
selection-reliabilityGate the next update
Before a new skill, memory or adapter ships, compare a shadow run with production. ALLOW, BLOCK or NEED MORE DATA, with exit codes for CI.
reliability gate (API, MCP, selrel)Fix the selector, not the skills
Put each candidate's scores on one scale, defer close calls to a verifier or a person, and find the smallest shortlist a verifier needs. Everything is fitted on half your log and judged on the other half.
selector-tuneAudit a judge or reward model
Measure how often a wrong answer outscores every right one, where Best-of-N stops paying, and whether the verifier only looks good on problems it was trained on.
exploitability-auditHow it works
- 01
Log
One JSON line per selection: what was chosen, from how many, the score of each candidate and, when known, the right answer. selection/v1, or your existing OpenTelemetry/OpenInference spans, Anthropic tool search or Mem0 search results.
- 02
Probe
Force the right candidate on a few hundred queries and record whether the task succeeds. `selrel probes` picks which ones.
- 03
Measure
Selection error ε per selector and pool size, with intervals, plus a forgetting vs. mis-selection verdict on each regression.
- 04
Act
Gate the update, tune the selector, cap the memory or tool library, or retrain only when forced accuracy actually fell.
Where it runs
- In the browser: paste a log into the research tools. Nothing is uploaded.
- Locally: the open
selrelPython package, standard library only, with a CLI that fits in CI. - Over the API or MCP: metered calls for agents and pipelines; see developers.
- Hosted: send decision events and watch ε per selector from your account.
Design partners
We are taking two or three teams that run agents in production for an eight-week program at no cost: memory and retrieval layers, platforms that ship frequent model or skill updates, and teams with large tool libraries or reward models. You share decision logs around one update and a few hundred forced probes. You get the hosted monitor, the gate and the audits, founder-led support, and a written verdict, including "no selection problem here" if that is what the data says.
Get in touchWhere forgetting is mis-selection, and where it isn't
"Forcing" a skill means skipping the agent's choice and handing the query straight to the skill, tool, document or expert that is known to be right. If the forced run still succeeds, the capability survived and the agent chose wrong. That test only makes sense where there is a separate choice to skip.
Where the claim holds
Systems with a separate selection step, where the old skill sits in a component that later training doesn't overwrite:
- Class-incremental learning with per-task experts or heads (our Split-MNIST result, and Kim et al.'s decomposition).
- Agents choosing among tools, skills or adapters, for example frozen LoRA adapters.
- Retrieval over external memory, where the memory is still there and the retriever picks the wrong item.
- Mixture-of-experts models where the experts are frozen or change little and the router drifts.
Where it doesn't
One network whose shared weights are fine-tuned on task A, then task B. Training on B can genuinely overwrite what A needed: that is real forgetting, and there is no separate component to force. Work on mixture-of-experts continual tuning also finds the experts themselves drift, not only the router. The fix there is retraining, rollback or regularisation, and our verdict reports it as forgetting.
The middle ground
Even in a single network, much measured forgetting is shallower than it looks: retraining only a simple readout on the network's internal features often recovers the old task (Davari et al., 2022), so the knowledge is still in the representation and the output layer no longer reads it out. That is an analogue of mis-selection, tested with a readout probe rather than by forcing a component, and it is a weaker claim.
What we claim, and what we don't
Splitting accuracy into choosing the right expert and the expert being right is prior art (Kim et al., NeurIPS 2022). What we add is one calibrated selection error across routers, retrievers and judges, measured on the pool each actually faces, and a gate built on it. Production agents are increasingly modular (tool libraries, memory stores, routed models, adapters), which is where the claim is strongest. Our own evidence is still small-scale: selection error rose as tasks grew on Split-MNIST (0.21 to 0.57 from 2 to 10 tasks) and, with frozen experts, on Split-CIFAR-100 (0.53 to 0.79 from 5 to 20), where the whole accuracy loss was selection. The agent-scale study is the next piece of work. See how this compares and the research.
