NeuroGenesysHow it compares
Everyone scores tool selection. Few calibrate it.
Agent evaluation and observability platforms already trace calls and score them with LLM judges. Use them. Selection reliability sits on top: it turns those logs into a calibrated error rate per selector and says why an agent got worse. It complements a tracing platform rather than replacing it.
Common approaches
| Approach | What it reports | What it leaves open |
|---|---|---|
LLM-judge tool-selection score e.g. Galileo Tool Selection Quality | Share of “yes” votes from repeated judge calls on whether the right tool was chosen. | No stated judge accuracy, calibration or confidence interval; no link between the score and a regression's cause. |
Canaries, regression sets and release gates e.g. Arize continual-learning guidance | Whether quality on a standing set drops after an update. | Does not say whether the drop is forgetting or mis-selection, so the fix (retrain vs. repair the selector) is a guess. |
Trajectory precision and recall e.g. Vertex AI agent evaluation | How closely the agent's tool calls match a reference trajectory. | Needs reference trajectories, and does not say whether a drop is forgetting or mis-selection. |
Tool-correctness metric e.g. DeepEval Tool Correctness | Whether the expected tools were called. | Checks which tools were called, not whether the tool would have succeeded if chosen, so capability and choice stay mixed. |
Based on each product's public documentation in November 2026. Products change; tell us if something here is out of date.
What selection reliability adds
- Calibrated selection error ε
- per router, retriever or judge, with 95% intervals, on the pool it actually faces
- Attribution
- forced accuracy vs. ε: a regression is forgetting, mis-selection, both, or not yet attributable
- Scaling
- ε against pool size, and the largest tool, skill or memory library that keeps accuracy on target
- Gate
- ALLOW / BLOCK / NEED MORE DATA for an update, usable in CI
- Judge exploitability
- how often a wrong answer outscores every right one, and the Best-of-N budget past which more samples hurt
- Selector repair
- standardised scores, a deferral margin and a verifier shortlist, judged on held-out decisions
Why not build it in-house?
You can, and the open selrel package is a start. The hard parts are the statistics (intervals at the small error rates and counts selection logs produce, judge-error correction, estimating a filter on data it was not fitted to) and neutrality: a vendor grading its own router or memory is not an independent number.
