# selrel: selection reliability (v0.2) When an agent gets worse after it learns something new, the skill is often still there and the selector picked wrong. `selrel` measures how often the selector picks wrong (ε), on the pool it actually faces. It tells forgetting from mis-selection before anyone pays to retrain, and it sets safe budgets. It works for any selector: a skill or adapter router, a retriever over a memory, a tool picker, or a verifier choosing among candidates. It uses the standard library only (Python 3.9+), and its estimators are ports of the hosted service's, checked to give the same numbers. pip install selrel # installs the `selrel` and `selrel-cloud` commands selrel demo # milestone one in miniature, on a synthetic library The PyPI distribution is named `selrel` (the import name too). If it is not on PyPI yet, or you want exactly the version deployed on the site, install the tarball built from it: pip install https://neurogenesys.ai/tools/selection-reliability/selrel-latest.tar.gz From a checkout of this folder, `pip install .` works too. ## The model A_routed ≈ (1 − ε) · A_forced + ε · A_wrong - `A_forced` is accuracy when the correct candidate is forced. A drop here is **forgetting**: the skill itself degraded. - `ε` is the rate at which the selector picks something other than the target. A rise here, with `A_forced` holding, is **mis-selection**. The fix is the router, the retriever or the budget, not the weights. Forgetting is always judged on the candidates that existed at baseline. When a library grows, new skills change the mix, and pooled forced accuracy can move with no skill degrading at all. ## Logging decisions One JSON object per line, one line per selection: ```python from selrel import DecisionLog log = DecisionLog("decisions.jsonl") log.record(chosen=skill, pool_size=len(library), target=gold_skill, routed_ok=passed, forced_ok=passed_when_forced, tag="lib-v7") ``` | field | needed for | |---|---| | `chosen`, `pool_size` | always | | `target` | ε (labelled traffic or evals) | | `routed_ok` | routed accuracy | | `forced_ok` | telling forgetting from mis-selection; probe a sample of queries | | `scores` | score-level ε, which moves before the argmax starts flipping | | `tag` | splitting windows (model, library or memory version) | Spend forced probes on the skills that existed at baseline: a fixed probe set of old skills gives the forgetting check most of its power. ## Commands selrel report decisions.jsonl ε and accuracy, overall and by pool size, with the ε(K) trend selrel diagnose base.jsonl current.jsonl verdict: mis-selection, forgetting, both, stable selrel diagnose log.jsonl --baseline v6 --current v7 selrel gate before.jsonl after.jsonl ALLOW / BLOCK / NEED MORE DATA for a skill or memory update (anytime-valid) selrel gate before.jsonl after.jsonl --paired --mode fixed same probe queries replayed; one-look intervals selrel ppi events.jsonl ε from a few gold labels plus many judge verdicts (PPI++) selrel plan --eps 0.05 --change 0.02 --paired --flip-rate 0.04 labelled decisions a gate needs selrel pull langfuse --since 2026-11-01T00:00:00Z --out events.jsonl events from Langfuse observations selrel pull phoenix --project default --out events.jsonl events from Arize Phoenix spans selrel budget candidates --p .1 --dprime 2 --eps .01 safe Best-of-N budget N*(ε) selrel budget library decisions.jsonl --target 0.8 largest library that keeps routed accuracy on target selrel simulate --mode forgetting --out sim.jsonl selrel audit candidates.jsonl is the verifier exploitable? measured ε, ε*, empirical N* selrel tune decisions.jsonl --target-eps .02 --out selector.json fix the selector, judged on held-out data selrel tune decisions.jsonl --defer-cost 1 --error-cost 20 cost-optimal deferral instead of a target ε selrel probes decisions.jsonl --budget 200 --baseline v6 --p-base .9 which queries to force-probe, and how many selrel run-probes decisions.jsonl --queries q.jsonl --agent my_agent:run --out probed.jsonl run them and record forced_ok selrel report decisions.jsonl --html report.html standalone one-page report to share selrel simulate --mode bon --out bon.jsonl (also --mode overconfident, for tune) ## Auditing a verifier (`audit`) Log every scored candidate of a Best-of-N run, one JSON object per line: {"problem": "q17", "score": 3.2, "correct": false, "pool": "prefilter-a", "seen": false} `problem`, `score` and `correct` are required. The audit measures everything on your data instead of assuming a score model: - **problem ε**: the share of problems with both right and wrong answers on which some wrong answer scores at least as high as every right one; - **candidate ε**: the share of wrong answers scored at or above their problem's best right answer, the exploiter rate of the verifier-bottleneck model, compared against **ε\* = p/(1−p)**, below which consensus among high scorers holds; - **the Best-of-N curve**, exact over random N-subsets of each problem's candidates. Its peak is the safe budget N\*, and the drop to the largest N comes with a paired 95% interval over problems. A drop whose interval clears zero gives the verdict MORE SAMPLES HURT. Ties count against the verifier. ε is reported per `pool`, because it belongs to a verifier and the pool it faces. If `seen` marks problems the verifier was trained or tuned on, the audit compares seen against unseen problems. An audit on the verifier's own training data understates ε. ## Tuning a selector (`tune`) `tune` reads an ordinary decision log whose records carry `scores` (which must include the `target`). It fits three repairs to the selector, not to the skills: 1. **Per-candidate standardisation**: z-score each candidate's scores with its own mean and spread, so a habitually confident expert stops winning picks it should lose; 2. **Deferral**: the smallest top-two margin whose accepted picks keep the Wilson upper bound on ε at or below `--target-eps`; below it, defer to a verifier, a larger model or a person; 3. **A shortlist**: the smallest k whose top-k holds the target often enough, so a verifier judges k candidates instead of the whole pool. Records alternate between a calibration half and a held-out half in log order. Every choice is fitted on calibration and every reported number comes from held-out decisions: a filter must be estimated, never trained on the data it judges. `--out` writes the fitted selector as JSON. Load it with `Selector(**...)` and call `select(scores)`, which returns `(choice, margin)` and gives `None` for a deferred pick. With `--defer-cost` and `--error-cost`, `tune` picks the margin that minimises deferral cost plus the cost of accepted mistakes on the calibration half, and reports both costs per decision on the held-out half. ## Planning probes (`probes`) Forced probes are what separate forgetting from mis-selection, and they cost a second run each. `probes` lists unprobed, labelled queries to probe next. It uses only targets that existed in the `--baseline` window and spreads the budget so the least-probed targets come first. With `--p-base` (baseline forced accuracy) it also prints how many probes per window detect a drop of `--delta` at the requested power. ## Running probes (`run-probes`) `run-probes` runs the planned probes against your agent and fills in `forced_ok`. You don't write a harness. You mark where the agent selects, once, and selrel replays each planned query with the selector bypassed and the target pinned. Mark the selection point with whichever hook fits your stack. Outside a probe, every hook is a no-op: ```python from selrel import pinnable, tool_choice, pinned_tools, pin_retrieval @pinnable # your own router: returns the chosen skill def route(query, library): ... # Anthropic tool use: force the pinned tool client.messages.create(model=..., tools=tools, messages=msgs, tool_choice=tool_choice("anthropic", default={"type": "auto"})) # OpenAI Chat Completions ("openai-responses" for the Responses API) client.chat.completions.create(model=..., tools=tools, messages=msgs, tool_choice=tool_choice("openai", default="auto")) # Frameworks that take a tool list (LangGraph, agent SDKs, an MCP client's tools/list) agent = create_agent(model, tools=pinned_tools(tools)) # A retriever over a memory: return only the pinned item search = pin_retrieval(memory.search, fetch=memory.get) ``` Then store the inputs of your labelled queries, one per line: {"query_id": "q17", "input": {"messages": [...]}, "expected": "..."} and run: selrel run-probes decisions.jsonl --queries queries.jsonl --agent my_agent:run \ --baseline v6 --budget 200 --workers 8 --out probed.jsonl - **Calling the agent.** `--agent` names a function that takes one query's `input` and returns the agent's output. It can be sync or async. - **Grading.** `--grade` names a function `(output, case) -> bool`. Without it, the output is compared to `expected` (strings trimmed and case-folded). Return `None` to leave a probe ungraded. - **Choosing probes.** They are chosen exactly as `probes` chooses them, with `--budget` and `--baseline`. `--dry-run` shows the plan without running. - **Writing results.** The merged log goes to `--out`, with `forced_ok` written onto each probed line, matched by `query_id` and `target`. Every other field is kept as written, `event_id` and other selection/v1 fields included, so `diagnose`, `gate` and the hosted service read it directly. - **Checking each probe.** A probe only counts if the agent actually consulted a pinned hook. Otherwise it is reported as `pin_unused` and left out, because its outcome says nothing about the target. Errored and ungraded probes stay unprobed, so the next run picks them up. `--results` writes one line per probe with its status. - **Concurrency.** Probes run in their own context, so `--workers` can run them concurrently without pins leaking between them. With several selectors, pass `--selector router` (and the hook's name) to pin only one of them. From Python, the same steps are `cases_from_log`, `run_probes` and `merge_file` (or `merge_results` on `Decision` lists). Hosted projects store events append-only, so run the probes on the local log first, then `selrel-cloud push probed.jsonl`. `report`, `diagnose`, `gate`, `ppi`, `plan`, `audit` and `tune` take `--json`. To send decision events to the hosted monitor instead of keeping them in a local file, see [docs/CLOUD.md](docs/CLOUD.md). ## Gating an update (`gate`) `gate` decides whether a router, retriever, tool or memory update can ship, from labelled decisions before and after it. It is a non-inferiority test on ε with tolerance `--delta` (default 0.02): - **ALLOW** if the interval for the rise in ε lies below `delta`; - **BLOCK** if the whole interval is above zero (ε rose); - **NEED MORE DATA** otherwise. Two choices matter: - `--mode anytime` (the default) uses empirical-Bernstein confidence sequences (Waudby-Smith & Ramdas 2024). They stay valid however often you re-run the gate as logs arrive, which is what CI and monitors do. `--mode fixed` uses Newcombe intervals (or a paired normal interval), valid for one look only and narrower. - `--paired` compares the same probe queries replayed before and after. It needs far fewer labels. Decisions are matched by `event_id`, then `query_id`, else by line order (which then needs equal counts). When both logs carry `forced_ok`, forced accuracy on the candidates that existed before is checked in the same mode: a significant fall beyond `--forced-tol` BLOCKs, and an interval that can't rule one out turns ALLOW into NEED MORE DATA. The ε verdict alone is the hosted gate's (the `reliability_gate` API op and `selrel-cloud gate`); the forced check is an addition of the local command. `--json` prints the whole result. `gate` exits 0 for ALLOW, 2 for BLOCK and 3 for NEED MORE DATA, so it can sit in CI in front of an update (see [GitHub Action](#github-action)). v0.1's gate was fixed-sample only, with `--eps-tol` (still accepted as an alias of `--delta`). Pass `--mode fixed` for one-look intervals. In Python, `gate(before, after, mode=..., paired=...)` takes `Decision` lists and `gate_eps(before_flags, after_flags, ...)` takes mis-selection booleans. ## Judge-assisted ε (`ppi`) Gold targets are expensive. An LLM judge's verdict that a pick was wrong is cheap but biased. Log it as `judge_wrong` (0 to 1, a selection/v1 field) on as many decisions as you like, and add `target` to a random sample of those same judged decisions: {"chosen": "doc-7", "pool_size": 40, "judge_wrong": 0.1, "target": "doc-7"} gold + judge {"chosen": "doc-2", "pool_size": 40, "judge_wrong": 0.8} judge only `selrel ppi events.jsonl` estimates ε with PPI++ (Angelopoulos, Duchi & Zrnic 2023). The judge's weight λ is tuned from the data and clipped to [0, 1], so the interval stays valid whatever the judge's quality. A good judge narrows it. A useless one gets λ ≈ 0, and the estimate falls back to the gold labels. The output also shows the gold-only interval and how many gold labels the judge is worth. The gold sample must be random among the judged decisions for the correction to hold. The hosted report shows this estimate from 20 gold-and-judge decisions; locally it runs from 2 and warns below 20. In Python, use `ppi_rate(gold_pairs, judge_only)` or `judge_eps(events)`. ## Planning labels (`plan`) selrel plan --eps 0.05 --change 0.02 two independent windows: 2213 per window selrel plan --eps 0.05 --change 0.02 --paired --flip-rate 0.04 paired replay: 783 probe queries These are fixed-sample sizes for a two-sided test at `--alpha` with `--power`. Anytime-valid gating typically needs 1.3 to 2 times more. For paired replay, `--flip-rate` is the share of probe queries whose outcome differs between versions. It defaults to min(1, 2ε + change), which is conservative. ## Diagnosis and attribution `diagnose` also splits the modelled routed accuracy, (1 − ε)·A_forced + ε·A_wrong, exactly into a selection part, −Δε·(Ā_forced − Ā_wrong), and a capability part, (1 − ε̄)·ΔA_forced + ε̄·ΔA_wrong, with midpoint averages. It does this when both windows have labels and forced probes. In Python it is `attribute(Window(eps, a_forced, a_wrong), Window(...))`. ## Zero-code connectors ### Live: OpenTelemetry span processor If your agent already emits OpenTelemetry spans, add one processor. It needs no extra package: `selrel` doesn't import `opentelemetry`, and the class just has the `on_start` / `on_end` / `shutdown` / `force_flush` methods the SDK calls. ```python from opentelemetry.sdk.trace import TracerProvider from selrel import DecisionLog, SelectionSpanProcessor provider = TracerProvider() provider.add_span_processor(SelectionSpanProcessor( DecisionLog("decisions.jsonl"), tag="lib-v7", target_fn=lambda span: span["attributes"].get("eval.expected_tool"), # optional pool_size_fn=lambda span: len(TOOLS), # optional )) ``` Each finished span that records a selection becomes a selection/v1 event: OTel GenAI `retrieval`, `search_memory` and `execute_tool` spans, and OpenInference `RETRIEVER`, `RERANKER` and LLM-with-tools spans. Other spans are ignored, and errors are counted, never raised into your code. The event id is the span's trace and span id. ### Pull: Langfuse and Arize Phoenix export LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... # LANGFUSE_HOST for US/self-hosted selrel pull langfuse --since 2026-11-01T00:00:00Z --until 2026-11-08T00:00:00Z --tag lib-v7 --out events.jsonl export PHOENIX_API_KEY=... # only if auth is on; PHOENIX_BASE_URL, default http://localhost:6006 selrel pull phoenix --project my-agent --since 2026-11-01T00:00:00Z --out events.jsonl - **Langfuse.** This uses the public API (`GET /api/public/v2/observations`, basic auth, cursor pagination). It reads RETRIEVER, TOOL and GENERATION observations by default (`--type` to change). - Spans that reached Langfuse over OTLP are converted from their OTel GenAI or OpenInference attributes, which Langfuse keeps in `metadata.attributes`. - Native RETRIEVER observations use their output document list. - GENERATION observations use the first tool call against the `tools` in their input. - TOOL observations need `--pool-size`. - Self-hosted Langfuse v3 has only the older v1 route: pass `--api v1`. - **Phoenix.** This uses `GET /v1/projects/{project}/spans` with cursor pagination and a bearer token if one is set. It reads RETRIEVER, RERANKER, LLM and TOOL spans (`--span-kind` to change). TOOL spans need `--pool-size`, because they don't list the tools on offer. Event ids are `langfuse:` or `phoenix:`, so pulling an overlapping window twice and pushing to the hosted service doesn't double-count. In Python, use `pull_langfuse(...)` and `pull_phoenix(...)`, which return `(events, scanned)`, or `from_langfuse_observation` and `from_phoenix_span` on records you already have. The connectors were tested against fixtures in the documented response shapes, not against live Langfuse or Phoenix servers. `selrel/connectors.py` lists exactly what was checked and what wasn't. Like every adapter, they can't supply `target`. Add it from evals, user corrections or a judge (`judge_wrong`, then `selrel ppi`). The same limits apply as for the adapters in [docs/CLOUD.md](docs/CLOUD.md#adapters). ## GitHub Action `action/action.yml` is a composite action. It does four things: - installs selrel (`pip install selrel`, falling back to the site tarball); - runs `selrel gate` on two logs; - writes the verdict, the ε change and its interval to the job summary; - fails the job on BLOCK, and on NEED MORE DATA if you set `fail-on-more-data: true`. **Where to put it.** Reference it from the public repository, [aw3-technology/selrel](https://github.com/aw3-technology/selrel), pinned to a release tag (`uses: aw3-technology/selrel/action@v0.2.0`). To vendor it instead, copy `action/action.yml` to `.github/actions/selrel-gate/action.yml` in your repository and use `uses: ./.github/actions/selrel-gate`. ```yaml # .github/workflows/selection-gate.yml name: selection gate on: pull_request jobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # Produce the two logs however you evaluate: replay a fixed probe set # against the base and the PR version, writing selection/v1 JSONL. - run: python eval/replay.py --version base --out before.jsonl - run: python eval/replay.py --version pr --out after.jsonl - id: gate uses: aw3-technology/selrel/action@v0.2.0 with: before: before.jsonl after: after.jsonl delta: "0.02" mode: anytime # or fixed paired: "true" # same probe queries before and after fail-on-more-data: "false" - run: echo "verdict ${{ steps.gate.outputs.decision }}, change ${{ steps.gate.outputs.d-eps }}" if: always() ``` | input | default | | |---|---|---| | `before`, `after` | required | JSONL logs (selection/v1 or v0.1) | | `delta` | `0.02` | largest rise in ε to accept | | `mode` | `anytime` | `anytime` or `fixed` | | `paired` | `false` | same probe queries before and after | | `forced-tol` | `0.02` | largest forced-accuracy drop to accept, when both logs carry `forced_ok` | | `fail-on-more-data` | `false` | also fail on NEED MORE DATA | | `package` | `selrel` | what to `pip install` | | `python-version` | `3.12` | for `actions/setup-python` | Outputs: `decision`, `d-eps`, `d-eps-lo` and `d-eps-hi`. The gate step was tested locally with the same commands and environment variables; the action itself has not yet been run on GitHub's runners. ## Statistics `report` and `diagnose` use 95% intervals: Wilson for rates, Newcombe for differences. A change counts only when its interval clears the tolerance (`--tol`, default 0.02). `gate`, `ppi` and `plan` are ports of `lib/reliability/stats.ts`, the hosted estimators. `tests/test_parity.py` checks them against values computed by that TypeScript, to 1e-12 relative, on fixed inputs. ## What the demo shows The simulator builds a library of skills whose router keys crowd together as the library grows from 4 to 128 skills. With skills intact, forced accuracy on the original skills holds, ε climbs from about 0.003 to 0.088, routed accuracy falls, and the verdict is MIS-SELECTION. With skills that also decay as later ones are added, forced accuracy falls from 0.92 to 0.61 on the original skills, and the verdict is FORGETTING + MIS-SELECTION. This is milestone one's prediction and its kill criterion, on synthetic data. It tests the instrument, not the thesis. ## Limits - ε needs labelled targets. On unlabelled production traffic, use evaluation sets, user corrections or a judge, and say which. A judge alone is biased: pair it with a random gold sample and use `selrel ppi`. - Anytime-valid intervals cost width. At a few thousand decisions per window an unpaired anytime gate often says NEED MORE DATA where one fixed look would ALLOW. Paired replay is the cheap way out. - The library budget extrapolates a logit-linear ε(ln K) fit. Trust it only within about one doubling of the largest pool you have measured. - The candidate budget uses the verifier-budget score model (Gaussian scores). Under heavy-tailed scores that model is wrong; measure ε directly. - Splitting class-incremental error into within-task and task-selection parts is prior art (Kim et al., NeurIPS 2022). What this adds is one ε across routers, retrievers and verifiers, estimated on the live pool, and turned into gates and budgets. Related tools and papers: https://neurogenesys.ai/research/tools License: MIT.