A better router helps, but on CIFAR-100 the experts were the weak link

A system built from frozen experts, one per task, has to decide which expert answers each input. Our router synthesis argued that the decision should not rest on the experts' own confidence, and that a router should instead score every expert in one shared feature space with a simple closed-form rule. Earlier, that design had been measured only on Split-MNIST. We pre-registered six predictions and tested them at a harder scale.

The setup: Split-CIFAR-100 divided into 5, 10 or 20 tasks, one small ResNet expert trained per task, ten seeds each. That is 30 expert libraries, about 16 GPU-hours, with every expert's outputs and features cached once. Routers were then fitted offline on training images only and scored once on the test set. Differences are paired by seed with bootstrap intervals and a Holm correction across the six tests (ledger section 15).

The confidence-free rule did win, but without pretraining only barely. Routing on the first expert's own features, the only shared space a continual learner has without outside data, cut selection error from 0.627 to 0.614 at 10 experts. Our pilot had predicted a tie. The router still picked the wrong expert more than 60% of the time. Without a good representation, a better rule does little.

With a strong representation the picture changes completely. Using ImageNet ViT-B/16 features, the same rule cut selection error to 0.178. As the library grew from 5 to 20 experts, it rose by only 0.030, against 0.257 for routing on the experts' self-report. ImageNet's classes overlap CIFAR-100's, so this arm shows what a router can do given good features, not that a continual learner could build them from its own stream.

Then the control we added out of caution became the headline. A plain 100-way linear readout of those same ViT features, with no experts at all, scored 0.803, against 0.671 for the best routed system. Once routing is good, experts get their hard inputs as well as their easy ones, and they were right on only 76–85% of the inputs routed to them. Self-report had looked better on that measure, 89–98%, only because it routes each expert the inputs it already finds easy. At this scale the experts, not the router, are the bottleneck.

One prediction failed. On MNIST, abstaining on an independent score had beaten abstaining on expert confidence (ledger 9.3). Here the router's margin between its top two choices ranked errors slightly worse than the chosen expert's own confidence. The MNIST comparison was with confidence read across experts; within one expert, confidence still carries information. Top-2 mixing helped by only 0.3 points with the strong router, and a router trained on a replay buffer was clearly worse than the closed-form rule in both feature spaces.

The practical lesson is to check the readout first. Before building a router over weak experts, test whether a readout of the shared representation already wins; here it did by 13 points. Routing earns its place where experts know something the shared space does not. That points the programme's next investment toward features rather than gating. The language-model arm of the study, which routes requests among 64 skills, is queued.