How many tools can your agent handle?

There is no hard limit, but accuracy falls as the list grows. On BFCL-V4 (Toollery, arXiv 2609.22218), small open models given every tool in the prompt dropped from 85–97% accuracy with 10 tools to 25–39% with 457, while stronger models held around 77–81%. On a 584-tool enterprise catalog (arXiv 2606.17519), GPT-5.4's routing F1 fell from 58.2% at 51 tools to 42.1% at 584.

It helps but moves the problem. Shortlisting with embeddings recovered about 10 points of F1 at 584 tools in the enterprise study, yet even a perfect shortlist lost 10 points as the catalog grew, because similar tools get confused. Retrieval also misses: BM25 left the right tool out of a 10-item shortlist about 3–7% of the time at roughly 450 tools and 43% at 79,000.

It depends on the catalog. At 3,616 production cloud tools (arXiv 2607.15593), hybrid BM25 plus embeddings kept about 97.6% of correct tools in a 15-tool shortlist, against 83.1% for embeddings and 68.8% for BM25 alone. In the enterprise study, BM25 shortlisting did worse than no shortlist, because the tools shared vocabulary.

Solid lines cover library sizes someone actually measured. Beyond one doubling of the largest measured size, and below the smallest, the line is an extrapolation of a logit-linear fit, and real curves often bend. The spread band shows how much the published systems differ from each other.

Log each choice the agent makes, how many candidates it chose from, and the correct tool when you know it, from evals, user corrections or a judge. The selection-reliability tool and hosted reliability projects turn those logs into error by library size and tell you whether a regression is mis-selection or forgetting.