A Pooled Accuracy Is Not a Scaling Verdict: What Tool-Calling Benchmarks Show About Whether Larger Models Call Tools Better, Why Per-Token Price Misranks Cost Only in One Cost Regime, and the Per-Skill, Per-Task Accounting a Router Needs

Abstract

Life of Research

Two claims about tool-using language models now circulate together. The first is that larger or more expensive models do not reliably call tools better than smaller or cheaper ones. The second is that per-token price is a poor guide to what a request costs, because some cheap models spend many tokens. A 2026 router for agentic tool calling reported both, from eight models scored on five function-calling benchmarks, and a position paper argues on related grounds that small models are the future of agentic systems. In this analysis the author reviews and synthesises published results to show that both claims are true only under conditions that the evidence offered for them rarely states. On the first claim, about 90 percent of the items behind the router's pooled accuracy come from two public fine-tuning corpora of short, cheap calls; by the author's arithmetic on the study's own per-group table, both of its inversions reverse on the multi-turn subset it calls hardest, a subset that is about 1 percent of queries and roughly three-quarters of spend for most models. The reversal on that small subset is within sampling noise, so the router's data neither establish nor refute the inversion outside the pooled mix. Across other studies, call accuracy rises with size within a model family; the decision not to call a tool does not improve reliably with size and moves far more with targeted training; and switching on reasoning raised tool hallucination in every model pair tested. On the second claim, the router's own cost ordering matches list price for its main pool, and output verbosity moves per-request cost by a factor of 0.78 to 1.31, while conversation length moves it by two orders of magnitude. The author consolidates the measured values in Table 1 and Figure 1, re-weights the router's published results in Table 2, separates the tool-calling sub-skills in Table 3, states a per-skill, per-task accounting as Algorithm 1, and names the measurements that would settle the open part.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 147 papers by Pranay Mahendrakar, or his ORCID record.