A Hit Rate Is a Retention Policy: Prefix-Cache Reuse in Agentic Coding Traces Compounds the Workload With the Server's Eviction Rule and the Harness's Invalidations, Which Serving Assumptions the Traces Actually Break, and the Rent-or-Buy Rule That Prices the Turn Boundary

Abstract

Life of Research

Coding agents are becoming a major class of language-model traffic, and the first production-scale characterisation of that traffic, 13.5 million GitHub Copilot agent sessions, concludes that its findings "challenge assumptions underlying current LLM-serving systems and provide an empirical foundation for agent-native infrastructure." Its central evidence is the prefix-cache hit rate: about 90 percent within an agent turn, far lower across turn boundaries, 8 percent after a model switch. Other 2026 studies of agent traffic report hit rates from above 98 percent to below 17 percent. In this analysis the author reviews and synthesises published results to show that a measured hit rate is the product of three factors that the studies vary separately: the reuse the workload makes available, the probability that cached state survives until it is reused, which is set by the server's retention rule, capacity and load, and the probability that the agent harness has not invalidated the prefix by switching models or compacting context. Holding the workload and model fixed, one study's hit rate falls from 98.85 to 16.78 percent when concurrency rises from 15 to 20 agents; in another, sweeping the eviction timeout from one minute to one hour moves the achievable rate from 85.4 to 98.6 percent. The paper audits which serving assumptions the traces break, finds that statelessness was abandoned for multi-turn chat by 2024 and that context rewriting which invalidates the prefix predates agents, while the size of the state at stake, model switches made by automatic routing and LLM-written compaction are new, and separates three senses in which agent serving is said to be dominated by input, by decode or by cache charges. It states the turn-boundary retention decision as a rent-or-buy problem with a hazard-rate threshold, works a keep-alive break-even from published prices, and specifies the policy-held-fixed comparisons that would settle what is specific to agents.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 104 papers by Pranay Mahendrakar, or his ORCID record.