Cache Hit Ratio for LLM Spend: A Practical Guide
Learn what cache hit ratio means for LLM costs, how provider caching signals work, and how to surface and improve hit rates to cut OpenAI and Anthropic spend.
A higher cache hit ratio can make an LLM system slower and less profitable. A 2024 Carnegie Mellon paper found that increasing hit ratio can reduce throughput and worsen response time in some caching systems, especially as hardware gets faster and workloads become more parallelized. The broader lesson is uncomfortable but useful: cache hit ratio isn't a success metric until you connect it to latency, token volume, provider pricing, and the bill. The paper's analysis of caching metrics is a good reminder that one dashboard percentage can hide the outcome you care about.
For LLM pipelines, that outcome is usually money saved. A provider cache can reuse a stable system prompt, tool definition, or retrieved context instead of charging full input pricing for every request. But request counts alone won't tell you whether the cached material is large enough to change your invoice. The practical unit is the token, and the practical destination is a finance-readable cost line.
Table of Contents
- Why Cache Hit Ratio Matters More for Your AI Bill Than Your Backend
- What Cache Hit Ratio Actually Measures
- How Provider Caching Works in OpenAI and Anthropic
- Sample Calculation From Tokens to Dollars Saved
- Reading Cache Metrics in SpendLens AI
- When a Higher Hit Ratio Can Still Cost You Money
- Optimization Patterns That Lift Hit Rate
- A Short Checklist for Honest Cache Metrics
Why Cache Hit Ratio Matters More for Your AI Bill Than Your Backend
Teams often treat cache hit ratio as a latency metric first and a financial metric second. For LLM workloads, that order is backwards.
A conventional Redis dashboard might celebrate a high hit ratio because fewer requests reach a database. That still matters. A cache hit ratio guide from Gcore defines the metric as cached requests divided by total requests and explains the operational chain: more hits mean fewer origin fetches, lower backend load, and less egress. But the ratio says nothing by itself about how many bytes or tokens each request contains.
That distinction changes the economics of AI. A request with a small reusable prefix can count as a hit while saving almost nothing. A single cache hit containing a large system prompt and knowledge base can be worth far more than many short hits. In other words, the invoice follows cached tokens, not the number printed in a request-level dashboard.
Consider a team looking at a $48,000 monthly OpenAI invoice. A 20-point improvement in effective input-token cache coverage could move that bill by roughly $9,000, while the same prompt-routing work might shave only 80ms from p95 latency. Those figures are a planning example, not a universal result. The exact delta depends on model, token mix, cache-write behavior, and workload shape, so the measurement has to come from provider usage fields rather than a generic infrastructure graph.
The financial interpretation
Treat each uncached token as a full-price liability and each cached token as a discounted input. The discount varies by provider and model, and cache creation can introduce its own charge. That means a cache hit ratio becomes useful only after you answer three questions:
- What was cached? System instructions, tool schemas, examples, retrieved documents, or some combination?
- How many tokens did it represent? Request-level coverage can flatten this difference.
- What would those tokens have cost without reuse? The answer belongs in the same report as the ratio.
A Redis layer costing $40 per month can be operationally successful with a high hit rate, but LLM prompt caching sits on a different cost curve because every repeated uncached token is billed again. SpendLens AI is designed around that distinction, mapping provider cache counters into workload-level spend instead of treating cache efficiency as abstract engineering hygiene.
The right question isn't “Did the ratio go up?” It's “How much of the bill moved from full-price input to discounted cached input, and did the saving exceed the cost of maintaining the cache?”
What Cache Hit Ratio Actually Measures
A cache is like a vending machine that has already loaded the snack you want. The same prompt prefix is the snack, the provider cache is the loaded machine, and a cache hit means the provider can reuse eligible context rather than processing it as fresh input.
The classic formula is:
Request-level cache hit ratio = cached requests ÷ total requests
Total requests normally means hits plus misses. This is easy to explain and easy to chart, which is why most infrastructure dashboards start there. It works well when requests have roughly similar sizes and similar economic value.
LLM prompts rarely behave that neatly. One request might contain a short instruction, while another includes a long system prompt, tool definitions, and retrieved context. Both requests count once, but they don't contribute equally to provider spend.
Two ratios, two decisions
The second formula looks at tokens:
Token-level cache hit ratio = cached tokens ÷ total input tokens processed
This is the ratio that maps more directly to discounted provider usage. If a workload processes a large static prefix on every call, token-level coverage can reveal substantial savings even when request-level reporting looks ordinary. The reverse is also true. A dashboard can show a strong request-level ratio while most expensive tokens remain uncached.
| Dimension | Request-Level Hit Ratio | Token-Level Hit Ratio |
|---|---|---|
| Denominator | Total requests | Total input tokens |
| Best use | Operational coverage and routing health | Provider cost and discount analysis |
| Main weakness | Treats short and long requests equally | Requires accurate token counters |
| Financial meaning | Indirect | Directly connected to cached spend |
| Example question | How many calls found a cache entry? | How many billable input tokens avoided full-price processing? |
A broader caching reference makes the same distinction for content systems: request hit ratio measures request coverage, while byte hit ratio may be necessary when object sizes vary significantly. Gcore's explanation of cache hit ratio and byte coverage applies cleanly to LLMs when “bytes” become “tokens.”
Practical rule: Keep request-level and token-level ratios side by side. Use the first to debug behavior, and use the second to explain the bill.
The denominator also needs a scope. Separate response caching from provider prompt caching, and distinguish total prompt tokens from cached tokens. A 99% hit rate on a tiny prompt segment can coexist with high spend if the rest of the prompt changes on every request. A good cache hit ratio is therefore a scoped measurement, not a universal grade.
How Provider Caching Works in OpenAI and Anthropic
OpenAI and Anthropic expose different usage signals, but both reward the same architectural behavior: put stable content at the front of the prompt and keep changing content after it.
OpenAI returns cached usage through the cached_tokens field inside usage.prompt_tokens_details. Automatic prefix matching applies above the provider's 1024-token threshold, so a short prompt may be ineligible even when it repeats. Cached input is billed at roughly 10% of the regular input price for relevant pricing tiers, while cache creation can be charged at the full input rate.
Anthropic exposes cache_creation_input_tokens and cache_read_input_tokens in usage. The provider uses a 1024-token minimum cache block, a default five-minute TTL, a 90% discount on cache reads, and a 25% surcharge on cache writes. The economics depend on how often the prefix is reused before expiration.
| Signal Field | OpenAI | Anthropic |
|---|---|---|
| Cached-read field | usage.prompt_tokens_details.cached_tokens |
usage.cache_read_input_tokens |
| Cache-creation signal | Provider-specific cached input details and creation pricing | usage.cache_creation_input_tokens |
| Eligibility threshold | Automatic prefix matching above 1024 tokens | 1024-token minimum block |
| Read pricing behavior | Roughly 10% of input price on relevant tiers | 90% discount on cache reads |
| Write pricing behavior | Full input rate for cache creation on the described tier | 25% surcharge over base input pricing |
| Matching model | Reusable prompt prefix | Reusable prompt prefix with TTL behavior |
System prompts, developer instructions, tool definitions, examples, and long retrieved context can qualify when they form a stable prefix. User-specific content, timestamps, changing retrieval results, and dynamic tool state can break the match when inserted too early.
Teams integrating these fields should also understand how the provider client reports usage, rather than relying on a generic middleware counter. The OpenAI Completions API reference is useful when checking where usage details appear in an existing integration.
The central constraint is prefix matching. You can't append a stable block after volatile content and expect the provider to discover it as an independent reusable segment. Prompt order is part of the cache design.
Sample Calculation From Tokens to Dollars Saved
Take a customer-support assistant processing 12 million input tokens per day. Its static system prompt and knowledge base contain 40,000 tokens, and Anthropic pricing is used as the working example: $3 per million tokens for full-price input, $0.30 per million for cache reads, and $3.75 per million for cache writes. These values and the resulting calculations should be treated as a model, not a quote for every Anthropic workload.
The input-only baseline without caching is:
12 million × $3 = $36 per day
Across 30 days, that becomes $1,080 per month. The calculation below applies the cache-write surcharge to the leading edge of each five-minute window and excludes output tokens, which are billed separately.
| Hit Rate | Anthropic Monthly Cost | OpenAI Monthly Cost | Savings vs 60% Baseline |
|---|---|---|---|
| 60% | $792 | $594 | Baseline |
| 75% | $630 | $495 | $162 Anthropic, $99 OpenAI |
| 90% | $468 | $405 | $324 Anthropic, $189 OpenAI |
The OpenAI comparison uses $1.25 per million cached input tokens versus $2.50 per million full-price input tokens for the relevant model. Those figures show why provider multipliers matter. The same token-level improvement produces a different dollar outcome because the gap between full-price and cached input differs.
At 60% effective coverage, 40% of the input remains full price. At 90%, only 10% does. The 30-day delta between those scenarios is $324 for Anthropic and $189 for OpenAI in this input-only example. The result is measurable value, but only if the workload produces the stated token volume and stable prefix behavior.
The operational saving isn't only on the invoice. If prompt restructuring raises coverage without forcing a retrieval rewrite, the team also avoids spending engineering time refactoring the codebase. That time saved has real value, but it shouldn't be assigned an invented hour count. Track the actual implementation time in your project system, then add it to the provider-cost delta. The LLM cost-savings calculation guide gives teams a place to formalize that model.
Reading Cache Metrics in SpendLens AI
A useful cache dashboard starts with provider response data, not with a request counter. SpendLens AI's instrumentation aggregates OpenAI cached_input_tokens and Anthropic cache_creation_input_tokens from observed responses, then presents a blended cache hit ratio weighted by spend. That weighting prevents a large number of cheap, short requests from hiding an expensive workload with weak reuse.
Start with the overview
The overview should answer a financial question quickly: how much input spend came from cached versus uncached tokens during the selected billing period? A blended ratio provides orientation, but it shouldn't be the last view you inspect.
Open the per-workload breakdown next. Each route, agent, or prompt template gets its own hit-rate line, which makes unstable prefixes visible. A support agent may reuse its instructions consistently, while a retrieval-heavy research agent may sit near zero because changing context appears before the reusable material.
Find the missed opportunities
The wasted-opportunities panel estimates what the bill could have looked like if eligible prefixes had been cached. The savings panel turns that difference into a dollar figure tied to the billing period, so an engineer can prioritize work without translating tokens into provider pricing by hand.
Use filters for model, team, and environment. A production workload deserves a different decision from a development experiment, and a batch route may have a different token mix from an interactive route. Metrics that update near real time help diagnose a deployment or prompt change, while the nightly billing reconciliation should be treated as the accounting checkpoint.
SpendLens AI also provides workload attribution and AI observability views through its developer-focused AI observability platform. The useful practice is to compare cache behavior with spend, latency, prompt versions, and workload ownership in one investigation.
When a Higher Hit Ratio Can Still Cost You Money
A cache write isn't free. Anthropic charges a 25% surcharge over the base input rate for cache creation, and OpenAI can apply a separate cache-creation rate on relevant pricing tiers. If a prefix changes before enough requests reuse it, the write premium can exceed the read discount.
Consider a 2,000-token prefix rewritten every third request at low traffic volume. The resulting effective hit rate can reach 66%, yet the workload can cost more than sending the same prompts without caching. The reason is that each new version pays for cache creation, while the following reuse doesn't generate enough discounted reads to recover that premium. The exact outcome depends on provider pricing and the rest of the prompt, so the example illustrates the decision pattern rather than a universal bill.
| Scenario | Requests per 5-min Window | Effective Hit Rate | Cost vs No Cache |
|---|---|---|---|
| Prefix rewritten every request | 1 | 0% | Higher or equal, with no useful reuse |
| Prefix rewritten every third request | 3 | 66% | Can exceed no-cache cost |
| Stable prefix with repeated reads | Multiple | Higher | Usually favorable when read savings exceed writes |
Recent prompt-caching analysis notes that a workload may need roughly 1.4 reads per cached write to break even, and that below about a 30% hit rate caching can cost more than it saves under the described Anthropic economics. The prompt-caching break-even discussion also emphasizes that TTL, pricing, and prefix stability determine the result.
There is a second trap. Aggressive caching can hide retrieval regressions because stable cached instructions may continue to hit while the knowledge base becomes stale or the dynamic context stops reflecting production. A high ratio can therefore coexist with incorrect answers, poor retrieval, or no meaningful reduction in total prompt spend.
Disable caching when the prefix changes too frequently, the write premium isn't recovered by subsequent reads, or correctness requires fresh context on every call. The right target isn't 100%. It's positive net savings with acceptable freshness and latency.
Optimization Patterns That Lift Hit Rate
Prompt caching responds to three variables: prefix length, prefix stability, and request routing. The highest-value changes usually happen in prompt architecture, not in a new cache product.
Put the largest stable block first
Place developer messages, tool schemas, examples, and long instructions at the beginning. Push user content, retrieved context, timestamps, and other runtime variables toward the end. Providers match a fixed prefix, so moving a stable block above volatile data can turn repeated full-price processing into reusable cached input.
Keep the structure deterministic. Standardize whitespace, field ordering, serialization, and punctuation. A timestamp embedded near the front can invalidate the entire suffix, and inconsistent trailing punctuation can create needless misses.
Keep similar work together
Route repeated workloads to the same model and region where the provider's caching behavior supports it. After deployment, a background request can pre-warm high-traffic prefixes, but don't pre-warm low-volume or unstable prompts just to improve a dashboard number.
Avoid treating “cache this” as a magic instruction. If the prompt changes before the provider can reuse the prefix, an explicit request won't repair the underlying design.
Measure the expensive route
Track hit ratio by workload, not only in aggregate. A 95% hit rate on a chatty free-tier endpoint says little about the batch endpoint that drives most of the bill. Rank opportunities by cached-token volume and missed-cache spend, then test one prompt-template change at a time.
Prefix-heavy workloads can reasonably aim for the 70% to 90% band within two weeks, but that range is a target for a specific workload, not a universal benchmark. The financial outcome still depends on provider rates and cache-write frequency. A successful change should show both higher cached-token coverage and a lower input-cost line.
SpendLens AI can help associate those changes with workload spend and prompt versions through its token cost optimization guidance. The time saved comes from testing the prompt boundary directly instead of refactoring retrieval pipelines blindly, while the money saved appears only after provider counters confirm that the eligible prefix is being reused.
A Short Checklist for Honest Cache Metrics
A cache dashboard earns trust when finance can read it without translating token counters. Start with the parser, then validate the denominator, then connect the result to a workload and billing period.
Confirm provider fields: Capture
prompt_tokens_details.cached_tokensfor OpenAI andcache_read_input_tokensfor Anthropic. Silent parsing failures can make a dashboard report clean-looking ratios without showing the actual provider discount.Weight by tokens and cost: Keep request-level coverage for operational debugging, but calculate the financial ratio from cached tokens divided by total input tokens. Weight the result by spend so short prompts don't dominate the average.
Split workloads: Report each route, agent, prompt template, team, model, and environment separately. A broad average can hide the endpoint that caches nothing.
Set a spend-based target: A 70% weighted hit ratio can be a useful production target when the workload is prefix-heavy, but review it against net savings every week rather than treating the percentage as a goal by itself.
Question 100% claims: Cached tokens still carry a discounted charge, and cache creation can cost more. A 100% hit ratio isn't free.
Validate eligibility: Confirm that the stable prefix reaches the provider minimum of 1024 tokens for OpenAI and 1024 tokens for Anthropic before counting expected savings. Gcore's cache metric reference also explains why request coverage alone doesn't represent total bandwidth or saved work.
Version the prompt: Log a prompt-version hash with every request. That lets you compare prompt edits against cache behavior instead of guessing which formatting change caused misses.
Export the monthly row: Give finance cached spend, missed-cache spend, hit ratio, and net saving for every workload in one report. The value shown should be a dollar amount, supported by the token evidence behind it.
The most honest metric is the one that survives reconciliation. If the ratio rises but net spend stays flat, investigate prefix size, dynamic context, write frequency, and the denominator before declaring success.
SpendLens AI records provider cache counters across OpenAI and Anthropic workloads, attributes cache efficiency to the routes and prompt templates responsible for spend, and turns missed-cache opportunities into billing-period savings estimates. Visit SpendLens AI to connect cache hit ratio to the dollars your AI workloads consume.