Why Token Counts Alone Don't Tell the Full Story

Token counts are one of the first metrics teams reach for when an artificial intelligence bill starts to grow. That makes sense: every large language model request has an input, an output, and a measurable number of tokens moving through the system.

But a token count is a measurement of volume, not an explanation of value or cost.

Two workloads can consume the same number of tokens and have completely different prices, performance requirements, quality thresholds, and business outcomes. If engineering and finance teams look only at aggregate token usage, they may optimise the wrong feature, move a critical workflow to an unsuitable model, or miss a much easier saving hidden in caching and repeated context.

Effective LLM cost optimisation begins with tokens. It cannot end there.

The same token count can produce a different bill

Tokens are not priced uniformly. Input and output tokens often have different rates, and those rates vary substantially between models and providers. Cached input may receive a discount, while specialised features or long-context requests may change the economics again.

Consider two features that each process one million tokens per month. One is a classification workflow with short outputs on a smaller model. The other is a research assistant that generates long answers on a frontier model. Their usage totals look identical in a token dashboard, but their costs may be far apart.

Even the split between input and output matters. A workflow that repeatedly sends a long system prompt has a different optimisation opportunity from one that generates unnecessarily verbose responses. The first may benefit from prompt caching or context reduction. The second may need tighter output limits or better instructions.

This is why LLM cost tracking must connect token usage to model-specific pricing and token type. A total token number without those details is closer to a traffic counter than a financial metric.

Workload context turns usage into an explanation

Aggregate provider usage can tell you that consumption increased. It cannot tell you why.

Perhaps a customer-support feature attracted more users. Perhaps an agent began making five model calls for every task instead of three. Maybe a product change added a large retrieved document to every request, or a retry loop started repeating failed calls. Each situation needs a different response.

Workload-level attribution connects usage to the product feature, endpoint, customer journey, or background job that created it. With that context, teams can answer practical questions:

• Which AI feature drives the most spend?
• Is cost growth caused by adoption or inefficiency?
• Which workloads have unusually high output-token usage?
• Where are repeated prompts creating avoidable input costs?
• Which customer or project has changed its usage pattern?

These answers are much more useful than knowing that the organisation processed 80 million tokens last month. They explain whether an increase is healthy growth, accidental waste, or a technical regression.

Cache efficiency can matter more than raw volume

Many production AI applications send the same content repeatedly: system instructions, tool definitions, reference material, policies, examples, or document prefixes. Counting those tokens is necessary, but treating every processed token as economically identical hides the impact of caching.

A high-volume workload with strong cache utilisation may be efficient. A lower-volume workload that continually reprocesses identical context may deserve attention first.

Teams should evaluate cache reads, cache writes, eligible repeated content, and the actual discount applied by the provider. They should also inspect why expected cache hits are not happening. Small changes in prompt structure, unstable prefixes, or frequently reordered tool definitions can reduce cache effectiveness.

Token reduction is not the only goal. Sometimes the better move is to make repeated tokens cheaper.

Quality and latency belong in the cost conversation

The cheapest request is not automatically the best request. A smaller model may reduce the price per token but create more retries, longer workflows, manual review, or lower customer satisfaction. A model that produces a correct result on the first attempt can be more economical than a cheaper model that needs several attempts.

This is where simplistic model switching becomes risky. Before recommending a lower-cost model, teams should define the quality threshold for the workload and test representative examples. They should also measure latency, reliability, structured-output compliance, tool-use success, and escalation rates.

For a routine extraction task, a smaller model may deliver excellent results and meaningful savings. For a complex reasoning workflow tied to a high-value customer decision, the more capable model may have better unit economics despite its higher price.

AI FinOps is not about minimising every token. It is about finding the lowest sustainable cost that still meets the workload's requirements.

Business value changes what “expensive” means

A $500 monthly workflow that supports $50,000 in revenue may be a stronger investment than a $50 workflow nobody uses. Raw usage and total cost cannot make that distinction.

Useful AI cost metrics should therefore sit beside product and business signals. Depending on the feature, teams may track cost per resolved support ticket, completed report, qualified lead, active customer, generated asset, or successful agent task.

Unit economics reveal whether spending is producing value. They also make prioritisation clearer. A costly feature with strong margins may need monitoring, not aggressive cuts. A low-cost feature with poor adoption or weak outcomes may need redesign or retirement.

This perspective helps finance, product, and engineering discuss AI spend in the same language. Instead of asking only, “How many tokens did we use?”, they can ask, “What outcome did that usage create, and could we produce it more efficiently?”

A better framework for LLM cost optimisation

Token counts remain an essential foundation. The mistake is treating them as the whole system. A useful AI observability and cost-management practice combines:

• Usage: input, output, cached, and reasoning tokens where available
• Price: the actual model and provider rate applied to each request
• Workload: the feature, task, project, or customer responsible for usage
• Performance: latency, retries, errors, and the number of calls per outcome
• Quality: whether the result met the workload's acceptance criteria
• Value: the business or customer outcome created by the request

Together, these dimensions show where optimisation is both financially meaningful and operationally safe.

Measure tokens, then ask better questions

Token counts are valuable because they make AI activity measurable. They expose growth, help forecast capacity, and provide a common unit across requests. But they do not tell you whether a workload is appropriately priced, efficiently designed, reliable, or worth running.

The fuller story appears when token data is connected to cost, caching, model choice, application context, performance, and business value.

That is the difference between monitoring AI usage and managing AI economics.

Use the OpenAI and Anthropic savings calculator to estimate potential savings, or start tracking workload-level AI costs with SpendLens AI.