How to estimate LLM API costs

An LLM cost estimate that begins with model price usually misses the expensive part: the workload.

You need request volume, prompt and output distributions, cache behavior, tool loops, retries, quality fallbacks, and growth.

The useful unit is cost per completed user task. Token price is one input to that number.

Count user tasks before API calls

Define the product action you are pricing.

Examples include answering one support question, reviewing one pull request, extracting one invoice, or completing one research report.

One task may trigger several model calls:

Route or classify the request.
Retrieve or summarize context.
Generate a draft.
Call tools and inspect results.
Verify or grade the answer.
Retry after a failure.

Count the full call graph.

If you budget each endpoint independently, shared retries and agent loops can disappear between teams.

Measure token distributions, not one average

Collect billed input, cached input, cache writes, and output for real requests.

Use percentiles or buckets. A mean hides a small group of very long and expensive tasks.

A useful set of buckets might be:

Short chat: under 2,000 input tokens.
Document task: 2,000 to 20,000 input tokens.
Repository or research task: above 20,000 input tokens.

Use boundaries that match your product, not these examples.

Keep output separate. Output often has a higher unit price and can vary with reasoning, tool use, and verbosity.

Write the base token formula

For one model call:

Input cost equals uncached input tokens divided by one million, multiplied by the input rate.

Cache-read cost uses cached tokens and the cache-read rate.

Cache-write cost uses written tokens and the write rate.

Output cost uses output tokens and the output rate.

Add them:

Call cost = input cost + cache-read cost + cache-write cost + output cost + tool charges.

Task cost is the sum of call costs plus expected retry and fallback costs.

Use the provider's billed token fields. A local tokenizer estimate is useful before a request, but the invoice follows provider accounting.

Build one worked example

Suppose a support task makes two calls.

The first call classifies 1,000 uncached input tokens and produces 50 output tokens.

The second call uses 8,000 input tokens, of which 6,000 are cache reads, and produces 600 output tokens.

At an example rate of $2 per million uncached input, $0.20 per million cache reads, and $12 per million output:

First call input: 1,000 / 1,000,000 x $2 = $0.002.
First call output: 50 / 1,000,000 x $12 = $0.0006.
Second call uncached input: 2,000 / 1,000,000 x $2 = $0.004.
Second call cache reads: 6,000 / 1,000,000 x $0.20 = $0.0012.
Second call output: 600 / 1,000,000 x $12 = $0.0072.

The token total is $0.015 before cache-write charges, tools, retries, and infrastructure.

At one million tasks, that small per-task number becomes $15,000.

Price cache writes against reuse

Caching is an investment. A write costs more than ordinary input on some current models, while later hits cost less.

Let B be the ordinary input price for the cached prefix.

If a write costs 1.25B and each hit costs 0.1B, writing once and reading once costs 1.35B.

Sending the same prefix uncached twice costs 2B.

In that simple case, one full hit pays for the higher write cost.

Real caches have partial matches, expiry, invalidation, routing, and changed prefixes.

Track:

Eligible tokens.
Tokens written.
Tokens read from cache.
Hit rate by endpoint and tenant.
Reuse count before expiry.
Cost saved after write cost.

The OpenAI prompt caching guide and Anthropic prompt caching guide publish current rules.

Output control is a budget control

Many current APIs price output above input.

Measure output that users read, output discarded by parsers, hidden reasoning billed by the provider, and tokens produced before cancellation.

Use clear stopping conditions. Set a sensible maximum output for each endpoint.

Structured output can reduce retries and parsing failures, but schemas and tool definitions add input.

Do not force every task into a short answer. A cheap wrong answer has poor unit economics.

The aim is the shortest output that meets the task's quality contract.

Add tool charges and tool-induced tokens

Web search, file search, code execution, image processing, and remote tools may have separate prices.

They also add content to the model context.

For each tool, record calls per task, direct tool price, result tokens, and downstream model calls.

A search result that adds 20,000 tokens can cost more through model input than through the search call.

Reduce tool payloads before reducing evidence. Select relevant fields, cap logs, and summarize only after preserving the facts the model needs.

Fanout's back-of-the-envelope lab is a useful place to estimate these fan-out effects.

Retries need an expected-cost term

Let p be the probability that a task needs one full retry.

If the normal task costs C and the retry repeats the whole path, expected token cost is roughly C x (1 + p), before multiple retries.

That estimate is too optimistic when failures occur after expensive work.

Track where the retry begins. A parser retry may repeat only the final call. A timeout may repeat the entire agent run.

Distinguish provider errors, rate limits, timeouts, invalid output, quality fallback, and user-requested regeneration.

Set retry limits and idempotency rules. A loop without a budget is a cost incident waiting for traffic.

Model routing can save or waste money

A router can send easy tasks to a cheaper model and hard tasks to a stronger one.

The router itself costs money and makes mistakes.

Let the cheap path cost C, the strong path cost S, and the fraction escalated be e.

Ignoring router cost, expected model cost is C + eS if escalation repeats the task on the stronger model.

If a bad cheap answer reaches the user, the cost includes support, trust, and rework that token accounting does not show.

Evaluate routing on the same quality rubric as direct use. Price the escalation path and false confidence.

Batch and flex tiers change the latency contract

Both OpenAI and Anthropic currently publish 50 percent token discounts for supported batch processing.

OpenAI also publishes Flex pricing for eligible workloads.

Move offline work only when its deadline, ordering, cancellation, and result handling fit the tier.

Good candidates include evaluations, nightly extraction, embeddings, catalog enrichment, and document classification.

Bad candidates include interactive turns and jobs whose value disappears after a short deadline.

Use official OpenAI pricing and Anthropic pricing because model and tier rates change.

Long context needs its own bucket

Some providers or models charge different rates beyond a context threshold.

Even when the rate stays constant, long prompts increase latency, memory pressure, and the chance that irrelevant context distracts the model.

Create a separate cost line for long-context tasks.

Ask whether retrieval, compaction, prefix caching, or state outside the prompt can reduce repeated input.

Do not truncate blindly. Removing the evidence that prevents a wrong answer can increase retries and quality failures.

The prefill vs decode guide explains why long prompts change more than the invoice.

Convert the estimate into a monthly budget

For each task type, calculate:

Monthly cost = tasks per month x expected cost per task.

Add tool fees, storage, vector databases, observability, compute you host, and a safety margin.

Build low, expected, and high cases.

Vary traffic, token lengths, cache hit rate, retry rate, escalation rate, and price changes.

Do not hide all uncertainty inside one 20 percent buffer. Show which assumption moves the budget.

Fanout's latency numbers lab can help connect throughput assumptions to the capacity required.

Track cost per successful task in production

Tag usage with tenant, feature, task type, model, tier, and outcome without putting sensitive content into analytics.

Join token usage with product success signals.

Useful measures include:

Cost per accepted answer.
Cost per resolved ticket.
Cost per merged code change.
Cost per document processed without review.
Cost of retries and abandoned streams.
Cache savings after write charges.

Review the long tail. A handful of agent loops or giant documents may own a large share of spend.

A cost review checklist

Before launch, answer:

What is the user task and quality bar?
How many model and tool calls can one task make?
What are input and output distributions?
Which content is cached, and how often is it reused?
Which failures retry, and from what point?
Which tasks can use batch or flex?
What happens when the monthly budget or per-task limit is reached?
Which metric proves the cheaper configuration still works?

Update the model rates from official sources, run the estimate against a real trace, and keep the assumptions beside the number.

For a current vendor snapshot, see OpenAI vs Anthropic API pricing. The comparison is useful only after the workload model exists.