Token Costs: The Arithmetic Nobody Shows You

The arithmetic, in one place

Every price in this guide is yours to supply, deliberately. Provider rates, cache discounts and tier structures change without notice, so any figure printed here would be wrong within months. The structure of the calculation and the places tokens leak out of it do not change, and that is the useful part.

Tokens are not words, and your spreadsheet knows it

A token is a unit produced by a model's tokenizer, roughly a frequently occurring sequence of characters. Common English words often come out as one token, while rare words, proper nouns, identifiers and misspellings split into several. That is why a word count times a fudge factor is a bad estimator, and why the factor that worked on your English prose fails on your JSON payload.

Never commit to a number derived from a word count. Take a hundred real inputs, including the longest ones your system has seen, push each through one real call, and read the usage fields on the responses. The gap between estimate and billed count is routinely large enough to change a build decision.

The six meters every language model system runs

A system that looks like one API call bills on six meters at once. A naive estimate reads one of them, usually the payload. Name all six, measure each separately, and the invoice stops surprising anyone.

Output tokens are the ones that hurt

Output tokens are usually priced above input tokens, often by a multiple, and they are generated one at a time rather than processed in parallel. So output length is your cost problem and your latency problem at once. Check the current ratio for your model before you decide how hard to work on this, because the value of everything below scales with that ratio.

• Return identifiers, not prose. If the model is choosing among retrieved passages, have it return chunk ids and assemble the answer in your own code. Otherwise you pay generation rates to have a model retype text you already hold.
• Return a patch, not the document. A rewritten four thousand token document bills four thousand output tokens. A list of targeted edits bills a fraction, and it is easier to review and to roll back.
• Constrain the schema. A tight output schema with enumerated values and length limits produces shorter output than free prose, and removes most of the retry meter at the same time. The mechanics are in structured output from LLMs.
• Stop paying for narration. Asking the model to explain its reasoning on every call bills that explanation at the output rate on every call. If you want reasoning for debugging, sample it on a small share of traffic.
• Set a maximum output length as a circuit breaker, and log hitting it as an error rather than a normal truncation. An unbounded generation loop is the fastest route to a bill nobody predicted.
• Budget reasoning tokens explicitly where your model exposes that control. Extended reasoning left on its default is a standing charge on every request, including the easy ones that never needed it.

One trade-off, stated honestly: shorter output genuinely reduces accuracy on some tasks, because the model uses generated tokens as working space. Run both variants against the same test set and compare cost per accepted answer, the only comparison that accounts for the rework a worse answer creates.

Prompt caching rewrites the order of your prompt

Prompt caching bills a repeated leading section of the prompt at a reduced rate. The match is on an exact prefix, which is the single most important property to design around and the one most teams discover late. One changed character anywhere before the cache boundary makes the entire request a miss.

Put the static part first

Order the prompt so everything stable comes before everything variable: system prompt, tool schemas, few shot examples, then the retrieved payload, then the user turn. Teams often build the reverse order because it reads better in code. Retrieved content placed above the fixed instructions guarantees a unique prefix on every request, which turns the discount off without turning the feature off.

Keep everything unique out of the prefix

A timestamp, a request id, a session id, a user name or a randomised greeting inside the system prompt makes every prefix unique. Nothing matches, every call pays the full input rate, and the configuration still says caching is on. Personalisation belongs after the cached section. So does any prompt rebuilt by interpolation in a way that reorders keys between requests.

Two properties decide whether caching is worth anything to you. Providers usually price a cache write differently from a cache read, and entries expire, so a low traffic endpoint can pay the write premium repeatedly and never collect a read. Caching is a volume play. And a deploy that changes one word in the system prompt resets the cache for every user at once, which makes batching prompt changes into scheduled releases a cost decision too.

Retrieval is where the money actually goes

In a retrieval system the payload meter dominates, and its size is set by two numbers: how many chunks you retrieve and how large each chunk is. Both get tuned for answer quality in one afternoon and neither gets re-costed afterwards. Doubling the retrieved chunk count doubles the payload on every request from that moment on.

Look at which term dominates the input line. If the retrieved payload is more than half of it, retrieval quality work is cost work, and the change that pays most is retrieving wide and sending narrow: pull forty candidates, rerank, send the handful that survive. Chunk size interacts with this directly, covered in chunking strategies.

The meters nobody bills for until the invoice arrives

Every line here is real spend that never appears in a per request estimate, because none of it is triggered by a user. The fix has the same shape each time: give the traffic its own key or metadata tag so it lands as a separate line instead of dissolving into the total.

Put a cost line on every request

Provider dashboards aggregate at exactly the wrong level, by key and by day, when what you need is by unit of work and by meter. Emit a cost line per request into the same store as your traces. It is a small piece of code and it repays itself the first time somebody asks why the bill moved.

# Prices live in config and carry the date they were checked. Fill these
# from your provider's current pricing page before you trust any output,
# and set them per million tokens in whatever currency you are billed in.
PRICES = {
    "checked_on": "2026-08-19",
    "unit": "per million tokens",
    "models": {
        "small": {"in": 0.0, "cached_in": 0.0, "out": 0.0},
        "large": {"in": 0.0, "cached_in": 0.0, "out": 0.0},
        "embed": {"in": 0.0, "cached_in": 0.0, "out": 0.0},
    },
}

METERS = ("prefix", "payload", "growth", "output", "retry", "offline")


def cost_of(usage, model, meter, prices=PRICES):
    """usage must be the provider's own usage object from the response.
    A client side tokenizer estimate will not match what you are billed."""
    p = prices["models"][model]
    cached = usage.get("cached_input_tokens", 0)
    # providers differ on whether input_tokens already excludes cached tokens.
    # verify against one real invoice line before you trust this subtraction.
    fresh = max(0, usage["input_tokens"] - cached)
    out = usage["output_tokens"] + usage.get("reasoning_tokens", 0)
    return {
        "model": model,
        "meter": meter,
        "cached_in": cached,
        "fresh_in": fresh,
        "out": out,
        "cost": (fresh * p["in"] + cached * p["cached_in"] + out * p["out"]) / 1_000_000,
        "price_checked_on": prices["checked_on"],
    }


def request_ledger(request_id, unit_of_work, calls, accepted):
    """calls: [{"usage": {...}, "model": "large", "meter": "payload"}, ...]"""
    lines = [cost_of(c["usage"], c["model"], c["meter"]) for c in calls]
    total = sum(l["cost"] for l in lines)
    billed_in = sum(l["cached_in"] + l["fresh_in"] for l in lines) or 1
    return {
        "request_id": request_id,
        "unit_of_work": unit_of_work,          # one answered question, one parsed invoice
        "lines": lines,                        # one line per model call, tagged by meter
        "cost_total": round(total, 6),
        "cost_per_accepted": round(total, 6) if accepted else None,
        "cache_hit_ratio": sum(l["cached_in"] for l in lines) / billed_in,
        "retries": sum(1 for c in calls if c["meter"] == "retry"),
        "accepted": accepted,
    }

Three fields do the real work. The meter tag turns a total into a diagnosis, because a jump in the offline meter and a jump in the payload meter have different causes and different fixes. The cache hit ratio catches a broken prefix the day it breaks rather than at month end. The checked date stops a stale rate propagating quietly into a business case.

Terms to be precise about

Definitions worth getting exactly right

Token

The unit a model's tokenizer produces from text, usually a frequently occurring sequence of characters rather than a word. Token counts are specific to a model family, so a count taken from one provider's tokenizer does not transfer to another, and only the usage fields returned with a response reflect what you are actually billed.

Prompt cache hit

A request whose leading section exactly matches a previously cached prefix and is therefore billed at a reduced input rate. The match is on an exact prefix, so a single changed character anywhere before the cache boundary turns the whole request into a miss.

Output token

A token the model generates. On most providers it is priced above an input token, and because generation is serial rather than parallel, output length drives latency as well as cost. Any control over response length is therefore a control over two budgets at once.

Reasoning tokens

Output a model generates as intermediate working, billed at the output rate whether or not it is ever shown to a user. A system that enables extended reasoning without setting a budget is paying generation rates for text nobody reads.

Effective input rate

The blended price actually paid per input token once cached and fresh tokens are separated. Quoting a single input rate for a system with prompt caching overstates cost on cached traffic and understates it on cold traffic, sometimes badly enough to invert a build decision.

Cost per accepted outcome

Total token spend for one unit of work divided by the fraction of outputs accepted without rework. It is the only per unit figure that survives a comparison between two models with different accuracy, because a cheaper model that is wrong more often is billed twice and delivers once.

Before you quote a number to anyone