The cost of an LLM in production depends on three variables: the context tokens resent with every request, the real number of requests per day, and the model chosen for each step. The levers that move the bill are prompt caching on the stable part of the prompt, routing simple steps to smaller models, and retrieval that sends few relevant passages instead of many.
The pilot is always cheap. Four people from the team ask an internal assistant twenty questions each over two weeks, the invoice comes in at around twelve euros, and everyone concludes that AI costs nothing. Then the thing gets opened to forty users who use it for actual work, and the second invoice starts a conversation with the finance office.
Nothing went wrong. The pilot simply measured a different system. Real usage changes two variables at once: the number of requests and the size of each request, because a question asked in earnest carries documents, history and instructions that a demo question does not.
Why the pilot lies
During a pilot people ask short, clean questions and stop after one answer. In production they ask follow-ups, and every follow-up resends the whole conversation. A five-turn exchange does not cost five times the first question, it costs closer to fifteen, because turn five carries turns one to four with it.
The second thing that grows is the prompt itself. It starts as three lines of instructions and ends up as two pages, because every wrong answer during testing gets patched by adding a rule. That page and a half of accumulated instructions is paid for on every single request, forever.
Where the bill comes from
You pay for tokens in and tokens out, and output tokens cost several times more than input ones. That asymmetry is worth remembering: an assistant that answers in three sentences instead of three paragraphs is not just nicer to read, it is materially cheaper, and asking for brevity in the instructions is the least effort you will ever spend on cost control.
The dominant cost, though, is usually not the question. It is the context. An assistant that answers over internal documents typically sends eight to ten thousand tokens of retrieved material per question. At three hundred questions a day that is roughly three million input tokens a day, before anyone has written a word of reply. Multiply by the working days in a month and you have your real number, which is almost never the number from the pilot.
Then there are the invisible items. Retries after a timeout, requests where the model is called twice because a validation step rejected the first answer, background jobs that reprocess documents every night whether they changed or not. In the projects we audit, this category is often a quarter of the total and nobody has ever looked at it, because it does not appear in any interface.
Three levers that actually move it
Cache the stable part of the prompt. System instructions, tool definitions and any document set that does not change between requests can be cached by the provider, and cached input costs a fraction of fresh input. This requires exactly one thing from the code: keep the stable material at the beginning of the prompt and the variable material at the end. Teams that interleave the two pay full price for everything.
Use the right model for each step. Most AI features are several steps wearing one coat: classify the request, pull the relevant data, produce an answer, check it. Only one of those steps needs the expensive model. Routing the classification and the extraction to a small model, and keeping the large one for the final answer, tends to cut the bill by more than half without a visible difference to the user.
Send fewer, better documents. The habit of retrieving twenty passages and letting the model sort it out is expensive and, more annoyingly, makes answers worse: the relevant passage gets diluted. Three good passages usually beat twenty mediocre ones on both counts. Improving retrieval is the one optimisation that reduces cost and increases quality at the same time, which is why we start there.
Latency is a cost that never appears on the invoice
An assistant that takes nine seconds to answer gets abandoned, and an abandoned feature costs whatever it cost to build. Streaming the answer as it is generated changes the perception more than any speed-up: the user reads while the model writes. Steps that do not depend on each other should run in parallel, and anything that can be computed before the user asks, such as a nightly summary of yesterday's tickets, should not be computed while they wait.
What to measure from day one
Log one row per request: which model, input tokens, output tokens, whether the cache was hit, latency, which feature called it, which user. It is half a day of work and it is the difference between managing a cost and receiving it. Without those rows, the only lever anyone can pull when the invoice grows is switching everything to a cheaper model, which is the crudest possible answer.
The number worth watching is not the monthly total. It is the cost per resolved case: per answered question, per processed invoice, per handled ticket. That figure can be compared with what the same work costs done by hand, and it is the only version of the conversation that a management team can actually decide on. A daily spend alert on top of it prevents the classic surprise of a loop that ran all weekend.
When not to use a model at all
A surprising share of what gets sent to an LLM is deterministic work. Extracting a VAT number from a structured document, deciding which of five categories a request belongs to when the rule is written in a manual, converting a date format: these are cheaper, faster and more reliable as ordinary code. Using a model for them costs money on every call and introduces the one thing the task did not have before, which is variability.
The useful question is never how much AI costs. It is how much this specific answer costs and what it is worth. On the AI projects we take on, we ask for that number before writing anything, because a feature nobody can price is a feature nobody will keep.
