The Real Cost of Generative AI in Production

Cheaper tokens, bigger bills

Two facts about generative AI costs look contradictory. The price of a given level of capability has collapsed: Epoch AI measured declines of between 9 and 900 times per year depending on the task. Yet McKinsey's 2026 State of AI survey found that about one in five organizations is limiting its use of AI because of operating costs, including token costs.

Both are true because enterprises do not buy a fixed level of capability, and they do not consume a fixed number of tokens. Each new generation of applications sends more context, runs more steps, calls more tools and wraps more human and automated checking around the model. Price per token falls; tokens per outcome rise; and the costs that are not tokens at all grow with every system put into production.

The thesis of this article: manage generative AI on fully loaded cost per completed outcome, and fix the architecture before negotiating the price. In a typical production system, design choices about context, caching, loops and routing move cost by multiples, while list-price differences between comparable models move it by percentages. And the largest line items are often people: the reviewers who check output and the engineers who keep the system running.

The FinOps Foundation's guidance for AI makes a similar point in its own terms: the basic equation of price times quantity still applies, but AI services are charged inconsistently and in many variants, so unit metrics are essential. This article extends those unit metrics to the business outcome.

What you are actually billed for

All prices in this article are standard list prices from the providers' official pages as of 1 October 2026, in US dollars per million tokens. Enterprise agreements may differ.

A model call is billed on several meters, not one:

• Input tokens: instructions, retrieved documents, conversation history and tool definitions sent with every call.
• Output tokens: typically priced at five to six times input. Claude Sonnet 5.5 lists $2 input and $10 output; OpenAI's GPT-6.1 Sol lists $2 and $10; Google's Gemini 3.5 Flash lists $1.50 and $9.00.
• Cached input: a stable prompt prefix that the provider stores and re-reads at a steep discount.
• Tool overheads: Anthropic documents that enabling tools adds a system prompt of a few hundred tokens per call on current models; its computer-use toolset adds about 4,500 input tokens and its browser toolset about 6,600, before any screenshots or page content. Server-side web search on the Claude API costs $10 per 1,000 searches on top of tokens.
• Modifiers: discounts for asynchronous batch processing, and premiums for long context, faster output or guaranteed data location.

Two lessons follow from that table. First, the discounts are larger than the differences between vendors. Batch processing halves the price at all three providers; a cache read costs 10% of the input price on most Claude models, 5% on Opus 5.5 and 5% on GPT-6.1 Sol. Second, the premiums are easy to trigger by accident. Sending more than 272,000 tokens of context to GPT-5.5 doubles its input price; regional endpoints for Claude on Amazon Bedrock and Google Cloud carry a 10% premium.

Token counts themselves are not stable. Anthropic notes that the tokenizer introduced with Claude Opus 4.7 produces roughly 30% more tokens for the same text. And a newer model may finish the same task in fewer steps: Anthropic reports that on its agentic coding tasks, Opus 5.5 matched Opus 5's quality in about half the turns and output tokens, cutting the cost of that workload by 40 to 50% even though list prices fell only 20%. That is vendor material, but the principle is general: compare models on cost per completed task on your own evaluation set, never on price per token.

The multipliers that matter more than list price

Five design choices determine how many tokens each outcome consumes.

Context size. Every call re-sends its instructions, retrieved passages and history. A retrieval step that returns twenty passages when five would do multiplies input cost on every request, indefinitely.

Loop depth. Agents call the model repeatedly, and each step typically re-sends the growing context. Anthropic reported that in its own systems agents used about four times the tokens of chat interactions, and multi-agent systems about fifteen times, and that multi-agent designs only make sense where the value of the task is high enough to pay for them.

The arithmetic is unforgiving. In the illustration above, a ten-step agent that starts with 8,000 tokens of context and adds 2,000 per step sends 170,000 input tokens in total, more than twenty times a single call. Caching the stable prefix cuts the input cost of that run by about 70%, which is why caching is the first optimization to make in any agentic system.

Retries and failures. Failed tool calls, malformed outputs and timeouts are billed. A 10% retry rate is a 10% surcharge.

Verbosity and reasoning effort. Output tokens cost five to six times as much as input on the price lists above. Asking for structured, concise output, and setting reasoning effort to what the task needs, is a direct cost control.

Model choice per step. Not every step needs the most capable model. Classification, extraction and routing can often run on a small model at a fraction of the price, with the large model reserved for the hard cases.

The costs that are not tokens

A production system carries costs that never appear on a model provider's invoice:

• Evaluation: running regression suites on every prompt or model change, and sampling production outputs for automated or human grading.
• Human review: people checking uncertain or high-risk outputs before they reach a customer. Often the largest variable cost in the system.
• Platform: vector stores, gateways, logging, observability, guardrails and the security controls required by your risk tier.
• Engineering and operations: the team that maintains prompts, retrieval, integrations and evaluations, handles incidents and re-qualifies the system when models change.
• Change: migrating to a new model when the old one is retired, which requires re-running evaluations and often re-tuning prompts.

None of these are optional in a governed enterprise system, and all of them should be in the unit-cost model from the first business case.

A worked example: cost per resolved conversation

Consider an illustrative customer-service assistant handling 200,000 conversations a month. Each conversation takes four model calls of about 12,000 input tokens and 600 output tokens. Half of each call's input is a stable 6,000-token prefix of instructions, policies and tool definitions. Prices are list prices as of 1 October 2026; all volumes, rates and salaries are assumptions chosen to show the mechanism.

Tokens, including retries, are about 18% of the total. Human review and the engineering team are about 73%. Halving the review rate from 5% to 2.5%, by improving the evaluation set and the routing of uncertain cases, would save more than switching every call to a model at half the price.

Note the denominator. Per conversation handled, the system costs $0.34. Per conversation resolved, which is the outcome the business is paying for, it costs $0.49. The comparison that matters is that figure against the fully loaded cost of a resolution today, measured on the same definition. A business case that cannot show both numbers at production volume is, as argued in From AI Experimentation to Enterprise Value, not yet a business case.

Caching, batch and routing: what each actually saves

Applied in sequence to the token line of the same example:

• Baseline: every call on Claude Sonnet 5.5 with no caching costs $0.120 per conversation in tokens.
• Prompt caching of the 6,000-token prefix brings it to $0.077, a 36% saving. A cache read costs $0.20 per million tokens against $2 for fresh input. The writes needed to populate the cache are ignored here because, at this volume, the shared prefix is re-used thousands of times per write.
• Routing 60% of conversations to Claude Haiku 4.5, at half the price of Sonnet 5.5, plus a small classification call, brings it to $0.055, a total saving of 54%.
• Batch processing does not apply to live conversations, but halves the cost of everything that can wait: evaluation runs, back-office summarization, document classification and nightly enrichment.

The order matters. Caching is nearly free to implement and has no quality risk. Routing requires an evaluation set to prove the small model is good enough for the cases sent to it; without one, it saves money by quietly degrading quality. Neither depends on negotiating a lower price.

Self-hosting is a utilization decision

Running open-weight models on your own GPUs swaps a variable cost for a fixed one. Whether that pays depends almost entirely on how many tokens you can push through the hardware, around the clock, at acceptable quality.

As a reference point, Lambda lists on-demand NVIDIA H100 capacity at $3.99 per GPU-hour on its 8-GPU instances as of 1 October 2026. An eight-GPU node running all month costs about $23,300. Add one engineer to operate the serving stack, at the same $15,000 a month used above, and the fixed cost is about $38,300 a month before storage, networking, redundancy or evaluation.

The break-even question is how many tokens a month the node must serve to beat the API alternative. The answer depends on which alternative you compare against:

• Against a mid-tier API model at list price, blended at three input tokens per output token ($4.00 per million for Sonnet 5.5), the node must serve about 9.6 billion tokens a month, a sustained 3,600 tokens per second, every second.
• Against the effective price in the worked example after caching and routing, about $1.07 per million, the requirement rises to about 36 billion tokens a month, or 13,600 tokens per second.
• Against a small API model such as OpenAI's GPT-6 Luna, at $0.20 per million blended, it rises to about 190 billion tokens a month, or 73,000 tokens per second.

Whether a given node can sustain those rates, for your model and prompt lengths, is something only a load test will tell you. But the structure of the decision is clear. Self-hosting competes with the cheapest API option that meets your quality bar, after your own optimizations, not with the flagship list price. Traffic that peaks in business hours and idles overnight raises the effective cost per token proportionally.

Fine-tuned and custom models change this arithmetic again, since they add training, re-qualification and often hosting costs of their own; the trade-offs are covered in RAG, Fine-Tuning, Continued Pretraining, or a Custom LLM?.

There are good reasons to self-host that are not about price: data that must stay in a specific jurisdiction, latency, offline operation, or control over model versions. In the Gulf, residency is often the deciding factor. In February 2026 AWS made Claude models available from its UAE and Bahrain Regions through global cross-Region inference, which it says does not store customer data in the destination Region but may process requests elsewhere. Where Saudi or UAE rules, or a regulator's guidance for a sector, require in-country processing, the choice may narrow to in-region endpoints or self-hosting, and the business case should say so plainly rather than present the decision as a cost optimization.

A cost-control checklist

What to do in the next 90 days

1. Build a unit-cost model for each production AI system, with every line in the worked example above, and agree the outcome definition with the business owner.
2. Instrument cost per completed outcome and the human review rate, alongside token spend, on a monthly dashboard.
3. Turn on prompt caching for every system with a stable prefix and monitor the hit rate.
4. Move non-interactive workloads to batch processing, starting with evaluation and back-office jobs.
5. Pilot routing on the highest-volume system, using an evaluation set to confirm the smaller model meets the quality bar for the cases routed to it.
6. Set token budgets and alerts per application and per agent before scaling any agentic system.
7. Re-run the self-hosting break-even with your own effective prices and a load test before any GPU commitment.

Frequently asked questions

What does it cost to run generative AI in production?

It depends far more on architecture and operations than on token price. In an illustrative customer-service example at list prices as of 1 October 2026, tokens were under a fifth of the fully loaded cost per conversation; human review and engineering were nearly three-quarters.

How much does prompt caching save?

On current price lists a cache read costs a small fraction of fresh input: 10% on most Claude models and 5% on Claude Opus 5.5 and OpenAI's GPT-6.1 Sol. In the worked example, caching a 6,000-token prefix cut token cost by 36%; in a ten-step agent loop, it cut input cost by about 70%.

Why do AI agents cost more than chatbots?

Each step re-sends a growing context, calls tools and may retry. Anthropic reported that agents in its systems used about four times the tokens of chat, and multi-agent systems about fifteen times. Budgets should be set per agent and per task.

When does self-hosting an LLM become cheaper than an API?

Only when sustained utilization is high enough to beat the cheapest API option that meets the quality bar, after caching and routing. For an eight-GPU node, that can mean thousands to tens of thousands of tokens per second around the clock. Residency or latency requirements are often better reasons to self-host than price.

Sources and further reading

• The state of AI in 2026: On the road to ROI, McKinsey & Company, August 2026.
• Pricing, Anthropic documentation, accessed 1 October 2026.
• Pricing, OpenAI API documentation, accessed 1 October 2026.
• Gemini Developer API pricing, Google AI for Developers, last updated 1 October 2026.
• Introducing Claude Opus 5.5, Anthropic, 22 September 2026. Vendor material.
• How we built our multi-agent research system, Anthropic Engineering, 13 June 2025.
• LLM inference prices have fallen rapidly but unequally across tasks, Epoch AI, 12 March 2025.
• FinOps for AI Overview, FinOps Foundation, 17 February 2026.
• GPU cloud pricing, Lambda, accessed 1 October 2026.
• Amazon Bedrock global cross-Region inference for Claude in the Middle East Regions, AWS Machine Learning Blog, 24 February 2026.