AI Cost Governance: Why Token Spend Is the New Cloud Cost Problem

A year ago, most companies barely looked at their AI bill. Today, that bill is landing on the CFO’s desk, and executives are asking the obvious question: how did this get so big, so fast?

The numbers are hard to ignore: Uber, according to reporting from The Wall Street Journal, burned through its entire 2026 AI coding budget in four months, while Google now processes 3.2 quadrillion tokens per month (seven times what it processed a year earlier).

None of this is happening because businesses are being wasteful. If anything, it's happening because the tools work: adoption is real, and workflows that used to be slow or manual are moving faster because of them. What's happened is more structural: the way AI gets billed has collided with the pace at which people are using it, and the infrastructure most companies built for the cloud era was not designed for the economics of AI at production scale.

Where AI cost governance starts, and why cloud FinOps alone isn't enough

Before going further, a definition worth pinning down: a token is the small unit of text that AI models charge for, and every word sent to a model, along with every word it sends back, is broken into tokens and metered like electricity. For a sense of scale, a short sentence might be twenty tokens, while a complex agent completing a multi-step task can consume hundreds of thousands.

AI cost governance is the discipline of understanding, allocating, and controlling how those tokens get used across a company. That sounds close to FinOps, the practice teams already use to manage cloud spend, but there is a meaningful difference in how the two behave.

A meaningful share of cloud costs comes from infrastructure that sits idle and avoidable: servers left on when nobody is using them, storage that piles up, environments running through the weekend. That kind of waste is visible after the fact, which makes it relatively straightforward to clean up once teams start looking. 

AI consumption behaves differently: every prompt is a live economic decision, invisible until the invoice arrives, and heavily influenced by how the AI feature or application generating that prompt was built weeks or months earlier. 

That gap is exactly why AI cost governance is hardening into its own discipline, separate from cloud FinOps. The signals are already institutional: the Linux Foundation launched the Tokenomics Foundation in mid-2026 specifically to standardize how AI usage gets billed, and FinOps X, the industry's largest FinOps conference, made token economics its central theme this year for the first time. Neither of those events changes what shows up on the invoice. What they signal is that this stopped being a side effect of adopting new technology and started being something companies are expected to have a handle on, the same way FinOps itself became a baseline expectation once cloud spend got large enough to matter. 

The paradox behind exploding AI bills

Something counterintuitive has been happening in the market: the price per token charged by AI providers has dropped roughly 280 times in two years, yet total AI spend at large companies keeps climbing, in many cases doubling every few months. Across the enterprise landscape, three dynamics tend to explain most of that gap: 

1. The shift from chatbots to agents. A traditional chatbot answers one question with one response, and its cost is easy to estimate. An agent is an AI system designed to complete a full task without step-by-step human input: it checks its own work, coordinates across different tools, and sends the model request after request before landing on a final answer. Goldman Sachs has projected that enterprise token consumption will multiply roughly 24 times between 2026 and 2030 as agentic systems become common, and the mechanic behind that projection is straightforward: a single prompt from a user can trigger dozens of requests to the model behind the scenes, and every one of them shows up as consumption. None of this argues against building agents. They are precisely where AI is producing the most operational value right now. It does argue for designing them with cost in mind from the start, which is a solvable problem.
2. A cultural pattern that has surprised finance teams. Some companies have encouraged their employees to use AI as intensively as possible, in some cases with internal leaderboards ranking who consumes the most tokens. CIO.com reported one case where a single Disney employee interacted with Claude 460,000 times in nine days. Enthusiasm from teams is a good sign that adoption is real, but at that scale, without any tracking, it stops being adoption and turns into a budget event nobody planned for.
3. Retry storms and silent iteration. The third dynamic is quieter, and it's the one that surprises finance teams most in production reviews. Modern AI systems retry, refine, and iterate on their own responses, especially when they run into ambiguity, so a single user request can quietly produce twenty model requests before the user sees anything on screen. Well-designed systems reduce this behavior through what's called context engineering, which is the practice of shaping the inputs, constraints, and guardrails the model works within so it doesn't have to iterate blindly. Without that discipline, the behavior compounds fast.

This is the same logic we laid out in The Cost of Decision Friction When Exceptions Run the Workflow: in most production workflows, only a small share of cases are actually ambiguous enough to need a model's judgment, while the rest follow a pattern the system already recognizes. Sending every case to the model treats all of them as equally hard, which is expensive and unnecessary. Routing the repeatable share to fixed logic, and reserving the model for what genuinely needs it, keeps cost tied to complexity instead of volume. 

What token spend actually moves inside the P&L

For a mid-market company, this stops being an IT topic the moment it starts changing the P&L. When a product uses AI to serve customers, every request becomes a variable cost of goods sold, which has direct implications for margin, particularly on features that used to look almost free to operate. Cash flow becomes harder to forecast, because consumption scales with usage in ways that traditional software licensing never did, and annual budgets locked in during the fourth quarter often look outdated by the end of the first.

Board conversations are shifting to match. The question is no longer "are we using AI?" but something closer to "what did the AI spend this quarter, in dollars, net of what we paid for it?" Productivity gains that used to satisfy leadership no longer close the loop. We wrote recently about how CFOs are approaching AI proposals, and the pattern is consistent: the ROI conversation has moved from "hours saved" to "impact on a specific line of the financials, net of infrastructure and token cost."

What sits behind that shift is a harder demand: turning AI investment into ROI that a board or a sponsor can actually defend. Faster response times only count if they translate into retained revenue or higher conversion. Throughput faces the same test: processing documents in seconds is worth celebrating only when it shows up as reduced headcount cost or expanded capacity without new hires. AI-generated content clears the bar the moment it moves a metric someone was already tracking. In every case, the dollar figure has to trace back to a specific line of the financials, or it doesn't count.

There is a positive version of this shift. Companies that develop the discipline to answer those questions calmly build real credibility with the people who matter most: their own leadership, their board, and, for PE-backed businesses, the sponsor. Cost visibility is turning into a value creation lever, not just a control mechanism.

The bill is decided long before it arrives

Most teams treat AI consumption as something finance will optimize later, but by the time the invoice arrives, much of the cost has already been shaped by architectural decisions made months before. A dashboard can show where the money went after the fact, yet it cannot change the design choices that created that spend in the first place. In AI, the bill starts forming at the design stage, which is also where some of the biggest cost-saving opportunities tend to hide until a team takes a closer look.

Four practices tend to make the difference between AI systems that scale affordably and systems that don't. These are the ones we build into every agentic system we take to production:

1. Model tiering. Not every task needs the most powerful model available. Sending everyday tasks to smaller, cheaper models, and saving the top-tier one for the questions that genuinely need it, typically cuts spend by more than half without a visible effect on output quality. It's the highest-leverage decision in most systems and one of the most often skipped.
2. Structured workflows instead of prompt-everything. Many production systems today send every request to a large model, even when most of the work is routine. A structured design uses the model only where reasoning is actually needed, and handles the rest with fixed logic that costs a fraction as much. We covered this in detail in a recent piece on structured AI workflows.
3. Semantic caching. When users ask similar questions repeatedly, the answer can be reused instead of regenerated. Caching sounds like a small optimization but at production volume, it's usually the difference between predictable costs and margin surprises.
4. Prompt hygiene. Long, verbose prompts consume tokens on every call. Cleaning them up sounds trivial and rarely gets prioritized, but the compounding effect across millions of requests turns out to be meaningful.

Nothing about these is technically exotic. What tends to happen is that AI systems start as quick proofs of concept, using the shortest path from prompt to answer, and by the time usage grows the shortest path has become the production path. Refactoring later is possible but costly. The teams that avoid the surprise are the ones that treat cost as a design decision from day one.

How this looked in practice: Doppler's ECO IA

Here's how this played out in one of our recent projects. Doppler is one of Latin America's most established digital marketing platforms, with over 20 years in the market and thousands of businesses managing campaigns across email, SMS, WhatsApp, and other channels through their tools. In 2026 they set out to embed an AI assistant, ECO IA, directly into their platform, so users could get diagnostics on their account performance and generate campaign drafts through natural conversation.

For a product with thousands of users interacting with it daily, the cost profile of a customer-facing AI feature is not a detail. It's the difference between a product that scales and one that quietly compresses margins as adoption grows. We approached the build the same way we've been describing throughout this article: model tiering, caching, prompt hygiene, and real-time monitoring baked into the design, not added afterward.

The production beta launched in eight weeks. Prompt caching hit a 96% cache hit rate, which means only about four of every hundred requests were paying full inference cost. Time from insight to campaign draft dropped 4x compared to the previous manual flow, and the beta ran with zero errors.

What made the difference was applying model tiering, caching, prompt hygiene, and monitoring together, before the first user ever saw the product. That's usually the pattern: the numbers look impressive at the end, but the decisions that produced them were made at the start. 

Visibility, guardrails, and unit economics

Once the architecture is doing its job, the operational side becomes much more manageable. The pieces that matter at this layer, and that we usually put in place with clients as usage scales, are:

• Tagging every AI call by team, feature, and environment, so each cost has a name attached to it and someone accountable for it.
• Real-time budget alerts, so a runaway process gets caught in hours rather than at the end of the month.
• Tracking cost per outcome (per resolved ticket, per processed document, per completed transaction) instead of only cost per token.
• ROI thresholds that new AI initiatives have to clear before rollout, so pilots don't quietly turn into permanent line items.

Cost governance is one dimension of a broader AI governance framework that also covers risk, quality, and accountability. Treating them as separate exercises tends to duplicate work, while treating them as one system, with cost as one lens among several, produces cleaner outcomes and fewer surprises at the leadership level.

What a mid-market approach looks like

Most of the AI cost governance conversation today is being written by and for hyperscalers, which is a useful signal but leaves out the reality of a company between USD 50 million and USD 500 million in annual revenue. Absolute AI spend at that scale is smaller than at a Google or a Microsoft, but sensitivity to every dollar is higher, dedicated FinOps teams don't exist yet, and there is no bandwidth to build one before AI starts producing measurable value.

For companies in that position, the sequence that tends to work has three moves.

1. Mapping: Get a clear picture of where AI is actually being used today, formal or not. Most mid-market companies discover the real footprint is broader than the version on paper.
2. Designing: Take the higher-leverage use cases into a structured build that bakes in model tiering, caching, and guardrails from day one, so the cost profile is defensible before rollout rather than after the surprise.
3. Operating: Governance, tagging, and unit economics layer on as usage matures, giving the finance side the visibility it needs without slowing the engineering side down.

Sequenced that way, it's less overwhelming than it sounds.

This is the terrain our work on Agentic AI Solutions covers. The systems we design are meant to reach production and stay there, which means engineering around cost, quality, and coverage together rather than treating cost as a downstream concern that gets solved later. For teams still figuring out where to begin, our piece on how PE-backed companies choose their first AI initiative walks through the earlier decision.

Three questions to ask about your AI spend this quarter

If you want a fast diagnostic, three questions tend to reveal how much attention this area deserves in the next quarter:

1. What percentage of our current token spend is tied to a business outcome we can actually measure? If most of it is exploratory or hard to attribute, you have less signal than the invoice suggests.
2. What's the ratio between what we pay AI providers and the value delivered to the end customer? In cost-sensitive features, that ratio is the difference between margin expansion and margin compression.
3. What happens to our AI budget if usage doubles next quarter, which it probably will? If the answer involves a scramble, this is the right moment to invest in cost governance rather than wait for the surprise.

The companies that come through this transition ahead are the ones that stop thinking about AI consumption as an infrastructure expense and start treating it as a product design decision. That shift, from downstream cost to upstream choice, is where structural advantage builds. Because when the design is right, the invoice stops being a surprise. It becomes a variable you already know how to move.

If your AI spend is climbing faster than expected, or you're about to scale a feature and want to get the cost profile right before rollout, that's a conversation worth having.