Measuring the Business Value of Enterprise AI

Why the usual evidence does not hold up

Ask how an enterprise knows its AI tools are working and the answer is usually a survey. Users are asked how much time they save; the answer is multiplied by headcount and a loaded salary; the result goes into a board pack. The problem is that people are poor judges of their own productivity with AI, and the errors are large.

The clearest demonstration comes from a randomized controlled trial run by METR, a research nonprofit, and published in July 2025. Sixteen experienced open-source developers worked on 246 real issues in their own repositories, with each issue randomly assigned to allow or disallow AI tools. Before starting, the developers expected AI to speed them up by 24%. Afterwards, they estimated it had sped them up by 20%. Measured against the randomized comparison, they took 19% longer with AI.

That study concerns one population, one period and early-2025 tools, and it should not be read as a verdict on AI in software engineering. Its lesson is about measurement: a self-reported 20% gain and a measured 19% loss can describe the same work.

The same gap appears at the scale of a national economy. Humlum and Vestergaard linked Danish adoption surveys to administrative records covering earnings and hours. Workers reported productivity gains and most employers in exposed occupations had adopted chatbot initiatives, yet the authors found precise null effects on earnings and recorded hours, ruling out effects larger than 2% two years after ChatGPT's launch. Their interpretation is that employers absorbed AI through task reorganization rather than through changes visible in pay or hours. Either way, the self-reported gains did not show up where an economist, or a CFO, would look for them.

The argument of this article is simple: treat every material AI deployment as an experiment whose measurement is designed before launch. That means five decisions, made in advance and signed off by the business owner and finance:

1. What outcome counts as value, and how it is defined.
2. What the baseline is, measured on the old process.
3. What the comparison group is.
4. Which leading indicators will be watched while the lagging result accumulates.
5. What the fully loaded cost per outcome is.

Choosing which initiatives to pursue is covered in How to Select Enterprise AI Use Cases. Deciding whether an initiative deserves further funding at each stage is a separate question, covered in From AI Experimentation to Enterprise Value. This article is about producing evidence those decisions can rely on.

Define value as a chain, not a number

A single headline metric invites argument. A chain of metrics, from system behaviour to financial result, lets each party see where value is created or lost, and gives early warning long before the financial result is visible.

The chain has four links:

• System quality (days): accuracy on an evaluation set, groundedness, escalation rate, latency. These tell you whether the tool works.
• Adoption and behaviour (weeks): share of eligible work routed through the tool, acceptance or edit rate of suggestions, overrides. These tell you whether people use it.
• Operational outcome (one to three months): throughput, cycle time, first-contact resolution, error and rework rates, customer satisfaction. These tell you whether the work changed.
• Financial outcome (a quarter or more): cost per case, revenue, losses avoided, working capital. These tell you whether the business changed.

The first two are leading indicators; the last two are lagging. A program that reports only system quality is reporting engineering progress. A program that waits for the financial result without watching the leading indicators will discover problems a quarter too late.

Two rules make the chain usable. First, fix definitions before launch: what counts as "resolved", over what window, excluding which case types. Changing definitions after launch is the most common way an AI result becomes unfalsifiable. Second, measure quality alongside speed. The customer-support study discussed below measured resolutions per hour, but also customer sentiment, escalation requests and agent attrition. A speed gain bought with more rework is not a gain.

Measure the baseline properly

A baseline is not last month's average. It needs to capture what the metric looks like across a normal cycle of the business, including its variance.

• Cover at least one full business cycle. Month-end, seasonal peaks and policy changes all move operational metrics. A baseline taken in a quiet month will flatter almost any intervention.
• Record the distribution, not just the mean. If handling time varies widely between agents or case types, a small average improvement may be indistinguishable from noise.
• Use the same data source and definition that will be used after launch. Comparing a manually sampled baseline with system telemetry afterwards is comparing two different things.
• Note what else is changing. New hires, a revised process, a pricing change, a system migration. These are the confounders the comparison group will need to absorb.

For software engineering, the equivalent baseline is delivery performance: deployment frequency, lead time, change failure rate and recovery time, recorded before any coding assistant is rolled out, as discussed in Claude Code vs. OpenAI Codex for enterprise engineering.

Build a counterfactual

The central question of AI measurement is not "did the metric improve?" but "how much better is it than it would have been without AI?" Answering that requires a comparison. The options form a ladder, from strongest to weakest.

Randomized assignment. Users, teams or cases are randomly assigned to have AI or not. This is the gold standard and is more feasible than it sounds for internal tools: Google ran a randomized trial with 96 of its own software engineers on an enterprise-grade task and estimated that AI shortened time on task by about 21%. The authors note the confidence interval was wide, a reminder that a small sample can only detect a large effect.

Staggered rollout. If everyone will get the tool eventually, roll it out in waves and compare early groups with those not yet switched on. This is how Brynjolfsson, Li and Raymond studied a generative AI assistant used by 5,172 customer-support agents, published in The Quarterly Journal of Economics in 2025. The staggered introduction let them estimate a 15% average increase in issues resolved per hour, with larger gains for less experienced workers and small gains in speed with small declines in quality for the most experienced. Most enterprises roll out in waves anyway, for training and support reasons. Deciding the wave order deliberately, and recording it, costs almost nothing and turns a deployment into evidence.

Matched comparison. Where randomization or staggering is impossible, compare against similar teams, regions or periods that did not receive the tool, matched on baseline performance and mix of work. This is weaker because the groups may differ in ways you cannot see.

Before and after. Comparing the metric before and after launch, with no comparison group, is the weakest design. It attributes to AI everything else that changed in the same period. It is acceptable only for small investments and should be labelled as such.

Two practical rules apply at every rung. Decide the minimum effect worth detecting before launch and check the pilot is large enough to see it; otherwise a null result is uninformative. And keep the comparison group long enough to cover the period when novelty effects fade.

A worked example: the difference a comparison group makes

Consider an illustrative claims operation with four regional teams. Two teams receive an AI triage and drafting assistant in month one; the other two receive it in month four. In the meantime, all four teams also adopt a new triage rule unrelated to AI.

The early teams' throughput rises from 20.0 to 23.0 claims per handler per day, a 15% improvement. A before-and-after report would claim all of it for AI. But the later teams, who had no AI yet, also improved, from 19.0 to 20.0, because of the new triage rule. The difference in differences, the early teams' gain minus the later teams' gain, is 2.0 claims per day, or 10%. A third of the headline gain was not AI.

That is still a strong result, and now it is one that survives scrutiny. When the later teams switch on in month four, the effect can be checked again on a second group, which is the cheapest replication an enterprise will ever run.

Attribution when several things change at once

AI is rarely deployed alone. It arrives with a process redesign, new training, sometimes a reorganization. Three rules prevent double counting and endless argument:

• Attribute to the initiative, not the technology. If AI and a workflow redesign were deployed together, report the combined effect and do not split it artificially. Value comes from the redesign that AI enables, and pretending otherwise invites the claim that the redesign alone would have done it.
• Keep one ledger across initiatives. When three programs each claim the same reduction in contact volume, the enterprise counts it three times. A single value ledger owned by finance, with each gain claimed once, prevents this.
• Separate one-time from recurring effects. A backlog cleared once with AI help is real value, but it does not recur. Report it separately from run-rate improvements.

Count cost per outcome, not cost per token

The denominator matters as much as the numerator. Token spend divided by requests tells you the cost of computation. It does not tell you the cost of a result. The FinOps Foundation's guidance for AI recommends unit metrics such as cost per inference and cost per token; for business measurement, carry the unit one step further, to fully loaded cost per completed outcome: per resolved case, per approved claim, per qualified lead.

The numerator should include model and platform costs, evaluation, human review of AI output, and the people who operate the system. The denominator should count only outcomes that meet the quality standard, not every attempt. A tool that is cheap per request but generates rework is expensive per outcome, and only this measure will show it. Comparing cost per outcome with and without AI, over the same period and against the same comparison group, gives finance a number it can reconcile.

Getting finance to sign before the build

Finance sign-off at the end of a project is a negotiation. Sign-off at the start is a contract. A one-page measurement plan, agreed before build, should state:

Most of this is the discipline an analytics team already applies to pricing tests or marketing campaigns. AI programs have simply been exempted from it. Where an organization already runs controlled experiments in marketing, pricing or operations, the same analysts and tooling can usually be pointed at AI with little new investment.

What to do in the next 90 days

1. Audit current AI value claims and classify the evidence behind each: randomized, staggered, matched, before-and-after or self-reported. Expect most to fall in the last two categories.
2. Write a one-page measurement plan for every AI initiative entering production, using the eight elements above, signed by the business owner and finance.
3. Turn the next rollout into a staggered one with a recorded wave order, even if the whole population will be live within a quarter.
4. Fix metric definitions and baseline them over a full business cycle before launch, with the same data source that will be used afterwards.
5. Instrument the leading indicators (adoption, acceptance rate, escalation, quality on an evaluation set) on a weekly dashboard beside the lagging outcome.
6. Introduce cost per completed outcome as the standard unit for every production AI system, reported monthly.
7. Create a single AI value ledger owned by finance, so each gain is claimed once.

Frequently asked questions

How do you measure the ROI of enterprise AI?

Define the outcome metric and its baseline before launch, compare against a control or not-yet-treated group rather than relying on before-and-after figures or user surveys, track leading indicators such as adoption and quality while the financial result accumulates, and calculate fully loaded cost per completed outcome.

Are user surveys a reliable way to measure AI productivity?

Not on their own. In METR's 2025 randomized trial, experienced developers believed AI had sped them up by 20% when they were measured as 19% slower. Surveys are useful for adoption and sentiment, but productivity claims need a measured comparison.

What is a staggered rollout and why does it help measurement?

A staggered rollout introduces an AI tool to groups in waves. Groups not yet switched on act as a comparison for those already using it, which separates the AI effect from other changes over the same period. It was the design used in the 2025 Quarterly Journal of Economics study of 5,172 customer-support agents.

What are leading and lagging indicators for AI value?

Leading indicators move within days or weeks: system quality, adoption, acceptance rates and escalations. Lagging indicators take months: operational outcomes such as throughput and rework, and financial outcomes such as cost per case or revenue. Watch the leading indicators to catch problems before the lagging result is in.

Sources and further reading

• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 10 July 2025.
• Generative AI at Work, Erik Brynjolfsson, Danielle Li and Lindsey Raymond, The Quarterly Journal of Economics 140(2), 889–942, 2025. Working-paper version: NBER Working Paper 31161.
• Still Waters, Rapid Currents: Early Labor Market Transformation under Generative AI, Anders Humlum and Emilie Vestergaard, NBER Working Paper 33777, May 2025, revised March 2026.
• How much does AI impact development speed? An enterprise-based randomized controlled trial, Elise Paradis and colleagues, Google, arXiv, November 2024.
• FinOps for AI Overview, FinOps Foundation, 17 February 2026.