Did the AI Pilot Work? Before-and-After vs. Difference-in-Differences, Side by Side

An analyst stands before a wall of glowing bar charts where one tall coral bar casts a misleading glow over the room, illustrating how common AI pilot measurements overstate ROI

The pilot report you will be asked to sign

In August 2025, MIT's NANDA initiative reported that about 95% of enterprise generative AI pilots showed little or no measurable impact on profit and loss. The figure came from 150 leader interviews, a survey of 350 employees and a review of 300 public deployments, so it is a rough read on the market, not a controlled study. It still matches what I see when a pilot reaches the budget meeting. Most decks show a before-and-after chart, or the pilot teams next to everyone else, and both can mislead the people deciding whether to scale.

Perception is worse. In METR's 2025 randomized trial, 16 experienced open-source developers expected AI tools to make them 24% faster. After the study they believed they had been 20% faster. Measured against randomly assigned tasks without AI, they took 19% longer. A pilot judged by how people feel about the tool can report a gain that does not exist.

This piece runs the three common ways to measure an AI pilot on the same data, where the true effect is known because I simulated it. The tool raises productivity by 6%. Before-and-after says 11.4%. Comparing pilot teams with everyone else says 21.9%. Difference-in-differences says 4.6%, with a 95% range of 2.4% to 6.8% that contains the truth. Turned into first-year ROI, those answers become 144%, 369% and a range of minus 49% to plus 45%, against a true 28%.

You get the one equation that explains all three answers, runnable Python for the simulation and the estimators, a pre-trend check and why it often misses, and the ROI arithmetic that turns an effect into a number a CFO can sign. Every figure here can be reproduced in the live AI pilot ROI demo, which runs the same model in your browser.

Did the AI Pilot Work? Three Ways to Measure It · 1 min 13 s

Video transcript

Same AI pilot. Same data. Three answers. Before vs. after: 11.4%. Pilot vs. everyone else: 21.9%. Difference-in-differences: 4.6%. The true effect is 6%. Only one of these methods gets close to the truth.

Answer 1 of 3, before vs. after: it credits the tool for the whole quarter. A new knowledge base, a process fix or a slow season lands in launch week. In the simulation, a 7% company-wide change arrived with the tool's 6%. Across 1,000 simulated pilots this method averaged 13.4%, more than double the truth.

Answer 2 of 3, pilot vs. everyone else: it credits the tool for who volunteered. Pilot teams were already 12% ahead before the tool arrived. This method averaged 19.1%, three times the real effect, and the gap never closes.

Answer 3 of 3, difference-in-differences: subtract what would have happened anyway. Comparison teams changed by 7.0% and pilot teams by 13.4%; the difference, about 6%, is the tool. The company-wide change and the head start both cancel. Across 1,000 simulated pilots it averaged 6.0%, and its 95% range held the truth 93.3% of the time.

Why the method matters: same pilot, very different first-year ROI. The true effect is worth an ROI of 28%. Before vs. after implies 144%, pilot vs. everyone else implies 369%, and difference-in-differences gives a range of minus 49% to plus 45%. The honest answer is the last one: too close to break-even to call from 12 teams. Widen the pilot before signing a company-wide contract.

The check that often misses: when pilot teams were already improving 1.5% a month faster, the standard pre-launch test flagged it only 38.9% of the time. Assign the tool at random when you can.

Try it with your own numbers at vector1.ai/demos/pilot-roi. It is free and runs in your browser. The full article, Python code and ROI model are on vector1.ai.

One equation behind all three answers

Write each team's weekly productivity in logs, so effects read as percentages:

y_it = α_i + λ_t + δ G_i Post_t + ε_it

Here y is log tickets resolved per agent per week for team i in week t. The term α is the team's own level, which differs because some teams are simply stronger. The term λ is everything that hits every team in a given week: seasonality, a new knowledge base, a reorganization. G is 1 for pilot teams and 0 otherwise, Post is 1 after launch, and δ is the tool's effect, the only number the pilot exists to find. ε is noise.

Each measurement method is a different average of this equation, and each one lets in a different extra term. Before-and-after averages the pilot teams' change, so it returns δ plus the company-wide change Δλ. Comparing pilot teams with everyone else after launch returns δ plus the head start the pilot teams had, the difference in α. Difference-in-differences subtracts the comparison teams' change from the pilot teams' change, so both extra terms cancel and it returns δ, as long as the two groups would have moved in parallel without the tool.

The simulation in this piece, and in the demo, uses a true effect of 6%, pilot teams that were already 12% ahead (volunteers, or teams hand-picked by a confident manager), and a 7% company-wide improvement at the same time as launch. The equation predicts the bias before any data exists. Before-and-after should land near 1.06 × 1.07 − 1 ≈ 13.4%. The cross-section should land near 1.06 × 1.12 − 1 ≈ 18.7%. Difference-in-differences should land near 6%. Across 1,000 simulated pilots of 12 teams per group and 12 weeks on each side of launch, the three methods averaged 13.4%, 19.1% and 6.0%. The single run in the opening (seed 3 in the demo) gave 11.4%, 21.9% and 4.6%, which is what one real pilot looks like: the right method still misses by a point or two, and the wrong ones miss by a lot more.

Method 1, before and after: the tool gets credit for the quarter

Before-and-after is the default because it needs no comparison group. Take the pilot teams' average productivity for the weeks before launch, take it again for the weeks after, and report the change. In the equation, that difference is δ + Δλ. Whatever else changed between the two windows rides along.

Something else almost always changes. AI pilots tend to launch alongside a new knowledge base, a process cleanup or a fresh round of training, because the team building the pilot fixes the obvious problems on the way in. Seasonality does the rest: a support center that launches in late January compares its post-holiday lull with its holiday rush. In the simulation, a 7% company-wide improvement lands in the same week as the tool. Before-and-after reports 11.4% on this run and 13.4% on average, roughly double the true 6%.

Two other effects push the same way. If a team was chosen for the pilot after a bad quarter, part of its rebound is regression to the mean, which would have happened without any tool. And a team that knows it is being watched often works differently for a few weeks. Neither shows up as a separate line in a before-and-after chart. Both get booked as AI.

This is the simulation behind every number in the piece. It is plain Python with no dependencies, and it reproduces the demo exactly (same random number generator, same model).

"""pilot_sim.py: simulated AI pilot with a known true effect (Brian Curry | Vector1 Research)."""
import math


def imul(a, b):
    """32-bit integer multiply, matching JavaScript's Math.imul."""
    r = ((a & 0xFFFFFFFF) * (b & 0xFFFFFFFF)) & 0xFFFFFFFF
    return r - 0x100000000 if r & 0x80000000 else r


def rng(seed):
    """Seeded standard-normal generator (mulberry32 + Box-Muller), identical to the browser demo."""
    a = seed & 0xFFFFFFFF

    def u():
        nonlocal a
        a = (a + 0x6D2B79F5) & 0xFFFFFFFF
        t = imul(a ^ (a >> 15), a | 1) & 0xFFFFFFFF
        t ^= (t + imul(t ^ (t >> 7), t | 61)) & 0xFFFFFFFF
        return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296

    def normal():
        x = u() or 1e-12
        return math.sqrt(-2 * math.log(x)) * math.cos(2 * math.pi * u())

    return normal


def simulate(effect=0.06, head_start=0.12, company_change=0.07, trend_gap=0.0,
             teams=12, weeks=12, seed=3):
    """Weekly log productivity per team. Pilot teams are the first `teams` entries.

    effect          true effect of the tool (0.06 = 6%)
    head_start      how far ahead pilot teams already were (selection)
    company_change  change that hits every team at launch time
    trend_gap       extra monthly growth of pilot teams before and after launch
    """
    normal = rng(seed)
    lvl, com, eff = math.log1p(head_start), math.log1p(company_change), math.log1p(effect)
    slope = math.log1p(trend_gap) / 4.33          # monthly rate to weekly
    out = []
    for j in range(2 * teams):
        pilot = j < teams
        base = math.log(40) + 0.15 * normal()      # about 40 tickets per agent per week
        series = []
        for t in range(-weeks, weeks):
            post = t >= 0
            y = base + (lvl + slope * (t + weeks) if pilot else 0.0)
            y += com if post else 0.0
            y += eff if (pilot and post) else 0.0
            series.append(y + 0.06 * normal())     # weekly noise
        out.append((pilot, series))
    return out

The team-level noise (0.15 in logs) means teams differ by about 15% for reasons that have nothing to do with the tool, and the weekly noise (0.06) is what a busy support queue looks like week to week. The head start enters as a level for pilot teams only, and the company-wide change enters for everyone after launch. Change the arguments and the three methods move exactly as the equation says they should.

Method 2, pilot vs. everyone else: the tool gets credit for the volunteers

The second common chart puts the pilot teams next to the teams that didn't get the tool, after launch. It fixes the seasonality problem, because both groups live through the same weeks. It replaces it with a worse one. In the equation, the difference is δ plus the gap in α between the groups, and pilot groups are rarely a random draw.

Volunteers tend to be the teams with the strongest managers, the cleanest processes and the most patience for a new tool. Hand-picked teams tend to be the ones a sponsor expects to succeed. In the simulation, pilot teams start 12% ahead. The cross-section reports 21.9% on the demo run and 19.1% on average across 1,000 runs, three times the true effect. The longer the pilot runs, the more confident this chart looks, because the gap never closes.

Selection can also bias the answer down, which is a reason to take it seriously even when the number looks modest. In Brynjolfsson, Li and Raymond's study of 5,179 customer support agents, an AI assistant raised issues resolved per hour by about 14% on average, by about 34% for novice and lower-skilled agents, and by very little for the most experienced. A pilot staffed with your best people can understate what the tool would do for a new hire. Either way, the cross-section measures who you picked as much as what the tool did, and nothing in the chart tells you which.

Method 3, difference-in-differences: subtract what would have happened anyway

Difference-in-differences takes the pilot teams' change and subtracts the comparison teams' change over the same weeks:

δ̂ = (ȳ_P,post − ȳ_P,pre) − (ȳ_C,post − ȳ_C,pre)

P is the pilot group and C the comparison group. The first bracket contains δ plus the company-wide change. The second bracket contains the company-wide change alone, so subtracting removes it. The head start sits in both of the pilot group's averages, before and after, so it cancels inside the first bracket. What is left is the tool's effect plus noise, provided the comparison teams show what the pilot teams would have done without the tool. That proviso is the parallel trends assumption, and the next section is about checking it.

The standard error is where most hand-rolled pilot analyses go wrong. Weekly observations from the same team are not independent: a strong team is strong every week. Bertrand, Duflo and Mullainathan showed that treating them as independent can make placebo policies look significant up to 45% of the time at the 5% level. Their simplest remedy works well here: collapse each team to a single number, its average after launch minus its average before, and compare those numbers across teams. With n teams per group and team-level changes that vary with standard deviation s, the 95% range is the estimate plus or minus 1.96 times the square root of (s_P² / n_P + s_C² / n_C).

"""estimate.py: three pilot estimators on the same data, plus a pre-trend check."""
import math
from statistics import mean, variance

from pilot_sim import simulate


def team_changes(data, weeks):
    """Collapse each team to one number: mean log productivity after minus before."""
    pilot, comparison = [], []
    for is_pilot, y in data:
        change = mean(y[weeks:]) - mean(y[:weeks])
        (pilot if is_pilot else comparison).append(change)
    return pilot, comparison


def estimate(data, weeks):
    dp, dc = team_changes(data, weeks)
    post_p = [mean(y[weeks:]) for p, y in data if p]
    post_c = [mean(y[weeks:]) for p, y in data if not p]
    did = mean(dp) - mean(dc)
    se = math.sqrt(variance(dp) / len(dp) + variance(dc) / len(dc))
    pct = math.expm1
    return {
        "before_after": pct(mean(dp)),
        "pilot_vs_rest": pct(mean(post_p) - mean(post_c)),
        "did": pct(did),
        "did_low": pct(did - 1.96 * se),
        "did_high": pct(did + 1.96 * se),
    }


def slope(y):
    """OLS slope of a series against its index (per week)."""
    n = len(y)
    xbar, ybar = (n - 1) / 2, mean(y)
    return sum((i - xbar) * (v - ybar) for i, v in enumerate(y)) / sum((i - xbar) ** 2 for i in range(n))


def pre_trend(data, weeks):
    """Monthly gap in pre-launch growth, pilot minus comparison, with a 95% range."""
    sp = [slope(y[:weeks]) for p, y in data if p]
    sc = [slope(y[:weeks]) for p, y in data if not p]
    gap = (mean(sp) - mean(sc)) * 4.33
    se = math.sqrt(variance(sp) / len(sp) + variance(sc) / len(sc)) * 4.33
    return gap, gap - 1.96 * se, gap + 1.96 * se


if __name__ == "__main__":
    weeks = 12
    data = simulate(weeks=weeks, seed=3)
    for name, value in estimate(data, weeks).items():
        print(f"{name:14s} {value:+.1%}")
    gap, lo, hi = pre_trend(data, weeks)
    print(f"pre-trend gap  {math.expm1(gap):+.1%} per month ({math.expm1(lo):+.1%} to {math.expm1(hi):+.1%})")

Run it and you get the numbers from the opening: before-and-after 11.4%, pilot vs. the rest 21.9%, difference-in-differences 4.6% with a range of 2.4% to 6.8%. Across 1,000 simulated pilots the difference-in-differences estimate averaged 6.0%, with a run-to-run spread (standard deviation) of 1.0 point, and its 95% range contained the true 6% in 93.3% of runs, close to the 95% it promises. That is what a trustworthy estimator looks like: not exact on any single pilot, but honest about how far off it might be.

Check the assumption, and know what the check can't see

Difference-in-differences removes a head start in level. It does not remove a head start in growth. If the pilot teams were already improving faster than the comparison teams before launch, that momentum continues after launch and gets credited to the tool. You can't observe the after-launch counterfactual, but you can look at the months before launch, which is what pre_trend in the code does: it fits each team's pre-launch slope and compares the average slope across groups.

On the demo's default run the gap is minus 0.2% a month, with a range of minus 1.1% to plus 0.8%, so nothing is flagged and the estimate stands. Now give the pilot teams a real 1.5% a month head start in growth (trend_gap=0.015). Difference-in-differences reports 9.0% with a range of 6.7% to 11.3%, which excludes the true 6%. On this run the pre-trend check catches it: the gap comes back at 1.3% a month, range 0.3% to 2.3%.

The uncomfortable part is how often it doesn't. Across 1,000 simulated pilots with that same 1.5% a month gap, the check flagged the problem only 38.9% of the time, while the 95% range around the estimate contained the true effect in just 2.2% of runs. With a smaller 0.5% a month gap, the check flagged 8.8% of runs, barely above its 6.2% false-alarm rate when there was no gap at all, while the estimate drifted from 6.0% to 7.5% on average. Twelve teams and twelve weeks of history are not enough to see a modest difference in trend.

Jonathan Roth made this point formally: conventional pre-trend tests often have low power, and reporting results only when the pre-test passes can make the bias worse and the confidence intervals too narrow. The practical response is to treat a passed check as weak evidence, not proof. Choose comparison teams whose pre-launch trajectories already match the pilot teams, collect more weeks of history where you can, and report how much the estimate would move under a plausible trend gap. The cleanest fix is design, not diagnosis: when the business lets you assign the tool to teams at random, the trend difference goes away by construction.

From effect to ROI: the number the CFO actually signs

An effect size is not a business case until it becomes dollars. This is a proposed first-year ROI model that keeps the two assumptions people usually skip in plain view:

ROI = (k g W − (S + F)) / (S + F)

g is the measured productivity gain, W the annual labor cost of the people using the tool, and k the share of freed-up time the business actually puts to use. S is the annual seat cost and F the one-time rollout and training cost.

The capture rate k matters more than most pilots admit. Time saved turns into money only when someone decides what to do with it: absorb growth without hiring, cut overtime, clear a backlog. Humlum and Vestergaard linked surveys of about 25,000 Danish workers at 7,000 workplaces to payroll records and found average time savings of about 3% from AI chatbots, with only 3% to 7% of the productivity gains showing up in earnings. Saved minutes that nobody redeploys are not savings.

The demo's defaults: W = $12 million (the teams using the tool), k = 0.6, S = 120 seats at $60 a month, or $86,400 a year, and F = $250,000. Total first-year cost is $336,400. At the true 6% gain, the captured value is 0.06 × 0.6 × $12 million = $432,000, and ROI is 28%. The break-even gain is $336,400 / $7.2 million, about 4.7%.

Now run the three measured effects through the same formula. Before-and-after (11.4%) implies an ROI of 144%. Pilot vs. the rest (21.9%) implies 369%. Difference-in-differences (4.6%) implies minus 2%, and its range of 2.4% to 6.8% implies an ROI anywhere from minus 49% to plus 45%. The honest answer on this pilot is "probably positive, possibly not, and too close to break-even to call from 12 teams." That sentence is more useful to a budget owner than either of the confident wrong answers, because it says what to do next: extend the pilot or widen it.

The range narrows with the square root of the number of teams. In the simulation, the 95% range was about 4.1 points wide on average with 12 teams per group and about 2.1 points wide with 48. Quadrupling the pilot halves the uncertainty, which is often cheaper than a wrong decision on a tool the whole company will pay for.

Where it breaks

The simulation is kind to difference-in-differences in four ways that real pilots are not.

Staggered rollouts. Most companies don't flip a switch for every pilot team on one date; they add teams month by month. The usual regression for that case, with team and week fixed effects, quietly uses teams that already have the tool as the comparison for teams that get it later. Goodman-Bacon showed the estimate is a weighted average of every two-group, two-period comparison in the data, and it can be badly biased when the effect grows or fades over time. Callaway and Sant'Anna's estimator, which compares each rollout cohort only with teams not yet treated, is the standard fix, and open-source implementations exist in R and Python. With a single launch date, as in this piece, the simple version is fine.

Spillovers. Comparison teams are not sealed off. Agents share prompts in the hallway, a team lead forwards the good outputs, and some people use a personal AI account anyway. Every spillover shrinks the measured gap and biases the estimate toward zero. Comparison teams in a different site or shift help, and so does asking about tool use directly.

Novelty and learning. Twelve weeks may catch an early burst that fades, or miss a gain that builds as people learn the tool. METR, for example, has said its early-2025 developer results are out of date for later tools. Report the effect by month after launch, not just the average, and keep a small holdout group after the pilot ends so you can see whether the gain lasts.

The metric itself. A tool can raise tickets per hour by closing easy tickets faster and leaving the hard ones, or by resolving issues that reopen next week. Name a quality measure (reopen rate, customer satisfaction, error rate) next to the productivity measure before the pilot starts, and check it the same way. Finally, none of these methods fixes a survey. If the evidence is how people say they feel about the tool, the METR result says to expect a confident answer with the wrong sign.

The path forward: what to do with the next pilot

Start before the tool arrives. Write down the outcome metric, the quality metric beside it and the capture plan (what the saved hours will be used for) before anyone logs in. Pull at least eight to twelve weeks of history for every team, so there is a pre-period to compare and a trend to check.

Assign by team, not by person, so pilot and comparison agents don't share a supervisor or a queue. If you can, randomize which teams go first. If the business insists that everyone gets the tool, stagger the rollout so later teams serve as the comparison for earlier ones, and analyze it with an estimator built for staggered timing.

When the data is in, report the difference-in-differences estimate with its range, the pre-trend check with its range, and the ROI as a range using a stated capture rate. If the ROI range straddles zero, say so and size the next phase to narrow it, rather than rounding to the answer the sponsor hoped for.

You can try all of this on the live AI pilot ROI demo. Move the head start, the company-wide change and the trend gap, and watch the three methods pull apart while the pre-trend check tries to keep up. It runs the same model as the code above, in your browser, with nothing sent anywhere.

Sources

MIT NANDA, The GenAI Divide: State of AI in Business 2025, as reported by Fortune, August 2025
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 2025
Brynjolfsson, Li and Raymond, Generative AI at Work, NBER Working Paper 31161; Quarterly Journal of Economics, 2025
Humlum and Vestergaard, Large Language Models, Small Labor Market Effects, Becker Friedman Institute Working Paper 2025-56, 2025
Bertrand, Duflo and Mullainathan, How Much Should We Trust Differences-in-Differences Estimates?, Quarterly Journal of Economics, 2004
Roth, Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends, American Economic Review: Insights, 2022
Goodman-Bacon, Difference-in-Differences with Variation in Treatment Timing, Journal of Econometrics, 2021
Callaway and Sant'Anna, Difference-in-Differences with Multiple Time Periods, Journal of Econometrics, 2021