Demo · AI strategy & economics
Did the AI pilot work, or did you just pick good teams?
Most AI pilots are judged with a before-and-after chart or a comparison against teams that didn't get the tool. Both are easy to fool: the volunteers were often your strongest teams, and other things changed while the pilot ran. This demo simulates a support organization where the tool's true effect is known, measures it three ways, and shows how far each answer moves the ROI.
Three answers to "did it work?"
Check the assumption before you trust the answer
Difference-in-differences assumes the pilot and comparison teams were on parallel paths before launch. You can't prove that for the period after launch, but you can check the months before it.
How each method works
Before and after compares pilot teams' productivity before and after launch; it also counts anything else that changed at the same time. Pilot vs. everyone else compares the two groups after launch; it also counts however far ahead the pilot teams already were. Difference-in-differences takes the pilot teams' change and subtracts the comparison teams' change over the same weeks, which removes both, as long as the two groups were on parallel paths. Its 95% range comes from the spread across teams. First-year ROI counts the productivity gain as labor capacity worth that share of annual labor cost, times the share of freed-up time actually put to use (time saved that nobody redeploys isn't a saving), against the tool's annual seat cost plus the one-time rollout cost.
All data is simulated: each team has its own baseline, weekly noise and the effects set on the left; productivity is measured as tickets resolved per agent per week. Real pilots also need to watch for the tool changing what gets measured (for example, faster but lower-quality resolutions).
Watch: the 73-second version
Video transcript
Same AI pilot. Same data. Three answers. Before vs. after: 11.4%. Pilot vs. everyone else: 21.9%. Difference-in-differences: 4.6%. The true effect is 6%. Only one of these methods gets close to the truth.
Answer 1 of 3, before vs. after: it credits the tool for the whole quarter. A new knowledge base, a process fix or a slow season lands in launch week. In the simulation, a 7% company-wide change arrived with the tool's 6%. Across 1,000 simulated pilots this method averaged 13.4%, more than double the truth.
Answer 2 of 3, pilot vs. everyone else: it credits the tool for who volunteered. Pilot teams were already 12% ahead before the tool arrived. This method averaged 19.1%, three times the real effect, and the gap never closes.
Answer 3 of 3, difference-in-differences: subtract what would have happened anyway. Comparison teams changed by 7.0% and pilot teams by 13.4%; the difference, about 6%, is the tool. The company-wide change and the head start both cancel. Across 1,000 simulated pilots it averaged 6.0%, and its 95% range held the truth 93.3% of the time.
Why the method matters: same pilot, very different first-year ROI. The true effect is worth an ROI of 28%. Before vs. after implies 144%, pilot vs. everyone else implies 369%, and difference-in-differences gives a range of minus 49% to plus 45%. The honest answer is the last one: too close to break-even to call from 12 teams. Widen the pilot before signing a company-wide contract.
The check that often misses: when pilot teams were already improving 1.5% a month faster, the standard pre-launch test flagged it only 38.9% of the time. Assign the tool at random when you can.
Try it with your own numbers at vector1.ai/demos/pilot-roi. It is free and runs in your browser. The full article, Python code and ROI model are on vector1.ai.