Twelve months. Nine separate pilots, if you count the abandoned one from January that nobody talks about anymore. The steering committee at a mid-market insurer I worked with sat through a full year of AI demos in a conference room with a view of a parking lot nobody had ever fixed the lighting for, watching a chatbot draft claims summaries, an agent triage support tickets, and a fraud model flag patterns the old rules engine had missed for years. Every demo landed. Heads nodded around the table, more than once. Then the CFO, because it's always the CFO who asks this, wanted to know which of the nine had moved a number the board actually tracked. The room went quiet. Not one hand.
I've told versions of that story to at least a dozen founders and CIOs since, and the reaction is always the same: a wince, then a nervous laugh, because everyone in AI right now has sat in that room at least once. My first instinct, and I'll admit this outright, was to blame the model. Maybe the LLM hallucinated too often on edge cases. Maybe latency killed adoption. I was wrong, and MIT's own research says so in numbers too large to argue with.
What did MIT's GenAI Divide study actually find?
MIT's Project NANDA published "The GenAI Divide: State of AI in Business 2025" in July 2025, built from over 300 public AI deployments, 52 structured interviews, and 153 executive surveys, and the headline number is blunt: 95% of organizations have seen no measurable financial return on their generative AI investment (Fortune, August 18, 2025). Enterprises had already sunk an estimated $30 billion to $40 billion into generative AI tools and systems by the time researchers went looking for the return, and for the overwhelming majority, that money produced pilots, decks, and press releases instead of profit (Digital Commerce 360, August 25, 2025). Lead researcher Aditya Challapally didn't point at the models. Not the model. The gap, he said, is a "learning gap": tools that don't retain feedback or adapt to a team's actual workflow, and quietly get abandoned, not because the technology failed, but because nobody validated what "working" meant before anyone started building.
Talent gaps are real. Integration debt too. I'm not pretending otherwise; plenty of pilots genuinely stumble on a messy data pipeline, an under-resourced team, or a rollout nobody scoped properly before the first sprint. But MIT's researchers controlled for company size, industry, and AI maturity on purpose, and the divide held anyway: pilots inside companies with strong technical talent failed at nearly the same rate as pilots inside companies without it. That's the detail that should worry a CFO more than the 95% headline ever will.
Why do teams blame the model when the ROI never shows up?
Blame the model instead. That's easier than admitting nobody defined what success looked like before the build started, and easier still than telling a steering committee the real failure was a rushed discovery phase, not a weak transformer. Model-blaming has a familiar shape: swap the vendor, bump the version, add a bigger context window, and hope the number finally moves. It doesn't. Not because the tooling is bad. Because the model was never the actual constraint.
Stanford's Digital Economy Lab tested this directly. Researchers studied 51 AI deployments that actually worked, across 41 organizations, nine industries, and seven countries, and for 42% of them, the specific model in use was fully interchangeable; swapping it wouldn't have changed the outcome (Stanford Digital Economy Lab, "The Enterprise AI Playbook," March 2026). Same technology. Wildly different outcomes, across companies running the exact same foundation models underneath. The paper's own line still sticks with me weeks after I first read it.
The model was never the differentiator. The organization was.
So the next time a pilot stalls and someone in the room suggests trying a different model, ask a sharper question first: did anyone validate the requirement this pilot was supposed to solve, or did it just seem like a good demo? Nine times out of ten, that's where the real answer lives. It usually is.
What did the 5% that worked actually do differently?
They scoped one problem, picked a lane, and didn't wander. MIT's own researchers found the successful 5% shared a pattern: they picked a single, well-defined pain point instead of chasing a broad transformation, and they overwhelmingly bought or partnered for the capability rather than building it from scratch. Vendor partnerships in the study succeeded roughly two times as often as internal builds (Digital Commerce 360, August 2025). Back-office automation, the unglamorous stuff, eliminating outsourced business processes, cutting external agency spend, produced the biggest returns of anywhere in the survey. No press release. Just a finance team that noticed the invoice was smaller.
I almost used Morgan Stanley's DevGen.AI as the proof point for "buy beats build" too. Wrong pairing. Worth admitting, too. Morgan Stanley built DevGen.AI in-house, a tool that reviewed nine million lines of legacy Perl and Cobol, translating it into specifications a developer could work from, saving roughly 280,000 developer hours in its first five months (Entrepreneur, 2025). That cuts against MIT's own finding that partnerships beat internal builds roughly two to one. Fine. What it actually proves is the other half of the argument: the team scoped it to one task, not "modernize the whole platform." Mike Pizzi, the bank's global head of technology and operations, didn't announce a sweeping transformation. He announced a tool that did one job, on nine million specific lines, and measured the hours it saved.
The winners knew one thing cold: exactly what "done" meant, and exactly which number would prove it, before a single line of code or a single vendor contract got signed. The other 95%? Mostly didn't. No model upgrade was ever going to fix that gap.
How does requirements intelligence turn a pilot into a bet with a defined "done"?
It replaces a demo-driven pilot with a validated one, forcing the scoping conversation to happen before the build, not three months after the CFO asks for a number nobody can produce. Requirements intelligence runs structured discovery: multiple expert perspectives interrogating the same problem, stakeholders surfaced and interviewed instead of assumed, conflicts between departments resolved on paper instead of in production. Not a deck. A validated, specific, measurable requirement a pilot can actually be built against, one a CFO can hold someone accountable to.
This is the same governance gap I wrote about when Gartner projected more than 40% of agentic AI projects would be cancelled by 2027, in Why 40% of Agentic AI Projects Will Be Cancelled by 2027. Cancelled projects and zero-ROI pilots are cousins; both trace back to teams that could not answer "what was this supposed to do, and who validated that?" I found the same gap in the adoption numbers too: Deloitte's 2026 survey found 74% of organizations expect to use agentic AI within two years, yet only 21% have a mature governance model ready, a mismatch covered in the agentic AI adoption gap. Same root cause, three reports, three years. That's the pattern.
Specira runs that discovery layer before the pilot gets built, not as an audit after it stalls. Five specialist agents, four analysts plus a Red Team Critic built to argue against the group's own conclusions, interrogate a requirement from multiple angles before anyone commits a dollar of budget to building it. Not a faster demo. A pilot with a defined finish line, one a CFO can actually check on the way out.
What does a scoped AI pilot look like in practice?
It looks boring, almost disappointingly so: one department, one measurable pain point, a success metric everyone agreed on in writing before the first sprint, and a kill switch if the metric doesn't move by a set date. No press tour. No department-wide rollout on day one. Just a scoped bet with a number attached to it.
Two pilots can use the identical foundation model and land in completely different places. One starts broad ("improve customer service with AI"), skips validation, and gets built by whoever's available that sprint. It demos beautifully in week three and produces nothing measurable by week twelve, because nobody ever defined what "improved" meant in a number the finance team could check. The other starts narrow ("cut average claims-summary drafting time from eleven minutes to under three, for auto claims specifically, without adding a review step"), gets validated against the actual claims team before a line of code exists, and ships with a metric baked in from day one. Same technology. Same budget range, probably. A wildly different fifth month, though, and a wildly different conversation with the board.
The 95% isn't a model problem. It's a scoping problem.
MIT's own research says 95% of enterprise generative AI pilots show no measurable P&L impact, despite $30 to $40 billion already invested, and the lead researcher points at a "learning gap," not a capability gap. Stanford's follow-up study of 51 successful deployments found the model was interchangeable in 42% of cases. The organization was the differentiator. Not the model.
The 5% that worked scoped one validated problem, agreed on a measurable "done" before building anything, and mostly bought or partnered rather than building from scratch. Requirements intelligence is what makes that scoping conversation happen on day one. Not at the postmortem.
None of this requires a bigger model, a longer pilot, or a braver CFO. Smaller ask. It requires validating the requirement before anyone opens a build ticket, a much cheaper conversation to have in month one than a postmortem to run in month eleven.
What are the most common questions about AI pilot ROI?
