· Software
The Only Randomized Study on AI Coding Speed Found Developers Got Slower
Before you let "AI writes the code now" shrink a build estimate, look at the one study that measured the claim instead of asking developers how they felt about it.
I wrote in September that AI coding tools were producing a real but modest gain, somewhere around 7 to 15 percent faster throughput, based on data from the engineering analytics firm DX. That is still the most cited observational number in this debate. It is also not the strongest evidence available, and the strongest evidence says something more uncomfortable.
The case for assuming the discount
Start with why so many people, including me a month ago, take the speedup for granted. AI coding tools improved fast through 2025 and 2026, and adoption among professional engineers is now close to universal. DX, which instruments and surveys large engineering organizations for a living, has built a business on the premise that the gain is real and measurable, and its own commentary on the research below leans toward nuance rather than denial: it argues the technology still delivers value in the right settings and that organizations should target it at specific tasks rather than assume a blanket discount. That is a reasonable, good-faith position, and it is the one nearly every custom software quote I have reviewed this year quietly assumes.
What the randomized trial actually found
METR, a nonprofit that evaluates AI systems, ran the one study in this space built to rule out self-deception instead of measure it. Sixteen experienced open source developers, each with years of history on large repositories averaging over 22,000 stars and a million lines of code, worked on 246 real issues from their own projects. Each issue was randomly assigned to allow or disallow AI tools, mostly Cursor Pro running Claude 3.5 and 3.7 Sonnet, the frontier models at the time. The developers predicted AI would speed them up by 24 percent. When allowed to use it, they took 19 percent longer to finish the same kind of work. Afterward, even having just been measured taking longer, they still believed AI had made them about 20 percent faster. METR checked the result against different outcome measures and found it held up, and it had enough statistical power across 246 issues to rule out zero effect in either direction.
That gap between what developers felt and what a stopwatch recorded is the whole problem with pricing a build on vibes. A client's engineering team will tell you, honestly, that they are faster now. The one experiment designed to check whether that feeling matches reality found it did not, at least for experienced developers doing real work on codebases they already knew well.
METR did not stop at the convenient headline
Here is the part that actually earns trust. METR ran a larger follow-up starting in August 2025 with newer models, expecting to either confirm or overturn the first result. Instead, so many developers refused to take on tasks without AI assistance that the study's no-AI comparison group fell apart, and a pay cut partway through likely skewed who stuck around. In a post this spring, METR said plainly that its own new data gave an unreliable signal and that it was redesigning the experiment rather than publish a number it did not trust. I have not found a single randomized trial, as opposed to a survey or a vendor's own usage data, that shows AI tools reliably speeding up experienced developers on real production code. That may well change. It has not changed yet in any result I can verify.
What this means for your build-or-rent decision
If a custom software quote assumes 20 or 30 percent fewer engineering hours because "the team uses AI now," ask to see before-and-after time data on comparable work, not a feeling. Price the labor you can verify, and treat anything shakier as upside, not a given.
The other side of the comparison has not stood still either. Zylo's 2026 SaaS Management Index found SaaS spend rose almost 8 percent in a single year, and 61 percent of organizations cut a project or initiative because of an unplanned SaaS cost increase. But Common Paper's 2026 benchmark, drawn from more than 16,000 signed cloud agreements, found automatic fee increase clauses nearly halved, from 23 percent of contracts to 13 percent, as buyers pushed back at the negotiating table. The market is already disciplining its own worst habit, without anyone needing to mandate it.
Run both halves of this with real numbers before you decide anything. Price a build on labor hours your own team can document, not an AI-era discount nobody has proven, and negotiate your next SaaS renewal like the automatic hike is optional, because the data says it increasingly is. Our custom software team runs exactly that comparison with clients, and we recommend keeping the subscription more often than you would expect from people who build software for a living. If you want a second set of eyes on your own numbers, get in touch.
Sources
References used in this article. Links also appear alongside the relevant claims.
Let's make it happen.
You don't need a polished brief. A couple of lines about where you want your company to go is plenty, and we'll come back with what we'd tackle first.
We scope the work around your goals and timeline before quoting anything, so you know exactly what you're getting.