A randomized study found AI made expert coders slower

Start here

A senior engineer who has used an AI coding assistant for months feels confident it saves time on their own well-known codebase. A 2025 randomized trial tested exactly that scenario and found the opposite of what participants believed. The nonprofit AI research group METR reported a randomized controlled trial of 16 experienced developers working in mature open-source projects they knew well, averaging around five years of prior experience on those specific repositories.

What the documents say

The design randomly assigned 246 real repository issues, bug fixes, features and refactors, averaging about two hours each, to be done with or without AI tool access, mostly Cursor Pro paired with Claude 3.5 or 3.7 Sonnet, models current at the time. This is a randomized trial on real work, not a benchmark or a survey. The paper and blog post agree on the headline result: allowing AI tool use increased completion time by 19 percent. Before starting, developers expected AI to make them 24 percent faster; even after living through the slower outcome, they still estimated afterward that AI had sped them up by about 20 percent. Economists and machine learning researchers surveyed separately had forecast even larger speedups, around 38 to 39 percent. The authors tested roughly 20 possible explanations for the slowdown, including project familiarity and task type, and state the slowdown held up across those checks.

Check this

A reader can check their own sense of being helped against a stopwatch, not a feeling, which is exactly the substitution this study made: screen recordings and objective timestamps, compared against what participants believed had happened. The gap between a 24 percent expected speedup and a 19 percent measured slowdown, on the same people and the same repositories, is the concrete mechanism worth naming: perceived helpfulness and measured time are different things, and one does not confirm the other.

What holds and what fails

METR states plainly that this result does not show AI tools lack value in general; the trial specifically measured experienced developers working in large, high-quality codebases they already knew intimately, using tools and models from early 2025, and the authors flag their own small sample and possible sampling bias among volunteers as open questions. It fails as evidence about newcomers, unfamiliar codebases, or later tool versions, none of which this trial tested. Extending '19 percent slower' into a general verdict on AI coding tools is editorial overreach the authors explicitly warn against in their own limitations.

• Time a task with and without an assistant on your own familiar work before trusting a felt sense of speed.
• Note which model and tool version a study used, since METR named specific 2025 tools that will not stay current.
• Treat a 16-person trial as a signal to test on your own work, not a verdict for every developer.

The study's most transferable lesson may be procedural rather than numerical: when perception and measurement can diverge this far even for careful, experienced people, checking with a timer is worth more than trusting the feeling of being faster.