A timed trial found Copilot developers were 56% faster

Start here

A hiring manager wonders whether an AI pair programmer will make a team measurably faster or just feel that way. One randomized trial isolated the effect on a single, well-defined task. In the study, researchers from Microsoft Research, GitHub and MIT Sloan recruited 95 professional programmers through the freelance platform Upwork and asked each to build an HTTP server in JavaScript as quickly as possible.

What the documents say

This was a randomized controlled trial, not a survey of opinions: 95 accepted freelancers were randomly split into 45 given access to GitHub Copilot, with a short introductory video, and 50 working without it but free to use search and Stack Overflow as normal. Task completion time was measured objectively through a coding-classroom platform that timestamped each submission's first fully passing run. The full paper reports the treated group completed the task 55.8 percent faster, with a 95 percent confidence interval spanning 21 to 89 percent, and that developers with less experience, older programmers, and those who coded more hours per day benefited most. Thirty-five participants per group finished the task and survey used in the main analysis. Tellingly, both groups' own guesses about the size of the speed gain averaged only 35 percent, underselling the 55.8 percent actually measured.

Check this

A reader weighing a similar claim can check what task and population it was measured on. This trial used one task, a fairly standard HTTP server, in one language, and participants recruited from a freelance marketplace, mostly in their 20s and 30s and based in India and Pakistan, rather than a random sample of professional developers everywhere. The paper's own conclusion states results may vary across tasks and languages and explicitly did not examine effects on code quality, only speed.

What holds and what fails

The speed effect holds for a bounded, well-specified coding task similar to the one tested, using GitHub Copilot as it existed in mid-2022, and the paper's own confidence interval shows real uncertainty even within that scope. It fails, by the authors' own statement, as a claim about code quality, about unfamiliar or ambiguous tasks, or about languages and codebases unlike the one tested; the paper never measured those. Treating a single timed exercise as proof of durable team-level productivity gains is a step the authors do not take themselves.

• Ask what specific task and language a cited coding-productivity number was measured on before applying it to your own work.
• Remember this trial did not measure code quality, only time to a passing test suite.
• Notice that self-estimated speed gains in the study undershot the measured gain, a reminder that guesses about your own AI-assisted speed can run either direction.

The result is real evidence for one bounded claim, this task, this tool, this population, in 2022, and the paper's caution about generalizing beyond that is as much a part of the finding as the 55.8 percent itself.