A writing experiment found ChatGPT narrowed the skill gap

Start here

A marketer wonders whether letting ChatGPT help with a press release will save time, hurt quality, or both. A randomized experiment answers this for one slice of professional writing directly. In the working paper that MIT economists Shakked Noy and Whitney Zhang later published in Science, 444 college-educated professionals in occupations including marketing, grant writing, consulting, data analysis, human resources and management were each assigned two occupation-specific writing tasks, such as press releases, short reports and delicate emails.

What the documents say

The design was a preregistered, randomized online experiment: after a first task set a baseline, participants were randomly split so half were told to sign up for ChatGPT before their second task, while the control group signed up for the unrelated LaTeX editor Overleaf. Each output was graded blind by experienced professionals in the same occupation. The paper's own abstract reports ChatGPT access lowered time taken by 0.8 standard deviations and raised output quality by 0.4 standard deviations, with lower-ability writers, judged by their first-task score, benefiting more than stronger ones, compressing the spread between them. The authors describe ChatGPT as mostly substituting for the writer's own drafting effort rather than adding skills the writer lacked, shifting time toward idea generation and editing and away from rough drafts. A Crossref record confirms the study appeared in Science, volume 381, issue 6654, pages 187 to 192, in July 2023, after peer review.

Check this

A reader can check whether a writing task resembles this study's conditions before assuming the same effect: occupation-specific, time-boxed at 20 to 30 minutes, judged by an expert rubric, with a real incentive to do well. The paper's funding notice cites an Emergent Ventures grant, the George and Obie Shultz Fund, and a National Science Foundation fellowship, and the study was registered in advance at the AEA RCT Registry, letting a skeptical reader confirm the analysis plan was not adjusted after seeing results.

What holds and what fails

The compression of the gap between weaker and stronger writers held for mid-level professional writing tasks under time pressure with an expert grader, using a single model in early 2023. It is editorial, not the paper's claim, to say this generalizes to every writing task; the authors note the effect looked like substitution for effort rather than skill transfer, so a writer who leans on the tool every time may not build the ability it stands in for. The study also does not test months of repeated use, only a single sitting.

• Compare your own writing task to the study's conditions, time pressure, a clear rubric, an occupation-specific brief, before assuming a similar gain.
• Notice whether an assistant is substituting for your drafting effort or genuinely teaching you a technique you lacked.
• Check a working paper's preregistration when a study's headline number is being repeated as settled fact.

The clearest lesson is not the size of the time saved but the direction of the inequality effect: the tool did more for the person starting from a weaker position than for the person already skilled, which is not something every automation technology has done.