A support-agent study found AI helped novices most, not experts

Start here

A call-center manager wonders whether adding an AI assistant will make every agent faster by the same amount. The best available field evidence says no: gains concentrate where the assistant substitutes for experience the newest agents have not yet built. A study described in the NBER working paper examined 5,179 customer support agents at an anonymous software firm who were given staggered access to a generative AI assistant while handling ordinary chat-based support conversations.

What the documents say

This is a usage analysis built around the tool's staggered rollout across the firm, not a lab experiment: agents were given access to the assistant at different times, letting the researchers compare productivity before and after access rather than randomly assigning it. The paper's own abstract states access to the tool raised productivity, measured as issues resolved per hour, by 14 percent on average, including a 34 percent improvement for novice and lower-skilled workers, but minimal impact on already experienced, highly skilled agents. The authors describe the assistant as spreading practices that its best agents already used, functionally handing newer workers a version of senior agents' accumulated know-how. The Quarterly Journal of Economics later published the same authors' peer-reviewed version under the same title, confirming the study cleared academic review rather than remaining a preprint claim.

Check this

A reader evaluating a similar rollout at their own workplace can check whether the claimed gain is being measured against a genuine before-and-after baseline for the same agents, or only against a cross-section of different agents who happen to have different tenure. The paper's design leans on the staggered introduction precisely because comparing an agent to their own earlier self controls for skill differences a simple average would hide. Anyone citing '14 percent' without naming who benefited is dropping the paper's central finding, which is about the shape of the gain across the workforce, not one number.

What holds and what fails

The finding holds for one company's customer-support chat operation using one, unnamed, generative AI tool in a period around 2023, a single-firm study, however large its worker sample. It is editorial to extend 'AI helps novices most' into a universal law of AI and skill; the paper documents this pattern in one service context where a written record of good responses already existed for the model to draw from, and it does not test tasks without that structure. Where no accumulated expert practice exists for an assistant to surface, this mechanism has nothing to transfer.

• Ask whether a productivity claim compares workers to their own earlier performance or to a different group entirely.
• Look for whether an AI tool in your own work draws on an existing body of expert practice, as this study's did.
• Treat a single-company average as a starting hypothesis for your own workplace, not a guarantee.

The paper's most durable idea is not the 14 percent headline but the mechanism behind it: an assistant trained on good examples can compress the gap between a new worker and an experienced one faster than most training programs, in the one setting this study measured.