Chain-of-thought prompting worked because scale let it

Start here

Anyone told to ask a model to show its work is repeating a specific 2022 finding, not a folk remedy. The paper that named the technique, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, tested giving a model worked examples that included the reasoning steps, not just the final answer, before asking it a new question. Submitted 28 January 2022 by Jason Wei and eight co-authors, it reports the method significantly improves the ability of large language models to perform complex reasoning on arithmetic, commonsense and symbolic tasks.

What the documents say

The paper's headline number: prompting a 540-billion-parameter model with eight chain-of-thought examples reached state-of-the-art accuracy on GSM8K, a benchmark of grade-school math word problems, surpassing a version of GPT-3 that had been separately fine-tuned for the task with a verifier rather than only prompted. The authors tested three large language models and describe the effect as emerging naturally in sufficiently large language models, tying the gain to model scale rather than treating it as a wording trick that works everywhere. A second paper, Large Language Models are Zero-Shot Reasoners, submitted months later by Takeshi Kojima and colleagues, found that adding just the phrase 'let's think step by step,' with no worked examples, raised accuracy on InstructGPT (text-davinci-002) from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K, with similar-sized gains on a 540-billion-parameter PaLM model. Neither paper is a vendor claim; both are independent research reports naming their benchmarks and models, not a general claim about AI reasoning.

Check this

The mechanism to check is scale-dependence: try the same step-by-step instruction on a small, locally run model and on a large hosted one, on the same arithmetic problem, and compare whether the smaller model's accuracy improves, stays flat, or gets worse. The original paper's framing predicts the technique should help less, or not at all, below some model size, since it describes the ability as emerging at scale rather than existing uniformly.

What holds and what fails

The finding holds as a description of two 2022 experiments on named benchmarks and models; it is not a promise that any request phrased as 'think step by step' improves any model's answer on any task, a stretch neither paper makes. Current reasoning models now build extended step-by-step processing in by default, an editorial connection: vendor guides describe separate instructions for reasoning models, consistent with the technique moving from a prompting trick into a built-in mode rather than replacing the need to check the answer.

• Ask a model to show intermediate steps on a multi-part problem and check each step, not only the final answer.
• If using a known small or older model, test whether step-by-step prompting actually helps before relying on it.
• Treat a reasoning model's built-in step-by-step process as related to this technique, not identical to it.

The two papers together separate the instruction from the capability: telling a model to reason in steps only pays off where the underlying model is large enough to make the steps meaningful, a distinction easy to lose once the phrase becomes a habit.