Chain of Thought Prompting

Introduction to Chain of Thought

If you ask a Large Language Model (LLM) a complex math or logic problem (e.g., “John has 5 apples, gives 2 to Mary, buys 3 times as many as he currently has, and drops 1. How many does he have?”), standard LLMs will often confidently output the wrong answer.

This happens because of how the neural network generates text. It attempts to predict the final answer instantly in a single, massive calculation. Because the intermediate mathematical steps are complex and require tracking state, the single-shot calculation fails.

In 2022, Google researchers discovered a breathtakingly simple Prompt Engineering technique that dramatically increased the reasoning capabilities of LLMs: Chain of Thought (CoT) Prompting.

By simply forcing the AI to “think out loud,” the model’s accuracy on complex logic tasks skyrockets.

How Chain of Thought Works

Chain of Thought works by breaking a multi-step reasoning problem into intermediate steps, forcing the model to write out its logic before it is allowed to output the final answer.

Because LLMs are autoregressive (they predict the next word based on all the previous words they just generated), generating the intermediate steps physically adds those steps to the model’s Context Window. The model then uses its own generated logic as the context to calculate the final answer.

Zero-Shot Chain of Thought

The most famous and widely used implementation of CoT requires adding exactly eight words to the end of your prompt:

Prompt: John has 5 apples, gives 2 to Mary, buys 3 times as many as he currently has, and drops 1. How many does he have? Let’s think step by step.

By adding the magic phrase “Let’s think step by step,” the model shifts its probability distribution. Instead of trying to guess the final number, it starts generating the intermediate math:

1. John starts with 5.
2. Gives 2 to Mary: 5 - 2 = 3.
3. Buys 3 times his current amount: 3 * 3 = 9.
4. Total is now 3 + 9 = 12.
5. Drops 1: 12 - 1 = 11.
6. Final answer: 11.

Because the model generated the number 12 in step 4, the math to get to 11 in step 5 became computationally trivial.

Few-Shot Chain of Thought

For highly complex, proprietary tasks, developers use Few-Shot CoT. In the system prompt, they provide the AI with 3 or 4 examples of how it should think. Instead of just providing [Question -> Answer], the developer provides [Question -> Step-by-Step Logic -> Answer]. The model mimics the provided logical structure to solve unseen problems.

The Evolution: ReAct and O1

Chain of Thought is the foundational concept behind modern Agentic AI. Frameworks like ReAct (Reasoning and Acting) build upon CoT by forcing the AI to generate a “Thought,” use a tool to take an “Action” (like searching Wikipedia), observe the “Result,” and then generate the next “Thought.”

Furthermore, model builders are now baking CoT directly into the neural network architecture. Models like OpenAI’s o1 do not require the user to type “Let’s think step by step.” The model has been explicitly trained (via reinforcement learning) to generate thousands of invisible “thinking tokens” in the background for 10 to 20 seconds before it ever begins typing the final answer to the user, mimicking deep human contemplation.

Conclusion

Chain of Thought Prompting proved that Large Language Models possess latent reasoning capabilities that are bottlenecked by their autoregressive nature. By simply altering the prompt to give the AI the temporal space to generate intermediate logic, developers can transform a model that fails at basic arithmetic into a robust reasoning engine capable of solving complex, multi-step algorithmic problems.