Chain-of-Thought & Reasoning Prompts

Chain-of-thought (CoT) prompting asks the model to show its reasoning before giving a final answer. This deceptively simple technique can dramatically improve accuracy on tasks that require multi-step reasoning - but it adds tokens, latency, and cost, and is actively harmful for simple lookup tasks.

Why CoT Works

Language models generate tokens sequentially. Each token is conditioned on all previous tokens. When you prompt a model to write out reasoning steps, those steps become conditioning context for the final answer - the model is less likely to skip logical steps it has already committed to in writing. The scratchpad forces the model to "think" more carefully before concluding.

Basic CoT: "Think Step by Step"

Adding "Let's think step by step" or "Think through this carefully before answering"to a prompt is the simplest form. It costs a few extra tokens but substantially improves accuracy on arithmetic, logic, and multi-step reasoning tasks.

Few-Shot CoT

Providing worked examples with reasoning shown is more powerful than the zero-shot "think step by step" trigger. Show 2–3 examples where the reasoning chain leads to the correct answer, then pose the target question. The model learns the expected reasoning format from the examples.

ReAct: Reasoning + Acting

ReAct interleaves reasoning (Thought) with tool calls (Action) and observations. It is the standard pattern for tool-using agents:

Thought: I need to check the current stock price of AAPL.
Action: search("AAPL current stock price")
Observation: AAPL is trading at $213.42 as of 2026-06-26.
Thought: Now I can answer the user's question.
Answer: AAPL is currently $213.42.

ReAct helps agents avoid hallucinating facts they could verify via tools, and gives you a human-readable audit trail of why the agent took each action.

When Not to Use CoT

CoT vs Extended Thinking

Modern reasoning models (o3, Claude with extended thinking, Gemini 2.5 Pro) perform CoT internally as part of their architecture. You do not need to prompt for "think step by step" - the model already does it. These models generate a hidden scratchpad before producing the visible response. For tasks where reasoning models are appropriate, standard CoT prompting may be redundant.

Practical Tips

• Add CoT for tasks with >2 logical steps; skip it for single-step lookups
• Use few-shot CoT over zero-shot when you can provide 2–3 good worked examples
• For tool-using agents, structure the ReAct loop explicitly in your system prompt
• Tell the model to "put the final answer in a clearly marked section" so you can extract it without parsing the reasoning
• Measure accuracy with and without CoT on your actual task - the improvement is highly task-dependent