Correction — November 5, 2026: An earlier version presented an unsupported monthly cost illustration and overstated how broadly the results apply. We have withdrawn that illustration and the general cost-saving promise, qualified the latency and model claims, and removed an unvalidated clinical deployment example. The benchmark results should be read in the context of the tested models, tasks and prompting setup.
This article reviews the 2025 research; the correction checks the revised paper (v2, March 3, 2025). The audio overview below predates this correction and has not been revised. Refer to the corrected text for the qualifications.
Audio Overview
The Evolution of AI Reasoning
Large Language Models (LLMs) have revolutionized AI capabilities through their remarkable reasoning abilities. Chain-of-thought (CoT) prompting has played a key role in this rapid advancement which, enabled LLMs to tackle complex reasoning tasks with step-by-step articulation. This technique has led to breakthroughs in mathematical reasoning, common-sense understanding, and symbolic problem-solving.
However, this enhanced reasoning capability comes at a significant cost. The verbose nature of CoT prompting results in substantial computational overhead, increased latency, and higher operational expenses. LLMs generate lengthy explanations with each reasoning step, so they consume more tokens. This leads to slower response times and elevated inference costs, which are particularly problematic for real-time applications and high-volume deployments. These challenges create barriers for industries that require fast, cost-effective, AI-powered decision-making.
Introducing Chain of Draft (CoD) Prompting
This is where Chain of Draft (CoD) prompting comes into play. Introduced by researchers from Zoom Communications in their February 2025 paper, CoD represents a significant shift in how LLMs generate reasoning outputs. Unlike traditional CoT methods that produce verbose human-like explanations, CoD draws inspiration from the way humans naturally solve problems—utilizing concise notes, shorthand, and minimal drafts that capture only the essential insights needed to progress toward a solution.
The paper reports shorter generated reasoning on its selected benchmarks. Accuracy must be compared for each model and task; token efficiency alone does not establish equivalent quality. The reported results offer several considerations:
- Latency: Measure the complete response time on the intended workload.
- Cost: Compare billed input and output tokens, including demonstrations.
- Accuracy: Check whether the reduction in response length changes the error rate.
- Implementation: The evaluated setup uses prompting and demonstrations; successful transfer to another model is not guaranteed.
The experiments motivate testing concise reasoning where token use and latency matter. They do not establish deployment costs, capacity or accuracy for a particular business workload.
Background on Chain-of-Thought (CoT) Prompting
Chain-of-thought (CoT) prompting, introduced in 2022 by Wei et al., emerged as a groundbreaking technique that significantly improved LLMs’ ability to tackle multi-step reasoning tasks. Before CoT, LLMs struggled with structured problem-solving—while they possessed vast knowledge, they lacked a systematic approach to apply it effectively to complex problems.
Why Chain-of-Thought Prompting Works:
- Mimics Human Cognitive Processes: CoT encourages LLMs to break down problems into sequential steps, similar to how humans analyze and solve intricate challenges.
- Improves Accuracy Across Diverse Domains: CoT has proven effective in arithmetic reasoning, logical reasoning, and commonsense understanding, enabling LLMs to reason through multi-step solutions rather than guessing.
In practice, CoT prompting involves giving models structured examples that illustrate reasoning steps. For instance, when addressing a mathematical word problem, the model doesn’t rush to the answer; rather, it undergoes a systematic thought process: identifying variables, establishing relationships, performing calculations, and reaching a final solution.
The influence of CoT on LLM performance has been significant. Latest reasoning-focused models, including OpenAI’s o1, DeepSeek’s R1, and Alibaba’s QwQ, have adopted CoT principles, reaching remarkable outcomes on benchmarks designed to evaluate complex reasoning. This achievement has established CoT as the standard for deriving structured reasoning from LLMs.
The Limitations of CoT:
While CoT significantly enhances reasoning, it comes at a substantial cost:
- Token Explosion: CoT increases token consumption—sometimes by an order of magnitude compared to direct prompting. More reasoning steps mean longer outputs, driving up inference costs.
- Slower responses: Longer generated explanations can increase latency; whether that is acceptable depends on the application.
- High Computational Overhead: Because commercial LLM APIs charge based on token usage, CoT dramatically raises inference costs for enterprises running large-scale AI systems.
The latency implications of CoT are particularly concerning. Research by Xu et al. demonstrated that models using CoT consistently exhibited higher latency than direct prompting methods. This delay is a major drawback for real-time applications like customer support chatbots, AI tutoring systems, and live decision-making tools.
At scale, the financial effect depends on input and output prices, prompt length, retries and the accuracy required. Neither CoT nor CoD has a universal cost advantage independent of those conditions.
These limitations have motivated researchers to explore alternative approaches that maintain CoT’s reasoning benefits while significantly lowering its computational footprint—paving the way for Chain of Draft (CoD) prompting.
Introducing Chain of Draft (CoD) Prompting
The inception of Chain of Draft (CoD) prompting stems from a fundamental observation about human problem-solving behavior. When humans tackle complex problems—whether solving equations, drafting essays, or writing code—we rarely articulate every detail of our thought process. Instead, we use concise notes, shorthand, and minimal drafts that capture only the essential insights needed to make progress.
These abbreviated thought externalities serve as cognitive scaffolding, allowing us to track key information while minimizing mental load.
This contrast between human efficiency and the verbose nature of CoT led researchers to ask: Could LLMs adopt a similarly minimalist approach to reasoning? Their findings confirm that they can—with CoD.
Why Chain of Draft LLM prompting is a significant shift:
- Enhanced Reasoning Efficiency – Rather than simply shortening responses, CoD restructures how LLMs process and represent their reasoning, focusing on essential steps without unnecessary elaboration.
- Compact & High-Value Output – Each reasoning step is condensed so that every token carries maximum semantic meaning, ensuring efficient communication of thought.
- Concise structure: CoD asks for shorter intermediate steps; correctness still needs evaluation.
For example, rather than explaining:
“To find how many lollipops Jason gave to Denny, I need to subtract the remaining lollipops from the initial number, so I calculate 20 – 12 = 8.”
CoD outputs:
“20 – 12 = 8.”
This minimalist representation retains all computational value but dramatically reduces token usage.
CoD is a prompting method that requires evaluation on the intended workload. Results depend on the model and demonstrations; shorter responses alone do not establish acceptable accuracy. Test accuracy, token use and latency together before choosing it over the existing prompt.
How Chain of Draft LLM Prompting Works
To understand the mechanics of Chain of Draft (CoD) prompting, let’s examine how it differs from other prompting approaches through a simple example from the original research. Consider the following arithmetic problem:
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
Comparing Prompting Strategies
Standard Prompting
With standard prompting, the LLM directly produces an answer without showing its reasoning:
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: 8
While this approach is token-efficient, it lacks transparency. We can’t see how the model arrived at its answer, which makes it difficult to verify correctness or identify reasoning errors. This black-box nature significantly limits reliability, especially for complex problems where step-by-step reasoning is critical.
Chain-of-Thought (CoT) Prompting
With CoT prompting, the model provides detailed reasoning:
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: Let’s think through this step by step:
1) Initially, Jason had 20 lollipops.
2) After giving some to Denny, Jason now has 12 lollipops.
3) To find out how many lollipops Jason gave to Denny, we need to calculate the difference between the initial number of lollipops and the remaining number.
4) We can set up a simple subtraction problem: Initial number of lollipops – Remaining number of lollipops = Lollipops given to Denny
5) Putting in the numbers: 20 – 12 = Lollipops given to Denny
6) Solving the subtraction: 20 – 12 = 8
7) #### 8
This response is comprehensive and transparent, showing each step of the reasoning process. However, it is unnecessarily verbose, containing repetitive information that adds no computational value. This verbosity significantly increases token usage, leading to higher latency and cost.
Chain of Draft (CoD) Prompting
With CoD prompting, the model provides only the essential reasoning:
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
A: 20 – x = 12; x = 20 – 12 = 8. #### 8
The CoD approach distills the reasoning to its mathematical essence. It abstracts away all contextual details, focusing solely on the core calculation needed to solve the problem. This representation maintains transparency while dramatically reducing token usage.
Implementing Chain of Draft Prompting
CoD is a prompting method that requires evaluation on the intended workload. Results depend on the model and demonstrations; shorter responses alone do not establish acceptable accuracy. Test accuracy, token use and latency together before choosing it over the existing prompt.
The researchers used the following system instruction together with manually written few-shot demonstrations:
Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator ####.
The five-word instruction is a guideline in the prompt, not an enforced limit.
Additionally, the researchers provided few-shot examples in their prompts to demonstrate the expected format. These examples showed how to transform verbose reasoning into concise, information-dense representations. Each few-shot example included both the original problem and a manually crafted Chain of Draft solutions, helping the model learn the desired response style.
The separator instruction (####) clearly demarcates the reasoning process and the final answer, facilitating easier extraction of the answer for downstream processing. This is an important consideration for practical applications.
CoD is a prompting method that requires evaluation on the intended workload. Results depend on the model and demonstrations; shorter responses alone do not establish acceptable accuracy. Test accuracy, token use and latency together before choosing it over the existing prompt.
Experimental Results of Chain of Draft LLM Prompting

To rigorously evaluate Chain of Draft (CoD) prompting, the researchers conducted comprehensive experiments across diverse reasoning tasks, adhering to the same evaluation methodology used in the original Chain-of-Thought (CoT) paper. Their experimental design concentrated on three major categories of reasoning tasks that traditionally benefit from step-by-step reasoning:
Experimental Setup
The researchers selected benchmark datasets for each reasoning category:
- Arithmetic Reasoning: GSM8k, a challenging dataset of 8,500 grade school math problems requiring multi-step solutions.
- Commonsense Reasoning: Date understanding and sports understanding tasks from BIG-bench, testing the model’s ability to reason about temporal relationships and sports rules.
- Symbolic Reasoning: Coin flip tasks, which require tracking the state of a coin through a sequence of conditional operations.
The main experiments evaluated these two specific model versions:
- GPT-4o (gpt-4o-2024-08-06) from OpenAI
- Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) from Anthropic
Each model was tested with three distinct prompting strategies:
- Standard Prompting: Direct answer generation without reasoning.
- Chain-of-Thought (CoT): Detailed step-by-step reasoning.
- Chain of Draft (CoD): Concise, minimal reasoning steps.
The evaluation focused on three key performance metrics:
- Accuracy: Correctness of the final answer.
- Token Usage: Number of tokens generated in the response.
- Latency: Time required to generate the complete response.
Arithmetic Reasoning Results
The GSM8k benchmark provided a rigorous test of arithmetic reasoning capabilities, requiring models to solve multi-step mathematical word problems.
- Standard Prompting:
- GPT-4o: 53.3% accuracy
- Claude 3.5 Sonnet: 64.6% accuracy
- These results highlight the limitations of direct answer generation for complex reasoning tasks.
- Chain-of-Thought (CoT) Prompting:
- GPT-4o: 95.4% accuracy
- Claude 3.5 Sonnet: 95.8% accuracy
- Substantial cost: ~200 tokens per response.
- Increased latency: 4.2s for GPT-4o, 3.1s for Claude 3.5 Sonnet.
- Chain of Draft (CoD) Prompting:
- GPT-4o: 91.1% accuracy
- Claude 3.5 Sonnet: 91.4% accuracy
- Efficiency boost: 80% reduction in token usage (~40 tokens per response).
- Latency improvements: 76.2% faster for GPT-4o, 48.4% faster for Claude 3.5 Sonnet.
Commonsense Reasoning Results
Researchers evaluated date understanding and sports understanding tasks from BIG-bench for commonsense reasoning.
- For Claude 3.5 Sonnet on date understanding, the paper reports CoD accuracy of 89.7% versus 87.0% for CoT, alongside shorter outputs and lower latency.
- In the sports understanding task, CoD reduced token usage from 189.4 to just 14.3 tokens—a 92.4% reduction.
- Despite this efficiency gain, CoD still maintained a higher accuracy than CoT (97.3% vs. 93.2%).
Symbolic Reasoning Results
For symbolic reasoning, the coin flip task required models to track the state of a coin across conditional operations.
- Standard Prompting: GPT-4o (73.2%), Claude 3.5 Sonnet (85.2%).
- CoT and CoD both achieved perfect 100% accuracy.
- However, CoD dramatically improved efficiency:
- GPT-4o: 16.8 tokens (vs. 52.4 for CoT, a 68% reduction).
- Claude 3.5 Sonnet: 18.9 tokens (vs. 135.3 for CoT, an 86% reduction).
Performance Summary
The results need to be read by task and model:
- Chain-of-Thought (CoT) significantly improves accuracy over Standard Prompting, confirming the value of step-by-step reasoning.
- CoD uses fewer output tokens in the reported few-shot comparisons, with accuracy that can be higher, equal or lower than CoT.
- The approximately 80% output-token reduction describes the GSM8K comparison; it is not a total-bill saving or a universal average across workloads.
- Latency reductions vary by task and model; the arithmetic results should not be generalized to all applications.
- Testing two flagship model versions does not establish the same tradeoff for every architecture or deployment.
These comparisons show an efficiency–accuracy tradeoff. The paper’s zero-shot and small-model limitations also matter when deciding whether to use the method.
Limits of the Reported Results
The revised paper reports weaker results without few-shot demonstrations and a larger accuracy gap against CoT on small models. On GSM8K, the main few-shot CoD results are less accurate than CoT despite using fewer tokens. See the paper, Tables 1, 5 and 6 and section 4.5. Shorter demonstrations can also reduce input tokens, so both input and output counts belong in a cost comparison.
Related Work in LLM Reasoning Optimization
The development of Chain of Draft (CoD) prompting exists within a broader ecosystem of research focused on improving LLM reasoning efficiency. Understanding this context highlights CoD’s novelty and significance.
Structured Reasoning Frameworks
Recent years have seen the emergence of specialized reasoning language models that leverage structured approaches to enhance problem-solving capabilities:
- OpenAI’s o1 model (2024): Optimized for complex reasoning tasks with extensive architectural refinements and training on reasoning-focused data.
- DeepSeek’s R1 model (2025): Uses reinforcement learning techniques to encourage robust, structured reasoning patterns.
- Alibaba’s QwQ model: Focuses on “reflecting deeply on the boundaries of the unknown”, training models to recognize knowledge gaps and adjust reasoning accordingly.
The presence of these reasoning models provides background, not evidence that CoD will improve them. Such a claim requires a comparison on the specific model and workload.
Chain-of-Thought (CoT) Enhancements
Several research efforts have aimed to refine CoT by improving accuracy and robustness:
- Self-Consistency CoT (Wang et al., 2022): Generates multiple independent reasoning paths and selects the most consistent answer—improving accuracy but dramatically increasing computational cost.
- ReAct (Reasoning + Acting): Integrates tool usage into the reasoning process, allowing LLMs to query external knowledge bases. While powerful, this further increases token usage and latency.
These approaches have different accuracy, token and latency costs. Their suitability depends on the workload rather than a blanket distinction between practical and impractical methods.
Latency Reduction Techniques
Several methods have been proposed to mitigate latency in LLM inference:
- Streaming Outputs: Generates incremental responses instead of waiting for a complete answer—improves perceived latency but does not reduce computational cost.
- Skeleton-of-Thought (SoT, Ning et al., 2023): First generates a skeletal outline of the response, then fills in details via parallel decoding—reduces latency but does not lower token usage.
- Layer-Skipping Approaches (Zhang et al., 2023): Generates low-quality draft responses at high speed and refines them in a second pass—improves efficiency but adds extra processing complexity.
CoD targets the length of generated reasoning. Its latency and token results should be assessed in the particular tested setup rather than treated as universal reductions.
Direct Competitors to Chain of Draft
CoD’s closest alternatives are Concise Thoughts (CCoT) and Token-Budget-Aware LLM Reasoning (TALE)—both of which aim to reduce verbosity in reasoning.
- Concise Thoughts (CCoT): Uses a fixed global token budget for reasoning steps. However, different tasks require varying token budgets, making this approach less adaptable than CoD.
- TALE (Token-Budget-Aware LLM Reasoning): Dynamically estimates a global token budget based on problem complexity. While more flexible than CCoT, it requires an extra LLM call to estimate complexity, adding computational overhead.
How Chain of Draft LLM Prompting Differs:
- Per-step guidance: CoD requests short reasoning steps; the paper does not enforce the five-word guideline.
- Dynamic Adaptability: CoD allows unlimited reasoning steps when necessary while still maintaining brevity and efficiency.
- Human-like reasoning aligns with how humans take notes, prioritizing essential information at each step rather than guessing how much detail is needed upfront.
The per-step guideline is one design choice to evaluate against other prompting strategies.
This comparison situates CoD among methods for reducing reasoning verbosity. It does not establish a generally superior method for deployment.
Practical Applications of Chain of Draft LLM Prompting
The following are hypothetical application ideas, not deployments or outcomes demonstrated by the paper. Each would require its own accuracy, latency, cost and operational evaluation.
Real-Time Customer Support
A customer-support team could test concise reasoning against its existing prompt, measuring:
- Troubleshooting accuracy and complete response latency.
- Correct handling of refund rules, policy decisions and eligibility.
- Token costs and capacity under representative traffic.
Latency depends on the model, task and serving conditions. Report the measured values for each experiment; these results do not establish a general sub-second response guarantee for deployed applications.
Mobile & Edge AI Applications
For mobile or edge use, assess whether shorter text offers a useful benefit in the actual system:
- Measure data transferred for cloud inference.
- Measure battery and computation costs rather than inferring them from response length.
- Verify that the chosen model and hardware support on-device inference; prompting alone does not establish that.
A mobile tutor is a possible evaluation setting, not evidence of on-device feasibility.
Educational Tools & AI Tutors
An educational application would need to test the balance between concise answers and explanations students can use:
- Correctness and feedback latency.
- Capacity and cost at the intended number of users.
- Whether concise reasoning provides an adequate explanation for the task.
The paper does not establish a reduction from minutes to seconds for a programming education platform.
Enterprise AI & High-Traffic Applications
For high-volume use, evaluate:
- Total input and output charges per successful query.
- Accuracy and response latency on representative requests.
- Capacity and retry costs in the serving system.
Financial applications would require domain-specific evaluation; the paper does not demonstrate loan, fraud or investment outcomes.
AI Reasoning in Resource-Constrained Environments
For constrained infrastructure, any accessibility benefit remains dependent on the model and deployment:
- Validate the intended task and the resources it requires.
- Compare operating costs at the required quality level.
- Measure bandwidth and hardware needs directly.
Expanding AI Accessibility Through CoD
Token efficiency can be useful, but the study does not establish equivalent accuracy or cost savings across industries and deployment environments.
The practical decision is whether a measured reduction in token use is worth any change in accuracy on the intended task.
Advantages of Chain of Draft LLM Prompting
The potential benefits below require the same model, task and cost qualifications as the benchmark results.
1. Reduced Latency
Latency depends on the model, task and serving conditions. Report the measured values for each experiment; these results do not establish a general sub-second response guarantee for deployed applications.
2. Cost Efficiency
Shorter responses can lower output-token charges, but an output-token reduction is not a total-bill saving. Compare input and output token counts for both prompting methods using the applicable prices: cost = input tokens × input price + output tokens × output price. Include the demonstrations and any caching, retries or other charges. The earlier $3,800-to-$760 example did not specify those assumptions and is withdrawn.
3. Accuracy Must Be Evaluated
CoD is a prompting method that requires evaluation on the intended workload. Results depend on the model and demonstrations; shorter responses alone do not establish acceptable accuracy. Test accuracy, token use and latency together before choosing it over the existing prompt.
4. Simplicity of Implementation
CoD is a prompting method that requires evaluation on the intended workload. Results depend on the model and demonstrations; shorter responses alone do not establish acceptable accuracy. Test accuracy, token use and latency together before choosing it over the existing prompt.
5. Recheck Each Model Version
A model update can change the accuracy, token use and latency tradeoff. Repeat the workload comparison rather than assuming efficiency gains will persist.
Related Articles
- Prompt Engineering – Unlock the Power of Generative AI
Explore the fundamentals of prompt engineering, its real-world applications, and its significance in today’s rapidly advancing AI landscape. ajithp.com - Qwen2.5-1M: Long-Context Models
Discover how Qwen2.5-1M models can accurately retrieve hidden numbers from documents containing up to 1 million tokens, with minimal errors observed in the 7B model. ajithp.com - AI Research Agents with MLGym and MLGym-Bench
Learn about the need for advanced architectures and training algorithms to enable LLM agents to go beyond baseline improvements and contribute to scientific discovery. ajithp.com - Enhancing RAG with Multi-Agent Reinforcement Learning & MAPPO
Understand how Llama-3-8B-Instruct was chosen as the foundational LLM for MMOA-RAG due to its strong instruction-following capabilities and its ability to generate high-quality text. ajithp.com - Test Time Compute in AI: Enhancing Real-Time Inference and Adaptive Reasoning
Dive into how Test Time computing is emerging as an important approach, prompting a rethink on enhancing machine reasoning beyond scaling up computational power.
Conclusion
Chain of Draft LLM prompting has potential to play a key role in AI reasoning, making sophisticated LLM applications more practical and efficient.
The Importance of Latency Reduction
Latency depends on the model, task and serving conditions. Report the measured values for each experiment; these results do not establish a general sub-second response guarantee for deployed applications.
Challenging Assumptions About Verbosity
The reported results motivate testing concise intermediate reasoning. They do not establish that fewer words preserve accuracy on every task.
Future Directions for Chain of Draft LLM Prompting
The success of CoD opens up exciting research directions, including:
- Hybrid Approaches: Combining CoD with techniques like parallel decoding or Skeleton-of-Thought for even faster inference.
- Training on Concise Reasoning: Developing LLMs specifically trained on efficient reasoning patterns.
- Task-Specific Optimization: Exploring how different reasoning tasks benefit from different brevity levels.
- Multimodal Applications: Extending CoD to text+visual reasoning models, making multimodal AI more efficient.
Implications for LLM Development
CoD’s success suggests a new direction for AI model training and evaluation:
- AI models could be optimized for reasoning efficiency alongside accuracy.
- Benchmarks might evolve to include performance-per-token metrics.
- AI reasoning could shift toward more structured, symbolic approaches that capture essential reasoning content without unnecessary language overhead.
Bridging Research & Practical AI Applications
CoD is a candidate for workloads where the measured token and latency savings justify the observed accuracy tradeoff.
Final Thoughts
Choose the prompting method using a comparison on the intended workload, with accuracy, complete response latency and total billed costs assessed together.
Key Links
Research Paper: Chain of Draft: Thinking Faster by Writing Less
Authors: Silei Xu, Wenhao Xie, Lingxiao Zhao, Pengcheng He
Discover more from Ajith Vallath Prabhakar
Subscribe to get the latest posts sent to your email.
30 Comments