When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
| Authors | Sailesh Panda et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2605.00817 |
| Download | |
| Categories | cs.CL |
Abstract
Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We study this question through a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic algorithm and two numeric inputs, and must return the final computed value. The benchmark uses simple arithmetic operations but increases complexity through algorithm length and look-back dependencies over intermediate variables. Across 14 models and 55 datasets, average first-answer accuracy drops from 61% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error, under-executed traces, and hallucinated extra steps. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful instruction execution.
Engineering Breakdown
Plain English
This paper introduces a diagnostic benchmark that tests whether large language models can faithfully execute step-by-step arithmetic procedures, not just get final answers right. The researchers gave 14 different models simple arithmetic algorithms with varying lengths (5 to 95 steps) and measured how often they correctly computed the final result. The key finding is stark: accuracy drops dramatically from 61% on 5-step procedures to just 20% on 95-step procedures, revealing that LLMs struggle with procedural faithfulness even on tasks with simple arithmetic operations. Analysis shows failures stem from missing intermediate answers, premature termination, and self-correction errors.
Core Technical Contribution
The paper's core contribution is a controlled, systematic diagnostic benchmark that isolates procedural execution capability from general reasoning ability. Unlike existing benchmarks that measure final-answer accuracy on complex reasoning tasks, this benchmark holds constant the arithmetic operations themselves and specifically increases complexity through algorithm length and look-back dependencies—forcing models to track and use intermediate variables. This diagnostic approach directly measures faithfulness to procedure specification rather than whether models can pattern-match to get correct answers, providing a precise lens for understanding where and why LLMs fail at step-by-step execution. The benchmark spans 55 datasets across 14 models, offering broad empirical coverage.
How It Works
The benchmark works by constructing sequences of simple arithmetic operations (addition, subtraction, multiplication, division) organized into multi-step algorithms that models must execute faithfully. Each test case provides the algorithm specification (in natural language or pseudocode) plus two numeric inputs, and the model must produce the final computed value after executing all intermediate steps. Complexity is systematically increased along two dimensions: algorithm length (the number of sequential steps, ranging from 5 to 95 steps) and look-back dependencies (requiring models to reference and use previously computed intermediate variables, not just the most recent result). The benchmark measures both whether the final answer is correct and traces generation-level failures—examining where in the execution sequence models diverge from the specified procedure, whether by skipping steps, stopping early, or self-correcting in ways that break procedure faithfulness.
Production Impact
For production systems, this work exposes a critical vulnerability in any pipeline relying on LLMs to execute procedural workflows reliably. If you're using an LLM to orchestrate multi-step data processing, complex calculations, or algorithmic workflows (e.g., financial computations, scientific simulations, or business logic), you cannot assume the model will faithfully follow your procedure just because it achieves high accuracy on similar reasoning benchmarks. The 41-point accuracy drop from 5-step to 95-step procedures means that moderately complex workflows—well within typical real-world requirements—face unacceptable failure rates without external verification. In practice, you would need to add explicit verification layers: breaking procedures into shorter sub-tasks with intermediate human or rule-based validation, implementing self-verification mechanisms where the model checks its own work, or using constrained generation (forcing the model to output intermediate steps in parseable format for automated checking). The trade-off is increased latency and API calls, but it's necessary to meet reliability requirements for mission-critical processes.
Limitations and When Not to Use This
The benchmark, while revealing, is limited to arithmetic operations on simple numeric inputs—it doesn't test whether findings generalize to procedural execution in natural language, symbolic reasoning, or more complex domain-specific algorithms. The paper abstracts away the effect of prompt engineering and model scaling on procedural faithfulness; it's unclear whether better prompting or much larger models substantially change the accuracy curves, or whether the degradation is fundamental to the architecture. The work also doesn't deeply investigate causal mechanisms—it identifies failure modes (missing answers, premature termination) but doesn't explain whether these stem from training-time biases toward short outputs, attention mechanisms that lose track of state, or fundamental limitations in how transformers represent and maintain procedural context. Finally, the benchmark doesn't measure the cost or feasibility of improving faithfulness through fine-tuning, RLHF, or other training interventions—only what the current models do out of the box.
Research Context
This paper builds on a growing body of work questioning whether LLM reasoning abilities are as robust as raw benchmark scores suggest. It extends concerns from reasoning benchmarks (like MATH, GSM8K) where final-answer accuracy can be misleading by introducing a controlled diagnostic setting specifically designed to isolate procedural execution. The work is aligned with the broader "interpretability through decomposition" research direction—breaking down complex reasoning into atomic steps to understand where models fail. It also connects to ongoing research on hallucination and faithfulness in LLMs, particularly work examining whether models are actually executing instructions or pattern-matching plausible outputs. The diagnostic benchmark approach may spawn follow-up work on other procedural domains (code execution, protocol following, workflow management) and investigations into architectural changes or training methods that improve procedural faithfulness.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::