Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation
| Authors | Richard J. Young |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.20172 |
| Download | |
| Categories | cs.CL, cs.AI, cs.LG |
Abstract
Recent work on chain-of-thought (CoT) faithfulness reports single aggregate numbers (e.g., DeepSeek-R1 acknowledges hints 39% of the time), implying that faithfulness is an objective, measurable property of a model. This paper demonstrates that it is not. Three classifiers (a regex-only detector, a two-stage regex-plus-LLM pipeline, and an independent Claude Sonnet 4 judge) are applied to 10,276 influenced reasoning traces from 12 open-weight models spanning 9 families and 7B to 1T parameters. On identical data, these classifiers produce overall faithfulness rates of 74.4%, 82.6%, and 69.7%, respectively, with non-overlapping 95% confidence intervals. Per-model gaps range from 2.6 to 30.6 percentage points; all are statistically significant (McNemar's test, p < 0.001). The disagreements are systematic, not random: inter-classifier agreement measured by Cohen's kappa ranges from 0.06 ("slight") for sycophancy hints to 0.42 ("moderate") for grader hints, and the asymmetry is pronounced: for sycophancy, 883 cases are classified as faithful by the pipeline but unfaithful by the Sonnet judge, while only 2 go the other direction. Classifier choice can also reverse model rankings: Qwen3.5-27B ranks 1st under the pipeline but 7th under the Sonnet judge; OLMo-3.1-32B moves in the opposite direction, from 9th to 3rd. The root cause is that different classifiers operationalize related faithfulness constructs at different levels of stringency (lexical mention versus epistemic dependence), and these constructs yield divergent measurements on the same behavior. These results demonstrate that published faithfulness numbers cannot be meaningfully compared across studies that use different classifiers, and that future evaluations should report sensitivity ranges across multiple classification methodologies rather than single point estimates.
Engineering Breakdown
Plain English
This paper challenges the common practice of reporting chain-of-thought faithfulness as a single, objective number by demonstrating that different evaluation methods produce dramatically different results on the same data. The authors tested three different classifiers (regex-only, regex-plus-LLM, and Claude Sonnet 4) on 10,276 reasoning traces from 12 open-weight models, finding faithfulness rates of 74.4%, 82.6%, and 69.7% respectively—with confidence intervals that don't overlap and per-model disagreements ranging from 2.6 to 30.6 percentage points, all statistically significant. This reveals that what researchers and practitioners treat as a measurable property of a model is actually highly dependent on how you measure it, undermining the validity of aggregate faithfulness claims like 'DeepSeek-R1 acknowledges hints 39% of the time.'
Core Technical Contribution
The paper's core insight is that chain-of-thought faithfulness metrics lack ground truth and are fundamentally classifier-dependent rather than model properties. Rather than proposing a new algorithm, the authors conduct a rigorous empirical study showing that three reasonable but different evaluation approaches produce non-overlapping confidence intervals on identical data across 12 models spanning 7B to 1T parameters. This is a meta-scientific contribution that decouples the observed faithfulness score from the true underlying behavior of the model, demonstrating that prior single-number faithfulness claims are artifacts of measurement methodology rather than objective facts. The work uses established statistical testing (McNemar's test with p < 0.001) to show these differences are not noise but systematic disagreements rooted in how each classifier interprets reasoning traces.
How It Works
The evaluation pipeline processes 10,276 influenced reasoning traces (outputs from models given hints or prompts designed to test faithfulness) through three parallel classifiers. The first classifier uses regex patterns only—simple string matching to detect whether the model acknowledged external hints or constraints in its reasoning. The second stage uses regex rules to filter candidates, then passes borderline cases to an LLM for judgment, combining pattern matching efficiency with semantic understanding. The third independently applies Claude Sonnet 4 as a judge on the same traces. For each model in the dataset, each classifier produces a binary label (faithful or not faithful) for every trace, enabling head-to-head comparison. The disagreements between classifiers are analyzed using McNemar's test to confirm they're statistically significant rather than random variation, revealing systematic differences in how each method interprets what constitutes faithfulness.
Production Impact
For teams building AI products that rely on chain-of-thought reasoning—like reasoning-focused code generation, scientific discovery tools, or multi-step problem solving—this paper exposes a critical measurement problem: you cannot confidently claim your model is X% faithful without acknowledging that changing your evaluation methodology could shift that number by 10-30 percentage points. This directly impacts model selection, safety audits, and user-facing claims about model trustworthiness. In practice, this means: (1) avoid publishing single aggregate faithfulness numbers without detailed methodology documentation, (2) evaluate candidate models using multiple independent classifiers and report the variance, (3) treat faithfulness claims as classifier-specific rather than universal properties, and (4) invest in developing more robust evaluation frameworks with explicit agreements on what 'faithful reasoning' means semantically before measuring it. The cost is additional evaluation complexity and multiple judgment passes, but the gain is honest reporting that doesn't overstate model capabilities.
Limitations and When Not to Use This
The paper identifies a measurement problem but does not propose a solution—it does not define what faithfulness should be or which classifier is most correct. The three classifiers tested were chosen pragmatically (regex cost-effective, LLM semantic, Claude as independent judge) but are not exhaustive; other reasonable evaluation approaches might produce yet different results, expanding the disagreement space further. The study is also limited to open-weight models and influenced reasoning traces specifically; whether similar disagreements occur for other model families (closed-source), reasoning styles, or evaluation settings remains unclear. Finally, while the paper convincingly shows that metrics are classifier-dependent, it doesn't provide actionable guidance on how practitioners should choose among conflicting evaluation methods or whether some classifiers are better at measuring what we actually care about in real applications.
Research Context
This work builds on recent interest in understanding chain-of-thought reasoning quality, following papers that report aggregate faithfulness metrics (e.g., DeepSeek-R1 studies). It contributes to the broader critique of benchmark-driven AI research by highlighting how measurement methodology can dominate results, similar to concerns raised about metrics in other domains (e.g., BLEU scores in NMT). The paper sits at the intersection of interpretability research (understanding what models actually do during reasoning) and evaluation methodology, pushing back against the false precision of single-number faithfulness claims. This opens a research direction toward developing ground-truth definitions of reasoning faithfulness and classifier-agnostic metrics that remain stable across different evaluation approaches, potentially informing how the field reports model capabilities more honestly.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::