Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness
| Authors | Tomer Ashuach et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2604.12373 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation. We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage. On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes. We hypothesize this is due to high inter-model agreement of answer correctness. To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions. Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning. We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.
Engineering Breakdown
Plain English
This paper investigates whether large language models have privileged knowledge about their own answer correctness—internal information that wouldn't be visible to external observers. The researchers train correctness classifiers using representations from a model's hidden states versus external models, hypothesizing that self-probes would outperform peer-model probes. On standard benchmarks, they find no performance advantage for self-probes, likely due to high inter-model agreement on correctness. However, when evaluating on disagreement subsets where models produce conflicting predictions, they discover genuine privileged knowledge emerges, suggesting LLMs do possess internal understanding of their own correctness that diverges from what external models can observe.
Core Technical Contribution
The core novelty is the discovery and measurement methodology for identifying privileged knowledge in LLMs by isolating disagreement cases between models. Rather than assuming all privileged information should improve correctness classifiers uniformly, the authors recognize that high inter-model agreement masks genuine private knowledge. They introduce the disagreement subset evaluation strategy as a lens to reveal this hidden information—a simple but powerful insight that high consensus in standard settings obscures model-specific understanding. This reframes the question from 'do LLMs have privileged knowledge' to 'how do you isolate it when consensus exists,' opening a new evaluation paradigm for studying internal model representations.
How It Works
The methodology trains two parallel correctness classification systems: (1) self-probes that extract features from a target model's hidden states (representations at intermediate layers), and (2) peer-probes that use features from external models' hidden states or outputs as input. For each question-answer pair, these classifiers predict whether the model's answer is correct. The key innovation is partitioning evaluation data into two subsets: agreement cases where all models make the same correctness prediction, and disagreement cases where models conflict. On agreement cases, both probe types perform similarly because the signal is unambiguous. On disagreement cases, self-probes should outperform peer-probes if the model has private knowledge about its own correctness that external observers cannot access. This disagreement-based evaluation reveals the hidden advantage of self-representations by removing the confounding factor of universal agreement.
Production Impact
For production LLM systems, this research enables better uncertainty quantification and confidence scoring without relying on ensemble methods or external models. If integrated into a production pipeline, you could use self-probes during inference to generate calibrated confidence estimates alongside predictions, helping downstream applications decide when to route queries to humans or reject low-confidence outputs. This reduces the need for expensive peer-model ensembles since a single model's internal states can capture meaningful confidence information. The trade-off is modest: you need to fine-tune a lightweight correctness classifier on your model's hidden states (manageable compute cost, typically <5% inference overhead), but you gain interpretable uncertainty without deployment of multiple models. For safety-critical applications like code generation or medical QA, this could reduce hallucination exposure by surfacing model doubt early in decision pipelines.
Limitations and When Not to Use This
The paper evaluates primarily on standard benchmarks where inter-model agreement is already high, limiting the size of disagreement subsets available for testing—this may not generalize to niche domains or tasks where model diversity is lower. The approach assumes that privileged knowledge manifests primarily in disagreement cases, but some internal information might be useful even when models agree, leaving potential gains unexplored. The correctness classifiers are trained on labeled data (correct/incorrect answers), which requires ground truth labels; this dependency limits applicability to open-ended generation tasks without clear right answers. Finally, the paper does not deeply explore why models develop different internal representations despite training on similar data, leaving the mechanistic understanding of privileged knowledge formation as an open question for future work.
Research Context
This work builds on the broader research into interpretability and introspection in large language models, connecting to prior work on probing tasks and representation analysis. It extends recent discussions about what LLMs 'know' versus what they 'show,' similar to research on hidden knowledge and jailbreaking. The disagreement-based evaluation strategy parallels ensemble diversity analysis in classical machine learning and relates to work on model calibration and uncertainty quantification. The paper opens a new research direction: using model disagreement as a lens to isolate genuinely private information, which could inform future work on mechanistic interpretability, adversarial robustness (where privileged knowledge might reveal model vulnerabilities), and consensus-based explanation methods.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::