Semantic Invariance in Agentic AI
| Authors | I. de Zarzà et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2603.13173 |
| Download | |
| Categories | cs.AI, cs.CL |
Abstract
Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance.Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.
Engineering Breakdown
Plain English
This paper addresses a critical gap in LLM agent evaluation: standard benchmarks test accuracy on fixed problem formulations, but don't capture whether agent reasoning remains stable when inputs are semantically equivalent but syntactically different. The authors propose a metamorphic testing framework that applies eight semantic-preserving transformations (paraphrase, fact reordering, expansion, contraction, and others) to systematically stress-test LLM reasoning robustness. The core finding is that current LLM agents show significant performance variance under these benign transformations, suggesting they lack true semantic invariance despite high benchmark scores. This work provides both a diagnostic methodology and evidence that deployment of LLM agents in high-stakes applications requires explicit invariance testing beyond standard accuracy metrics.
Core Technical Contribution
The primary innovation is formalizing semantic invariance as a measurable and testable property of agentic LLMs, and developing a metamorphic testing framework to systematically evaluate it. Rather than treating LLM agent robustness as a monolithic concept, the authors decompose it into eight distinct semantic-preserving transformations, each exposing different failure modes in reasoning stability. This shifts the evaluation paradigm from point-accuracy benchmarking to trajectory-based robustness assessment, where the same logical problem in different linguistic forms should yield identical or near-identical agent behavior. The framework is agnostic to the underlying LLM architecture, making it broadly applicable to any agentic system that reasons over natural language inputs.
How It Works
The framework operates by: (1) taking a baseline task or problem statement as input, (2) applying one of eight semantic-preserving transformations (identity as control, paraphrase via synonym substitution, fact reordering to reorganize argument sequence, expansion with elaborated context, contraction with condensed phrasing, and others), (3) running the same LLM agent on the transformed input while preserving the underlying logical structure, and (4) measuring divergence in agent outputs across transformations using both exact-match and semantic similarity metrics. For each transformation, the framework captures not just final accuracy but intermediate reasoning steps, allowing fine-grained diagnosis of where semantic drift occurs in the agent's chain-of-thought. The key insight is that metamorphic testing doesn't require a gold-standard oracle; instead, it leverages the property that semantically equivalent inputs should produce equivalent reasoning trajectories. The framework aggregates results across all eight transformations to produce an invariance score, providing quantitative evidence of whether an agent's reasoning generalizes beyond its training distribution.
Production Impact
For engineers deploying LLM agents in consequential domains (medical diagnosis, financial advisory, scientific research), this framework becomes a critical pre-deployment validation step. Instead of relying solely on held-out test accuracy, teams can now run invariance tests to detect whether agents will degrade in customer-facing scenarios where input phrasing varies unpredictably (e.g., different physicians phrasing the same clinical query differently, or analysts reformulating the same business question in different ways). This catches a failure mode that static benchmarks miss: an agent might achieve 95% accuracy on canonical problem phrasings but drop to 70% when customers naturally rephrase questions. Integration into a CI/CD pipeline is straightforward—metamorphic tests can run automatically before each model release, adding moderate computational overhead (roughly 8× the cost of single-run evaluation, since you run 8 transformations per test case). The trade-off is runtime cost and the need to instrument agents to expose intermediate reasoning, but for high-stakes deployments, this is justified.
Limitations and When Not to Use This
The framework assumes that semantic equivalence can be reliably preserved through syntactic transformations, which breaks down for subtle semantics where context ordering or phrasing genuinely affects meaning (e.g., in legal or medical contexts, subtle word choice can alter interpretation). The paper does not address adversarial perturbations or out-of-distribution inputs that aren't semantically equivalent; it only covers variations that preserve meaning, leaving other robustness concerns unresolved. Scaling to multi-agent or long-horizon reasoning tasks becomes computationally expensive, and the framework's ability to pinpoint root causes (e.g., is it a tokenizer issue, attention mechanism fragility, or training data bias?) remains limited—it diagnoses the symptom but not always the underlying failure. Finally, the eight transformations are heuristically chosen and may not comprehensively cover all semantically-preserving variations that matter in practice, and results likely vary significantly by task domain, model architecture, and scale.
Research Context
This work builds on decades of metamorphic testing research in software engineering (Weyuker, 1982; Zhou et al., 2016) and recent concerns about LLM robustness under distribution shift (Hendrycks & Gimpel, 2017; Robey et al., 2024). It complements existing robustness evaluation frameworks (adversarial attacks, OOD generalization) by focusing specifically on semantic stability, which is orthogonal to traditional accuracy metrics. The paper positions itself within the broader agenda of trustworthy AI agents, alongside work on agent alignment (Hubinger et al., 2023), verifiability (Lewkowycz et al., 2022), and reasoning reliability. By formalizing invariance testing, it opens a research direction toward compositional robustness evaluation—testing not just individual agent steps but entire reasoning chains—and may influence how enterprise AI platforms certify agent reliability before deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::