ROSE: An Intent-Centered Evaluation Metric for NL2SQL
| Authors | Wenqi Pei et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2604.12988 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.
Engineering Breakdown
Plain English
ROSE is a new evaluation metric for NL2SQL (Natural Language to SQL) systems that addresses critical flaws in the standard Execution Accuracy (EX) metric. The paper identifies that EX is unreliable because it's sensitive to syntactic variations in SQL, ignores that natural language questions can have multiple valid interpretations, and fails when ground-truth SQL annotations are incorrect. The authors propose ROSE, which evaluates whether a predicted SQL actually answers the user's intent rather than matching reference SQL exactly, using an adversarial two-stage cascade where a Prover assesses semantic correctness and a Refuter challenges judgments using ground-truth as evidence.
Core Technical Contribution
The core novelty is an intent-centered evaluation paradigm that decouples semantic correctness from reference-dependent string matching. Rather than comparing predicted SQL to ground-truth SQL syntactically, ROSE introduces a Prover-Refuter adversarial cascade architecture where the Prover independently judges whether a predicted SQL answers the user's question correctly, while an Adversarial Refuter uses the ground-truth SQL as external evidence to validate or challenge that judgment. This two-stage approach handles the fundamental problem that multiple SQL queries can correctly answer the same natural language question, and it's robust to errors in ground-truth annotations because the Refuter stage explicitly incorporates contradictory evidence. The shift from reference-dependent to intent-centered evaluation is a significant methodological contribution that moves beyond simple string-matching metrics.
How It Works
ROSE operates in two sequential stages. In stage one, the SQL Prover takes the user's natural language question and the predicted SQL as input and generates a semantic judgment about whether the predicted SQL correctly answers the user's intent, without reference to any ground-truth SQL. This judgment includes reasoning about what the question is asking and whether the predicted query's semantics align with that intent. In stage two, the Adversarial Refuter receives the Prover's judgment along with the ground-truth SQL as additional evidence and attempts to challenge or refute the Prover's assessment—for instance, if the ground-truth SQL produces different results that better match the intent, the Refuter flags this as evidence that the Prover was wrong. The final ROSE score is computed by reconciling these two stages, creating a robust evaluation that accounts for multiple valid SQL formulations while remaining sensitive to truly incorrect predictions. The cascade architecture ensures that both semantic understanding and ground-truth consistency inform the final verdict.
Production Impact
Adopting ROSE in production NL2SQL systems would fundamentally change how you measure model performance and validate system quality. Currently, teams using EX metric often waste effort optimizing for syntactic SQL similarity when different queries answer equally well; ROSE directly measures semantic correctness, reducing false negatives where good predictions are marked wrong and false positives where wrong predictions pass due to lucky syntax matches. For data annotation and ground-truth collection, ROSE reduces the impact of human errors in SQL annotations—a Refuter stage automatically catches cases where the ground-truth SQL was incorrectly written. The practical trade-off is increased computational cost: the two-stage Prover-Refuter cascade likely requires multiple LLM calls per evaluation, potentially 2-5x slower than simple string matching, which matters at scale when evaluating millions of predictions daily. For teams already running model evaluation pipelines with LLM-based metrics, the overhead is manageable, but this is not a lightweight metric suitable for real-time serving feedback.
Limitations and When Not to Use This
ROSE assumes you have access to an LLM-based Prover and Refuter capable of semantic reasoning about SQL and natural language intent, which means it's not applicable to teams without API access to strong language models or those running fully offline systems. The paper doesn't address how to handle ambiguous user intents where even humans would disagree on the correct SQL—ROSE may not converge when both the Prover and ground-truth SQL represent equally valid but different interpretations. There's also a dependency on ground-truth SQL quality for the Refuter stage: while ROSE claims robustness to errors, a severely corrupted or systematically biased ground-truth annotation set could mislead the Refuter into accepting incorrect predictions. The evaluation framework itself needs validation on diverse domains and SQL complexity levels; the abstract doesn't specify which benchmarks were tested, so generalization to edge cases (complex joins, subqueries, aggregations with filters) remains unclear.
Research Context
This work builds on decades of NL2SQL research including semantic parsing and database querying systems, directly addressing known limitations of Execution Accuracy that have been criticized in recent work on code generation and program synthesis evaluation. The paper contributes to a broader shift in the research community away from reference-dependent metrics (like BLEU for translation or exact match for QA) toward intent-focused evaluation paradigms, similar to recent work on evaluating code generation where syntactic variation matters less than functional correctness. ROSE advances the evaluation methodology for NL2SQL benchmarks like Spider, WikiSQL, and BIRD, potentially enabling more accurate leaderboard rankings and better identification of which models genuinely improve semantic understanding versus those that just fit reference SQL better. The adversarial Prover-Refuter framework opens research directions into other program synthesis tasks (code generation, query optimization, logic programming) where ground-truth is noisy or non-unique.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::