SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints
| Authors | Rocky Klopfenstein et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2603.04334 |
| Download | |
| Categories | cs.DB, cs.AI, cs.LO, cs.PL |
Abstract
We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a constraint-mining pipeline that combines rule-based specification mining over example databases with LLM-based validation. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.
Engineering Breakdown
Plain English
SpotIt+ is an open-source evaluation tool for Text-to-SQL systems that uses bounded equivalence verification to find cases where generated SQL queries differ from ground-truth queries. Instead of relying on static test sets, it actively searches for database instances that expose discrepancies between the two queries. The key innovation is a constraint-mining pipeline that combines rule-based specification mining with LLM validation to ensure that discovered counterexamples reflect realistic, practically-relevant database constraints. On the BIRD dataset, this approach finds significantly more valid discrepancies than standard test-based evaluation while keeping the generated databases realistic.
Core Technical Contribution
The paper introduces a novel constraint-mining pipeline that combines automated rule extraction from example databases with LLM-based validation to generate realistic differentiating databases for SQL evaluation. Unlike prior work that either uses fixed test sets or generates arbitrary database instances, SpotIt+ ensures counterexamples satisfy domain-relevant constraints, making evaluation results more actionable for practitioners. The core insight is that constraint-aware verification can uncover many more meaningful bugs in generated SQL while avoiding false positives from unrealistic database states. This bridges the gap between black-box test-based evaluation and the theoretical ideal of complete equivalence checking.
How It Works
SpotIt+ takes as input a generated SQL query and a gold-standard ground-truth query, along with a seed set of example databases from the evaluation dataset. The constraint-mining pipeline first applies rule-based specification mining over the examples to extract likely database constraints (foreign keys, uniqueness, check conditions, etc.), then uses an LLM to validate and refine these candidate constraints. The core verification engine then performs bounded equivalence checking: it systematically searches the constraint-satisfying database space for instances where the two SQL queries produce different results, treating this as a satisfiability problem. When a counterexample database is found, it confirms a genuine discrepancy; the mined constraints ensure this counterexample is realistic and reflects actual database design patterns rather than pathological edge cases.
Production Impact
For teams building Text-to-SQL systems, SpotIt+ provides a more reliable way to measure progress beyond standard benchmarks, enabling you to catch subtle semantic bugs that pass simple test suites. The constraint-mining approach reduces engineering friction: instead of manually specifying database schemas and constraints for evaluation, you can bootstrap them automatically from your training data, scaling to new domains more easily. Integration into a CI/CD pipeline is straightforward—run SpotIt+ as a regression test to validate new model versions, and use discovered discrepancies to debug generation failures. The trade-off is computational cost: bounded equivalence checking is more expensive than running fixed test cases, so evaluation cycles will be slower (minutes to hours per large model depending on query complexity and constraint count), and you'll need to maintain the constraint-mining pipeline infrastructure.
Limitations and When Not to Use This
SpotIt+ assumes that the mined constraints accurately represent the true database design patterns; if your domain has implicit constraints not visible in the example data, the tool may miss realistic counterexamples or generate invalid ones. The approach is bounded equivalence checking, meaning it cannot prove a query is correct—only find counterexamples—so you still need a deployment verification strategy for production queries. The LLM-based validation step introduces potential brittleness: if the LLM misclassifies a constraint as valid when it isn't, subsequent counterexamples may be rejected by real databases, undermining the evaluation's practical relevance. The paper evaluates only on BIRD; generalization to other SQL domains, highly complex schemas with hundreds of tables, or specialized constraint types (temporal constraints, complex triggers) remains unclear and likely requires re-tuning the mining pipeline.
Research Context
This work advances the Text-to-SQL evaluation literature by moving beyond fixed test-based metrics (Exact Match, Execution Accuracy) toward automated, constraint-aware equivalence checking. It builds on prior work in semantic equivalence verification for databases and schema-aware code generation, but is the first to combine constraint mining with LLM validation for practical SQL evaluation. The research opens a new direction: using specification mining and LLMs together to make evaluation both more thorough and more realistic, a pattern applicable to other code generation tasks (program synthesis, SQL optimization). The BIRD dataset serves as the primary benchmark, and SpotIt+ will likely become a standard tool for comparing SQL generation models, similar to how HumanEval dominates Python code generation evaluation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::