SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables
| Authors | Sungho Park et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23286 |
| Download | |
| Categories | cs.CL, cs.AI, cs.DB, cs.IR |
Abstract
Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated - and therefore error-prone - and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question-answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. Our benchmark, construction code, and baseline models are available at https://github.com/pshlego/SPARTA/tree/main.
Engineering Breakdown
Plain English
SPARTA is an automated framework for generating large-scale Table-Text question answering benchmarks that require multi-hop reasoning across tables and documents. The paper addresses a critical gap: existing QA benchmarks are small (manually curated), error-prone, and contain mostly shallow questions that require only 1-2 reasoning steps and rarely involve complex operations like aggregation or grouping. SPARTA reduces annotation time by 75% compared to HybridQA (the prior state-of-the-art benchmark) while automatically generating questions that demand deeper reasoning — multiple hops, aggregations, and advanced analytical operations. The framework enriches source tables with grounding tables and uses lightweight human validation to ensure quality at scale.
Core Technical Contribution
The core innovation is a fully automated benchmark construction pipeline that eliminates the labor bottleneck in creating Table-Text QA datasets. Rather than having annotators manually write questions, SPARTA generates complex multi-hop questions by: (1) building a reference fact database from table enrichment, (2) automatically constructing tree-structured reasoning paths that connect tables and text, and (3) translating these paths into natural language questions with corresponding answers. This approach is principled because it explicitly models reasoning as traversals through a structured graph, ensuring questions correspond to verifiable logical operations rather than arbitrary semantic variations. It differs from prior work like HybridQA (manual, small-scale) by making the generation process systematic and scalable while maintaining quality through strategic, lightweight human validation rather than full annotation.
How It Works
SPARTA operates in three main stages: (1) Table enrichment and fact database construction — each source table is augmented with grounding tables that provide additional context and enable cross-table relationships; (2) Reasoning path generation — the system builds tree-structured execution plans where nodes represent operations (selection, aggregation, joining, grouping, sorting) and edges represent data flow across tables and text passages; (3) Natural language generation and validation — these logical paths are converted to natural language questions with ground-truth answers derived from executing the path, then passed through lightweight human validation to filter errors. The input is a corpus of tables and accompanying documents; the transformation involves graph construction and traversal to enumerate valid reasoning chains; the output is a large-scale QA dataset where each question has a verified answer grounded in an explicit execution path. The framework uses the tree structure to ensure questions have well-defined complexity (measured in hops) and explicit operation types, making it possible to generate diverse, complex questions at scale.
Production Impact
For teams building QA systems over hybrid document-table corpora, SPARTA dramatically reduces the cost and time to build evaluation benchmarks — 75% faster annotation than manual alternatives means you can iterate on model improvements with fresh benchmark data rather than relying on stale, small public datasets. This is particularly valuable in enterprise settings (financial documents, regulatory filings, knowledge bases combining tables and prose) where you need domain-specific benchmarks but can't afford months of annotation. A production pipeline adopting this approach would substitute manual annotation with semi-automated generation followed by targeted validation, allowing you to scale benchmark creation alongside model development. The trade-off is that you need labeled source data (tables + text) and domain expertise to design the enrichment strategy; the computational cost of generation is negligible compared to annotation savings. The structured reasoning paths also provide interpretability benefits — you can understand exactly what operations a model needs to perform, making it easier to diagnose failures.
Limitations and When Not to Use This
SPARTA assumes source tables and documents are already paired and semantically related — it doesn't solve the harder problem of discovering relevant document-table combinations from unstructured corpora. The framework's effectiveness depends heavily on the quality of table enrichment and the choice of which grounding tables to add; poor enrichment strategy will propagate through all generated questions, introducing systematic bias. The paper doesn't address compositional generalization (can models trained on SPARTA questions generalize to new table schemas or operation types?) or evaluate robustness to adversarial variations. Additionally, since the framework generates questions by traversing predefined operation trees, it may miss natural question phrasings that don't follow the template structures, potentially creating a distribution mismatch with real user queries.
Research Context
SPARTA builds directly on HybridQA (2020), which pioneered Table-Text QA by manually annotating questions over Wikipedia tables and prose but suffered from scale and quality issues. The work follows a broader trend in NLP of automating benchmark construction (similar to how programs like T5 and FLAN use task templates) and specifically addresses the gap between small supervised benchmarks and the scale needed to train large language models. This research advances the intersection of structured QA (over tables/SQL) and unstructured QA (over documents) — a critical area as real-world information is increasingly stored in hybrid formats. The principled, tree-structured approach to question generation opens research directions in controllable question complexity, explicit reasoning supervision for weak-to-supervised learning, and benchmark contamination analysis.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::