ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery
| Authors | Shahar Levy et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.09237 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Many disciplines pose natural-language research questions over large document collections whose answers typically require structured evidence, traditionally obtained by manually designing an annotation schema and exhaustively labeling the corpus, a slow and error-prone process. We introduce ScheMatiQ, which leverages calls to a backbone LLM to take a question and a corpus to produce a schema and a grounded database, with a web interface that lets steer and revise the extraction. In collaboration with domain experts, we show that ScheMatiQ yields outputs that support real-world analysis in law and computational biology. We release ScheMatiQ as open source with a public web interface, and invite experts across disciplines to use it with their own data. All resources, including the website, source code, and demonstration video, are available at: www.ScheMatiQ-ai.com
Engineering Breakdown
Plain English
ScheMatiQ is a system that automatically generates database schemas and extracts structured data from large document collections by leveraging LLM calls, eliminating the need for manual annotation schema design and exhaustive labeling. Given a natural language research question and a document corpus, the system produces both a structured schema and a grounded database, then offers a web interface where domain experts can iteratively steer and refine the extraction results. The authors validated the approach in real-world settings across law and computational biology, demonstrating that ScheMatiQ outputs support genuine analytical workflows without the traditional slow and error-prone manual labeling process. The system is released as open source with a public web interface, inviting cross-disciplinary adoption.
Core Technical Contribution
The core novelty is an end-to-end workflow that uses backbone LLM calls to infer appropriate database schemas directly from a question and corpus, bypassing the traditional requirement to hand-design annotation schemas before extraction. This shifts the paradigm from schema-first (manual design) to question-driven schema generation, where the LLM infers structure based on the analytical intent and document content. The key architectural contribution is integrating LLM-generated schemas with an interactive web interface that enables human-in-the-loop refinement, allowing domain experts to validate, correct, and steer extraction results without retraining models. This human-in-the-loop design addresses a critical gap in prior work that either fully automated extraction (losing precision) or required complete manual schema design upfront.
How It Works
The system takes two inputs: a natural language research question and a document collection. First, the backbone LLM is prompted with the question and a sample of documents to generate a candidate database schema (table structure, column definitions, data types). Next, the LLM processes the full document collection using the inferred schema, extracting structured records that populate the database. The extracted data is then surfaced through a web interface where domain experts can inspect extraction results, identify errors, and provide corrections or refinements. These human corrections are fed back into the system, allowing iterative improvement of both the schema and the extraction quality without requiring complete re-labeling or model retraining. The final output is a grounded database—a collection of extracted records validated by human expertise—ready for downstream analysis.
Production Impact
For teams doing information extraction from specialized document collections (legal contracts, scientific papers, regulatory filings), ScheMatiQ eliminates weeks of upfront schema design and manual annotation labor. Instead of hiring domain experts to design schemas blindly and then label thousands of documents, engineers can now generate initial schemas automatically and use the web UI for targeted human review, dramatically reducing time-to-first-insights. The approach is particularly valuable when analytical requirements change mid-project: instead of scrapping months of labeled data and restarting, teams can revise the schema and re-extract with minimal overhead. The main trade-offs are LLM API costs (each document may require inference), latency for initial schema generation and full-corpus extraction, and the need for at least some domain expert involvement in the steering phase—this is not a fully automated, hands-off solution, but rather a hybrid human-AI pipeline that shifts expert effort from upfront schema design to post-hoc validation.
Limitations and When Not to Use This
The paper does not address how extraction quality scales with corpus size or document heterogeneity; while law and biology were validated, results may not transfer to highly specialized domains with unusual document structures or domains with extreme data volume. The approach assumes the backbone LLM can infer meaningful schemas from limited samples, which may fail if the question is ambiguous, the domain requires deep expert knowledge beyond LLM training, or the corpus is highly diverse in structure and language. The system requires human-in-the-loop refinement to reach production quality, making it unsuitable for fully automated, real-time extraction pipelines where latency or human involvement is prohibitive. Additionally, the paper lacks analysis of failure modes—when does the LLM generate incorrect schemas that human review cannot catch, or when does the web interface become a bottleneck rather than an acceleration mechanism. Finally, the cost and token efficiency of repeated LLM calls over large corpora is not characterized, leaving unclear the practical budget required at scale.
Research Context
ScheMatiQ builds on a growing body of work in LLM-powered information extraction (BERT-based sequence tagging, GPT-3/4 few-shot extraction) and human-in-the-loop machine learning, combining automatic extraction with interactive refinement. It extends prior schema-induction work by coupling schema generation with extraction in a single pipeline and adding the interactive web layer—prior work either separated these stages or assumed humans would design schemas manually. The system addresses a real bottleneck in applied NLP: the cost of preparing labeled datasets for domain-specific extraction tasks. This work opens a direction toward question-driven, iteratively refined extraction systems that adapt to new domains and analytical needs without retraining, and invites broader questions about how LLMs can reduce data annotation burden in specialized fields beyond law and biology.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::