Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
| Authors | Vishnu Murali et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.09470 |
| Download | |
| Categories | cs.CL |
Abstract
Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves natural-language mentions of categorical values through embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the benchmark, all agent transcripts, and evaluation code to support reproducibility.
Engineering Breakdown
Plain English
This paper introduces Jackal, the first large-scale benchmark for translating natural language queries into Jira Query Language (JQL), containing 100,000 validated NL-JQL pairs tested against a live Jira instance with 200,000+ issues. The core problem is that standard LLMs fail on JQL generation because they cannot discover which field values (component names, versions, etc.) actually exist in a specific Jira instance, and they cannot verify their generated queries against live data. To solve this, the authors propose Agentic Jackal, a tool-augmented agent that gives LLMs access to live query execution and semantic value grounding capabilities, allowing them to interactively discover valid values and validate queries. The benchmark enables evaluation of execution-based approaches on a real-world system, establishing baselines for future work on instance-specific, context-aware query generation.
Core Technical Contribution
The paper's core novelty is combining live query execution with semantic value grounding in an agentic framework for text-to-JQL translation, rather than treating it as a single-pass code generation task. Agentic Jackal equips the LLM with tools that allow it to query a live Jira instance to discover what categorical values (components, versions, statuses) actually exist, then validate its generated JQL against the real data before returning results. This execution-based approach is fundamentally different from prior work on text-to-SQL or code generation, which assume the database schema is known and static; here, the agent must handle instance-specific variations and resolve ambiguity through active interaction. The Jackal benchmark itself is a major contribution—the first public, large-scale, execution-validated dataset for this task, enabling reproducible measurement of progress on an understudied but practically important problem.
How It Works
The system operates in a multi-turn agentic loop: (1) the LLM receives a natural language query and an optional schema description of available JQL fields, (2) it generates an initial JQL query or decides it needs to discover valid categorical values by calling exploration tools, (3) the agent tools query the live Jira instance to fetch all valid component names, fix versions, statuses, and other enumerations, returning these to the LLM's context, (4) the LLM generates or refines the JQL query based on the discovered values, (5) the system executes the query against the live instance and returns results or error messages, (6) if execution fails or the result set seems wrong, the agent iterates by calling tools again or asking for clarification on ambiguous predicates (e.g., 'recent' issues mapped to date ranges). The key architectural insight is that semantic value grounding—knowing which exact strings are valid in the Jira instance—is solved by live queries rather than memorization, making the agent adaptive to any instance without retraining.
Production Impact
For engineers building query-translation systems, this work directly improves accuracy on real-world Jira instances where field values change frequently and are not known at training time. Production adoption would require integrating a stateful agentic loop with LLM inference, adding latency (multiple LLM calls + Jira API queries, likely 2-5 seconds per request compared to single-pass ~500ms) but dramatically improving precision and recall on ambiguous or paraphrased queries. The execution-based validation approach is a significant reliability win: instead of deploying an LLM that returns plausible-looking but invalid JQL, Agentic Jackal only returns queries that execute successfully, eliminating a major class of user-facing failures. The trade-off is that this approach requires live access to a Jira instance and API rate-limit budgets; it scales poorly if you need sub-second latency or if your Jira instance is unavailable. For internal tools (help desks, platform teams, analytics dashboards), this is a strong fit; for low-latency public APIs, you'd need caching or a hybrid approach with a fast fallback model.
Limitations and When Not to Use This
The benchmark and approach are specific to Jira's query language and may not generalize to other domain-specific query languages (SQL, Datalog, MongoDB, etc.) without substantial re-engineering, though the agent framework is conceptually transferable. The paper assumes that resolving ambiguity via live query exploration is always feasible—this breaks down in scenarios where you have slow or unreliable external APIs, or where categorical discovery is expensive (e.g., querying millions of possible values). The approach requires the LLM to understand JQL syntax and semantics well enough to generate valid queries after value grounding; if the model is weak, discovery alone won't help, and no amount of tool access fixes fundamental syntax errors. Finally, the 100,000 benchmark pairs are collected from a single Jira instance; generalization to other organizations' instances with different field schemas, naming conventions, or issue structures remains unexplored, and adversarial paraphrases or multi-step complex queries may still exceed the agent's capabilities.
Research Context
This paper builds on a growing body of work on tool-augmented LLMs (ReAct, Toolformer, ToolBench) and semantic parsing, but applies it to a previously under-explored domain: domain-specific query languages in commercial software. It extends text-to-SQL research (Spider, SParC) by addressing the 'live data grounding' problem—discovering which values are valid in a particular data instance rather than inferring from static schema—which is a practical concern often ignored in academic benchmarks. The benchmark contribution parallels the value of Spider and BIRD for SQL; Jackal provides a large, validated, execution-based dataset that future work can use to measure progress on text-to-JQL and similar instance-specific query tasks. This opens a research direction toward more adaptive, context-aware semantic parsing systems that interact with live data rather than memorizing all possible outputs during training.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::