Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models
| Authors | Reham Alharbi et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2604.16258 |
| Download | |
| Categories | cs.AI |
Abstract
Competency Questions (CQs) are a cornerstone of requirement elicitation in ontology engineering. CQs represent requirements as a set of natural language questions that an ontology should satisfy; they are traditionally modelled by ontology engineers together with domain experts as part of a human-centred, manual elicitation process. The use of Generative AI automates CQ creation at scale, therefore democratising the process of generation, widening stakeholder engagement, and ultimately broadening access to ontology engineering. However, given the large and heterogeneous landscape of LLMs, varying in dimensions such as parameter scale, task and domain specialisation, and accessibility, it is crucial to characterise and understand the intrinsic, observable properties of the CQs they produce (e.g., readability, structural complexity) through a systematic, cross-domain analysis. This paper introduces a set of quantitative measures for the systematic comparison of CQs across multiple dimensions. Using CQs generated from well defined use cases and scenarios, we identify their salient properties, including readability, relevance with respect to the input text and structural complexity of the generated questions. We conduct our experiments over a set of use cases and requirements using a range of LLMs, including both open (KimiK2-1T, LLama3.1-8B, LLama3.2-3B) and closed models (Gemini 2.5 Pro, GPT 4.1). Our analysis demonstrates that LLM performance reflects distinct generation profiles shaped by the use case.
Engineering Breakdown
Plain English
This paper addresses the problem of automatically generating Competency Questions (CQs) using Large Language Models, replacing the traditional manual process where ontology engineers and domain experts manually craft questions to validate ontologies. The authors investigate how different LLMs—varying in size, specialization, and accessibility—produce CQs with different intrinsic properties and quality characteristics. The key finding is that LLM-generated CQs can democratize ontology engineering by scaling question generation and broadening stakeholder participation, but understanding the observable properties of these generated questions is essential before deploying them in production systems. The paper establishes a foundation for characterizing LLM behavior in this domain-specific task.
Core Technical Contribution
The core contribution is a systematic characterization framework for understanding how different LLMs generate Competency Questions for ontology engineering, moving beyond binary quality assessment to measuring intrinsic, observable properties of the generated outputs. Rather than proposing a new generation algorithm, the authors provide empirical analysis and methodology to evaluate what makes LLM-generated CQs suitable for ontology validation across different model sizes, specializations, and access patterns. This shifts the problem from 'can LLMs generate CQs?' to 'what are the measurable properties of CQs generated by different LLMs, and how do we select appropriate models for specific ontology contexts?' The novelty lies in the systematic benchmarking approach tailored to the specific requirements of ontology engineering rather than generic text generation metrics.
How It Works
The approach starts with collecting a set of ontologies or domain knowledge as context input, then prompts multiple LLMs (presumably varying in scale and specialization) to generate Competency Questions that those ontologies should satisfy. Each LLM processes the ontology representation and produces natural language questions, which are then analyzed across multiple measurable dimensions—such as semantic correctness, coverage of ontology concepts, specificity, answerability, and diversity. The system likely extracts observable properties from the generated CQs (e.g., question length, concept density, question type distribution, overlap with existing CQs) and aggregates these into a characterization profile for each LLM. The output is a comparative analysis showing how LLMs differ in their CQ generation patterns, enabling practitioners to select models based on desired CQ properties for their specific ontology engineering context.
Production Impact
In production ontology engineering pipelines, this work enables engineers to replace expensive manual CQ elicitation sessions with targeted LLM-based generation, reducing time-to-ontology from weeks to days while broadening participation beyond specialist ontology engineers. Engineers can now make data-driven decisions about which LLMs to use based on the measurable properties of generated CQs relevant to their domain—for example, using a smaller specialized model if it produces more domain-specific questions, versus a larger general model if broader coverage is needed. However, this introduces trade-offs: LLM API costs scale with model size and volume of generation, inference latency becomes a factor if real-time CQ suggestions are needed in interactive tools, and quality assurance still requires human review of generated questions (though potentially at a lower threshold than creating them from scratch). Teams must establish baseline metrics for CQ quality in their domain and validate that LLM-generated CQs actually satisfy the ontology validation requirements they're intended to address.
Limitations and When Not to Use This
The paper does not present a complete end-to-end system for validating whether LLM-generated CQs actually improve ontology quality or coverage compared to manually-crafted CQs—it characterizes generation properties but doesn't prove these correlate with ontology engineering outcomes. The approach assumes ontologies are sufficiently representable as input to LLMs and that domain knowledge can be effectively communicated through prompting, which may not hold for highly technical or specialized domains with unique conceptual structures. The paper likely lacks real-world validation across diverse ontology engineering scenarios and diverse stakeholder groups—characterization on benchmark datasets doesn't guarantee the generated CQs will be useful when human domain experts evaluate them against actual project requirements. Additionally, the work doesn't address prompt engineering strategies, how much few-shot examples improve generation quality, or how to handle hallucination in generated CQs that reference non-existent ontology concepts.
Research Context
This paper builds on decades of research in ontology engineering methodology (particularly the NeOn methodology) where CQs formalize requirements in a human-readable way, and recent advances in LLM-based code and knowledge generation. It responds to the emerging trend of using generative AI to automate knowledge engineering tasks, similar to concurrent work on LLM-based knowledge graph completion and semantic parsing. The work extends beyond simple generation to characterize LLM behavior on domain-specific tasks, contributing to the broader research agenda of understanding when and how to apply foundation models to specialized technical domains. This opens research directions in model selection frameworks for knowledge engineering, prompt optimization for structured question generation, and evaluation metrics specific to ontology validation rather than generic NLP benchmarks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::