Short answer

When using LLMs for requirement elicitation, consider the specific use case and the LLM's known characteristics to optimize the quality and relevance of generated competency questions.

Field: Innovation & Design
Source: arXiv preprint (2026)
Method: Empirical study
Evidence: Moderate effect

Generative AI can automate the creation of competency questions, democratizing ontology engineering and broadening stakeholder engagement. This innovation & design research insight is drawn from a 2026 study published in arXiv preprint. Using Empirical study, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When using LLMs for requirement elicitation, consider the specific use case and the LLM's known characteristics to optimize the quality and relevance of generated competency questions.

Study

Innovation & DesignNew This WeekModerate effect

LLM-Generated Competency Questions: A Cross-Domain Analysis for Enhanced Requirement Elicitation

Generative AI can automate the creation of competency questions, democratizing ontology engineering and broadening stakeholder engagement.

arXiv preprint · 2026

Key Findings

01LLMs can generate competency questions at scale, democratizing the process.
02The performance and generation profiles of LLMs vary depending on the specific use case.
03Quantitative measures can effectively compare LLM-generated competency questions across dimensions like readability and structural complexity.

Application

Design takeaway

When using LLMs for requirement elicitation, consider the specific use case and the LLM's known characteristics to optimize the quality and relevance of generated competency questions.

How to apply

In your design project, experiment with using LLMs to generate initial sets of user requirements or competency questions, then critically evaluate and refine them with domain experts.

Project actions

01When using AI tools for research, clearly document which tools were used and how.
02Always critically analyze AI-generated content for accuracy, relevance, and bias.

Method & Evidence

AimTo characterize the properties of competency questions generated by Large Language Models (LLMs) across different domains and model types.

MethodEmpirical study

ProcedureThe study generated competency questions using various open and closed LLMs based on defined use cases and requirements. These generated questions were then analyzed using a set of quantitative measures to assess properties such as readability, relevance, and structural complexity.

ContextOntology engineering and requirement elicitation

Variables

IV["Type of LLM (open vs. closed)","Use case/domain"]

DV["Readability of competency questions","Relevance of competency questions","Structural complexity of competency questions"]

CV["Defined use cases and requirements","Quantitative measures used for analysis"]

Strengths & Limitations

Strengths

+Cross-domain analysis provides broader applicability.
+Comparison of both open and closed LLM models.

Limitations

The effectiveness of AI-generated questions can depend heavily on the quality of the input prompts and the specific LLM used, which might not be readily apparent.

Reliability & validity

Reliability would be assessed by repeating the LLM generation process multiple times to check for consistency. Validity would be addressed by comparing LLM-generated questions against those created by human experts.

Think critically

How might the 'black box' nature of some LLMs impact the designer's ability to fully understand and trust the generated requirements?

Design Principles

"Automate repetitive or time-consuming aspects of design research where appropriate, while maintaining critical human oversight and validation."

Leveraging LLMs for competency question generation can significantly streamline the requirement elicitation process, making it more accessible and scalable. This allows design teams to gather and validate requirements more efficiently, especially in complex domains.

What This Means for Your Design

Computers that can write can help us ask better questions about what we need to design, but we need to check their work because they work differently depending on the job.

How to use in your project

1.Reference this study when discussing the use of AI tools for requirement gathering or user research in your design project.

Add to My Project

Quick Cite

Paragraph starter

The use of Large Language Models (LLMs) presents an opportunity to automate and scale aspects of requirement elicitation, as demonstrated by studies characterizing LLM-generated competency questions. While these tools can democratize processes like ontology engineering, it is crucial to understand that their output profiles vary significantly based on the specific LLM and the context of the design problem, necessitating careful selection and critical evaluation of AI-generated content.

Source

arXiv preprint

Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

journal · 2026

View source

Questions About This Research

What does the research say about llm-generated competency questions: a cross-domain analysis for enhanced requirement elicitation?: When using LLMs for requirement elicitation, consider the specific use case and the LLM's known characteristics to optimize the quality and relevance of generated competency questions. Evidence: arXiv preprint (2026).
Why does "LLM-Generated Competency Questions: A Cross-Domain Analysis for Enhanced Requirement Elicitation" matter for design?: Leveraging LLMs for competency question generation can significantly streamline the requirement elicitation process, making it more accessible and scalable. This allows design teams to gather and validate requirements more efficiently, especially in complex domains.
How can designers apply this research?: When using LLMs for requirement elicitation, consider the specific use case and the LLM's known characteristics to optimize the quality and relevance of generated competency questions.
What were the main findings?: LLMs can generate competency questions at scale, democratizing the process.. The performance and generation profiles of LLMs vary depending on the specific use case.. Quantitative measures can effectively compare LLM-generated competency questions across dimensions like readability and structural complexity.
What research method was used?: Empirical study.
How strong is the evidence?: Evidence strength is rated Moderate effect, based on a 2026 journal from arXiv preprint.
What should I do differently in my next project?: In your design project, experiment with using LLMs to generate initial sets of user requirements or competency questions, then critically evaluate and refine them with domain experts.
What are the limitations?: The study's findings may be specific to the LLMs and use cases tested; generalizability to all LLMs and domains requires further investigation.