AI Learning Tools Show Mixed Results in Skill Development
While AI-based learning tools effectively enhance knowledge acquisition and student affect, their impact on developing complex cognitive processes and skills is inconsistent.
International Journal of Educational Technology in Higher Education · 2025
Key Findings
- 01AI tools generally improve cognitive knowledge acquisition and affective outcomes.
- 02Effectiveness of AI tools in developing cognitive processes and skills varies significantly.
- 03AI is primarily used for assessment/evaluation, personalized feedback/recommendations, and intelligent tutoring.
- 04Approximately half of the reviewed studies used publicly available AI systems, while the other half developed proprietary tools.
- 0526 studies used AI to generate and deliver multimodal learning materials.
Application
Design takeaway
When designing AI-powered educational tools, prioritize features that actively engage users in skill-building activities and critical analysis, rather than solely focusing on content delivery or basic knowledge recall.
How to apply
When developing or selecting AI learning tools, evaluate their potential to support higher-order thinking and skill development, not just knowledge retention. Consider how the tool's features encourage analysis, synthesis, and application.
Project actions
- 01When designing an AI learning tool, think about how it will help users develop skills, not just memorize information.
- 02Consider how to measure if your AI tool is actually improving users' ability to think and solve problems.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Comprehensive systematic review covering a decade of research.
- +Categorization of AI roles and design features provides a structured overview.
Limitations
The effectiveness of AI tools can depend heavily on the specific subject matter, the students' prior knowledge, and how the AI is integrated into the overall curriculum.
Reliability & validity
The systematic review methodology enhances reliability by using a structured approach to literature selection and analysis. Validity is supported by the breadth of studies included, but may be limited by the heterogeneity of reported outcomes and methodologies within the reviewed papers.
Think critically
Given the mixed findings on skill development, what specific design interventions within AI learning tools could be implemented to consistently foster critical thinking and complex problem-solving abilities?
Design Principles
"Educational AI should be designed to scaffold complex cognitive processes and skill acquisition, not just to deliver information."
Designers of educational technologies must recognize that AI's utility extends beyond content delivery. A nuanced approach is required to ensure these tools actively foster higher-order thinking and practical skills, rather than just information recall.
What This Means for Your Design
AI tools in college can help you learn facts and feel good about learning, but they might not be as good at teaching you how to solve hard problems or think critically.
How to use in your project
- 1.Use this research to justify why focusing on skill development is important when designing your AI-based learning solution.
- 2.Refer to the mixed findings to explain potential challenges or areas for further investigation in your own design project.
Add to My Project
Quick Cite
(2025). Design and assessment of AI-based learning tools in higher education: a systematic review. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-025-00540-2 Retrieved from https://designdex.org/study/bae24ed8-4b8f-4b44-90ee-af83f466b599/ai-learning-tools-show-mixed-results-in-skill-development
Paragraph starter
This systematic review highlights that while AI-based learning tools show promise in enhancing knowledge acquisition and affective outcomes in higher education, their impact on developing cognitive processes and skills is inconsistent. This suggests that design efforts should focus not only on content delivery but also on features that actively promote higher-order thinking and practical skill development, acknowledging that the efficacy of AI in these areas is context-dependent.
Source
International Journal of Educational Technology in Higher Education
Design and assessment of AI-based learning tools in higher education: a systematic review
journal · 2025
View sourceQuestions about this research
- What does the research say about ai learning tools show mixed results in skill development?
- When designing AI-powered educational tools, prioritize features that actively engage users in skill-building activities and critical analysis, rather than solely focusing on content delivery or basic knowledge recall. Evidence: International Journal of Educational Technology in Higher Education (2025).
- Why does "AI Learning Tools Show Mixed Results in Skill Development" matter for design?
- Designers of educational technologies must recognize that AI's utility extends beyond content delivery. A nuanced approach is required to ensure these tools actively foster higher-order thinking and practical skills, rather than just information recall.
- How can designers apply this research?
- When designing AI-powered educational tools, prioritize features that actively engage users in skill-building activities and critical analysis, rather than solely focusing on content delivery or basic knowledge recall.
- What were the main findings?
- AI tools generally improve cognitive knowledge acquisition and affective outcomes.. Effectiveness of AI tools in developing cognitive processes and skills varies significantly.. AI is primarily used for assessment/evaluation, personalized feedback/recommendations, and intelligent tutoring.. Approximately half of the reviewed studies used publicly available AI systems, while the other half developed proprietary tools.
- What research method was used?
- Systematic Literature Review with 63 peer-reviewed articles.
- How strong is the evidence?
- Evidence strength is rated Mixed findings, based on a 2025 journal from International Journal of Educational Technology in Higher Education.
- What should I do differently in my next project?
- When developing or selecting AI learning tools, evaluate their potential to support higher-order thinking and skill development, not just knowledge retention. Consider how the tool's features encourage analysis, synthesis, and application.
- What are the limitations?
- The review's findings on skill development are varied, suggesting that the effectiveness of AI tools is highly dependent on their specific design and implementation context.
- Is there evidence that learning tools affects design outcomes?
- AI learning tools are good at helping students learn facts and feel more engaged, but they aren't always successful at teaching students how to think critically or apply what they've learned in practical ways. Designers of educational technologies must recognize that AI's utility extends beyond content delivery. A nuan Source: International Journal of Educational Technology in Higher Education (2025).
- Where does this content delivery research apply?
- Higher Education It sits within user-centred design research on designdex.org.
Related research topics
learning tools design research · evidence on learning tools · does learning tools improve design outcomes · content delivery studies for designers · learning tools and content delivery findings · user-centred design research evidence