Study
Human FactorsNew This Week

Automated assessment of creativity in programming environments can achieve 68% correlation with human judgment.

Computational methods, including LLMs, can objectively and efficiently assess creativity in digital design projects, mirroring human expert evaluations.

Journal of Learning Analytics · 2025

01

Key Findings

  • 01The ECD-based automated approach showed a moderate correlation (r = 0.48) with human evaluations.
  • 02The LLM-based approach (ChatGPT-4) demonstrated a stronger correlation (r = 0.68) with human evaluations.
  • 03The LLM-based approach exhibited greater consistency across different training examples (r = 0.82).
02

Application

Design takeaway

Consider implementing or developing AI-driven tools for assessing creative aspects of design projects to ensure objectivity and efficiency.

How to apply

Use AI-powered platforms or develop custom scripts to analyze student projects for elements like novelty, complexity, and efficiency, comparing the automated scores to established human rubrics.

Project actions

  • 01When assessing creative elements of your design project, explore if computational tools can offer objective metrics.
  • 02Consider how AI could be used to analyze user-generated content or design iterations for specific creative qualities.
03

Method & Evidence

AimCan automated methods, specifically an evidence-centred design framework and an LLM-based approach, accurately assess creativity in primary students' music programming artifacts compared to human evaluations?
MethodComparative analysis and correlational study
ProcedureResearchers collected 383 music programming artifacts from 194 primary school students. They then applied two automated assessment methods: one based on an evidence-centred design (ECD) framework incorporating divergent thinking, complexity, efficiency, and emotional expressiveness, and another using ChatGPT-4 with few-shot learning, trained on human creativity ratings and ECD examples. Both automated scores were compared against human expert evaluations.
Sample194 participants (students)
ContextPrimary school students' music programming artifacts within a flow-based programming environment.

Variables

IV["Automated assessment approach (ECD-based vs. LLM-based)","Few-shot learning examples provided to LLM"]
DV["Correlation with human creativity ratings","Consistency of scores"]
CV["Age group of students (primary school)","Type of artifacts (music programming)","Human evaluation rubric/method (implied)"]
04

Strengths & Limitations

Strengths

  • +Utilized a large dataset of student artifacts.
  • +Compared two distinct automated assessment methodologies.
  • +Investigated a novel application of AI in creativity assessment within a STEM context.

Limitations

The accuracy of automated assessment depends heavily on the quality and relevance of the training data and the specific AI model used. It may struggle with nuanced or highly abstract forms of creativity.

Reliability & validity

The study demonstrates good reliability for the LLM-based approach (r=0.82 consistency) and moderate to strong validity through its correlation with human judgment (r=0.68). The ECD-based approach showed moderate validity (r=0.48).

Think critically

To what extent can automated assessment truly capture the subjective and often ineffable qualities of human creativity, and what are the ethical implications of relying on AI for such evaluations?

05

Design Principles

"Leverage computational intelligence to objectively measure and provide feedback on creative attributes within design outputs."

This research offers a pathway for designers and educators to leverage technology for more scalable and consistent evaluation of creative output. It addresses the limitations of subjective human assessment, enabling broader application of creativity metrics in design education and practice.

06

What This Means for Your Design

Computers can now be trained to judge how creative a student's digital project is, almost as well as a human teacher can, and much faster.

How to use in your project

  • 1.Reference this study when discussing the methods used to assess creativity or user engagement in your design project, especially if you are using digital tools or analyzing user-generated content.
07

Add to My Project

08

Quick Cite

(2025). Exploring Automated Assessment of Primary Students’ Creativity in a Flow-Based Music Programming Environment. Journal of Learning Analytics. https://doi.org/10.18608/jla.2025.8835 Retrieved from https://designdex.org/study/0d6c08a6-2e42-4977-a534-da9148aebf0c/automated-assessment-of-creativity-in-programming-environments-can-achieve-68-correlation-with-human-judgment

Paragraph starter

Automated assessment tools, particularly those employing advanced AI models like large language models, offer a promising approach to objectively and efficiently evaluate creative output in design projects. Research indicates that LLM-based methods can achieve significant correlations with human expert judgment, suggesting their utility in providing scalable and consistent feedback on creative attributes such as novelty and complexity, thereby complementing traditional subjective evaluation methods.

09

Source

Journal of Learning Analytics

Exploring Automated Assessment of Primary Students’ Creativity in a Flow-Based Music Programming Environment

journal · 2025

View source

Questions about this research

What does the research say about automated assessment of creativity in programming environments can achieve 68% correlation with human judgment?
Consider implementing or developing AI-driven tools for assessing creative aspects of design projects to ensure objectivity and efficiency. Evidence: Journal of Learning Analytics (2025).
Why does "Automated assessment of creativity in programming environments can achieve 68% correlation with human judgment." matter for design?
This research offers a pathway for designers and educators to leverage technology for more scalable and consistent evaluation of creative output. It addresses the limitations of subjective human assessment, enabling broader application of creativity metrics in design education and practice.
How can designers apply this research?
Consider implementing or developing AI-driven tools for assessing creative aspects of design projects to ensure objectivity and efficiency.
What were the main findings?
The ECD-based automated approach showed a moderate correlation (r = 0.48) with human evaluations.. The LLM-based approach (ChatGPT-4) demonstrated a stronger correlation (r = 0.68) with human evaluations.. The LLM-based approach exhibited greater consistency across different training examples (r = 0.82).
What research method was used?
Comparative analysis and correlational study with 194 participants (students).
What should I do differently in my next project?
Use AI-powered platforms or develop custom scripts to analyze student projects for elements like novelty, complexity, and efficiency, comparing the automated scores to established human rubrics.
What are the limitations?
The study focused on primary school students and a specific programming environment, which may limit generalizability to other age groups or creative domains. The reliance on LLMs also introduces potential biases inherent in the training data.
Is there evidence that automated assessment affects design outcomes?
Automated tools, particularly those using advanced AI like LLMs, can effectively and consistently evaluate creativity in digital creations, achieving results comparable to human experts. This research offers a pathway for designers and educators to leverage technology for more scalable and consistent evaluation of crea Source: Journal of Learning Analytics (2025).
Where does this creativity research apply?
Primary school students' music programming artifacts within a flow-based programming environment. It sits within human factors research on designdex.org.

Related research topics

automated assessment design research · evidence on automated assessment · does automated assessment improve design outcomes · creativity studies for designers · automated assessment and creativity findings · human factors research evidence