Short answer
When assessing design work, especially graphical outputs with varied solutions, consider using comparative methods like ACJ for more reliable evaluations.
- Field
- Modelling
- Source
- Research@THEA (2016)
- Method
- Comparative assessment study
- Sample
- 128 participants
- Evidence
- Strong effect
ACJ can provide a reliable and valid method for assessing graphical design work, even with diverse responses. This modelling research insight is drawn from a 2016 study published in Research@THEA. Using Comparative assessment study with 128 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When assessing design work, especially graphical outputs with varied solutions, consider using comparative methods like ACJ for more reliable evaluations.
Adaptive Comparative Judgements (ACJ) offer reliable assessment of graphical design capability.
ACJ can provide a reliable and valid method for assessing graphical design work, even with diverse responses.
Research@THEA · 2016
Key Findings
- 01ACJ demonstrated a high level of reliability in assessing graphical capability.
- 02Correlations between ACJ scores and criterion-referenced scores suggest validity, but also indicate potential influence from other variables on holistic judgments.
Application
Design takeaway
When assessing design work, especially graphical outputs with varied solutions, consider using comparative methods like ACJ for more reliable evaluations.
How to apply
When evaluating design portfolios or project outcomes, present evaluators with pairs of work and ask them to choose the 'better' option, rather than assigning absolute scores. This can be done iteratively to build a comparative ranking.
Project actions
- 01When assessing design solutions, consider using pairwise comparisons to gauge relative quality.
- 02Investigate how different criteria might implicitly influence comparative judgments.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Investigated a novel assessment method (ACJ) for design capability.
- +Employed a relatively large sample size for the study.
Limitations
The reliability of ACJ might be influenced by the number of items being compared and the complexity of the design task itself.
Reliability & validity
The study reports high reliability for ACJ. Validity is suggested by correlations with criterion-referenced scores, but the presence of other influencing variables indicates a need for careful interpretation.
Think critically
How might the 'additional variables' influencing ACJ judgments be identified and controlled for, or intentionally leveraged, in future design assessment frameworks?
Design Principles
"Comparative assessment methods can enhance the reliability of evaluating subjective or complex design outputs."
Accurate assessment of design outputs is crucial for feedback and development. ACJ offers a potentially more efficient and reliable alternative to traditional methods, especially for subjective or complex design tasks.
What This Means for Your Design
Comparing design work side-by-side (like ACJ) is a reliable way to judge how good it is, even if there are many different ways to do the design.
How to use in your project
- 1.You can use ACJ to assess the effectiveness of different design iterations or prototypes in your design project.
- 2.Reference this study when discussing the validity and reliability of your chosen assessment methods for design outcomes.
Add to My Project
Quick Cite
Paragraph starter
The assessment of graphical design capability can be reliably achieved through comparative methods such as Adaptive Comparative Judgements (ACJ), as demonstrated by Seery et al. (2016). This approach offers a robust framework for evaluating diverse design outputs, providing a more consistent measure of skill than traditional criterion-referenced methods alone, while also highlighting the potential for holistic judgments to be influenced by unstated variables.
Source
Research@THEA
The validity and reliability of adaptive comparative judgements in the assessment of graphical capability
journal · 2016
View sourceQuestions About This Research
- What does the research say about adaptive comparative judgements (acj) offer reliable assessment of graphical design capability?
- When assessing design work, especially graphical outputs with varied solutions, consider using comparative methods like ACJ for more reliable evaluations. Evidence: Research@THEA (2016).
- Why does "Adaptive Comparative Judgements (ACJ) offer reliable assessment of graphical design capability." matter for design?
- Accurate assessment of design outputs is crucial for feedback and development. ACJ offers a potentially more efficient and reliable alternative to traditional methods, especially for subjective or complex design tasks.
- How can designers apply this research?
- When assessing design work, especially graphical outputs with varied solutions, consider using comparative methods like ACJ for more reliable evaluations.
- What were the main findings?
- ACJ demonstrated a high level of reliability in assessing graphical capability.. Correlations between ACJ scores and criterion-referenced scores suggest validity, but also indicate potential influence from other variables on holistic judgments.
- What research method was used?
- Comparative assessment study with 128 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2016 journal from Research@THEA.
- What should I do differently in my next project?
- When evaluating design portfolios or project outcomes, present evaluators with pairs of work and ask them to choose the 'better' option, rather than assigning absolute scores. This can be done iteratively to build a comparative ranking.
- What are the limitations?
- The study identified potential for additional variables to influence holistic judgments, suggesting that ACJ might not capture all nuances of design quality without further refinement or contextualization.