Short answer

When designing digital educational assessments, prioritize user experience and ensure the digital medium genuinely supports, rather than complicates, the demonstration of intended skills, especially for complex tasks.

Field: Modelling
Source: UPT. Syiah Kuala University Library (Syiah Kuala University) (2010)
Method: Comparative analysis and design research exercise
Evidence: Moderate effect

Translating traditional mathematics assessments to digital formats can inadvertently alter task difficulty and what skills are actually being measured, potentially hindering the assessment of complex problem-solving abilities. This modelling research insight is drawn from a 2010 study published in UPT. Syiah Kuala University Library (Syiah Kuala University). Using Comparative analysis and design research exercise, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When designing digital educational assessments, prioritize user experience and ensure the digital medium genuinely supports, rather than complicates, the demonstration of intended skills, especially for complex tasks.

Study

ModellingHigh ImpactModerate effect

Digital Math Assessments May Hinder, Not Help, Problem-Solving Skills

UPT. Syiah Kuala University Library (Syiah Kuala University) · 2010

Key Findings

01Digitizing mathematics assessments can change task difficulty and the skills being assessed.
02Assessing 'process skills' and 'problem-solving' in mathematics via computer presents unique design challenges.
03Some students, particularly at the GCSE level, expressed skepticism towards computer-based examinations.
04The computer may not be a 'natural medium' for mathematics, potentially adding an extra cognitive load.

Application

Design takeaway

How to apply

Before developing a digital assessment, conduct user testing with the target audience to identify potential usability issues and ensure the interface does not interfere with the assessment of core mathematical skills.

Project actions

01When designing a digital tool for a specific subject, consider how the interface might affect the user's ability to perform the core task.
02If comparing digital and physical prototypes, ensure the core functionality is equivalent, but acknowledge that the medium itself is a variable.

Method & Evidence

AimTo investigate the challenges and implications of designing computer-based assessments for mathematics, particularly concerning the measurement of process skills and problem-solving, and to compare these with traditional paper-based methods.

MethodComparative analysis and design research exercise

ProcedureThe study involved analyzing existing mathematics assessment tasks, comparing paper and computer versions of a commercial test, and undertaking a design research exercise to create, implement, and trial a computer-based assessment system for similar tasks. This included analyzing equating studies and gathering student feedback.

ContextMathematics education (primary and secondary)

Variables

IVFormat of assessment (computer-based vs. paper-based)

DVTask difficulty, skills assessed (process, problem-solving), student performance, student perception/attitude

CVMathematical content of the tasks, student age/educational level (within specific comparisons)

Strengths & Limitations

Strengths

+Direct comparison of paper and computer versions of a commercial test.
+Inclusion of a design research exercise to build and trial a new system.

Limitations

The study focused on mathematics; the impact of digitizing assessments in other subjects might differ. Also, student attitudes towards technology can vary widely.

Reliability & validity

The study's validity relies on the careful comparison of equivalent tasks and the analysis of equating studies. Reliability would be enhanced by larger, more diverse student samples and repeated testing.

Think critically

To what extent does the 'naturalness' of a medium for a particular task influence the validity of assessments conducted within that medium?

Design Principles

"The digital medium's affordances and constraints must be intentionally designed to align with the learning objectives and assessment goals, rather than simply replicating existing analog formats."

Designers creating digital educational tools must be aware that the medium itself can influence user performance and the validity of the assessment. Simply digitizing existing paper-based tests without careful consideration of interface design and interaction can lead to inaccurate evaluations of student capabilities.

What This Means for Your Design

Making math tests on computers can be tricky. Sometimes, the computer itself makes the test harder or measures the wrong things, not just the math skills. It's important to design computer tests carefully so they actually test what they're supposed to.

How to use in your project

1.Use this research to justify the importance of user testing and iterative design when developing digital assessment tools, emphasizing how the medium can influence outcomes.

Add to My Project

Quick Cite

Paragraph starter

The transition of traditional assessment formats to digital platforms requires careful design consideration, as evidenced by research suggesting that computer-based mathematics assessments can inadvertently alter task difficulty and the skills being measured. This highlights the need for a user-centered design approach that accounts for the unique affordances and potential cognitive load introduced by the digital medium, ensuring that the technology serves to enhance, rather than impede, the accurate assessment of student capabilities.

Source

UPT. Syiah Kuala University Library (Syiah Kuala University)

On computer-based assessment of mathematics

journal · 2010

View source

Questions About This Research

What does the research say about digital math assessments may hinder, not help, problem-solving skills?: When designing digital educational assessments, prioritize user experience and ensure the digital medium genuinely supports, rather than complicates, the demonstration of intended skills, especially for complex tasks. Evidence: UPT. Syiah Kuala University Library (Syiah Kuala University) (2010).
Why does "Digital Math Assessments May Hinder, Not Help, Problem-Solving Skills" matter for design?: Designers creating digital educational tools must be aware that the medium itself can influence user performance and the validity of the assessment. Simply digitizing existing paper-based tests without careful consideration of interface design and interaction can lead to inaccurate evaluations of student capabilities.
How can designers apply this research?: When designing digital educational assessments, prioritize user experience and ensure the digital medium genuinely supports, rather than complicates, the demonstration of intended skills, especially for complex tasks.
What were the main findings?: Digitizing mathematics assessments can change task difficulty and the skills being assessed.. Assessing 'process skills' and 'problem-solving' in mathematics via computer presents unique design challenges.. Some students, particularly at the GCSE level, expressed skepticism towards computer-based examinations.. The computer may not be a 'natural medium' for mathematics, potentially adding an extra cognitive load.
What research method was used?: Comparative analysis and design research exercise.
How strong is the evidence?: Evidence strength is rated Moderate effect, based on a 2010 journal from UPT. Syiah Kuala University Library (Syiah Kuala University).
What should I do differently in my next project?: Before developing a digital assessment, conduct user testing with the target audience to identify potential usability issues and ensure the interface does not interfere with the assessment of core mathematical skills.
What are the limitations?: The study's findings regarding student skepticism might be context-specific and could evolve with increased digital literacy and familiarity with computer-based testing.