Study
Human FactorsHigh ImpactModerate effect

Symptom reporting tools for concussion require refinement for clarity and consistency

Standardized concussion symptom checklists, while useful, contain ambiguities and redundancies that can affect the accuracy of assessment.

Scholarship@Western (Western University) · 2019

01

Key Findings

  • 01Both adult and child SCAT5 versions demonstrated reliability but showed poor fit to the Rasch model due to multidimensionality, item redundancy, and response dependency.
  • 02A qualitative survey revealed that six items in the adult SCAT5 were interpreted inconsistently with their original wording.
  • 03MCID estimates for the adult SCAT5 were significant, but only a subset of items in the child SCAT5 yielded significant MCID estimates.
02

Application

Design takeaway

Refine the wording and structure of symptom checklists to ensure consistent interpretation and accurate data capture, particularly for vulnerable populations.

How to apply

When designing any form of user-reported data collection, conduct thorough user testing to identify and resolve ambiguous terminology and ensure questions are interpreted as intended.

Project actions

  • 01When designing surveys or questionnaires, test your questions with potential users to ensure they understand them clearly.
  • 02Consider using a mix of qualitative and quantitative methods to get a full picture of how users interact with your design.
03

Method & Evidence

AimTo evaluate the psychometric properties, interpretability, responsiveness, and minimal clinically important difference (MCID) of the adult and pediatric versions of the Sport Concussion Assessment Tool 5 (SCAT5).
MethodMixed-methods research, including Rasch analysis, qualitative survey, and clinical-anchor-based estimation.
ProcedureThe study employed Rasch analysis to assess the reliability and construct validity of the SCAT5 symptom checklists. A qualitative survey explored user interpretation of symptom items. Clinical-anchor methods were used to estimate responsiveness and MCID for individual items and total scores.
ContextSports medicine and concussion management.

Variables

IVWording and structure of symptom checklist items.
DVUser interpretation of symptoms, reliability, construct validity, responsiveness, MCID.
CVParticipant demographics (adult vs. child), specific concussion symptoms being assessed.
04

Strengths & Limitations

Strengths

  • +Utilized multiple research methods (Rasch analysis, qualitative survey, clinical-anchor) for a comprehensive evaluation.
  • +Investigated both adult and pediatric versions of a widely used tool.

Limitations

The findings might be specific to the context of concussion assessment and may not directly apply to all types of symptom reporting.

Reliability & validity

The study assessed reliability through Rasch analysis and explored construct validity by examining item fit and dimensionality. Responsiveness and MCID were investigated to assess measurement validity.

Think critically

How might the identified ambiguities in the SCAT5 affect the diagnosis and management of concussions in real-world scenarios?

05

Design Principles

"Clarity and precision in user interface language are paramount for accurate data collection in diagnostic and assessment tools."

Designers of diagnostic tools and health monitoring systems must ensure that the language used is unambiguous and that the structure of the tool supports clear interpretation by users. This is crucial for accurate data collection and effective decision-making in critical health contexts.

06

What This Means for Your Design

The tools used to ask people about concussion symptoms aren't perfect. Some questions are confusing or worded in a way that people understand differently, which can make the results less reliable.

How to use in your project

  • 1.Reference this study when discussing the importance of clear user instructions and unambiguous terminology in your own design project's user research or testing phases.
07

Add to My Project

08

Quick Cite

(2019). Evaluation of the Psychometric and Measurement Properties of the SCAT5 and Child SCAT5. Scholarship@Western (Western University). Retrieved from https://designdex.org/study/b8f025b7-3003-4f0f-af17-f0d26fc2a140/symptom-reporting-tools-for-concussion-require-refinement-for-clarity-and-consistency

Paragraph starter

The evaluation of the SCAT5 highlights the critical need for precise language and unambiguous item construction in user-facing assessment tools. Issues identified, such as multidimensionality and inconsistent interpretation of symptom descriptions, underscore the potential for design flaws to compromise data integrity, a crucial consideration for any design project involving user reporting.

09

Source

Scholarship@Western (Western University)

Evaluation of the Psychometric and Measurement Properties of the SCAT5 and Child SCAT5

journal · 2019

View source

Questions about this research

What does the research say about symptom reporting tools for concussion require refinement for clarity and consistency?
Refine the wording and structure of symptom checklists to ensure consistent interpretation and accurate data capture, particularly for vulnerable populations. Evidence: Scholarship@Western (Western University) (2019).
Why does "Symptom reporting tools for concussion require refinement for clarity and consistency" matter for design?
Designers of diagnostic tools and health monitoring systems must ensure that the language used is unambiguous and that the structure of the tool supports clear interpretation by users. This is crucial for accurate data collection and effective decision-making in critical health contexts.
How can designers apply this research?
Refine the wording and structure of symptom checklists to ensure consistent interpretation and accurate data capture, particularly for vulnerable populations.
What were the main findings?
Both adult and child SCAT5 versions demonstrated reliability but showed poor fit to the Rasch model due to multidimensionality, item redundancy, and response dependency.. A qualitative survey revealed that six items in the adult SCAT5 were interpreted inconsistently with their original wording.. MCID estimates for the adult SCAT5 were significant, but only a subset of items in the child SCAT5 yielded significant MCID estimates.
What research method was used?
Mixed-methods research, including Rasch analysis, qualitative survey, and clinical-anchor-based estimation..
How strong is the evidence?
Evidence strength is rated Moderate effect, based on a 2019 journal from Scholarship@Western (Western University).
What should I do differently in my next project?
When designing any form of user-reported data collection, conduct thorough user testing to identify and resolve ambiguous terminology and ensure questions are interpreted as intended.
What are the limitations?
The study identified issues with multidimensionality and response dependency, suggesting the SCAT5 may not be a unidimensional measure as intended. Interpretations of 'minimal clinically important difference' can vary.
Is there evidence that symptom checklists affects design outcomes?
The SCAT5 symptom checklists are reliable but have issues with how symptoms are described and understood, potentially leading to inconsistent assessments, especially in younger individuals. Designers of diagnostic tools and health monitoring systems must ensure that the language used is unambiguous and that the structu Source: Scholarship@Western (Western University) (2019).
Where does this accurate data research apply?
Sports medicine and concussion management. It sits within human factors research on designdex.org.

Related research topics

symptom checklists design research · evidence on symptom checklists · does symptom checklists improve design outcomes · accurate data studies for designers · symptom checklists and accurate data findings · human factors research evidence