Short answer

When designing tools to measure user attitudes or behaviors, actively involve a diverse group of stakeholders in the conceptualization and validation phases to ensure the tool's relevance and accuracy.

Field: User-Centred Design
Source: OhioLink ETD Center (Ohio Library and Information Network) (2018)
Method: Mixed Methods (Concept Mapping, Psychometric Validation)
Sample: 190 STEM stakeholders
Evidence: Strong effect

Incorporating diverse stakeholder perspectives through concept mapping leads to more robust and representative instruments for measuring attitudes and persistence in STEM fields. This user-centred design research insight is drawn from a 2018 study published in OhioLink ETD Center (Ohio Library and Information Network). Using Mixed methods (concept mapping, psychometric validation) with 190 STEM stakeholders, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When designing tools to measure user attitudes or behaviors, actively involve a diverse group of stakeholders in the conceptualization and validation phases to ensure the tool's relevance and accuracy.

Study

User-Centred DesignHigh ImpactStrong effect

Stakeholder input significantly enhances the validity of STEM attitude and persistence instruments.

Incorporating diverse stakeholder perspectives through concept mapping leads to more robust and representative instruments for measuring attitudes and persistence in STEM fields.

OhioLink ETD Center (Ohio Library and Information Network) · 2018

Key Findings

01Concept mapping effectively organized stakeholder conceptualizations of attitude and persistence towards STEM.
02A participatory framework involving diverse stakeholders enhanced the development of a valid instrument (APT-STEM).

Application

Design takeaway

How to apply

When developing surveys, questionnaires, or assessment tools related to user attitudes, behaviors, or experiences, initiate a concept mapping exercise with representatives from all relevant user groups and expert communities.

Project actions

01When developing a research instrument, clearly define the stakeholder groups relevant to your design project.
02Consider using visual tools like concept mapping to synthesize diverse stakeholder input.

Method & Evidence

AimHow can a participatory concept mapping approach be used to develop and validate an instrument measuring attitudes and persistence towards STEM fields?

MethodMixed Methods (Concept Mapping, Psychometric Validation)

ProcedureA concept mapping methodology was employed with a broad range of STEM stakeholders to collaboratively develop an item pool for measuring attitudes and persistence. This pool was then rigorously analyzed using qualitative and quantitative psychometric techniques, including Classical Test Theory (CTT), Item Response Theory (IRT), and Rasch Measurement Theory (RMT), to validate the resulting instrument.

Sample190 STEM stakeholders

ContextSTEM education and workforce development

Variables

IVStakeholder participation and concept mapping methodology

DVValidity and representativeness of the APT-STEM instrument

CVPsychometric validation frameworks (CTT, IRT, RMT)

Strengths & Limitations

Strengths

+Utilized a mixed-methods approach for comprehensive data collection and analysis.
+Involved a broad and diverse range of STEM stakeholders.

Limitations

The time and resources required for extensive stakeholder engagement can be a constraint for smaller design projects.

Reliability & validity

The study employed multiple psychometric frameworks (CTT, IRT, RMT) to establish the reliability and validity of the developed instrument, indicating a strong methodological rigor.

Think critically

To what extent can the findings regarding STEM attitudes and persistence be generalized to other fields or non-cognitive skills?

Design Principles

"Participatory instrument development, informed by diverse stakeholder conceptualizations, yields more valid and reliable measures."

Designing effective interventions or educational programs in STEM requires accurate measurement of student attitudes and persistence. By involving a wide range of stakeholders, designers can ensure that the tools used for assessment truly reflect the multifaceted nature of these constructs, leading to more targeted and impactful design solutions.

What This Means for Your Design

To make sure a survey or test about liking and sticking with science and math subjects is good, it's best to ask lots of different people involved (like students, teachers, and people working in science jobs) what they think and use their ideas to build and check the test.

How to use in your project

1.Reference this study when explaining the importance of stakeholder consultation in the development of research instruments for your design project.

Add to My Project

Quick Cite

Paragraph starter

The development of robust measurement instruments, such as those used to assess user attitudes or behaviors, benefits significantly from a participatory approach. As demonstrated by Sunny (2018), employing methods like concept mapping with diverse stakeholder groups can lead to a more comprehensive conceptual domain and enhanced instrument validity, ensuring that the tool accurately reflects the intended constructs.

Source

OhioLink ETD Center (Ohio Library and Information Network)

Stakeholders' Conceptualization of Students' Attitudes and Persistence towards STEM: A Mixed Methods Instrument Development and Validation Study.

journal · 2018

View source

Questions About This Research

What does the research say about stakeholder input significantly enhances the validity of stem attitude and persistence instruments?: When designing tools to measure user attitudes or behaviors, actively involve a diverse group of stakeholders in the conceptualization and validation phases to ensure the tool's relevance and accuracy. Evidence: OhioLink ETD Center (Ohio Library and Information Network) (2018).
Why does "Stakeholder input significantly enhances the validity of STEM attitude and persistence instruments." matter for design?: Designing effective interventions or educational programs in STEM requires accurate measurement of student attitudes and persistence. By involving a wide range of stakeholders, designers can ensure that the tools used for assessment truly reflect the multifaceted nature of these constructs, leading to more targeted and impactful design solutions.
How can designers apply this research?: When designing tools to measure user attitudes or behaviors, actively involve a diverse group of stakeholders in the conceptualization and validation phases to ensure the tool's relevance and accuracy.
What were the main findings?: Concept mapping effectively organized stakeholder conceptualizations of attitude and persistence towards STEM.. A participatory framework involving diverse stakeholders enhanced the development of a valid instrument (APT-STEM).
What research method was used?: Mixed Methods (Concept Mapping, Psychometric Validation) with 190 STEM stakeholders.
How strong is the evidence?: Evidence strength is rated Strong effect, based on a 2018 journal from OhioLink ETD Center (Ohio Library and Information Network).
What should I do differently in my next project?: When developing surveys, questionnaires, or assessment tools related to user attitudes, behaviors, or experiences, initiate a concept mapping exercise with representatives from all relevant user groups and expert communities.
What are the limitations?: The study focused on a specific set of STEM stakeholders; generalizability to other fields or stakeholder groups may vary. The concept mapping process itself can be resource-intensive.