Short answer
Implement rigorous checks for response quality in any user research that utilizes self-report questionnaires, especially when assessing subjective or socially valued attributes like creativity.
- Field
- User-Centred Design
- Source
- The Journal of Creative Behavior (2015)
- Method
- Quantitative survey research with statistical analysis.
- Sample
- 204 participants
- Evidence
- Strong effect
Careless or inattentive participants inflate scores on self-report measures of creativity and arts engagement, distorting research findings. This user-centred design research insight is drawn from a 2015 study published in The Journal of Creative Behavior. Using Quantitative survey research with statistical analysis. with 204 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Implement rigorous checks for response quality in any user research that utilizes self-report questionnaires, especially when assessing subjective or socially valued attributes like creativity.
Inattentive Responding Skews Creativity and Arts Self-Report Data
Careless or inattentive participants inflate scores on self-report measures of creativity and arts engagement, distorting research findings.
The Journal of Creative Behavior · 2015
Key Findings
- 01High rates of inattentive responding were observed, with approximately 25% of the sample identified as potentially inattentive.
- 02Inattentive respondents scored higher on creativity and arts scales.
- 03Excluding inattentive participants strengthened the effect sizes of interest in the study.
- 04Social desirability showed no significant relationship with creativity and arts scales.
Application
Design takeaway
Implement rigorous checks for response quality in any user research that utilizes self-report questionnaires, especially when assessing subjective or socially valued attributes like creativity.
How to apply
When designing user surveys for creative products or services, include attention checks (e.g., 'Please select 'Strongly Agree' for this item') and monitor response patterns for inconsistencies.
Project actions
- 01When designing a survey for your project, include simple questions that check if the respondent is paying attention.
- 02Look for patterns in responses that seem too quick or too random.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Inclusion of multiple metrics for inattentiveness.
- +Use of latent class analysis to identify distinct respondent groups.
Limitations
It's hard to be 100% sure if someone is truly inattentive or just has a different way of thinking. Some attention checks might be too obvious and annoy participants.
Reliability & validity
The study's reliability is supported by the use of multiple inattentiveness metrics. Validity is addressed by demonstrating that inattentive responding leads to distorted results (higher scores), suggesting the scales are sensitive to response quality.
Think critically
How might the design of the survey itself (e.g., length, complexity, perceived importance) influence the likelihood of inattentive responding?
Design Principles
"Ensure data integrity by validating user input through embedded checks and quality control measures."
In design research, particularly when using surveys or questionnaires to understand user perceptions, behaviors, or preferences related to creative outputs or artistic engagement, inattentive responses can lead to inaccurate conclusions. This can impact design decisions, product development, and user experience strategies if based on flawed data.
What This Means for Your Design
When people fill out surveys about creative things, some don't pay attention and just guess or answer randomly. These careless answers can make it look like people are more creative or know more about art than they actually do, messing up the results.
How to use in your project
- 1.Cite this research when discussing the limitations of self-report data in your design project and explaining the methods you used to ensure data quality.
Add to My Project
Quick Cite
Paragraph starter
In user research for design projects, self-report measures can be susceptible to inattentive responding, as highlighted by McKibben and Silvia (2015). Their study found that a significant portion of participants provided careless responses, leading to inflated scores on creativity and arts scales. This underscores the importance of implementing robust data quality checks, such as attention-checking questions and response pattern analysis, to ensure the validity of user feedback and inform design decisions accurately.
Source
The Journal of Creative Behavior
Evaluating the Distorting Effects of Inattentive Responding and Social Desirability on Self‐Report Scales in Creativity and the Arts
journal · 2015
View sourceQuestions About This Research
- What does the research say about inattentive responding skews creativity and arts self-report data?
- Implement rigorous checks for response quality in any user research that utilizes self-report questionnaires, especially when assessing subjective or socially valued attributes like creativity. Evidence: The Journal of Creative Behavior (2015).
- Why does "Inattentive Responding Skews Creativity and Arts Self-Report Data" matter for design?
- In design research, particularly when using surveys or questionnaires to understand user perceptions, behaviors, or preferences related to creative outputs or artistic engagement, inattentive responses can lead to inaccurate conclusions. This can impact design decisions, product development, and user experience strategies if based on flawed data.
- How can designers apply this research?
- Implement rigorous checks for response quality in any user research that utilizes self-report questionnaires, especially when assessing subjective or socially valued attributes like creativity.
- What were the main findings?
- High rates of inattentive responding were observed, with approximately 25% of the sample identified as potentially inattentive.. Inattentive respondents scored higher on creativity and arts scales.. Excluding inattentive participants strengthened the effect sizes of interest in the study.. Social desirability showed no significant relationship with creativity and arts scales.
- What research method was used?
- Quantitative survey research with statistical analysis. with 204 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2015 journal from The Journal of Creative Behavior.
- What should I do differently in my next project?
- When designing user surveys for creative products or services, include attention checks (e.g., 'Please select 'Strongly Agree' for this item') and monitor response patterns for inconsistencies.
- What are the limitations?
- The study focused on undergraduate students, and findings may not generalize to other populations. The specific self-report scales used may have unique characteristics.