Short answer
When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference.
- Field
- User-Centred Design
- Source
- Aaltodoc (Aalto University) (2015)
- Method
- Experimental study with literature review
- Evidence
- Moderate effect
The presence of a moderator during usability testing can lead users to rate the system more favorably, even if it doesn't significantly alter the number of usability issues found. This user-centred design research insight is drawn from a 2015 study published in Aaltodoc (Aalto University). Using Experimental study with literature review, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference.
Moderator presence boosts user preference ratings in usability tests
The presence of a moderator during usability testing can lead users to rate the system more favorably, even if it doesn't significantly alter the number of usability issues found.
Aaltodoc (Aalto University) · 2015
Key Findings
- 01Relaxed 'think aloud' protocol did not significantly affect the number of usability problems or subjective ratings, but it did slow down user performance.
- 02The presence of a moderator led to significantly higher subjective preference ratings for the system compared to participants testing alone.
Application
Design takeaway
When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference.
How to apply
When planning usability tests, decide whether the goal is to capture raw, uninfluenced user opinion or to gather rich, contextual feedback that might be influenced by a moderator's presence.
Project actions
- 01When designing your usability test, clearly define the role of any observer or moderator.
- 02Consider running parallel tests with and without a moderator to compare subjective feedback.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Combines literature review with experimental data.
- +Addresses under-researched aspects of usability testing methodology.
Limitations
The specific product tested and the participants' prior experience could influence the results.
Reliability & validity
The study's validity relies on controlling extraneous variables and ensuring consistent application of the 'think aloud' and moderator conditions. Reliability would be assessed by the consistency of findings across similar studies.
Think critically
How might the cultural background of the moderator or the user influence the perceived effect of moderator presence?
Design Principles
"The social context of user research can influence observed outcomes, particularly subjective user perceptions."
Understanding how testing conditions influence user perception is crucial for accurately interpreting usability data. This insight suggests that the social dynamic of testing can impact subjective feedback, requiring designers to consider this when evaluating user sentiment.
What This Means for Your Design
When someone is watching you use a product, you might say nicer things about it, even if you still find the same number of problems.
How to use in your project
- 1.Reference this study when justifying your choice of usability testing methodology, especially regarding moderator presence or 'think aloud' protocols.
Add to My Project
Quick Cite
Paragraph starter
The presence of a moderator during usability testing has been shown to positively influence users' subjective ratings of a system (Riihiaho, 2015). While this can be beneficial for gathering immediate clarifying insights, designers must be aware that this social dynamic may inflate positive sentiment, potentially masking genuine usability concerns.
Source
Aaltodoc (Aalto University)
Experiences with usability testing: Effects of thinking aloud and moderator presence
journal · 2015
View sourceQuestions About This Research
- What does the research say about moderator presence boosts user preference ratings in usability tests?
- When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference. Evidence: Aaltodoc (Aalto University) (2015).
- Why does "Moderator presence boosts user preference ratings in usability tests" matter for design?
- Understanding how testing conditions influence user perception is crucial for accurately interpreting usability data. This insight suggests that the social dynamic of testing can impact subjective feedback, requiring designers to consider this when evaluating user sentiment.
- How can designers apply this research?
- When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference.
- What were the main findings?
- Relaxed 'think aloud' protocol did not significantly affect the number of usability problems or subjective ratings, but it did slow down user performance.. The presence of a moderator led to significantly higher subjective preference ratings for the system compared to participants testing alone.
- What research method was used?
- Experimental study with literature review.
- How strong is the evidence?
- Evidence strength is rated Moderate effect, based on a 2015 journal from Aaltodoc (Aalto University).
- What should I do differently in my next project?
- When planning usability tests, decide whether the goal is to capture raw, uninfluenced user opinion or to gather rich, contextual feedback that might be influenced by a moderator's presence.
- What are the limitations?
- The study did not find a significant effect of 'thinking aloud' on problem discovery, and the specific system tested may influence generalizability.