Short answer
When designing and evaluating MR/AR experiences, prioritize testing in environments that closely mimic the intended use case, including dynamic user movement and varied contextual conditions, rather than solely relying on static lab setups.
- Field
- Modelling
- Source
- arXiv (Cornell University) (2020)
- Method
- Systematic Literature Review
- Sample
- 458 papers reviewed, 248 user studies involving 5,761 participants
- Evidence
- Strong effect
A significant majority of mixed and augmented reality evaluations are conducted in controlled laboratory settings, potentially limiting the applicability of findings to real-world scenarios. This modelling research insight is drawn from a 2020 study published in arXiv (Cornell University). Using Systematic literature review with 458 papers reviewed, 248 user studies involving 5,761 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When designing and evaluating MR/AR experiences, prioritize testing in environments that closely mimic the intended use case, including dynamic user movement and varied contextual conditions, rather than solely relying on static lab setups.
Lab-based MR/AR evaluations often neglect real-world context and user mobility
A significant majority of mixed and augmented reality evaluations are conducted in controlled laboratory settings, potentially limiting the applicability of findings to real-world scenarios.
arXiv (Cornell University) · 2020
Key Findings
- 01A strong coupling exists between the types of MR/AR studies, their research topics, and evaluation scenarios.
- 02Most user studies (216/248) are conducted in laboratory settings, often with participants in static positions.
- 03A smaller but notable portion of studies (30/248) are conducted 'in-the-wild' with mobile participants.
Application
Design takeaway
When designing and evaluating MR/AR experiences, prioritize testing in environments that closely mimic the intended use case, including dynamic user movement and varied contextual conditions, rather than solely relying on static lab setups.
How to apply
When planning a design project involving MR/AR, allocate resources for user testing in realistic, dynamic environments, not just controlled laboratory conditions. Document the environmental factors and user interactions observed during these tests.
Project actions
- 01If your design project involves MR/AR, think about where and how users will actually use it.
- 02Try to test your prototypes in a setting that's as close as possible to the real-world environment.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Comprehensive review of a large number of papers over a decade.
- +Systematic categorization of evaluation methodologies and contexts.
Limitations
The findings are based on studies published before 2020, so newer MR/AR technologies and evaluation methods might not be included. Lab studies are easier to control and replicate, which is why they are common.
Reliability & validity
The systematic nature of the literature review enhances its reliability and validity by ensuring a consistent methodology for paper selection and categorization. However, the validity of generalizing findings to current MR/AR practices is limited by the review's cutoff date.
Think critically
To what extent do the findings from lab-based MR/AR evaluations generalize to 'in-the-wild' scenarios, and what specific design features or user behaviors might be missed by this approach?
Design Principles
"Contextual fidelity in evaluation is paramount for the successful adoption of MR/AR technologies."
Designers and researchers developing MR/AR systems must consider the environmental and contextual factors that influence user experience and performance. Over-reliance on lab studies can lead to designs that perform poorly or are less intuitive when deployed in dynamic, unpredictable environments.
What This Means for Your Design
Most studies testing new AR/VR tech happen in labs, which isn't always like real life. This means some tech might not work as well when people actually use it outside the lab.
How to use in your project
- 1.Reference this review when discussing the limitations of lab-based testing for your MR/AR design project and justify your choice of evaluation context.
Add to My Project
Quick Cite
Paragraph starter
This research highlights a common tendency in MR/AR evaluations to favor controlled laboratory settings, which may not fully capture the complexities of real-world user interaction. For instance, a study by Merino et al. (2020) found that the vast majority of MR/AR user studies are conducted in labs, often with participants in static positions. This contrasts with the dynamic and mobile nature of many intended MR/AR applications, suggesting a potential gap between research findings and practical usability in diverse environments.
Source
arXiv (Cornell University)
Evaluating Mixed and Augmented Reality: A Systematic Literature Review\n (2009-2019)
journal · 2020
View sourceQuestions About This Research
- What does the research say about lab-based mr/ar evaluations often neglect real-world context and user mobility?
- When designing and evaluating MR/AR experiences, prioritize testing in environments that closely mimic the intended use case, including dynamic user movement and varied contextual conditions, rather than solely relying on static lab setups. Evidence: arXiv (Cornell University) (2020).
- Why does "Lab-based MR/AR evaluations often neglect real-world context and user mobility" matter for design?
- Designers and researchers developing MR/AR systems must consider the environmental and contextual factors that influence user experience and performance. Over-reliance on lab studies can lead to designs that perform poorly or are less intuitive when deployed in dynamic, unpredictable environments.
- How can designers apply this research?
- When designing and evaluating MR/AR experiences, prioritize testing in environments that closely mimic the intended use case, including dynamic user movement and varied contextual conditions, rather than solely relying on static lab setups.
- What were the main findings?
- A strong coupling exists between the types of MR/AR studies, their research topics, and evaluation scenarios.. Most user studies (216/248) are conducted in laboratory settings, often with participants in static positions.. A smaller but notable portion of studies (30/248) are conducted 'in-the-wild' with mobile participants.
- What research method was used?
- Systematic Literature Review with 458 papers reviewed, 248 user studies involving 5,761 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2020 journal from arXiv (Cornell University).
- What should I do differently in my next project?
- When planning a design project involving MR/AR, allocate resources for user testing in realistic, dynamic environments, not just controlled laboratory conditions. Document the environmental factors and user interactions observed during these tests.
- What are the limitations?
- The review covers publications up to 2019, and the field of MR/AR has evolved significantly since then. The analysis of participant demographics was limited, with a notable underrepresentation of female participants in the reviewed studies.