Short answer
Trust the consensus found in open card sorting data; it is a stable reflection of user mental models and likely to be replicated across your entire user base.
- Field
- User-Centred Design
- Source
- Academic Publication (2019)
- Method
- Comparative Empirical Reliability Study
- Sample
- 140 participants
- Evidence
- Strong effect
The inherent mental models users hold for specific domains are stable enough that different groups of people independently converge on the same organizational patterns when using unconstrained sorting tasks. This user-centred design research insight is drawn from a 2019 study published in Academic Publication. Using Comparative empirical reliability study with 140 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Trust the consensus found in open card sorting data; it is a stable reflection of user mental models and likely to be replicated across your entire user base.
Replicating open card sort studies produces consistent information architecture structures across independent participant groups
The inherent mental models users hold for specific domains are stable enough that different groups of people independently converge on the same organizational patterns when using unconstrained sorting tasks.
Academic Publication · 2019
Key Findings
- 01High correlation in card-pair similarity scores across independent study groups
- 02Strong agreement in the final navigation schemes generated by different groups
- 03Consistent mental models emerged regardless of which specific participant group performed the task
Application
Design takeaway
Trust the consensus found in open card sorting data; it is a stable reflection of user mental models and likely to be replicated across your entire user base.
How to apply
When restructuring a site, use open card sorting with 15-20 participants. You do not need to repeat the study with a second group of 20 to 'verify' the results, as the consensus architecture is statistically likely to remain the same.
Project actions
- 01Use open card sorting early in your project to define your navigation menu.
- 02Don't worry if your participants don't know each other; their shared cultural/digital experiences make their logic similar.
- 03Focus more on the 'grouping' of cards rather than the exact names people give the categories.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Employs a robust sample size (140 participants) divided into multiple independent groups, enhancing the statistical power to detect consistency.
- +The comparative empirical design directly addresses the research question of cross-study reliability through replication.
- +Focuses on a practical design context (Information Architecture), making the findings highly relevant to real-world design challenges.
Limitations
If your project is about a very specialized or niche topic, your results might be less reliable than the general e-commerce results found in this paper.
Reliability & validity
The study demonstrates high reliability by replicating the card sorting process with independent groups and observing consistent outcomes, suggesting that the method reliably captures user-generated information architecture. Validity is high within the tested domains (travel, e-commerce) as it measures what it intends to – the consistency of IA structures derived from user sorting. However, external validity is limited to domains with established user schemas, as noted in the limitations.
Think critically
If card sorting is so reliable for existing website types, does this mean we are just repeating old designs instead of innovating? How can we use card sorting for a product that has no existing category?
Design Principles
"Mental Model Stability"
Designers often fear that small sample sizes or the subjective nature of card sorting might lead to 'random' or 'fluke' navigation structures. This research confirms that user-generated taxonomies are robust and repeatable, ensuring that IA decisions based on card sorts represent the broader user population's logic rather than a localized coincidence.
What This Means for Your Design
Even when you ask completely different groups of people to organize website links in their own way, they almost always end up grouping the same things together.
How to use in your project
- 1.Cite this to justify why your proposed sitemap is valid based on user data.
- 2.Use it to defend your navigation choices against 'subjective' criticism from peers or teachers.
Add to My Project
Quick Cite
Paragraph starter
According to Katsanos et al. (2019), open card sorting demonstrates high cross-study reliability, meaning the mental models captured in my user testing are likely consistent with the broader target audience.
Source
Academic Publication
Cross-study Reliability of the Open Card Sorting Method
journal · 2019
View sourceQuestions About This Research
- What does the research say about replicating open card sort studies produces consistent information architecture structures across independent participant groups?
- Trust the consensus found in open card sorting data; it is a stable reflection of user mental models and likely to be replicated across your entire user base. Evidence: Academic Publication (2019).
- Why does "Replicating open card sort studies produces consistent information architecture structures across independent participant groups" matter for design?
- Designers often fear that small sample sizes or the subjective nature of card sorting might lead to 'random' or 'fluke' navigation structures. This research confirms that user-generated taxonomies are robust and repeatable, ensuring that IA decisions based on card sorts represent the broader user population's logic rather than a localized coincidence.
- How can designers apply this research?
- Trust the consensus found in open card sorting data; it is a stable reflection of user mental models and likely to be replicated across your entire user base.
- What were the main findings?
- High correlation in card-pair similarity scores across independent study groups. Strong agreement in the final navigation schemes generated by different groups. Consistent mental models emerged regardless of which specific participant group performed the task
- What research method was used?
- Comparative Empirical Reliability Study with 140 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2019 journal from Academic Publication.
- What should I do differently in my next project?
- When restructuring a site, use open card sorting with 15-20 participants. You do not need to repeat the study with a second group of 20 to 'verify' the results, as the consensus architecture is statistically likely to remain the same.
- What are the limitations?
- The study focused on established domains (travel and e-commerce) where users may already have existing schema expectations; results may vary for highly novel or abstract product categories.