Short answer
Designers must actively facilitate dialogue and co-creation between AI developers and clinical end-users to ensure that explainability features are meaningful and actionable in a healthcare context.
- Field
- User-Centred Design
- Source
- npj Digital Medicine (2023)
- Method
- Longitudinal multi-method study
- Sample
- 112 participants
- Evidence
- Strong effect
Understanding and reconciling the differing mental models of developers and clinicians regarding explainable AI (XAI) is crucial for its effective implementation in healthcare decision support systems. This user-centred design research insight is drawn from a 2023 study published in npj Digital Medicine. Using Longitudinal multi-method study with 112 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Designers must actively facilitate dialogue and co-creation between AI developers and clinical end-users to ensure that explainability features are meaningful and actionable in a healthcare context.
Bridging Developer and Clinician Mental Models Enhances Explainable AI in Healthcare
Understanding and reconciling the differing mental models of developers and clinicians regarding explainable AI (XAI) is crucial for its effective implementation in healthcare decision support systems.
npj Digital Medicine · 2023
Key Findings
- 01Developers and clinicians hold three key differences in their mental models of XAI: opposing goals (model interpretability vs. clinical plausibility), different sources of truth (data vs. patient), and differing views on exploring new knowledge versus exploiting existing knowledge.
- 02Design solutions such as causal inference models, personalized explanations, and ambidexterity in exploration/exploitation mindsets can help bridge these gaps.
Application
Design takeaway
Designers must actively facilitate dialogue and co-creation between AI developers and clinical end-users to ensure that explainability features are meaningful and actionable in a healthcare context.
How to apply
When designing AI systems for specialized domains, conduct user research with both the technical creators and the intended end-users to map out their differing expectations and mental models of the system's functionality and explanations.
Project actions
- 01When researching AI applications, consider interviewing both the developers and potential users to uncover different perspectives.
- 02Think about how to present AI 'reasoning' in a way that makes sense to someone who isn't an AI expert but needs to trust and use the output.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Longitudinal study design allows for observation of evolving perspectives.
- +Multi-method approach provides a comprehensive understanding of complex issues.
- +Involvement of a large sample of both developers and clinicians.
Limitations
The specific mental models identified might be unique to healthcare AI. The study involved co-design, which might influence participant perspectives differently than a standard user study.
Reliability & validity
The study's reliability could be enhanced by using standardized protocols for data collection and analysis across all participants. Validity is strengthened by the longitudinal design and the triangulation of data from multiple methods, capturing a nuanced understanding of complex mental models.
Think critically
To what extent do the proposed design solutions (causal inference, personalized explanations, ambidexterity) truly resolve the fundamental differences in mental models, or do they merely offer a compromise?
Design Principles
"User-centered AI design requires a deep understanding and integration of the diverse mental models and goals of all stakeholders, especially when bridging technical and domain-specific expertise."
Designers of AI-powered tools, particularly in sensitive fields like healthcare, must recognize that end-users (clinicians) and creators (developers) may have fundamentally different interpretations of what constitutes a useful and trustworthy explanation. Failing to address these discrepancies can lead to systems that are not adopted or are misused, undermining the potential benefits of AI.
What This Means for Your Design
When building AI tools for doctors, it's important to understand that doctors and the people who build the AI might think about 'explaining' the AI's decisions very differently. This research shows how to design AI so both groups find it useful and trustworthy.
How to use in your project
- 1.Reference this study when discussing the importance of user research in AI design, particularly when addressing the need to align technical and end-user perspectives.
- 2.Use the findings to justify design choices aimed at improving the interpretability and usability of AI systems for specific professional groups.
Add to My Project
Quick Cite
Paragraph starter
This research underscores the critical need for user-centered design in AI development, particularly in specialized fields like healthcare. By identifying and addressing the divergent mental models between AI developers and end-users (clinicians) concerning explainability, design teams can create more effective and trustworthy AI solutions. The study's findings on differing goals (interpretability vs. plausibility), sources of truth (data vs. patient), and knowledge handling (exploration vs. exploitation) provide a framework for designing AI systems that bridge these gaps, leading to better adoption and utility.
Source
npj Digital Medicine
Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals
journal · 2023
View sourceQuestions About This Research
- What does the research say about bridging developer and clinician mental models enhances explainable ai in healthcare?
- Designers must actively facilitate dialogue and co-creation between AI developers and clinical end-users to ensure that explainability features are meaningful and actionable in a healthcare context. Evidence: npj Digital Medicine (2023).
- Why does "Bridging Developer and Clinician Mental Models Enhances Explainable AI in Healthcare" matter for design?
- Designers of AI-powered tools, particularly in sensitive fields like healthcare, must recognize that end-users (clinicians) and creators (developers) may have fundamentally different interpretations of what constitutes a useful and trustworthy explanation. Failing to address these discrepancies can lead to systems that are not adopted or are misused, undermining the potential benefits of AI.
- How can designers apply this research?
- Designers must actively facilitate dialogue and co-creation between AI developers and clinical end-users to ensure that explainability features are meaningful and actionable in a healthcare context.
- What were the main findings?
- Developers and clinicians hold three key differences in their mental models of XAI: opposing goals (model interpretability vs. clinical plausibility), different sources of truth (data vs. patient), and differing views on exploring new knowledge versus exploiting existing knowledge.. Design solutions such as causal inference models, personalized explanations, and ambidexterity in exploration/exploitation mindsets can help bridge these gaps.
- What research method was used?
- Longitudinal multi-method study with 112 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2023 journal from npj Digital Medicine.
- What should I do differently in my next project?
- When designing AI systems for specialized domains, conduct user research with both the technical creators and the intended end-users to map out their differing expectations and mental models of the system's functionality and explanations.
- What are the limitations?
- The study's findings might be specific to the particular clinical decision support system and the healthcare domain studied. Generalizability to other AI applications or industries may vary.