Short answer
Design AI collaboration tools not as static entities, but as adaptive partners whose perceived expertise and utility grow with the user over time, requiring careful management of initial impressions and ongoing feedback.
- Field
- User-Centred Design
- Source
- Academic Publication (2026)
- Method
- Within-subjects study
- Sample
- 12 participants
- Evidence
- Strong effect
UX evaluators' trust and reliance on AI assistants for usability analysis develop over multiple interactions, shifting from initial novelty to a more calibrated understanding of the AI's perceived expertise and efficiency. This user-centred design research insight is drawn from a 2026 study published in Academic Publication. Using Within-subjects study with 12 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Design AI collaboration tools not as static entities, but as adaptive partners whose perceived expertise and utility grow with the user over time, requiring careful management of initial impressions and ongoing feedback.
AI Expertise Perception Evolves: Building Trust in Collaborative UX Analysis
UX evaluators' trust and reliance on AI assistants for usability analysis develop over multiple interactions, shifting from initial novelty to a more calibrated understanding of the AI's perceived expertise and efficiency.
Academic Publication · 2026
Key Findings
- 01Participants initially experienced a novelty effect with the AI assistants.
- 02Trust in the AI dipped after the initial novelty but recovered over time.
- 03Evaluators shifted their analytical approach from a two-pass to a one-pass video inspection as they gained experience.
- 04The AI assistant perceived as more expert-like was rated significantly more efficient, trustworthy, and comprehensive by the end of the study.
Application
Design takeaway
Design AI collaboration tools not as static entities, but as adaptive partners whose perceived expertise and utility grow with the user over time, requiring careful management of initial impressions and ongoing feedback.
How to apply
When designing AI tools for expert users, plan for an iterative development of trust and efficiency. Introduce AI features gradually and provide mechanisms for users to understand and calibrate their reliance on the AI's suggestions.
Project actions
- 01Consider how users might interact with your design over multiple sessions.
- 02Think about how to build user trust in a new technology or feature.
- 03Explore how different levels of perceived expertise in a tool might affect user behavior.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Within-subjects design controls for individual differences.
- +Multi-session study captures longitudinal effects.
- +Mixed methods approach (behavioral, subjective, interview) provides rich data.
Limitations
The artificiality of a lab setting and the specific nature of UX evaluation tasks may limit generalizability.
Reliability & validity
The within-subjects design enhances internal validity by controlling for participant variability. Reliability could be assessed by examining the consistency of subjective ratings and behavioral measures across similar tasks within sessions. External validity might be limited by the specific context of UX evaluation and the controlled nature of the AI assistants.
Think critically
To what extent does the 'novelty effect' with AI assistants mask genuine usability issues or benefits, and how can designers mitigate its influence on early user feedback?
Design Principles
"Calibrated Human-AI Collaboration: Design AI systems that foster a balanced understanding of their capabilities and limitations, allowing users to build trust and adapt their workflows effectively over time."
Understanding how users build trust and adapt their strategies when collaborating with AI is crucial for designing effective human-AI partnerships. This insight informs the development of AI tools that can genuinely augment, rather than hinder, expert workflows.
What This Means for Your Design
When people work with AI helpers for tasks like finding design problems, they don't trust them right away. But after using them for a while, they start to trust them more and learn how to work with them better, especially if the AI seems knowledgeable.
How to use in your project
- 1.Use this research to justify designing for iterative user learning and trust-building in your design process.
- 2.Cite this study when discussing the importance of user adaptation to new technologies.
Add to My Project
Quick Cite
Paragraph starter
This study highlights that user trust and strategic adaptation are not immediate but develop over repeated interactions with AI assistants. For instance, UX evaluators in a multi-session study initially exhibited novelty effects and dips in trust, but these perceptions evolved as they became more familiar with the AI's capabilities, ultimately leading to increased efficiency and trust in AI perceived as expert. This underscores the importance of designing for iterative user learning and trust-building in collaborative systems.
Source
Academic Publication
“It Became My Buddy, But I’m Not Afraid to Disagree”: A Multi-Session Study of UX Evaluators Collaborating with Conversational AI Assistants
journal · 2026
View sourceQuestions About This Research
- What does the research say about ai expertise perception evolves: building trust in collaborative ux analysis?
- Design AI collaboration tools not as static entities, but as adaptive partners whose perceived expertise and utility grow with the user over time, requiring careful management of initial impressions and ongoing feedback. Evidence: Academic Publication (2026).
- Why does "AI Expertise Perception Evolves: Building Trust in Collaborative UX Analysis" matter for design?
- Understanding how users build trust and adapt their strategies when collaborating with AI is crucial for designing effective human-AI partnerships. This insight informs the development of AI tools that can genuinely augment, rather than hinder, expert workflows.
- How can designers apply this research?
- Design AI collaboration tools not as static entities, but as adaptive partners whose perceived expertise and utility grow with the user over time, requiring careful management of initial impressions and ongoing feedback.
- What were the main findings?
- Participants initially experienced a novelty effect with the AI assistants.. Trust in the AI dipped after the initial novelty but recovered over time.. Evaluators shifted their analytical approach from a two-pass to a one-pass video inspection as they gained experience.. The AI assistant perceived as more expert-like was rated significantly more efficient, trustworthy, and comprehensive by the end of the study.
- What research method was used?
- Within-subjects study with 12 participants.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2026 journal from Academic Publication.
- What should I do differently in my next project?
- When designing AI tools for expert users, plan for an iterative development of trust and efficiency. Introduce AI features gradually and provide mechanisms for users to understand and calibrate their reliance on the AI's suggestions.
- What are the limitations?
- The study involved a relatively small number of professional UX evaluators, and the specific context of usability analysis might not generalize to all AI-assisted collaborative tasks.