Short answer

Actively question and define the model of disability underpinning your AI design, and ensure diverse perspectives, particularly from disabled individuals, are integrated throughout the entire design lifecycle.

Field: User-Centred Design
Source: First Monday (2023)
Method: Conceptual analysis and framework development
Evidence: Strong effect

The fundamental definition of disability used in the early stages of AI design significantly influences subsequent data selection, application, and operational choices, leading to biased outcomes. This user-centred design research insight is drawn from a 2023 study published in First Monday. Using Conceptual analysis and framework development, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Actively question and define the model of disability underpinning your AI design, and ensure diverse perspectives, particularly from disabled individuals, are integrated throughout the entire design lifecycle.

Study

User-Centred DesignRecentStrong effect

Disability Models Shape AI Bias: Design Choices Amplify Inequities

The fundamental definition of disability used in the early stages of AI design significantly influences subsequent data selection, application, and operational choices, leading to biased outcomes.

First Monday · 2023

Key Findings

01Different definitions of disability lead to distinct design choices across problem formulation, data, use, and operational elements.
02Bias in AI can emerge from seemingly isolated design decisions, amplified by a lack of transparency and disabled participation.
03Historical models of disability are embedded in AI design, leading to varied and potentially harmful biases.

Application

Design takeaway

How to apply

Before commencing an AI design project related to disability, conduct a thorough review of existing disability models and their potential implications for bias. Establish a framework for continuous feedback and co-design with the target user group.

Project actions

01When researching a problem, consider the different ways 'disability' can be understood and how each understanding might influence your design choices.
02Think about who is being excluded by your design decisions and why.

Method & Evidence

AimHow do different models of disability influence the design decisions and inherent biases within AI technologies used for decision-making?

MethodConceptual analysis and framework development

ProcedureThe researchers analyzed how historical models of disability (e.g., medical, social) impact various AI design stages, including problem formulation, data selection, technology application, and operational design, to illustrate the mechanisms of bias.

ContextAI development and deployment in decision-making contexts (e.g., healthcare, employment, social benefits)

Variables

IVModels of disability (e.g., medical, social)

DVAI design decisions (problem formulation, data, use, operational elements) and resulting biases

CVTransparency in design, participation of disabled individuals

Strengths & Limitations

Strengths

+Provides a clear framework for understanding how abstract concepts influence concrete design outcomes.
+Highlights the critical need for participatory design in AI development.

Limitations

It can be challenging to definitively identify and measure the 'model of disability' influencing a design without explicit designer input. The impact of these models can be subtle and intertwined with other factors.

Reliability & validity

The validity of the findings relies on the logical coherence of the framework and the illustrative examples provided. Reliability is based on the consistent application of the conceptual models across different design stages.

Think critically

To what extent can AI designers truly be 'neutral' when their own understanding and definitions of complex social constructs like disability are inherently shaped by societal norms and historical perspectives?

Design Principles

"Bias mitigation in AI begins with critically examining and defining the core conceptual frameworks, such as models of disability, that inform design decisions."

Understanding how different conceptualizations of disability can embed bias into AI systems is crucial for designers and engineers. This insight highlights the need for critical examination of foundational assumptions to prevent the perpetuation of systemic inequities in AI-driven decision-making processes.

What This Means for Your Design

How you think about disability when you start designing an AI tool will affect the kind of data you use, how the tool works, and whether it's fair to people with disabilities. If you don't involve disabled people in the design, the tool might end up being biased.

How to use in your project

1.Use this research to justify your choice of user group and to explain why involving them in the design process is critical for avoiding bias.
2.Reference the paper when discussing the potential ethical implications of your design.

Add to My Project

Quick Cite

Paragraph starter

The design of AI technologies, particularly those impacting individuals with disabilities, is significantly influenced by the underlying conceptual models of disability employed during the initial design phases. As demonstrated by Newman-Griffis et al. (2023), choices made regarding problem formulation, data selection, and application are directly shaped by these foundational definitions, potentially embedding biases that can lead to inequitable outcomes. Therefore, a critical examination of these models and the active inclusion of disabled users throughout the design process are paramount to developing fair and effective AI solutions.

Source

First Monday

Definition drives design: Disability models and mechanisms of bias in AI technologies

journal · 2023

View source

Questions About This Research

What does the research say about disability models shape ai bias: design choices amplify inequities?: Actively question and define the model of disability underpinning your AI design, and ensure diverse perspectives, particularly from disabled individuals, are integrated throughout the entire design lifecycle. Evidence: First Monday (2023).
Why does "Disability Models Shape AI Bias: Design Choices Amplify Inequities" matter for design?: Understanding how different conceptualizations of disability can embed bias into AI systems is crucial for designers and engineers. This insight highlights the need for critical examination of foundational assumptions to prevent the perpetuation of systemic inequities in AI-driven decision-making processes.
How can designers apply this research?: Actively question and define the model of disability underpinning your AI design, and ensure diverse perspectives, particularly from disabled individuals, are integrated throughout the entire design lifecycle.
What were the main findings?: Different definitions of disability lead to distinct design choices across problem formulation, data, use, and operational elements.. Bias in AI can emerge from seemingly isolated design decisions, amplified by a lack of transparency and disabled participation.. Historical models of disability are embedded in AI design, leading to varied and potentially harmful biases.
What research method was used?: Conceptual analysis and framework development.
How strong is the evidence?: Evidence strength is rated Strong effect, based on a 2023 journal from First Monday.
What should I do differently in my next project?: Before commencing an AI design project related to disability, conduct a thorough review of existing disability models and their potential implications for bias. Establish a framework for continuous feedback and co-design with the target user group.
What are the limitations?: The analysis is conceptual and relies on historical models; real-world AI implementations may involve complex interactions not fully captured.