Short answer
When designing or implementing algorithmic decision support, focus on integrating the system thoughtfully into existing workflows and consider mechanisms for managing the balance between algorithmic recommendations and human judgment.
- Field
- Innovation & Design
- Source
- Strategic Management Journal (2023)
- Method
- Mixed-methods research, including a pilot study with an Inspections Department and interviews with multiple departments.
- Sample
- 55 departments interviewed, pilot study details not specified but implies a specific department.
- Evidence
- Moderate effect
The effectiveness of algorithms in improving decisions is significantly influenced by how decision authority is managed and the algorithm's integration into the existing organizational context, rather than solely by its technical sophistication. This innovation & design research insight is drawn from a 2023 study published in Strategic Management Journal. Using Mixed-methods research, including a pilot study with an inspections department and interviews with multiple departments. with 55 departments interviewed, pilot study details not specified but implies a specific department., researchers explored how this design variable affects real-world outcomes. The key design takeaway: When designing or implementing algorithmic decision support, focus on integrating the system thoughtfully into existing workflows and consider mechanisms for managing the balance between algorithmic recommendations and human judgment.
Algorithmic Sophistication Alone Does Not Guarantee Improved Decision-Making
The effectiveness of algorithms in improving decisions is significantly influenced by how decision authority is managed and the algorithm's integration into the existing organizational context, rather than solely by its technical sophistication.
Strategic Management Journal · 2023
Key Findings
- 01Both simple and sophisticated algorithms provided substantial prediction gains.
- 02Prediction gains did not consistently translate into improved decisions.
- 03Decision-makers often overrode algorithmic recommendations due to other organizational objectives, without necessarily improving outcomes.
- 04Organizations tend to grant considerable decision authority to human operators even when using data-driven tools.
Application
Design takeaway
When designing or implementing algorithmic decision support, focus on integrating the system thoughtfully into existing workflows and consider mechanisms for managing the balance between algorithmic recommendations and human judgment.
How to apply
When developing a predictive analytics tool, conduct thorough user research to understand existing decision-making processes and potential override behaviors. Design the interface to facilitate informed overrides rather than outright rejection, and consider how the tool can support, rather than dictate, decisions.
Project actions
- 01Consider how your design will interact with existing human decision-making processes.
- 02Think about the user's autonomy and how they might override your design's suggestions.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Combines quantitative pilot data with qualitative interview data for a richer understanding.
- +Examines a real-world organizational setting.
Limitations
The specific organizational culture and the nature of the 'other objectives' considered by inspectors could influence the results.
Reliability & validity
The study's reliance on interviews introduces potential for subjective interpretation, while the pilot study's specific context might limit generalizability. Triangulation of data from interviews and the pilot study enhances validity.
Think critically
To what extent should designers aim to minimize human override of algorithmic recommendations, and under what circumstances is such an approach detrimental?
Design Principles
"Algorithmic decision support systems should be designed for contextual integration and flexible human oversight."
This research highlights a critical gap between algorithmic prediction gains and actual decision improvement. For designers and engineers developing decision support systems, it underscores the need to consider the human element and organizational dynamics, not just the technical prowess of the algorithm.
What This Means for Your Design
Just making a smart computer program doesn't mean people will make better choices. How much power people have to ignore the program, and what else they care about in their job, matters just as much.
How to use in your project
- 1.Reference this study when discussing the importance of user autonomy and contextual factors in the evaluation of your design's effectiveness.
Add to My Project
Quick Cite
Paragraph starter
Research by Kim et al. (2023) indicates that the effectiveness of algorithmic decision support is not solely determined by its predictive accuracy or sophistication. Their findings suggest that significant gains in decision-making outcomes require careful consideration of how decision authority is structured and how algorithms are integrated into the broader organizational context, as users may override recommendations based on unarticulated organizational objectives.
Source
Strategic Management Journal
Decision authority and the returns to algorithms
journal · 2023
View sourceQuestions About This Research
- What does the research say about algorithmic sophistication alone does not guarantee improved decision-making?
- When designing or implementing algorithmic decision support, focus on integrating the system thoughtfully into existing workflows and consider mechanisms for managing the balance between algorithmic recommendations and human judgment. Evidence: Strategic Management Journal (2023).
- Why does "Algorithmic Sophistication Alone Does Not Guarantee Improved Decision-Making" matter for design?
- This research highlights a critical gap between algorithmic prediction gains and actual decision improvement. For designers and engineers developing decision support systems, it underscores the need to consider the human element and organizational dynamics, not just the technical prowess of the algorithm.
- How can designers apply this research?
- When designing or implementing algorithmic decision support, focus on integrating the system thoughtfully into existing workflows and consider mechanisms for managing the balance between algorithmic recommendations and human judgment.
- What were the main findings?
- Both simple and sophisticated algorithms provided substantial prediction gains.. Prediction gains did not consistently translate into improved decisions.. Decision-makers often overrode algorithmic recommendations due to other organizational objectives, without necessarily improving outcomes.. Organizations tend to grant considerable decision authority to human operators even when using data-driven tools.
- What research method was used?
- Mixed-methods research, including a pilot study with an Inspections Department and interviews with multiple departments. with 55 departments interviewed, pilot study details not specified but implies a specific department..
- How strong is the evidence?
- Evidence strength is rated Moderate effect, based on a 2023 journal from Strategic Management Journal.
- What should I do differently in my next project?
- When developing a predictive analytics tool, conduct thorough user research to understand existing decision-making processes and potential override behaviors. Design the interface to facilitate informed overrides rather than outright rejection, and consider how the tool can support, rather than dictate, decisions.
- What are the limitations?
- The study's findings may be specific to the context of inspections and managerial decision-making; generalizability to other domains might vary. The exact nature of 'other organizational objectives' was not deeply explored.