Short answer
When designing or evaluating systems that use algorithms to profile users, ensure that accuracy metrics are comprehensive and account for false positives and negatives, rather than relying on single, potentially inflated, figures.
- Field
- User-Centred Design
- Source
- Social Policy and Society (2023)
- Method
- Exploratory Review
- Evidence
- Strong effect
Standard reporting metrics for algorithmic profiling in public services can inflate perceived accuracy, leading to misallocation of resources and support. This user-centred design research insight is drawn from a 2023 study published in Social Policy and Society. Using Exploratory review, researchers explored how this design variable affects real-world outcomes. The key design takeaway: When designing or evaluating systems that use algorithms to profile users, ensure that accuracy metrics are comprehensive and account for false positives and negatives, rather than relying on single, potentially inflated, figures.
Algorithmic Profiling Overstates Accuracy, Potentially Misdirecting Support
Standard reporting metrics for algorithmic profiling in public services can inflate perceived accuracy, leading to misallocation of resources and support.
Social Policy and Society · 2023
Key Findings
- 01Current methods of reporting accuracy for ASPAs typically use a single percentage, which can be misleading.
- 02ASPAs often exhibit high false positive rates, incorrectly identifying individuals as at risk of long-term unemployment.
- 03This overestimation of accuracy can lead to inefficient allocation of active labour market policy interventions.
Application
Design takeaway
When designing or evaluating systems that use algorithms to profile users, ensure that accuracy metrics are comprehensive and account for false positives and negatives, rather than relying on single, potentially inflated, figures.
How to apply
When developing or assessing any system that uses predictive algorithms for user segmentation or resource allocation, demand and utilize reporting metrics that include false positive and false negative rates, alongside overall accuracy.
Project actions
- 01When discussing the success of any predictive tool in your design project, be specific about the metrics used (e.g., precision, recall, F1-score) and explain what they mean.
- 02Consider how the 'success' of your design might be measured and what potential misinterpretations could arise from simplified reporting.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Highlights a critical flaw in common reporting practices for algorithmic systems.
- +Provides a foundational critique for assessing the real-world impact of such technologies.
Limitations
This study is a review of reporting standards, not an empirical test of specific algorithms. The findings are based on the analysis of how accuracy is communicated.
Reliability & validity
The validity of the findings rests on the logical analysis of reporting methods and their potential to mislead. Reliability is high in terms of the critique of the reporting method itself, as the mathematical principles of false positives and negatives are well-established.
Think critically
How might the pressure to demonstrate the success of technological solutions influence the way their performance is reported, and what are the ethical implications for the users of these systems?
Design Principles
"Transparency in algorithmic performance reporting is crucial for effective and ethical design."
Designers and researchers must be critical of how the performance of automated systems is communicated. Misleading accuracy figures can lead to flawed decision-making, impacting user trust and the effectiveness of interventions.
What This Means for Your Design
Imagine a system that predicts who will get sick. If it says 90% of people are predicted to get sick, that sounds really good. But what if it's wrong about most of those people? This study shows that when systems predict who needs job help, the way they report their success can be misleading, making them seem better than they are and potentially sending help to the wrong people.
How to use in your project
- 1.Reference this study when discussing the limitations of quantitative performance metrics for algorithmic systems, especially in contexts involving user support or resource allocation.
Add to My Project
Quick Cite
Paragraph starter
The effectiveness of algorithmic profiling systems, particularly in public service contexts, can be misrepresented by standard reporting metrics. As demonstrated by Gallagher and Griffin (2023), a single accuracy percentage often inflates perceived performance by failing to adequately account for high false positive rates, leading to potential misallocation of resources and support. Therefore, when evaluating or designing such systems, it is critical to adopt a more comprehensive approach to performance measurement that includes detailed analysis of false positives and negatives to ensure user-centric and effective outcomes.
Source
Social Policy and Society
(in) Accuracy in Algorithmic Profiling of the Unemployed – An Exploratory Review of Reporting Standards
journal · 2023
View sourceQuestions About This Research
- What does the research say about algorithmic profiling overstates accuracy, potentially misdirecting support?
- When designing or evaluating systems that use algorithms to profile users, ensure that accuracy metrics are comprehensive and account for false positives and negatives, rather than relying on single, potentially inflated, figures. Evidence: Social Policy and Society (2023).
- Why does "Algorithmic Profiling Overstates Accuracy, Potentially Misdirecting Support" matter for design?
- Designers and researchers must be critical of how the performance of automated systems is communicated. Misleading accuracy figures can lead to flawed decision-making, impacting user trust and the effectiveness of interventions.
- How can designers apply this research?
- When designing or evaluating systems that use algorithms to profile users, ensure that accuracy metrics are comprehensive and account for false positives and negatives, rather than relying on single, potentially inflated, figures.
- What were the main findings?
- Current methods of reporting accuracy for ASPAs typically use a single percentage, which can be misleading.. ASPAs often exhibit high false positive rates, incorrectly identifying individuals as at risk of long-term unemployment.. This overestimation of accuracy can lead to inefficient allocation of active labour market policy interventions.
- What research method was used?
- Exploratory Review.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2023 journal from Social Policy and Society.
- What should I do differently in my next project?
- When developing or assessing any system that uses predictive algorithms for user segmentation or resource allocation, demand and utilize reporting metrics that include false positive and false negative rates, alongside overall accuracy.
- What are the limitations?
- The review focused on reporting standards and did not directly test the algorithms themselves. The specific algorithms and contexts of PES may vary.