Short answer

Design forecasting support tools that offer constructive feedback and guidance after significant errors, rather than solely relying on expert intuition.

Field: Commercial Production
Source: European Journal of Operational Research (2015)
Method: Empirical analysis of large-scale forecasting data.
Sample: Large multinational dataset (specific number not provided in abstract)
Evidence: Strong effect

Experts who make significant forecasting errors, termed 'big losses', tend to become less confident and potentially less effective in their subsequent judgmental adjustments. This commercial production research insight is drawn from a 2015 study published in European Journal of Operational Research. Using Empirical analysis of large-scale forecasting data. with Large multinational dataset (specific number not provided in abstract), researchers explored how this design variable affects real-world outcomes. The key design takeaway: Design forecasting support tools that offer constructive feedback and guidance after significant errors, rather than solely relying on expert intuition.

Study

Commercial ProductionHigh ImpactStrong effect

Experiencing 'big losses' in demand forecasting reduces expert adjustment confidence

Experts who make significant forecasting errors, termed 'big losses', tend to become less confident and potentially less effective in their subsequent judgmental adjustments.

European Journal of Operational Research · 2015

Key Findings

01Experts who made 'big losses' in judgmental adjustments showed a change in their subsequent behaviour.
02The experience of significant forecasting errors can impact an expert's confidence and decision-making in future adjustments.

Application

Design takeaway

Design forecasting support tools that offer constructive feedback and guidance after significant errors, rather than solely relying on expert intuition.

How to apply

When designing forecasting software, consider implementing features that flag or provide additional checks for adjustments that deviate significantly from statistical models, especially after a period of poor performance.

Project actions

01When analyzing expert performance, consider the impact of past errors on current decisions.
02Think about how system design can help users recover from mistakes.

Method & Evidence

AimTo investigate whether experiencing 'big losses' (significant negative impacts on forecasting accuracy due to judgmental adjustments) influences experts' subsequent forecasting behaviour.

MethodEmpirical analysis of large-scale forecasting data.

ProcedureThe study analyzed a multinational dataset of statistical demand forecasts for pharmaceutical products, expert adjustments made to these forecasts, and the actual sales figures. 'Big losses' were identified as judgmental adjustments that substantially decreased forecast accuracy compared to the baseline statistical forecast. The researchers then examined how these 'big losses' affected subsequent judgmental adjustments made by the experts.

SampleLarge multinational dataset (specific number not provided in abstract)

ContextDemand forecasting for pharmaceutical products within multinational corporations.

Variables

IVExperiencing a 'big loss' in judgmental forecasting.

DVSubsequent judgmental adjustments and forecasting behaviour.

CVBaseline statistical forecast accuracy, actual sales data, type of product forecasted.

Strengths & Limitations

Strengths

+Utilizes a large, real-world dataset.
+Investigates a critical aspect of expert decision-making in forecasting.

Limitations

The study's findings might be specific to the type of data and experts involved. Generalizing to all forecasting scenarios requires caution.

Reliability & validity

The study's reliance on a large dataset and empirical analysis suggests good external validity for the context studied. Internal validity is supported by the focus on specific behavioural changes following defined 'big losses'.

Think critically

How might the design of a forecasting system encourage experts to maintain confidence and accuracy after experiencing a 'big loss', rather than becoming overly cautious or making further errors?

Design Principles

"Mitigate the impact of negative feedback loops in expert decision-making by providing supportive and corrective system interventions."

Understanding how past forecasting failures impact expert decision-making is crucial for designing robust forecasting systems. It highlights the need for support mechanisms that can mitigate the negative psychological effects of errors, ensuring continued accuracy and reliability in commercial operations.

What This Means for Your Design

If a forecaster makes a really big mistake, they might become less sure of themselves and make more mistakes later.

How to use in your project

1.Use this research to justify the need for specific features in your forecasting system that manage user error and confidence.

Add to My Project

Quick Cite

Paragraph starter

This research by Petropoulos, Fildes, and Goodwin (2015) highlights that 'big losses' in judgmental forecasting adjustments can negatively impact subsequent expert behaviour. This suggests that design interventions within forecasting support systems should consider mechanisms to mitigate the psychological effects of significant errors, potentially through guidance or controlled restrictiveness, to maintain consistent forecasting accuracy.

Source

European Journal of Operational Research

Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour?

journal · 2015

View source

Questions About This Research

What does the research say about experiencing 'big losses' in demand forecasting reduces expert adjustment confidence?: Design forecasting support tools that offer constructive feedback and guidance after significant errors, rather than solely relying on expert intuition. Evidence: European Journal of Operational Research (2015).
Why does "Experiencing 'big losses' in demand forecasting reduces expert adjustment confidence" matter for design?: Understanding how past forecasting failures impact expert decision-making is crucial for designing robust forecasting systems. It highlights the need for support mechanisms that can mitigate the negative psychological effects of errors, ensuring continued accuracy and reliability in commercial operations.
How can designers apply this research?: Design forecasting support tools that offer constructive feedback and guidance after significant errors, rather than solely relying on expert intuition.
What were the main findings?: Experts who made 'big losses' in judgmental adjustments showed a change in their subsequent behaviour.. The experience of significant forecasting errors can impact an expert's confidence and decision-making in future adjustments.
What research method was used?: Empirical analysis of large-scale forecasting data. with Large multinational dataset (specific number not provided in abstract).
How strong is the evidence?: Evidence strength is rated Strong effect, based on a 2015 journal from European Journal of Operational Research.
What should I do differently in my next project?: When designing forecasting software, consider implementing features that flag or provide additional checks for adjustments that deviate significantly from statistical models, especially after a period of poor performance.
What are the limitations?: The study focuses on a specific industry (pharmaceuticals) and may not generalize to all forecasting contexts. The definition of 'big loss' might require context-specific calibration.