Short answer
Acknowledge the limitations of RCTs and integrate their findings with existing knowledge and theoretical understanding to achieve deeper insights into design effectiveness.
- Field
- Innovation & Design
- Source
- Social Science & Medicine (2017)
- Method
- Literature review and theoretical analysis
- Evidence
- Moderate effect
The paper argues that an overemphasis on Randomized Controlled Trials (RCTs) in research, particularly in social sciences, can impede cumulative scientific progress by discarding prior knowledge and failing to adequately explain 'why' interventions work. This innovation & design research insight is drawn from a 2017 study published in Social Science & Medicine. Using Literature review and theoretical analysis, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Acknowledge the limitations of RCTs and integrate their findings with existing knowledge and theoretical understanding to achieve deeper insights into design effectiveness.
Over-reliance on RCTs hinders cumulative scientific progress
The paper argues that an overemphasis on Randomized Controlled Trials (RCTs) in research, particularly in social sciences, can impede cumulative scientific progress by discarding prior knowledge and failing to adequately explain 'why' interventions work.
Social Science & Medicine · 2017
Key Findings
- 01Randomization in RCTs does not equalize all factors other than the treatment.
- 02RCTs do not automatically provide precise estimates of the Average Treatment Effect (ATE).
- 03RCTs do not eliminate the need to consider covariates (observed or unobserved).
- 04The property of unbiasedness from RCTs has limited practical value.
- 05RCT estimates are often limited to convenience samples, requiring justification for external validity.
Application
Design takeaway
Acknowledge the limitations of RCTs and integrate their findings with existing knowledge and theoretical understanding to achieve deeper insights into design effectiveness.
How to apply
When researching or justifying a design solution, consider how it builds upon or challenges existing knowledge, and whether the 'why' behind its effectiveness is understood, not just 'what' works.
Project actions
- 01When citing research for your project, critically evaluate the methodology and its limitations.
- 02Consider how your design builds on existing principles or knowledge, and articulate this connection.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Provides a critical perspective on a widely used research methodology.
- +Emphasizes the importance of understanding causal mechanisms ('why') in addition to empirical outcomes ('what').
Limitations
The paper's critique of RCTs might be too strong for certain contexts where RCTs are the most appropriate method. Generalizing findings from social science/medical RCTs to physical product design requires careful consideration.
Reliability & validity
The paper's arguments are based on theoretical analysis and literature review, making it a strong conceptual piece. However, it does not present new empirical data, so its findings are not directly testable in terms of statistical reliability or validity in the traditional sense.
Think critically
How can a designer balance the need for empirical evidence from trials with the importance of building upon existing theoretical knowledge and understanding causal mechanisms?
Design Principles
"Cumulative knowledge and mechanistic understanding are crucial for robust design innovation."
This insight is relevant to design as it highlights the importance of building upon existing knowledge and understanding the underlying mechanisms of design solutions, rather than solely relying on empirical 'what works' data. It encourages a more holistic approach to research and development.
What This Means for Your Design
Don't just trust studies that say something works; try to understand why it works and if that reason applies elsewhere, and remember that old knowledge is still valuable.
How to use in your project
- 1.Use in the 'Analysis of existing products' or 'Research' sections to critique the evidence base for certain design features or solutions.
- 2.In the 'Evaluation' section, discuss how your design's effectiveness is understood beyond just empirical results.
Add to My Project
Quick Cite
Paragraph starter
The reliance on Randomized Controlled Trials (RCTs) in research, as highlighted by Deaton and Cartwright (2017), presents a potential pitfall for cumulative scientific progress. While RCTs offer unbiased estimates, their limitations in generalizability and their tendency to discard prior knowledge necessitate a more integrated approach. For design, this means not only seeking empirical validation but also understanding the underlying 'why' and building upon existing theoretical frameworks to foster robust innovation.
Source
Social Science & Medicine
Understanding and misunderstanding randomized controlled trials
journal · 2017
View sourceQuestions About This Research
- What does the research say about over-reliance on rcts hinders cumulative scientific progress?
- Acknowledge the limitations of RCTs and integrate their findings with existing knowledge and theoretical understanding to achieve deeper insights into design effectiveness. Evidence: Social Science & Medicine (2017).
- Why does "Over-reliance on RCTs hinders cumulative scientific progress" matter for design?
- This insight is relevant to IB DT as it highlights the importance of building upon existing knowledge and understanding the underlying mechanisms of design solutions, rather than solely relying on empirical 'what works' data. It encourages a more holistic approach to research and development.
- How can designers apply this research?
- Acknowledge the limitations of RCTs and integrate their findings with existing knowledge and theoretical understanding to achieve deeper insights into design effectiveness.
- What were the main findings?
- Randomization in RCTs does not equalize all factors other than the treatment.. RCTs do not automatically provide precise estimates of the Average Treatment Effect (ATE).. RCTs do not eliminate the need to consider covariates (observed or unobserved).. The property of unbiasedness from RCTs has limited practical value.
- What research method was used?
- Literature review and theoretical analysis.
- How strong is the evidence?
- Evidence strength is rated Moderate effect, based on a 2017 journal from Social Science & Medicine.
- What should I do differently in my next project?
- When researching or justifying a design solution, consider how it builds upon or challenges existing knowledge, and whether the 'why' behind its effectiveness is understood, not just 'what' works.
- What are the limitations?
- The paper's focus is on research methodology in social sciences and medicine, and its direct application to all design fields may require adaptation.