Short answer
Design tools and processes that acknowledge and adapt to developers' actual testing behaviours, rather than relying on idealized assumptions.
- Field
- User-Centred Design
- Source
- Academic Publication (2015)
- Method
- Observational field study
- Sample
- 416 software engineers
- Evidence
- Strong effect
Software developers spend considerably less time on testing and utilize their IDEs for testing far less frequently than commonly assumed or believed. This user-centred design research insight is drawn from a 2015 study published in Academic Publication. Using Observational field study with 416 software engineers, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Design tools and processes that acknowledge and adapt to developers' actual testing behaviours, rather than relying on idealized assumptions.
Developer Testing Habits Deviate Significantly from Perceived Norms
Software developers spend considerably less time on testing and utilize their IDEs for testing far less frequently than commonly assumed or believed.
Academic Publication · 2015
Key Findings
- 01The majority of developers do not engage in testing as frequently as expected.
- 02Developers rarely execute tests directly within their IDEs.
- 03Test-Driven Development (TDD) is not widely adopted.
- 04Developers allocate only a quarter of their work time to engineering tests, despite believing they spend closer to half their time on it.
Application
Design takeaway
Design tools and processes that acknowledge and adapt to developers' actual testing behaviours, rather than relying on idealized assumptions.
How to apply
When designing new developer tools or features, conduct user research to understand current workflows and identify pain points, rather than assuming standard practices.
Project actions
- 01When designing a tool for developers, observe how they actually work first.
- 02Don't just assume developers will use a feature the way you expect them to.
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Large sample size providing statistical power.
- +Longitudinal monitoring of developer activity over an extended period.
Limitations
The study was conducted in 2015, and developer practices may have evolved since then. The specific IDEs used might influence the results.
Reliability & validity
The large sample size and long monitoring period contribute to reliability. Validity is supported by direct observation of behaviour, though self-reported perceptions might have lower validity.
Think critically
How might the evolution of IDE features and testing frameworks since 2015 have impacted the findings of this study?
Design Principles
"Align design interventions with observed user behaviour rather than assumed norms."
Understanding the actual practices and perceptions of developers regarding testing is crucial for designing more effective development tools and workflows. This insight highlights a gap between idealised testing practices and real-world behaviour, suggesting opportunities for improved user support and training.
What This Means for Your Design
Most programmers don't test as much as people think they do, and they don't use their computer tools for testing as much as they should.
How to use in your project
- 1.Use this research to justify why you need to observe your target users' actual behaviour when designing your product.
- 2.It can help explain why your initial assumptions about user needs might be wrong.
Add to My Project
Quick Cite
Paragraph starter
This research highlights a significant discrepancy between assumed and actual developer testing practices, indicating that design interventions should be grounded in empirical observation rather than generalized beliefs. For instance, the study found that developers spend considerably less time on testing than commonly perceived and rarely utilize their IDEs for test execution, suggesting that tools designed without understanding these actual workflows may not be effectively adopted.
Source
Academic Publication
When, how, and why developers (do not) test in their IDEs
journal · 2015
View sourceQuestions About This Research
- What does the research say about developer testing habits deviate significantly from perceived norms?
- Design tools and processes that acknowledge and adapt to developers' actual testing behaviours, rather than relying on idealized assumptions. Evidence: Academic Publication (2015).
- Why does "Developer Testing Habits Deviate Significantly from Perceived Norms" matter for design?
- Understanding the actual practices and perceptions of developers regarding testing is crucial for designing more effective development tools and workflows. This insight highlights a gap between idealised testing practices and real-world behaviour, suggesting opportunities for improved user support and training.
- How can designers apply this research?
- Design tools and processes that acknowledge and adapt to developers' actual testing behaviours, rather than relying on idealized assumptions.
- What were the main findings?
- The majority of developers do not engage in testing as frequently as expected.. Developers rarely execute tests directly within their IDEs.. Test-Driven Development (TDD) is not widely adopted.. Developers allocate only a quarter of their work time to engineering tests, despite believing they spend closer to half their time on it.
- What research method was used?
- Observational field study with 416 software engineers.
- How strong is the evidence?
- Evidence strength is rated Strong effect, based on a 2015 journal from Academic Publication.
- What should I do differently in my next project?
- When designing new developer tools or features, conduct user research to understand current workflows and identify pain points, rather than assuming standard practices.
- What are the limitations?
- The study focused on specific IDEs and may not generalize to all development environments or team structures. Self-reported perceptions could be subject to recall bias.