Citation counts are a flawed proxy for research impact.
While citation counts are widely used to assess the influence of research, studies reveal that citing behavior is driven by a complex mix of intellectual acknowledgment and other, often non-scientific, motivations.
Journal of Documentation · 2008
Key Findings
- 01Citing behavior is not solely motivated by acknowledging intellectual influences.
- 02Non-scientific factors also play a role in the decision to cite.
- 03Studies on citing behavior vary widely in design and have methodological weaknesses, making results difficult to replicate.
- 04Despite variations, citation phenomena may still retain a role as a measure of impact.
Application
Design takeaway
When evaluating the influence of design research or innovations, look beyond simple citation counts and consider a more holistic approach that includes qualitative feedback, adoption rates, and real-world impact.
How to apply
When reviewing literature for a design project, critically assess the motivations behind cited works and consider alternative ways to measure the significance of design contributions.
Project actions
- 01When researching existing designs or technologies, don't just look at how many times a paper or patent is cited.
- 02Try to understand *why* something was cited – was it a foundational idea, a critique, or something else?
Method & Evidence
Variables
Strengths & Limitations
Strengths
- +Provides a critical perspective on a commonly used metric.
- +Highlights the complexity of scholarly communication.
Limitations
The original studies reviewed have methodological issues, meaning the conclusions about citation behavior might not be entirely robust.
Reliability & validity
The reliability and validity of the findings are questioned due to the methodological weaknesses and design variations in the reviewed studies.
Think critically
If citation counts are not a perfect measure of impact, what alternative or complementary metrics could be used to evaluate the success and influence of design research and innovations?
Design Principles
"Impact assessment should be multi-dimensional, accounting for both quantitative and qualitative indicators."
Understanding the nuances of citation behavior is crucial for accurately evaluating the impact and value of design research and innovation. Over-reliance on citation metrics can lead to misinterpretations of influence and potentially overlook valuable contributions that are not heavily cited.
What This Means for Your Design
Just because a research paper or design project has lots of citations doesn't automatically mean it's the most important or influential. People cite for many reasons, not just because they agree or learned something important.
How to use in your project
- 1.Use this research to justify why you are using a variety of methods to assess the success or impact of your own design project, not just relying on one metric.
Add to My Project
Quick Cite
(2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation. https://doi.org/10.1108/00220410810844150 Retrieved from https://designdex.org/study/545dc51a-1058-48b3-98cb-730ce572011a/citation-counts-are-a-flawed-proxy-for-research-impact
Paragraph starter
The evaluation of existing design solutions should not solely rely on bibliometric data such as citation counts. Research suggests that citing behavior is influenced by a complex interplay of factors, including non-academic motivations, and that studies measuring this behavior often suffer from methodological weaknesses. Therefore, a more comprehensive assessment of impact requires considering qualitative evidence, real-world application, and user feedback alongside quantitative metrics.
Source
Journal of Documentation
What do citation counts measure? A review of studies on citing behavior
journal · 2008
View sourceQuestions about this research
- What does the research say about citation counts are a flawed proxy for research impact?
- When evaluating the influence of design research or innovations, look beyond simple citation counts and consider a more holistic approach that includes qualitative feedback, adoption rates, and real-world impact. Evidence: Journal of Documentation (2008).
- Why does "Citation counts are a flawed proxy for research impact." matter for design?
- Understanding the nuances of citation behavior is crucial for accurately evaluating the impact and value of design research and innovation. Over-reliance on citation metrics can lead to misinterpretations of influence and potentially overlook valuable contributions that are not heavily cited.
- How can designers apply this research?
- When evaluating the influence of design research or innovations, look beyond simple citation counts and consider a more holistic approach that includes qualitative feedback, adoption rates, and real-world impact.
- What were the main findings?
- Citing behavior is not solely motivated by acknowledging intellectual influences.. Non-scientific factors also play a role in the decision to cite.. Studies on citing behavior vary widely in design and have methodological weaknesses, making results difficult to replicate.. Despite variations, citation phenomena may still retain a role as a measure of impact.
- What research method was used?
- Narrative review of empirical studies.
- How strong is the evidence?
- Evidence strength is rated Mixed findings, based on a 2008 journal from Journal of Documentation.
- What should I do differently in my next project?
- When reviewing literature for a design project, critically assess the motivations behind cited works and consider alternative ways to measure the significance of design contributions.
- What are the limitations?
- The review's findings are based on studies with methodological weaknesses and varying designs, limiting the generalizability and reliability of the conclusions.
- Is there evidence that citation counts affects design outcomes?
- Research indicates that scientists cite papers for reasons beyond just acknowledging intellectual contributions, including other, sometimes non-academic, factors. However, the reliability of studies measuring this behavior is questionable due to design and methodological issues. Understanding the nuances of citation be Source: Journal of Documentation (2008).
- Where does this behavior research apply?
- Academic research and scholarly communication It sits within innovation & design research on designdex.org.
Related research topics
citation counts design research · evidence on citation counts · does citation counts improve design outcomes · behavior studies for designers · citation counts and behavior findings · innovation & design research evidence