What is “partial randomization” when it comes to research funding?
For many of society’s challenges, including the UN’s global sustainable development goals, research funding is crucial. We also know, however, that just throwing money at a problem, does not mean that the problem has been solved by the money. This is especially true when it comes to research funding. It is not only vital to have abundant research funding, but the funding must also be invested in researchers with the best ideas and the most relevant expertise so that we can achieve all 17 of the UN’s global sustainable development goals.
We recently spoke with Dr. Helen Woods from the Research on Research Institute (RoRI), a consortium of research funders, university partners and the research support industry from 13 different countries and regions, to learn more about the randomization of research funding decisions. Specifically, in RoRI’s RANDOMISATION project, a “lottery” element is introduced to the research funding decision-making process.
Traditionally, research funders have used peer-review as the way to distinguish and rank eligible researchers and their research proposals to be funded. Peer-review, in the broadest sense, is when experts are asked to review the proposals of other experts. There are many ways peer-review may “fail” to select the best researchers and best research proposals. Some examples of peer-review failing include when peer-reviewers favor more conservative projects over high-risk projects [1], when reviewers let their conscious and unconscious biases influence their decision-making [2, 3], or when they change their ratings on a proposal after seeing another expert’s review on the same proposal [4]. Given the shortcomings of peer-review combined with a growing number of applications and a stagnant research budget, funders have a clear need to innovate their research funding processes.
In RoRI’s RANDOMISATION project, several funders, including the Volkswagen Foundation, the Swiss National Science Foundation (SNSF), the Austrian Science Fund (FWF) and the Health Research Council of New Zealand, adapted the randomization concept to bring about improved funding outcomes for their research funding schemes.
At the heart of it, the randomization methodology involves randomization that is applied after traditional peer-review and may be used to complement existing selection methods. Projects and researchers are assessed for initial eligibility, ranked by peer review, and only the most qualified projects are selected for the final randomization to determine the “winners” of the funds. What may be the arguments for and against randomization? According to RoRI’s The Experimental Research Funder’s Handbook:
Arguments for partial randomization
- Randomization reduces bias and promotes diversity. By randomizing funding among all meritorious applications, it is argued that it would reduce the bias according to age, gender, ethnicity and affiliation;
- Randomization fosters innovation and creativity. Peer-reviewers are suspected to prefer conservative approaches over riskier research projects [1], therefore, it is argued that randomization may improve the funding success of riskier research proposals;
- Randomization addresses the “randomness” of peer-review. It is known that reviewers’ scores for the same applications can vary, randomization thereby removes the element of randomness introduced by the reviewers and formalizes it with a process whereby all meritorious applicants have the same odds of success;
- Randomization reduces reviewers’ burden. Peer-review of grant applications is a time-consuming process and having to compare and rank proposals adds a substantial workload. Randomization alleviates reviewer burden by asking reviewers to make a binary decision, which is less time-consuming.
Arguments against partial randomization
- Randomization undermines merit-based decision making. The main argument against partial randomization is that it goes against the use of merit in scientific decision making;
- Randomization may create stigma and reputational damage. As a corollary to removing decision-making solely based on merit, researchers who are funded through partial randomization may be perceived to be less worthy of their success and both funder and the recipient may suffer reputational damage as a result.
Randomization may be one way to improve the success of research funding outcomes
It seems that the concerns raised against partial randomization are related to the perception of scientific merit and that reputation could be somehow damaged if someone’s success was due to pure luck. While these concerns must be taken into consideration, we must also recognize that no human is infallible, and it is impossible to precisely predict the successful completion of research proposals. An element of randomization might just be what is needed to reduce bias and bring about fair distribution of research funding. For more insights on how the four funders implemented randomization in their research funding strategy, check out RoRI’s The Experimental Research Funder’s Handbook.
Acknowledgment
We thank Dr. Helen Woods from the Research on Research Institute (RoRI) for taking the time to speak with us and for approving the content of this blog post.
References
[1] Luukkonen, T. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation. 2012. 21: 48–60.
[2] Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., Kington, R. Race, ethnicity, and NIH research awards. Science. 2011. 333(6045): 1015-9.
[3] Severin, A., Martins, J., Heyard, R., Delavy, F., Jorstad, A., Egger, M. Gender and other potential biases in peer review: cross-sectional analysis of 38250 external peer review reports. BMJ Open. 2020. 10: e035058.
[4] Lane, J. N., Teplitskiy, M., Gray, G., Ranu, H., Menietti, M., Guinan, E. C., Lakhani, K. R. Conservatism gets funded? A field experiment on the role of negative information in novel project evaluation. Management Science. 2022. 68 (6): 4478–4495.
