Summary of Key Points
In recent years, research funding agencies in countries such as Switzerland, the UK, and Germany have begun to use lotteries to allocate a portion of their funding. The reason for this is that traditional peer review methods, near the “funding threshold” (the dividing line between getting funding and not getting it), show significant differences in scores given by reviewers, making it impossible to accurately distinguish the quality of applications. Forcing a ranking is merely an attempt to appear precise. There are several types of lottery systems in use: some only apply lotteries to applications that reviewers cannot distinguish between, others use lotteries for all applications that meet the quality criteria, and yet others have applicants fill out complete applications after the lottery has been held. This approach has increased the diversity of applicants (for example, by boosting the proportion of researchers from smaller institutions, ethnic minorities, and women), reduced the review burden, but it has also sparked controversy. Some argue that using luck to distribute taxpayer money is unfair, while others believe that lotteries ignore the connection between a project and the existing body of knowledge. The scientific community is still exploring the long-term effects of these lotteries.
Why Use Lotteries? — The Problem of “Pseudo-Precision” in Traditional Review
Traditional research funding reviews assume that experts can accurately rank applications, but in reality:
- Large Score Differences: A 2018 study in PNAS showed that 43 reviewers scored 25 US NIH applications, with almost no consensus on the results. A data team from the Swiss National Science Foundation (SNSF) found that most application scores clustered together, and reviewers could only make minor adjustments by adding or subtracting points, which might represent random errors rather than real differences in quality.
- The Gray Area Near the Funding Threshold: The SNSF allocates $1 billion annually, and the funding threshold passes right through this clustered area. For example, applications ranked 23rd and 27th might be of similar quality, but reviewers must choose one, which is like choosing a winner by chance, yet it is presented as an “expert decision.”
Therefore, agencies have decided to acknowledge the limitations and use lotteries to address the issues that traditional review methods cannot resolve.
How Do Lotteries Work? — Three Different Approaches
The rules for lotteries vary among agencies and can be roughly categorized into three types:
1. Conservative (SNSF, Switzerland): Reviews are conducted first to eliminate clearly good and bad applications, and then a lottery is used to decide who receives funding. This approach uses experts to filter out the obvious outliers and relies on luck for the remaining applications.
2. Aggressive (UK National Academy): All applications that meet the quality criteria are included in the lottery pool. For example, the number of applications for small grants increased by 70% (from 600 to 1100), as more applicants from smaller institutions saw a chance of getting funding.
3. Radical (StIL, Germany): A lottery is held first, and only the successful applicants are then asked to submit complete applications. This reduces the review burden by avoiding the need to review a large number of low-quality applications, and the proportion of female applicants increased from 40.8% to 45%.
Benefits of Lotteries
The results of several experiments are clear:
- Diversity of Applicants: In the UK, the proportion of Asian researchers applying for small grants increased from 13% to 19%, and the success rate rose from 11% to 17%. Many previously unsuccessful applicants from smaller institutions, such as Abertay University, received funding. In Germany, the proportion of female applicants who received funding increased from 38% to 47%.
- Increased Application Volume: In the UK, the number of applications nearly doubled because more applicants from smaller institutions felt they had a chance.
- Reduced Review Burden: In Germany, the use of lotteries reduced the number of unnecessary applications and review tasks, cutting costs by two-thirds.
- Reassurance for Rejected Applicants: The UK now informs applicants whether they did not meet the criteria or were not selected in the lottery. Those who were not selected can use this information to apply for other funds, which serves as a form of “quality endorsement.”
Opposition to Lotteries: Is It Really Fair?
Not everyone supports lotteries, with three main concerns:
1. Emotional Resistance: Similar to how lottery players prefer to choose their own tickets in a bingo hall (even though it’s random), researchers prefer their efforts to be recognized by experts rather than relying on luck. Therefore, a completely random lottery is less acceptable, and most systems use a combination of review and lottery.
2. Public Perception: Politicians and taxpayers may view the use of public money for lotteries as irresponsible, and agencies fear criticism for being incompetent. For example, Australian funding agencies were very skeptical when this suggestion was first made.
3. Academic Concerns: French philosopher Bedessem argues that lotteries ignore the connection between a project and the existing body of knowledge. The value of a project depends on its relevance to existing research and the appropriateness of the methods used, but lotteries only consider whether an application meets the basic criteria. Current experiments have not addressed the key question of the long-term quality of projects funded by lotteries.
The Future of Lotteries
It’s still uncertain whether lotteries will become the norm, but the trend is that more institutions are willing to try them:
- The UK has extended the trial period until 2028, and the German model is being promoted.
- More than a dozen institutions worldwide have tested lotteries, and the RoRI (Research Institute) has compiled a directory of these efforts.
- A recent survey of 4,000 researchers asked whether they would accept lotteries, and the results are not yet available, but it indicates that the scientific community is seriously considering this option.
As statistician Barnett said, “Lotteries are an ancient method of ensuring fairness, but modern people always think they can make the best choices, even when evidence shows otherwise.” In the future, more institutions may seek a balance between expert review and lotteries to ensure both quality and address the issue of “pseudo-precision” in review processes.
(The entire text is written in plain language to make it understandable to readers without a financial or research background.)