Reputable scientific organisations across the globe are starting to scrutinise their own processes. This is largely due to society’s urgent need for scientific responses to challenges such as climate change, prompting research financiers to accelerate the innovation process.
However, questions arise about how to boost innovation? Should research grants be optimised? Does the increasing mean age of scientists pose a problem? How should scientific publication be configured? What is the most effective approach to arranging research groups?
When faced with complex queries, scientists commonly resort to the scientific method. Research leads are no exception to this, as curious as top scientists, they’re “applying science to science itself”. They are deriving solutions through the collection of data, statistical analysis and experimentation, the regular scientific toolbox.
The deployment of scientific instruments to enhance funding structures isn’t novel. Throughout the mid-1990s, thorough experimentation reformed global development strategies. Abhijit Banerjee, Esther Duflo, and Michael Kremer, the trio economists behind this transformation, were honoured with the Nobel Prize in Economics in 2019. They were awarded for their revolutionary “experiment-based approach” to procuring trustworthy answers regarding optimal methods to combat worldwide poverty.
This method diverged significantly from traditional techniques. In the past, economists operated similarly to investigators hunting for hints. They would collate data – teacher qualifications, availability of textbooks, school meal programmes – and determine if these factors could be linked to good marks and test outcomes. They endeavoured to establish connections.
By contrast, Kremer and his team targeted more selective questions. Rather than querying, “Which input results in good grades?”, they devised field experiments to solve “Do educational resources enhance pupil outcomes?” Certain schools were randomly selected to get extra resources, while others received the standard support. The outcomes were assessed and the research group was able to generate direct conclusions applicable to educational policy.
Peer review processes can categorise proposals as ‘good’ or ‘bad’, however, they lack the capacity to rank them. Alternative options such as industry panels, lotteries and positive discrimination could be just as effective.
The outcome was astounding. Traditional studies suggested that student materials enhanced test scores, while the randomised experiments identified no significant connection. Further experiments unveiled that organising students by skill rather than age was significantly more effective in enhancing test scores. After these results were divulged, the “teaching at an appropriate level” policy has “amplified learning opportunities for over 60 million students across India and Africa”.
It should be acknowledged that the very basis of pharmaceutical research, randomised clinical trials, employ a methodology identical to the one being discussed. This experimental design, which yields results with a high degree of statistical certainty, enables researchers to establish a cause-and-effect relationship: a cure for a disease if a certain drug is administered or a fixed output Y given a specific input X. This methodology was creatively adopted by Kramer and fellow Nobel laureates and transferred from the confines of a lab to the broader research landscape. Consistently, research sponsors have now begun to transform their agencies into laboratories in the same way.
On the same note, there’s a surge among those who fund research to experiment with their project funding methods. One case in point is the US National Institutes of Health (NIH) which has launched a trial grant aimed at easing the administrative load of leading scientists. Presently, these high-calibre scientists in the US devote nearly half of their time to administrative duties and maintenance rather than conducting scientific research.
The Maximising Investigators’ Research Award (MIRA R35) is a pilot programme that’s designed to “boost the efficiency of funding by providing investigators with an improved sense of stability and flexibility”. The objective of this scheme is to amplify scientific productivity and elevate the likelihood of major discoveries.
MIRA is also addressing another issue confronted by the NIH: the upward trajectory of the average age of principal investigators, from 39 in 1980 to 51 in 2008. Considering the fact that 78 per cent of Nobel Prize recipients between 1980 and 2010 conducted their pivotal research before reaching 51, the rising age points to a displacement of groundbreaking research. To counter this, a pilot project, “MIRA for Early Stage Investigators”, has been put in place to benefit young, driven researchers.
One of metascience’s core pursuits is the comprehension of efficient resource allocation. The “science of science” can disclose the most effective strategy – whether it be funding individuals or entire projects, employing teams or individuals, or opting for rigorous applications or swift grants. The traditional practice of peer review, wherein scientists assess each other’s proposals and critique each other’s results, has been the basis for making these decisions. This too, is being reformed under the auspices of metascience.
The process of peer review, although influential, is not without its notable drawbacks. The system, originally created to differentiate between superior and inferior scientific studies, now also plays a pivotal role in classifying high and low-grade scientific discoveries, a function in which it thrives.
However, the complexity of the system comes to light when attempting to distinguish with subtlety. Fund-seeking projects generally fall into three classes: exceptional (funded), inadequate (dismissed), or uniquely ordinary. Classifying projects within these divisions poses a considerable obstacle for peer review, where a clear standpoint on a proposal can be diluted by the biases of the reviewing individuals. The influence their prejudices, dictated by factors such as age, gender, institutional background, or ethnicity, carry over the final verdict can be substantial.
Research into the science of science conducted by Mike Lauer, NIH’s deputy director for external research, precisely measured these deficiencies in peer review. He discovered that while peer review is effective at broadly categorising (as either praiseworthy or dismissible), it lacks the capability to grade nominations according to merit. Evidently, alternative strategies could be just as practicable. Business panels, random selection, and affirmative action all could potentially serve the same purpose.
This field, known as metascience, not only recognised this problem but also proposed possible resolutions. The Swiss National Science Foundation (SNSF) ventured into the unprecedented method of lottery-based decision-making for funding allocations in 2018. If funding requests received equal scores, a game of chance was employed. This method was initiated in their postdoctoral transfer programme. Post deliberations, the Swiss National Research Committee allowed evaluators to use the lottery system, when necessary, across their full array of funding programmes. By 2021, every deadlock was settled by chance. Prof. Matthias Egger, the Council’s president, notes that “when such situations arise, the lottery system is the most equitable solution as it is inherently impartial and inhibits biases.”
Another flaw in the peer review system comes into view when it appears to be more inclined towards gradual research over audacious yet uncertain concepts. This cautious nature is a result of the deep divides on peer review committees regarding the worth of significant proposals. Plans need to secure a dominant support base before funding can be granted. Visionary proposals with radical implications but minimal success probability usually only attract a single and inadequate supporter on the review panels.
Metascientists, once more, are testing potential solutions. The US National Science Foundation is cooperating with the Institute for Progress to examine the concept of “golden tickets” with the aim of guaranteeing the survival of high-risk, high-reward proposals during the peer review process. This system would allow each reviewer to override a majority decision once per funding cycle. Nature mentions that this model could theoretically fund risky, out-of-the-box proposals lacking unanimous approval from all referees. By conducting pilots and trials, similar to the international development experiments of Kramer et al, the effectiveness of this approach will be assessed.
Executing these trials wouldn’t be expensive. A yearly pilot involving around 24 researchers could generate significant data over time. At the conclusion of a five-year experiment, more than 100 data points could have been gathered, providing concrete insights. One of the key benefits of performing these trials is the improved comprehension of the entire research system, leading to more precise and impactful enhancements.
An observable increase in aspiration and investment, from a number of countries, in metascience from governments, funding bodies, and foundations has been pointed out by James Wilsdon, Executive Director of the Research on Research Institute (RoRI). Wilsdon further observes a growing number of researchers from diverse disciplines who are utilising innovative methods from their respective fields to assess and enhance the effectiveness of the research systems and cultures they operate in.
The RoRI has produced the “Experimental Research Funder’s Handbook”, a useful resource for metascience experiments, compiling information on international trials. The handbook explores the design and execution of experiments related to funding processes. A newly established metascience unit of the UK government, backed by £10 million, will use the guide across the country’s innovation ecosystem. Yet another article by Nature poses the question of when other nations will emulate this. Wilsdon suggests: “With the inauguration of Research Ireland, Ireland stands a prime chance to place itself at the forefront of the global metascience gathering”.
It’s recognised by these funding bodies and governments that metascience, by enhancing the effectiveness of scientific research, expedites progress towards resolving society’s urgent issues. Metascience is a priority to countries anchoring their future affluence to knowledge economies, a notable case in point being Ireland.
The future of science is driven not only by the subjects of our research but also the methodology. The potential rewards, for individual nations and global advancement, are too significant to be neglected.
Luke Fehily holds the position of innovation policy director at Progress Ireland, an autonomous think tank dedicated to linking Ireland with database-dependent policy resolutions around the globe. Before this assignment, he functioned as an open innovation engineer for Johnson & Johnson.