Your experiment is groundbreaking, but will it succeed a second time?

Scientific research is often not reproducible. “That it was going so badly was quite an eyeopener for me”, says Michiel de Boer, founder of the NL Reproducibility Network.

Science can sometimes sound so simple. You mix two substances and the test tube explodes. You give a sailor vitamin C and he doesn’t get scurvy. No matter how often you try, the outcome is always the same.

But it doesn’t always work like that. Scientific studies sometimes produce different results when repeated. And often you cannot even repeat them. Who checks our scientific knowledge and, above all: is it even possible to check it?

In 2023 a network was set up that focuses on the reproducibility of scientific research. It started in Groningen; by now almost all Dutch universities are involved.

It’s the start of a cultural shift, is the hope of founder Michiel de Boer (associate professor of epidemiology in Groningen) and coordinator Daniela Gawehns (PhD candidate in data science in Leiden).

Why is scientific research often difficult to reproduce?

De Boer: “As a scientist you should describe your approach as clearly as possible. That is how we were all taught: in principle someone else should be able to repeat your research. But that is possible less often than you would like.”

The problem sometimes lies already in the description of the method, he explains. If you don’t know exactly how the research was carried out, you cannot repeat it properly. Information is often missing.

De Boer: “Journals often impose a word limit on publications, so you can never fully explain what you did. And sometimes researchers simply do not make their data or the questionnaires they used available.”

A part of the experiments can be repeated. Does that usually yield the same result?

Not at all, says De Boer: “Replication studies show that less than half of repeated studies produce the same results.” But these are usually a limited number of replications within a particular discipline. He can’t say for the breadth of science as a whole.

De Boer: “That this is going so badly was quite an eyeopener for me.” Also, the effects found are generally weaker than in the original study. It is so difficult that people speak of a replication crisis.

Should every study be repeatable?

“In any case it’s important to be transparent. Don’t just share your method and data, but also your analysis techniques, for example. Reproducibility will never be 100 percent, but it should be higher than 50 percent. Let’s say 80 or 90 percent. It may partly depend on the discipline and the context of the research.”

Gawehns: “It also partly depends on your definition of a successful replication. You will never get exactly the same outcome, because there is always noise in your research. Is a replication only successful if you reach the same p-value, or does the detected effect only have to point in the same direction? The latter is easier. In some disciplines you can say: on replication, the graph should look reasonably the same.”

What is the danger if studies prove to be non-replicable?

Gawehns: “For example, there was promising research into drugs for dementia. Millions of euros went into that. But the research turned out to be fraudulent, so it could not be reproduced. Then it is really a waste of money.”

De Boer: “Science should be cumulative: you build on each other’s insights. But if the foundation is wrong, you have a problem. Sometimes it takes quite a long time before it becomes clear that we are heading in the wrong direction. And much research is funded with taxpayers’ money.”

When does this affect everyday life?

De Boer: “Especially in the medical field it can do real harm if the effectiveness of new drugs proves not to be reproducible or if they even have negative effects. During the Covid crisis hydroxychloroquine briefly seemed like a good drug, but the research was shaky. It did more harm than good. Details were omitted and it was not clear where the data came from.”

And is that why there should be more attention for replication?

De Boer: “It’s one piece of the puzzle. Of course it’s not the only thing that matters in science. Sometimes you might replicate the results of poor research, but then you simply have two instances of poor research.”

Are you seeing more attention for the topic?

Gawehns: “On paper, everyone supports us. And we also see initiatives within institutes and research groups. Think of PhD candidates reproducing each other’s work. But how do we ensure that all those small projects and good intentions lead to reproducibility becoming part of our daily work?”

Does it, for example, play a role when you apply for a research grant?

De Boer: “There is some attention, but the system is certainly not perfect. If you submit a proposal to research funder NWO, you often have to say something about open science: how will you ensure other researchers have access to your data? That is connected to the reproducibility of research. But you typically spend ten or fifteen lines on it and then that’s usually fine. Those assessing your proposal mainly look at whether they find the proposal interesting. Moreover, this all happens only beforehand. There is little monitoring once the project is underway. That is problematic.”

Is extra oversight the solution then?

De Boer: “Universities can certainly do more. Policy documents often contain good ideas, but at organisational level they’re hardly being implemented. Then you depend on researchers’ intrinsic motivation to do things properly. We do see pockets of excellence, as we call them, but outside such a department nothing happens.”

Does it not also depend a bit on the discipline? I can imagine this topic is more relevant to quantitative research than, say, historical literary studies.

De Boer: “Yet one of our priorities is reproducibility for the non-quantitative research disciplines. I know two historians who do replication studies with their students. They examine how someone actually did the research. Are the footnotes enough to trace how people proceeded? Is their approach transparent enough? That produces good discussions.”

A new cabinet is coming. What do you expect from politics?

De Boer: “Work pressure is an important issue for us. One thing that prevents researchers from working with sufficient transparency is that it takes time. At least that is the perception. It certainly helps if there is some breathing space in the system. That said, there are also things we can do within science ourselves.”

What can science do itself?

De Boer: “We might perhaps publish a little less. Researchers are often judged on their publication list, but some of the articles are simply not that valuable. We can be more selective and then have time left to work more carefully. That insight is slowly taking hold.”