Everyone knows them, the studies on Facebook and Nu.nl which provide food for thought at the coffee machine: if you don’t eat that biscuit, you’ll be better at your crossword, sticking a pencil in your mouth puts you in a better mood, and if you think of old people, you’ll automatically start walking more slowly. However, these studies, originating from social psychology, proved unrepeatable. The result: crisis.
Rolf Zwaan, Professor of Biological and Cognitive Psychology at the Faculty of Social Sciences, is exploring how social psychology can move on from this crisis. Besides his regular work as researcher and tutor, he uses replication studies to test how reliable previous research is. Furthermore, he is embroiled in the struggle between ‘old’ and ‘new’ social psychologists.
Pressure to perform
Let’s start at the beginning. What exactly is happening in social psychology? The issues emerged a few years ago. “In this domain, a number of high profile cases of fraud were discovered, with Diederik Stapel being the all time low,” says Zwaan, correcting himself immediately. “I should say partly suspected cases of fraud. Stapel was the only one to admit it.”
Rotterdam was affected too thanks to Dirk Smeesters. This Professor of Consumer Behaviour at Rotterdam School of Management no longer had his raw data available for two of his articles. “The former rector, Henk Schmidt, called me and said: ‘We’ve got our own Stapel’. He asked me whether I’d head the committee which was going to conduct enquiries into this case.”
And although cases of fraud are rare, they highlighted a structural problem. Many researchers apply — often unconsciously —statistical methods incorrectly. “Often, they looked for as many test subjects as possible and did as many experiments with them as possible. And then they said: “At the end of term, we’ll see what effects are significant.’ So they established the hypothesis afterwards. In other words: they fired first and then drew the target.” However, he doesn’t think that these scientists were all aware that they were being fraudulent. “In their environment, that was normal, that’s how you did things. They thought that they were doing the right thing.”
Rotten apple
How come? In the scientific world, there is huge pressure to perform, according to Zwaan. It’s all about publishing as much as possible. “The more papers you have, the better. Because the better your reputation, the higher your salary, etc.” Sometimes the production is even directly linked to the remuneration system. At Florida State University – where he worked for nearly thirteen years – the number of papers had a direct influence on your salary. “Are you in the top three of the department? 10 percent more salary. Lower? 5 percent. So one scientist had a big house with a lovely swimming pool while the other lived in a small apartment.”
Zwaan explains how it was when he left in 2007: “One person had a salary of 150 thousand dollars, while the other had 60 thousand. Yet they were doing the same job. That’s when you get people taking shortcuts when they analyse their data.” That perverse stimulus is reflected in every branch of society. Amused: “Even in bridge. You can earn money playing bridge too. Players cough in a certain way to let their partner know what cards they have. Cheating is a kind of doping.”
So is social psychology the rotten apple in science? Zwaan is adamant that this isn’t the case. The problems in social psychology are all on the surface, because it’s about research which is easy to replicate. Furthermore, ‘exciting’ conclusions are often reported in the media. “But there are also problems in neurosciences, politicology, biology, medical sciences, etc.” Research in these fields is just often more expensive and more difficult to repeat, so problems are less likely to come to light.
The Beatles
The crisis in social psychology has all the ingredients of what Zwaan calls ‘Shakespearean drama’. “Established social psychologists, who feel that their career or legacy is being negated just before they retire, are confronting the new kids on the block who see that the established order is winning awards in the wrong way.”
Luckily, there’s still hope. “A new generation is looking more critically at how research is conducted. So in a few years’ time, when the older generation has retired, the culture will change into one in which researchers are more careful about how they approach their work.”
That new generation will do a number of things differently, Zwaan summarises. The most important change appears to be a wide open door. “Draw up your hypothesis in advance and immediately record the criteria for when test subjects do and don’t count in your results.” As if Zwaan is trying to bang home his message, he repeats it a number of times in the conversation. Because reality proves to be stubborn. “There’s still a lot of resistance to this idea because it ‘could hamper creativity’.”
The remuneration system must also change. The magazine Psychological Science took an initial step in that direction. “That magazine had many articles which could not be replicated. So they changed their policy. Researchers must say in advance what their hypotheses are and get a special badge if they publish their data.”
He refers to the scientific philosopher Karl Popper, who once said: ‘An invention is only scientific if you can repeat it.’ “That must become more important, repeating research. The big magazines feel that research alone is not interesting enough. It is apparently nicer to be The Beatles than their cover band.”
On the one hand, there is lots of support for all these plans. For example, last summer a new scientific organisation was founded: The Society for the Improvement of Psychological Science. This or- ganisation already has over a thousand members.
On the other hand, some scientists become defensive when they find that their research can’t be repeated. “They say: those guys are incompetent and have bad intentions.” According to Zwaan, that’s never how replications are approached. “Even if I had negative intentions, I couldn’t do anything. Most replication work is outsourced to PhD researchers and student assistants. They don’t even know what I think about the study, whether I have an opinion. I only analyse the data.”
Football hooligan
He uses stern words when he talks about the ‘cultural change’ which is needed in science. But the sterner the words, the more he weighs them. Occasionally, there is a glimmer of doubt, then Zwaan reconsiders his words and takes them back. Sometimes, the professor utters a considered sneer, packaged in a mild tone towards the established order, which incidentally accuses the new kids in social psychology and relative outsiders like him of all kinds of things. The list of swear words is already long: “Bullies, nothing in our brains, nitwits, self-appointed sheriffs, data terrorists, you name it.”
You might think: why is cognitive psychologist Rolf Zwaan getting involved? “It started when I was questioned by the committee which was investigating the case of fraud at EUR. They needed someone from an adjacent subject area who had no prior interest in the story.” While investigating, he had a moment of insight. “I saw what was going wrong and how separate from reality science sometimes was.”
Zwaan illustrates that with Smeesters’ research, which was concerned with the phenomenon that you can answer quiz questions better if you see a photo of Einstein, than if you see a photo of a football hooligan. Smeesters continued along that vein. “If that experiment was in a red folder, people would do worse when they saw a photo of Einstein. If it was in a blue folder, it would go according to expectations. That was the moment when I thought: something needs to be put right here.”
Scepticism and nihilism
Of course, Zwaan could also be subject of replication attempts, but that doesn’t worry him. On the contrary. “I look forward to my work being replicated. Obviously, there will always be an error margin in my research. That’s how statistics are. And I’m no different from other cognitive psychologists. Because although the problem isn’t as big as in social psychology, not everything is replicated here either.”
What does this do to the person behind the researcher? Zwaan admits: “Yes, I have to be careful not to cross the boundary between scepticism and nihilism.” He’s found that he looks differently at studies, even those which don’t get in the news. “In the past, I used to read the results and thought: oh, good research, nice results. Now I also read the methods. Is everything correct?”
Zwaan started thinking about what science really meant to him. “What do I want to leave behind in the time that we are together here on Earth?” He states determinedly: “I’m no longer taking part in the race for who has the most papers. When I retire, I want to have left behind a better subject area.” Except that’s not always easy in an environment with a publication culture. He smiles. “I’m lucky in that I don’t have to focus too much on publishing.” Then frowning: “But I do worry about my subject area. I’m trying to improve it through education and new methods like publishing data.”
As many clicks
At EUR, things aren’t too bad, says Zwaan. “We don’t need to worry that we have a kind of Sodom and Gomorra here, although I obviously don’t know everything that happens.” As far as Zwaan can see, there isn’t anyone with an exaggerated drive to publish or who gets too much media attention. “Those are the main factors that make me sceptical.”
And finally: what should we do with all those studies on Facebook and Nu.nl? “It’s not just about the way in which science approaches things.” According to Zwaan, there are many more interests at stake. “A journalist wants as many clicks as possible and a university press department wants as much media attention as possible.” So, how much of what we read is true? Zwaan mentions figures, but takes them back later. “I have no idea. But approach them sceptically. If you are really interested, look at what the claims are based on.”
