Grading an academic – now that’s a challenge

Booking.com, Airbnb and TripAdvisor – they all have review systems. As a guest, host or entrepreneur, these systems allow you to share your satisfaction – or lack thereof – with the world. But what’s the deal with review systems at Uni?

How can you determine whether a student deserves a diploma and isn’t slacking off in the lecture hall? In the Netherlands, lecturers rely on a grading system, which assigns students marks from 1 to 10. This evaluation runs via the Course and Examination Regulations (OER) and the examining board. “The lecturers are our professional experts,” says Brian Godor, Assistant Professor of Pedagogy and Education. “They know which subjects are important in their field, and by marking the students, they take stock of what they’ve learned during the course.” A student who makes a complete mess of it gets a 1, while a student who puts in a perfect job earns a 10.

But it isn’t that simple. Take the range of the marks themselves. According to Godor, lecturers in the Netherlands hardly ever grade lower than 3.5, or higher than 8.5. “Chinese students who come here struggle with this custom. In China, it’s possible to earn a 10 with your work. But in the Netherlands, a 10 is for God, a 9 for the Dean and an 8 for outstanding students.” In addition, hardly anyone is ever handed a 1 or 2 – even when they’ve botched it. “Lecturers are afraid that such a low mark only demotivates the student, but this assumption actually has no basis in fact.”

And the awarded marks may have a different weight depending on where you’re enrolled. It becomes clear from the research that Godor performed this year that faculties with stronger students – in academic terms – grade less generously than other faculties. The fancy word for this phenomenon is ‘gate keeping’. And as a student, you’re out of luck if the person on top of you on the pile scores exceptionally high. Because students who come immediately after a high-scorer can count on a stricter review than students in a random part of the pile.

Regardless of how important marks are for assessing a student’s abilities, they don’t tell the whole story, in Godor’s view. “Above all, you want to teach a student how to think in-depth, strategically and conceptually.” And the mark shouldn’t be all-important in that context. “Because a mark should never be used to penalise: it actually serves as feedback for the student. But at the same time, the government does require students to deliver solid results – via performance agreements, for example.”

And that’s the crux of the matter: as a lecturer, you don’t want to spend too much time on those performance agreements. “If everyone learns a lot, but is nevertheless awarded low marks, this leads to questions. It’s very difficult, in other words, to make an effective, balanced test.”

To successfully climb the academic ladder, it is important for researchers to publish articles in scholarly journals. This in turn relies on peer review: a fellow scientist in your particular field – or peer – evaluates your research and the paper you have written. This is often done anonymously. The big advantage: a pair of critical eyes that take a fresh look at research that you have been slaving away at for years.

But peer review also has its drawbacks. Writing these evaluations takes a lot of time. And some people don’t feel like doing this – meaning you can’t always find a suitable peer for a particular research paper. “And when criticism is unfounded because the peer reviewer lacks the expertise for a solid evaluation, things can get very complicated,” says Jack Vromen, Professor of Theoretical Philosophy and Professor of Economics at ESE.

Another issue is that researchers often work in a small field. “Even though the review itself is anonymous, you often know who wrote it.” For example, in Professor Rolf Zwaan’s field of research, Biological and Cognitive Psychology, you had two professors who couldn’t stand each other. Zwaan remembers how they would swear at one another at conferences. He needed to evaluate an article written by one of the two. “I wrote a positive review; the other reviewer a negative one. ‘Oh,’ was the professor’s response, ‘that must be that bastard I’m always arguing with.’ ‘No, it’s someone else,’ I said. I should have kept my mouth shut, because in that case he knew who it had to be. Meaning he now had a quarrel with two people.”

To avoid such problems, Zwaan is in favour of an open review system. This would allow everyone to access the reviewer’s various considerations and arguments. “Reviewers would pay more attention to spelling mistakes and would work to improve the paper by presenting strong arguments.” Another advantage is that you wouldn’t have to deal with mean-spirited comments like ‘A first-year student could write a better article,’ or ‘This is so bad it makes me want to throw up.’ Zwaan: “But this approach does take more time, which isn’t rewarded with university credits. Even though it does contribute to science – if you do it right.” So we have much to gain from this change, in his view.

Many scholars are in favour of open peer review, concludes Zwaan, based on a Twitter poll among 200 academics. But some remain wary nonetheless. “Young scientists in particular are scared to put their name under a review because they are afraid of the repercussions. They believe that a negative review will ensure that a professor obstructs them in the rest of their career. In some cases this fear is justified. In other words, you shouldn’t make open reviews with the peer reviewer’s name mandatory.”

Do you leave the lecture hall with an excited mind and a spring in your step? Or does your professor sap your strength with twohour-long, bone-dry monologues? These and other aspects are reflected in student assessments of lecturers’ teaching skills, handled via a digital evaluation form. The results are discussed in the programme committee. But the answers to the questionnaires don’t necessarily lead to better lecturers, according to educationalist Brian Godor, who is currently researching these evaluation systems. “The questions often focus on students’ level of satisfaction. But what are you actually assessing in these cases? We want to find out about students’ learning process and growth.” In addition, satisfaction doesn’t always reflect where there may be room for improvement, says Godor. “A student can also feel dissatisfied due to a variety of external factors, like the schedule, or to which extent he was interested in the subject itself.”

How many hours of you’ve studied – another question on the evaluation form – isn’t particularly revealing either. “Putting in many study hours doesn’t automatically result in a higher mark.” In short, the evaluation forms don’t adequately capture what defines a good course or lecturer. “We should first ask ourselves what we actually consider good education, and then determine how we can verify this.”

Interpreting evaluation forms is also a very challenging task. For example, response rates tend to be very low. Students Lisanne Notenboom (21) and Pim Somerwil (22) have never filled in an evaluation form because they always receive the email reminder during their finals. “And we don’t have time for them then,” says Somerwil. Notenboom: “And when you’re getting into a new subject, you no longer feel like doing it either.” Psychology student Elena Bartre (21) gets an email asking her to complete an evaluation too often: every five weeks. There’s not much in it for her personally, she says. “We’re always complaining about the same things. Some of the sentences in the exams haven’t been translated correctly into English. It never changes.” And she doesn’t believe that questions like ‘Describe the atmosphere in class” and ‘How many hours did you study’ make much sense. “I think it would be more useful if they asked me to which extent I learned something from the subjects that were dealt with.”

Juliette van Trigt (20) likewise only completes a form every now and then. “I don’t always: it’s too much hassle.” Although there are quite a few things that could be improved in her view: “Some of the lecturers don’t speak very clearly, and use a lot of terminology. One time, we had a guest lecturer who was going on 70 and wasn’t completely up to date on the subject. That was a drama to watch. But they never ask you about that kind of thing. So I’m not sure how much use it is to fill in those forms.”