Teaching evaluations part 1: the criticism

Nearly all educational programmes in higher education want to know how their students feel about the education they’re receiving: was the lecturer knowledgeable, are they a good teacher, was everything well organised? It might be a worthy aim to ask students about their opinion, but a lot of lecturers take issue with this practice.

Apart from the abusive tirades that they are sometimes confronted with or the criticisms regarding their appearance, lecturers mainly have a problem with the system itself. Most of them don’t feel that the evaluations are useful in their current form. HOP (Hoger Onderwijs Persbureau – Press Agency for Higher Education) researched the issue, and went in search of the criticisms and rebuttals as well as the solutions that might improve the current system of teaching evaluations.

An example? Some educational programmes ask students to comment on the level of English of their lecturers. “If a lecturer has a Dutch accent, they’re harshly criticised for that – even if the lecturer’s English is otherwise impeccable,” says Floris van der Burg, university lecturer at Utrecht University. “On the other hand, an Italian colleague receives enormous appreciation for his cute accent. What’s the point of this type of feedback?”

Van der Burg isn’t working with dummies either. He teaches at University College Utrecht, a small-scale honours programme that selects students for their talents and motivations. He is highly critical of the evaluations. “I usually get really annoyed reading the evaluations. As I’m the head of the philosophy department, I also read the feedback concerning my colleagues. Sometimes a big fuss is made about a train ticket for an excursion. Or students feel that they should’ve passed a subject they failed, even though they didn’t attend lectures and read none of the relevant books.

There are many stories similar to this one. “We often hear about criticism directed at anonymous teaching evaluations,” says Marijtje Jongsma of academic union VAWO (Vakbond voor de Wetenschap). She is a senior lecturer at Radboud University Nijmegen herself. “Sometimes students complete surveys while they’re hungover after a night out. They provide a few unconnected observations and you end up having to deal with those. It’s nauseating, I have witnessed lecturers being crushed by these evaluations. As the surveys are anonymous, there’s no chance of defending yourself. I find this truly bizarre and wonder if it is at all beneficial to the quality of education.”

Educator and higher professional education lecturer René van Kralingen, who provides training sessions on various universities of applied sciences, is highly critical of the assessments. “In the surveys, students are forced to respond to statements such as ‘the lecturer is enthusiastic, the lecturer is inspiring, the lecturer organises the educational content in an effective manner’. If, for instance, a lecturer encounters a faulty printer just before the lecture starts, they won’t be able to print their handouts. Or perhaps they have to provide an ‘inspiring’ lecture in an overheated building. There’s not much you can do about those sorts of things, but they will definitely be commented on in the evaluations. Imagine that you’re teaching statistics to students who have hardly finished puberty. Statistics! Especially the students’ assessments of difficult subjects make for sad reading. Students have gained more and more influence and are sometimes too quick to say that something is not right. There are times when I have to refrain myself from just telling them to shut up.” 

Through the years, the criticisms surrounding the evaluations have occasionally flared up, even if it was only in newspaper columns. In order to find out how widespread the support for those criticisms is, a survey was conducted among nearly three hundred respondents. The findings were quite interesting: lectures are critical, but not cynical about teaching evaluations.

One of the questions was whether the evaluations were at all worthwhile. More than half of all lectures indicated that the evaluations might be worthwhile if they were conducted differently. Thirty percent of lectures are already satisfied with the way things are going. Over one in six (17.6 percent) would prefer it if the evaluations were abolished.

(Read a complete report of the findings here, including the differences between men and women, universities and universities of applied sciences, and more.)

The possible career consequences of the surveys are a real thorn in the sides of the lecturers. A third of all respondents say that their jobs depend on the survey results, especially if they’ve only recently started teaching. Or they feel that they might not lose their jobs, but that the survey results will prevent them from being promoted.

These fears are not completely unfounded. “Here at University College Utrecht, you have to achieve a point average of four out of five in your evaluations,” says lecturer Van der Burg. “It used to be three and a half. If a student is too passive and gets a bad mark, they will give a bad review and harm the course or even the lecturer. Management is not interested in the answers given to open-ended questions, they only look at the point average. If a lecturer is assessed for the first time and gets a 3.2 because a few students felt that they had to work too hard, you can be sure that that lecturer will not be asked to return to teach again.” This might not result in them losing their job at Utrecht University, Van der Burg adds, but it will prevent them from teaching at UCU ever again.

“Maybe we should try assessing the quality of the course managers,” says university of applied sciences lecturer Van Kralingen scornfully. “Some of them have never even taught before, but still they keep on confronting lecturers with results from written evaluations without ever thinking about the consequences. Lecturers just have to try and defend themselves. Course managers also forget to take into account the experience of lecturers: you wouldn’t confront a pilot with assessment results if he or she flies a Boeing 747 for the first time.”

Christine Teelken (Vrije Universiteit), who has researched the use of educational evaluations, responds in a more reflective manner. Lectures have a certain degree of autonomy, but they have to answer to managers; Teelken is interested in the relation between autonomy and governance in education, and evaluations are significant in that regard. Their significance has even grown in recent times.

Teelken: “Ten years ago or so, the lecturer would occasionally hand out some forms and that was that. This has changed dramatically in the past five years, as these evaluations are now all conducted online. These days everything is monitored systematically, and the results are sent directly to lecturers and coordinators. The number of students who complete the surveys is dwindling, however. It has gone from eighty to thirty percent. Who are the people who still take the time to complete the surveys? Dissatisfied students, that’s who. They want to vent their views in the evaluations.”

The main issue here is that the educational quality does not correspond with the student’s experience at the time of the evaluation. “To me, this is the most interesting finding from my research,” says Teelken. “Lecturers believe that the added value of education cannot simply be measured. That value can often only become apparent years after the fact, when students realise why a specific course was useful to them. This is especially true for difficult subjects. The relationship between evaluations and quality is just very weak.”

Teelken believes that degree programmes mostly introduce or expand evaluation systems because of the whole ‘circus of external review’. Once every six years, programmes have to show an external committee that they are in control of the educational quality, so that they are ‘accredited’ once more by quality controller NVAO (Nederlands Vlaamse- Accreditatieorganisatie – Accreditation Organisation of the Netherlands and Flanders). “Programmes are highly motivated to maintain a good relationship with the committee. The committee doesn’t focus on student satisfaction per se: its task is mainly to monitor the system surrounding quality control.”

Teelken distinguishes between the primary feedback loop (students have justified criticisms; the lecturer makes an adjustment in his or her next course) and the secondary feedback loop (the organisation checks whether the lecturer has adjusted the course). “That second feedback loop is becoming increasingly more important because of the external reviews and it limits the autonomy of lecturers. The work pressure is already high, so lectures are not about to protest. They just feel that they have to put up with it.”

All in all, the evaluations don’t tell the whole story and many lecturers are afraid that their jobs depend on them – especially if they’ve just gotten started.
So why have these evaluations in the first place? Because you can also use them in a meaningful way. In fact, rector magnificus Henk Kummeling of Utrecht University can hardly believe that anyone working at his university would be so sharp in their criticism of the use of teaching evaluations. “I am familiar with the age-old question: what’s the point of all those assessments? I hadn’t heard responses like: let’s abolish them once and for all.”

Nobody is about to lose their job because of a few evaluations, Kummeling reassures his staff. “I am not trying to say that we should take evaluations with a pinch of salt, but I do think that we should put them into perspective. A lecturer’s contract should never be terminated solely on the basis of educational evaluations. They are a part of broader system at best. Evaluations are simply a part of a healthy quality culture in which we ask ourselves if we are on top of things, how a course is doing, if we are on the right track. If a performance culture arises in which lecturers are punished based on evaluation results, the evaluations should seize immediately. That’s not what they’re for and it would be highly impractical.”

Kummeling would be ‘very surprised’ if a UU programme would respond to evaluations like that. “In that case we would have to move quickly and open up a conversation with the relevant dean. Students aren’t the final assessors of how a lecturer functions. Nor can they assess if someone is truly competent or not.”

Kees Boele, president of the executive board of the HAN University of Applied Sciences, feels the same. He thinks that evaluations are very important: “As a lecturer you have an inadvertent tendency to create your own little fiefdom. Evaluations aren’t always enjoyable, but they play an important part in keeping you on your toes. I don’t mind admitting that I only figured that out when I started working in a commercial business. One of the company’s partners said to me: ‘Kees, you are good at your job’ – I started beaming straight away – ‘but you’re not professional’. He was criticising me for never asking for feedback. It was a hard lesson to learn and it still turns my stomach to think about it, but since then I have integrated assessment as standard practice.”

The position that Kummeling and Boele take wins approval from someone who, among other things, earns his crust with educational evaluations: Paul de Weger of the company Amplixs. He believes that evaluations can be very useful. “We started doing assessments for HU University of Applied Sciences Utrecht five years ago, and are now working with the Amsterdam University of Applied Sciences and Business School Netherlands. We offer bundles of different questionnaires, from which the institutes can take their pick. They might, for instance, want to focus on the enthusiasm of lecturers, their competence or on another theme altogether. A methodologist always monitors the process: what are we looking for and are we asking the right questions? Will we ask first years or students who will be graduating shortly?”

According to De Weger, the approach should be tailor-made, and no one should ever believe that an evaluation is completely neutral. “We are well aware that emotions might fuel the student’s answers, but that’s the whole point. If lecturers are usually assessed positively, the students will award significantly higher marks, as they also like the subject itself. That means that there is always a bias. On the other hand, a faulty scale can still register a trend. If my scale says that I weigh three kilos more or less than I actually do, I can still see if I gained or lost weight.”

It does mean that we have to be precise in our analysis. Are students even capable of commenting on the competence of a lecturer in a meaningful way? They don’t really know anything about that, do they? “This is true,” says De Weger. “On the other hand, if you’ve never been to or worked in a Michelin-star restaurant, you are still capable of passing judgment on whether the food is good or not.”

De Weger understands the criticism that evaluations are mainly a weapon in the hands of management. “I think this is a completely valid objection. They shouldn’t be used for personnel management; the results should be an indication of the quality of education.” And they can really add to that, De Weger feels. “The fact of the matter is that educational institutes are quite cumbersome organisations. Change-oriented people always hit a wall in educational environments. In those kinds of situations, evaluations can be a practical tool, as they provide insights into what’s going on in people’s minds and hearts. It’s not the most accurate method, but it does provide an indication.”

How do the students themselves view the evaluations? Their representatives in the Intercity Student Consultation and the Dutch Student Union don’t experience any real issues and prefer to stress the positive aspects of the evaluations. Take another look at the first example in this text regarding the lecturer whose Dutch accent in English is criticised, while the Italian lecturer’s accent in English is judged completely differently. “I can imagine that some people would call that arbitrary.” says Student Union board member John van Harten. “However, it might also be an indication that those students cannot really follow the lectures because the accent distracts them too much. That would also be realistic conclusion.”

The ISC shares the views of the Dutch Student Union. “Those we represent really appreciate those evaluations,” says board member Luc Rullens. “We understand the concerns of the lecturer, but we also see real value in the evaluations. Managers should look at the average and take notice of the outliers.”

What about the personal insults and the negative performance culture that seem to emerge here and there? Lecturers should just be more thick-skinned. “I have also heard about examples of students writing ridiculous things in evaluations or even resorting to swear words,” says Dutch Student Union member Van Harten. “Things shouldn’t be like that, but on the other hand, lecturers should know to ignore those kinds of comments.”

Rector Kummeling isn’t overly bothered by these examples either: “Obviously, you will always get some extremely crude, disconnected remarks about a lecturer, about their appearance or whatever, but you shouldn’t really take those things seriously,” he opines. “Those instances aren’t pleasant, but you have to be aware of working in a professional organisation. Perhaps it’s a sign of the times that people start saying negative things about others when they’re anonymous. These comments might hurt, but should be irrelevant for the lecturers concerned.”

Still, a majority of the lecturers feel that evaluations could be worthwhile, but in a different way. They are advocating change, but what is the result they have in mind? That will be discussed in the second part of this diptych.