Teaching evaluations part 2: nine solutions

They are not cynical about evaluations. The vast majority of lecturers are open to hearing how students feel about the education they’re receiving: after all, this can be highly instructive. The reason for the lecturers’ resistance to evaluations is the way in which these are conducted: the lecturer’s resent all those anonymous surveys and generalised questions.

The overall criticisms can be summed up as follows. Some students use the surveys to hurl abuse at the lecturer because they received a low mark for their exam or because they don’t like the lecturer’s appearance. Other students remain civil, but even so: how much should their judgment be valued? Sometimes it takes years for the true value of a subject to become apparent, even though you hated it at the time. Last but not least: far too few students complete the questionnaires and the answers are therefore often not representative of how most of the students experienced a course or subject.

In the meantime, however, many lecturers are confronted with the management’s judgement of their scores as well as possible consequences. The outcomes of the evaluations don’t just influence the lecturers’ moods, but they also impact their careers, as they are a part of performance reviews. To add insult to injury, many lecturers believe that the evaluations don’t even improve the quality of education. In other words: according to a great number of lecturers a change is needed, but how can this be achieved successfully?

Obviously, the most radical solution is to abolish teaching evaluations altogether. 18 percent of the respondents in our survey have indeed become so disillusioned with the whole process that they want to have assessments abolished completely: they believe that they’re not worthwhile, and couldn’t be made to be worthwhile even if they were to be restructured. “Abolish them!!!” one lecturer writes, complete with exclamation marks. “Humiliation by a group of lazy youngsters. There should be more respect for lecturers. We would like to have a go at reviewing the attitude of the students, their handwriting and their intelligence, but sadly we’re not allowed to.”

There are more responses of that sort, but fair is fair: most lecturers assume a more nuanced position. There definitely are criticisms that might be levelled at the evaluations. That doesn’t mean that they have to vanish entirely.

One of the lecturers remarks: “The students’ judgements can sometimes be determined by small events. This makes teaching an uncertain business. Even so, the quality of education and the attitudes of lecturers were a lot poorer in the time before evaluations were introduced.”

‘Don’t throw out the baby with the bathwater’ seems to be the gist of the views of most respondents. Reflecting on education won’t do any harm.

Trainer and university of applied sciences lecturer René van Kralingen has a notion of how things might be improved. First of all, you shouldn’t ‘just bombard students with forms’, you should teach them how to assess something, he thinks. “I deliver one-day courses to class representatives at various universities of applied sciences. I explain to them that education is different from ordering chips. It’s not about ‘consuming’. You’re not a visitor of a cafeteria with an opinion about your chips: when it comes to education, you’re a producer. The student and lecturer are both a part of the educational process: you’re frying chips together. For a functional assessment, it’s therefore important to not just listen to opinions regarding lecturers, but you also need to ask students: what have you done to ensure quality education? What efforts have you made, did you prepare properly?”

“Sometimes there can be a real culture of complaining among students,” says Van Kralingen, “and it’s rewarded by management too, as managers are highly focused on student satisfaction.” But students should understand their own role, according to Van Kralingen. “I always start off by saying that they may assess my performance afterwards, but that I’m also going to be assessing their efforts.”

Van Kralingen’s training solution might be a step in the right direction, but it won’t reassure all lecturers. A sore point with most of them is that the surveys are anonymous. This makes many lecturers feel like a target for students who cannot be held accountable. “Dispensing venom anonymously” is how one of the respondents describes the evaluations.

Some lecturers therefore support getting rid of the anonymity of the evaluations altogether, and instead just putting the name of every respondent with their specific evaluation responses. Marijtje Jongsma of academic union VAWO (Vakbond voor de Wetenschap), also a senior lecturer at Radboud University Nijmegen, is in favour of this solution. She believes that anonymous surveys are a ‘strange tool’.

In that she is not alone, but the problem is obvious: will students dare to speak freely in their evaluation responses if they have to provide their names? Student organisations attach great value to the anonymity of the evaluations. “The evaluations certainly need to remain anonymous, otherwise students might feel that they will be punished for their responses later on,” warns John van Harten, board member of the Dutch Student Union. “They shouldn’t have to fear giving their honest opinion if they have, for instance, had a discussion with a lecturer.”

Van Harten might be underestimating how outspoken those he represents are, as the experiences of lecturers seem different to his. Lecturers really appreciate being able to look a student in the eye. Marijtje Jongsma shares her experiences: “Non-anonymous panel discussions do constitute a viable option. All students are welcome to sign up and provide criticism. That’s a constructive way of doing things. I really like a critical stance and truly value constructive criticism, as it makes for better education.”

At large universities of applied sciences like Inholland, things indeed seem to be moving towards similar solutions. Assessments of education modules are important, according to chair Jet de Ranitz, but she doesn’t completely recognise the picture we’re painting. “It’s also a constant consideration for us: how do you organise quality assessments that allow for the student to provide feedback properly?” She would prefer lecturers and students to sit down together and assess the quality of education through a discussion.

De Ranitz puts forward the educational programme Social Legal Services as an example: its programme coordinator organises meetings with the lecturer and class representative for the express purpose of discussing the education module. This is appreciated, according to De Ranitz, but this system has not been implemented for every programme. De Ranitz: “You can really go in-depth by entering into a conversation. It allows you to ask questions like: ‘What do you mean by that?’ This is the type of assessment we’re developing.”

Respondents to the HOP survey add that teaching evaluations might ‘work well if you can discuss the evaluations with the relevant students afterwards’. Someone suggests: “Only use evaluations as a reason to enter into a conversation. Not as a yardstick without context.”

That last remark is almost an invocation: just remember, don’t place too much value on evaluations, keep a sense of proportion. And who do these remarks address? Not the lecturers themselves, nor the students. They’re meant for the managers.

Most of the various criticisms of the evaluations indicate how easily they can be misused. “Evaluations should be used to improve the quality of education, not as a big stick that ensures that they can get rid of you if they want,” says one respondent. Another respondent takes a firmer line: “The use of evaluations in annual performance reviews should be banned.”

It doesn’t seem that managers want to get rid of the evaluations, however. They like to emphasise the fact that they’re very reasonable when looking at the outcomes of surveys. As Henk Kummeling, rector magnificus of Utrecht University, already said: “If a performance culture arises in which lecturers are punished based on evaluation results, the evaluations should seize immediately. That’s not what they’re for.”

In fact, the evaluations should lead to consequences more quickly, according to Kees Boele, president of the executive board of the HAN University of Applied Sciences.

“Listen, you shouldn’t be led by the vagaries of the ‘consumer’. Overly positive teaching evaluations by students can also be a warning sign, as quality and student satisfaction are not one and the same. But I do believe that we can tighten things up a bit. In the educational system, we are quite lenient towards our employees, and we find it hard to draw inferences from evaluations. However, if a lecturer is not functioning properly, and not improving after coaching and guidance, there comes a time when management have to draw the line. Managers who abuse the evaluations in the case of a labour dispute? Perhaps it happens every so often, but I would say these are exceptions to the rule. Isn’t it a clear warning signal when a lecturer is afraid of evaluations?”

Similarly to the slogan of the American gun lobby, guns don’t kill people, people kill people, you could argue that it’s not the evaluations themselves, but the use of them that’s the problem. If you use them sensibly, there’s no problem at all.

Many lecturers tend to agree with that statement. “I am lucky that I have a reasonable manager who doesn’t take blind hatred all that seriously, should it ever turn up in one of the evaluations,” writes one of our respondents. But what if you’re not as fortunate? “Evaluations are published with the name of the lecturer at the top,” one respondent complains. “There is absolutely no regard for the privacy of lecturers.” Another lecturer writes: “These [evaluations] can influence whether someone gets a permanent position or not.”

“When the time comes for the results to be interpreted, not enough attention is paid to student numbers and course goals,” says another. “Remarks are copied without alteration, and this can have bizarre results. One student remarked that a course in literary history was pointless – mind you, this survey was completed by someone studying French Language and Culture.” There’s no point in getting defensive, she adds. “Whenever a lecturer comments on the results, it always feels like a case of ‘qui s’excuse, s’accuse’ [he who excuses himself accuses himself – Ed.] to me, especially if an exam is being discussed that was well crafted, but for which the students didn’t prepare properly.”

The way in which evaluation results are processed also influences how lecturers treat the feedback, according to Floris van der Burg, university lecturer at University College Utrecht and head of the philosophy department there: “You know what’s so annoying about this whole situation? Lecturers whose scores are on the positive half of the spectrum don’t give the evaluations a second glance.

Indeed, some lecturers seem to be getting weary of assessments altogether. “Ah well,” laments one of them in the survey. “You get the same responses each year, whatever changes you make, however clearly you communicate the course goals and explain your approach. What clearly emerges from the evaluations is that students are increasingly expecting higher-level education to be just like school: we’re increasingly getting judged on how explicitly we announce and lay down the course content; students want to be taken by the hand, and if you don’t do that, that’s mainly what your evaluations will be about. It’s not without reason that students are using terms like ‘school’ and ‘lessons’ more and more these days.”

The disaffection among lecturers is also fuelled by the fact that not all students complete the surveys. If you’re teaching thirty students and only the five disgruntled students complete the survey, this complete skews the results. Researcher Christine Teelken, who has studied the use of educational evaluations: “At the Vrije Universiteit we have discussed whether completing the surveys should be a mandatory task: in that scenario, students would only get their mark after completing their evaluations. But we felt that was taking it a step to far, so this plan was never implemented.”

A number of the respondents also made mention of mandatory evaluations. “If these evaluations are so important, they should be mandatory,” argued one of the respondents. The low response rate diminishes the credibility of the results. “It’s a bit nonsensical, too few students complete the surveys,” says one respondent, before adding that students don’t even know what the purpose of the surveys is. Even a respondent who is in favour of the surveys (‘They often provide useful insights or tips’) describes the response rate as ‘often too low or biased’. She also adds that the surveys are conducted too often. Students grow ‘tired of surveys’.

When challenged, managers will always say that are aware of the limits of evaluations, but in reality they still like to use them as a tool to keep a finger on the pulse. If someone’s results are systematically quite poor, you have to start asking questions.

Van der Burg is sceptical about the managers’ defence against allegations of misusing the results. “If you ask the managerial types, they will say that they’re monitoring the situation across multiple semesters and that the evaluation of students is only one part of all their considerations. But what else are they looking at? In the fifteen years that I’ve been working here, I have never had anyone come into my class to assess me. It could be that they only do such a thing if a lecturer is getting bad reviews, but in that scenario you can already guess what the outcome is going to be. The tone has already been set by the mere fact of someone coming into the classroom because the lecturer is not doing well. Do you really believe that the lecturer would be able to perform well in that scenario?”

Van der Burg has a different suggestion that has also been made by other respondents: “We should get rid of the current evaluations and develop a new instrument that only contains open-ended questions,” he states. “Every lecturer is guided properly, and a peer review system is put in place. It might take time, but it seems like the best solution to me.”

Peer review is common practice in academia: academics review the work of other academics (their peers), for instance to safeguard the quality of publications in a scientific journal. The same could be done for education, although this is not as common.

VAWO chair Marijtje Jongsma is sympathetic to the idea. “Peer review is the gold standard in academics. Why shouldn’t we implement it for education too? Every colleague is welcome to examine my type of teaching in a qualitative manner.” This proposal also comes up in the survey responses.

The proposal certainly doesn’t seem doomed to fail, if even managers such as Kees Boele of the HAN University of Applied Sciences think it’s a good idea. In fact, it’s already happening here and there. “It’s a good tool, as it allows colleagues to keep each other on their toes. I would really appreciate it if that was common practice among lecturers. Actually, I would prefer it, especially if you can also have open discussions with students to hear what they think of the education – something that’s already happening a lot. It does require the willingness and ability to ask one another for feedback.”

That ‘willingness and ability’ is not always there in all lecturers. As far as Boele is concerned, lecturers have a thing or two to learn from the way things are done in the business sector, where it is much more common (according to him) to assess each other. This would require a different attitude from lecturers.

Critics think that management would be better off searching its own heart. At the end of the day, the solution to the problem cannot be found in the system itself, according to VU researcher Teelken. Whatever system of assessment is selected, the outcomes can always be misused against employees.

Teelken believes that a culture change should start at the top. “Management likes to have control over professionals, but that’s not really a feasible situation for the long term. You have to dare to surrender control, even though that’s difficult and can cause anxiety, especially these days when nearly everything can be measured. How much attention should you pay to the evaluations? There is an imbalance at present.”