NVAO quality assessor unsure whether degree programmes have control over AI

How do students use ChatGPT when writing their thesis? That question receives ‘strikingly’ little attention during the assessment of degree programmes, according to the education quality assessor itself.

Degree programmes must be accredited in order to be allowed to award a recognised hbo or wo degree. This is regulated by the NVAO, the Dutch-Flemish Accreditation Organisation. Panels of experts visit institutions to assess their facilities, plans and education programmes.

The ‘visitation reports’ they produce have now been analysed by the NVAO on one specific topic: generative artificial intelligence. How do degree programmes deal with AI? What do they want to teach students about it? Are there clear rules? And do lecturers understand how they should work with AI?

Struggling

Attention to AI has risen rapidly, the NVAO observes in a ‘start exploration’. At the end of 2022, ChatGPT went online and turned out to be astonishingly good at generating texts. Degree programmes have been struggling with the consequences ever since.

Or are some perhaps struggling too little? In a further analysis of 194 recent visitation reports, the NVAO sees that university programmes in particular pay attention to AI. Hbo programmes do so ‘less’, the NVAO notes, without further explanation. A caveat: the situation may now be better, as quite some time passes between the visit by the experts and the final approval by the NVAO.

In any case, the NVAO sees that attention is mainly focused on the consequences for assessment. Degree programmes pay far less attention to the influence of AI on students’ future professions. Which competences should students learn if AI soon becomes a standard part of their work?

Quality of graduation work

The NVAO also calls it ‘striking’ that visitation reports do pay attention to assessment in general, but not to the graduation thesis. Both the degree programmes and the visiting expert panels do ‘not explicitly’ focus on the risks of AI for final projects. “From the visitation reports it cannot be determined whether degree programmes and their examination boards have control over this and whether they have had this discussion with the panel.”

The NVAO does not draw any further conclusions from this. NVAO chair Arnold Jonk writes in his foreword: “The call for frameworks and certainty that is often heard is a logical one, but we will probably have to live with uncertainty for some time yet.”

The exploration states: “Panels are not obliged to enter into discussions with degree programmes about the use of generative AI, nor do they have to report on this.” Meanwhile, the quality assessor is considering whether procedures around quality assurance should be more focused on generative AI. And if so, what the expert panels would then need in order to be able to judge properly.

AI literacy

Last year, the Education Inspectorate also raised the alarm. It saw a ‘major risk’ to the quality of graduation work. Examination boards find it difficult to detect or prove fraud using AI. And they do not have enough time to train themselves in the risks.

The NVAO agrees with this. It calls on degree programmes to improve the AI literacy of lecturers and to think more carefully about the role of AI in their education.