AI and Assessments: How AI will transform high-stakes assessments for the better

Recently, there’s been a lot of discussion around AI and assessments and whether AI will diminish the validity of the written examination? Can this use of technology be viewed in a positive light within the field of high-stakes assessments? The rapid advancements of AI language generators like ChatGPT have certainly provoked a wide range of responses among educators.

On the one hand, the abilities of AI chatbots in replicating human-like language have raised concerns about the future of essay-based learning, with nearly 1 in 3 students having used ChatGPT to assist with assignments. The potential of this technology to automate tasks that have traditionally been performed by humans is another consideration, with some worrying about the possibility of their own jobs being taken over by algorithms.

On the other hand, AI has generated excitement around the ways it might be used to help learners and instructors achieve more. For assessments used to measure the competency of professional roles within high-stakes industries, such as finance and law, it’s certain to be transformative.

In February, researchers put ChatGPT through the United States Medical Licensing Exam (USMLE), and it passed or nearly passed all three components. Earlier this month, ChatGPT-4 passed the Uniform Bar Exam with a score that approaches the 90th percentile of all test takers. Such results may cause a short-term retreat from remote proctoring, as the fear of AI-powered cheating prompts suppliers to revamp their security protocols – and this, in turn, would undermine accessibility. In the long run, however, the rise of AI tools may change assessments in ways that benefit students, educators, employers and the general public.

As a leading assessment and qualification provider, we recognise that remaining agile and embracing the positive impact of AI is critical in this ever-evolving landscape. Namely, a movement away from written assessments, which appear to be falling short in their ability to accurately measure competence, and a shift towards a more practical, skills-based model.

At Kaplan Assessments, we are experts in designing, developing, and delivering professional assessments that are tailored to the specific needs of your organisation. Get in touch with our team today.

How AI and assessments can help to close the skills gap

Prior to the ChatGPT-inspired AI frenzy, assessments were already due for an overhaul – particularly those for highly regulated professions where reports of quality failures appear to be on the increase. We hear from employers time and again that employees are arriving without the day-one competencies their jobs require, despite passing their assessments. They may be brimming with descriptive knowledge — the principles and theories that underpin the profession — but they often lack the ability to apply knowledge and skills in a professional context in order to perform the job well. They’re like a quiz show contestant who’s memorised the encyclopedia but can’t seem to work the buzzer.

The responsibility to close this gap has long been laid at educators’ feet. If students graduate from professional programs lacking in competencies, the thinking goes, then educators must change what they teach and how they teach it. That common prescription ignores an important fact of professional education: Educators will teach to the test—in this case, a high-stakes assessment—whether they want to or not; indeed, their very livelihood depends on it! If we’re going to close the skills gap, we must first look at the assessments.

Many assessments rely on written tests that emphasise descriptive knowledge. It’s a format that’s more susceptible to AI language generators. But more importantly, it doesn’t assess for a host of crucial professional competencies such as engaging with clients, applying ethical concepts, or exercising effective judgment when problem-solving. Fortunately, those competencies are virtually impossible to fake with AI.

AI and assessments: Can the most important professional skills be faked with AI?

When it comes to high-stakes industries, the emphasis should always be on the practical application of knowledge and skills. These can’t be faked with an AI-generated exam answer. Revisiting assessments to protect against high-tech cheating may therefore shift the focus onto this real-world approach, allowing educators to spend more time preparing their students for on-the-job challenges and less time preparing them for essay questions. The students will be better prepared for their chosen professions; employers will be more confident in their new hires’ practical competencies; and, ultimately the public will enjoy access to better services.

The Solicitors Qualifying Examination (SQE), launched in England and Wales in 2021, is the kind of holistic assessment that is both resistant to AI-assisted cheating and poised to close the skills gap. Aspiring lawyers taking the SQE must go through a two-part assessment: a multiple-choice test that assesses the application of legal knowledge and simulation exercises, both oral and written, which test skills as well as knowledge. Candidates conduct interviews with actors who role-play as clients, make arguments to qualified solicitors role-playing as judges, and draft legal documents and research cases.

In presenting a scenario that simulates the professional environment, the SQE model takes a similar approach to clinical skills tests for physicians. They too use actors trained in live patient scenarios to test medical students and graduates on their skills in data gathering, questioning, information sharing, professionalism and rapport – essentially assessing how well they engage with patients. Applying such a model to the SQE enables examiners to test for a full range of relevant, practical skills in a way that virtually eliminates the possibility of AI cheating and encourages candidates and educators to refocus their preparation. The successful adaptation of this approach has promising potential for other industries which require professional assessment.

That’s where we come in. Our team partners with organisations of all shapes, sizes, and sectors to create tailored assessments that set the benchmark for quality in the workplace and promote ongoing professional development. Our focus is on addressing the real-life hurdles faced on a daily basis. After all, the ultimate goal of any professional assessment should be to prepare the candidate to perform their role efficiently, thereby raising the standards of the wider organisation. A well-written essay will only get employees so far…

AI language generators are already changing education, and it’s difficult to predict where all these transformations will take us. But if AI prompts us to refocus high-stakes assessments on applied knowledge and practical, on-the-job performance, then all stakeholders stand to gain.