In this week's episode, Tim Burnett chats with Sergio Araneda, Karl Hartley, Miia Nummela-Price, and Neil Wilkinson as they launch a groundbreaking Research Sprint exploring the impacts of using AI to author assessment items. Sergio is a Research Scientist at Caveon specialising in test security with a PhD in psychometrics. Karl is Director of AI Learning Strategy at Epic Learning in New Zealand, having just completed the country's largest research into generative assessments. Miia is a Fractional General Counsel and legal consultant advising on the intersection of technology, education and law. Neil is Director of Product and Innovation at Certiverse, bringing extensive experience from creating high-stakes assessments at Pearson. They discuss:
- Legal implications and copyright challenges: ownership of AI-generated items, liability considerations when items prove flawed, and the evolving regulatory landscape
- Quality assurance and the human element: how AI is transforming item authoring workflows, the critical ongoing role of Subject Matter Experts (SMEs), and why treating SMEs well remains essential despite automation
- Test security vulnerabilities: pre-exposure risks specific to AI-authored items, benchmarks and rational expectations for LLM capabilities, and the importance of testing non-deterministic AI outputs multiple times
The panel explores the practical realities organisations face today as AI tools are already being deployed in classrooms and examination halls worldwide. Karl shares insights from his 114-page research report on generative assessments, whilst Sergio emphasises the importance of understanding LLM training and benchmarks to set realistic expectations. Miia highlights that legal frameworks are still developing and these conversations will continue for the foreseeable future as case law evolves. Neil stresses the value of breaking down AI tasks into smaller, focused prompts rather than creating massive verbose instructions, and advocates for using AI in review roles alongside human authoring. The discussion emphasises that this isn't about replacing humans but augmenting their capabilities, with the human remaining firmly in the loop for validation, construct validity and cultural fairness considerations. Tim introduces the Research Sprint platform where the assessment community can continue the conversation asynchronously, share insights and collaboratively develop practical guidance on AI authoring impacts.
Connect with Sergio Araneda on LinkedIn: https://www.linkedin.com/in/sergio-araneda-13994a16/
Connect with Karl Hartley on LinkedIn: https://www.linkedin.com/in/karlhartley/
Connect with Miia Nummela-Price on LinkedIn: https://www.linkedin.com/in/miianummela-price/
Connect with Neil Wilkinson on LinkedIn: https://www.linkedin.com/in/neil-a-wilkinson/
Connect with Tim Burnett on LinkedIn: https://www.linkedin.com/in/tburnett/
Resources mentioned in this episode:
Join the Research Sprint: https://research.testcommunity.network/
Karl's Research on AI-Generated Assessments: https://concove.ac.nz/discovery-hub/ai-generated-assessments-for-vocational-education-and-training/
AI Adoption Playbook: https://testcommunity.network/landing/aiplaybook
Special thanks:
Stuart Martin: https://www.linkedin.com/in/sgamartin/
Episode Sponsor - Vretta: https://www.vretta.com/
Connect with the Test Community Network:
If you would like to join Tim on a future episode or sponsor the Test Community Network then get in touch with Tim to schedule a call: https://calendly.com/educationtech/15min-teams-chat
Events: https://www.testcommunity.network/upcoming-events
Podcasts: https://www.testcommunity.network/podcasts
Marketplace: https://www.testcommunity.network/marketplace
Subscribe to the Test Community Network here: https://www.testcommunity.network