From AI Detectors to Heartwarming Help: Two Tales of AI in a school

Series 8 Episode 11

In this episode of the AI in Education podcast, hosts Ray Fleming and Dan Bowen interview Martin O'Sullivan, a UK headteacher and lifelong friend of Dan. They talk about two contrasting stories from Martin's school—one highlighting the frustration in the application of AI when students were accused of cheating, and the other illustrating the compassionate use of AI to help a student undergoing cancer treatment participate in classes remotely via an AI-enabled robot. The discussion emphasises the complexities and ethical considerations of integrating AI in education.

00:00 Introduction

02:32 Martin's Background and School

03:50 AI in Education: Martin's Journey

05:59 Challenges with AI Detectors

09:23 Emotional Impact on Students

11:45 Reflections on AI in Assessment

We discussed two assessment topics that have been in previous episodes:

• Sydney University's approach to assessment with Lane 1 and Lane 2 assessments - in Series 8 Episode 2
• The AI Assessment Scale research - in Series 8 Episode 7

22:34 Heartwarming Story: AI Robot for Student

There's more reporting on this story on the BBC website

27:10 Conclusion and Final Thoughts

Another episode rounding up the latest news and research on AI in Education. The links below go straight to all the news stories and research papers discussed this week

NEWS Victorian "Generative Artificial Intelligence Policy" for government schools.

https://www2.education.vic.gov.au/pal/generative-artificial-intelligence/policy

Meeting the AI Skills Boom

https://techcouncil.com.au/wp-content/uploads/Meeting-the-AI-Skills-Boom-2024.v2.pdf

LAUSD shelves its hyped AI chatbot to help students after collapse of firm that made it

https://www.latimes.com/california/story/2024-07-03/lausds-highly-touted-ai-chatbot-to-help-students-fails-to-deliver

A class above: UNSW Sydney uses AI to power personalised paths to student success

https://news.microsoft.com/en-au/features/a-class-above-unsw-sydney-uses-ai-to-power-personalised-paths-to-student-success/

Research Detecting ChatGPT-Generated Essays in a Large-Scale Writing Assessment: Is There a Bias Against Non-Native English Speakers?

https://www.sciencedirect.com/science/article/abs/pii/S0360131524000848#bib23

GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education

https://arxiv.org/abs/2403.19148

Avoiding embarrassment online: Response to and inferences about chatbots when purchases activate self-presentation concerns

https://myscp.onlinelibrary.wiley.com/doi/10.1002/jcpy.1414

Navigating the Ethical Landscape of Multimodal Learning Analytics: A Guiding Framework

https://osf.io/preprints/edarxiv/adxuq

How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses

https://arxiv.org/pdf/2405.00970

AI Conversational Agent Design for Supporting Learning and Well-Being of University Students

https://osf.io/preprints/edarxiv/w4rtf

The Neglected 15%: Positive Effects of Hybrid Human-AI Tutoring Among Students with Disabilities

https://osf.io/preprints/edarxiv/y52ew

The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances

https://osf.io/preprints/osf/qy8zd

The Future of Feedback: Integrating Peer and Generative AI Reviews to Support Student Work

https://osf.io/preprints/edarxiv/x3dct

Is ChatGPT Transforming Academics' Writing Style?

https://arxiv.org/abs/2404.08627

Can AI Provide Useful Holistic Essay Scoring?

https://osf.io/preprints/osf/7xpre

Read the excellent article about this paper in the Heching Report

Best Practices for Using AI When Writing Scientific Manuscripts

https://pubs.acs.org/doi/epdf/10.1021/acsnano.3c01544

A real-world test of artificial intelligence infiltration of a university examinations system: A "Turing Test" case study

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0305354

Continuing our Series 8, focused on the topic of assessment, and the impact of AI, roving reporter Ray visits Fitzroy, Victoria, to interview Dan Ingvarson, a notable founder and executive consultant in the field of AI in education, whilst co-host Dan (Welsh Dan?) joins remotely. Dan (Ingvarson) discusses his background and extensive work in AI policy and education technology, touching upon his involvement in various global initiatives including the Edsafe Alliance and Educational Services Australia. The conversation looks into the complexities of modern educational assessment, focusing on how generative AI can transform traditional assessment practices. He highlights the obstacles teachers face with current assessment methods and how AI could help alleviate some of these challenges by offering more personalised and authentic evaluations. The podcast also touches on the potential for AI to support teachers in creating effective rubrics and improving the overall quality of education through better data analytics. Ray and (Welsh) Dan emphasise the importance of evolving curricula to incorporate these advanced AI methodologies while discussing broader implications for student learning and teacher workload. 00:00 Introduction and Welcome 00:47 Meet Dan Ingvarsson: AI Guru 02:26 The Evolution of AI in Education 03:46 Deep Dive into Assessment 06:46 Challenges and Opportunities in AI-Assisted Assessment 19:21 Ensuring Authenticity in Student Work 23:28 Future of AI in Education 31:31 Practical Tips for Teachers 33:47 Conclusion and Final Thoughts

Content warning! This episode talks about an academic research paper titled "ChatGPT is bulls**t", and we've not edited the word out - in fact, we've gone to town with it, talking about the different types of it (in the strictest academic sense). So you may not want to play this in the car on your school run!

The news item discussed is:

Student crafts elaborate AI scheme to pass university exam, gets arrested

https://cybernews.com/news/turkish-student-found-using-ai-arrested/

This week's papers discussed are:

Developing evaluative judgement for a time of generative artificial intelligence

https://www.tandfonline.com/doi/full/10.1080/02602938.2024.2335321

Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization

https://arxiv.org/abs/2404.04941

Working Alongside, Not Against, AI Writing Tools in the Composition Classroom: a Dialectical Retrospective

https://uen.pressbooks.pub/teachingandgenerativeai/chapter/working-alongside-not-against-ai-writing-tools-in-the-composition-classroom-a-dialectical-retrospective/

GPT versus Resident Physicians — A Benchmark Based on Official Board Scores

https://ai.nejm.org/doi/pdf/10.1056/AIdbp2300192

Evaluating General Vision-Language Models for Clinical Medicine

https://www.medrxiv.org/content/10.1101/2024.04.12.24305744v1

Re-evaluating GPT-4's bar exam performance

https://link.springer.com/article/10.1007/s10506-024-09396-9

Automated Social Science: Language Models as Scientist and Subjects

https://arxiv.org/abs/2404.11794

Large language models cannot replace human participants because they cannot portray identity groups

https://arxiv.org/abs/2402.01908

I also mentioned the article about the the Infinite Focus Group, which is here:

https://www.ai-mindset.ai/ai-mindset-newsletter/the-infinite-focus-group

The impact of large language models on university students' literacy development

https://www.tandfonline.com/doi/epdf/10.1080/07294360.2024.2332259?needAccess=true

Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays

https://www.sciencedirect.com/science/article/pii/S2666920X24000109

Feedback sources in essay writing: peer-generated or AI-generated feedback?

https://link.springer.com/article/10.1186/s41239-024-00455-4

ChatGPT is bullshit

https://link.springer.com/epdf/10.1007/s10676-024-09775-5?sharing_token=0CIhP_zo5-plierRq8kkDPe4RwlQNchNByi7wbcMAY77xTOWyddkW01qGFs1m5zuuoZGBctVlsJF8SbYqcxWi-XzgEYEPiw7xwWi4bMYXJ_1JARDrER9JGdWZOW-UGSkrk_tXPjPh-XWvFNoiFzNlnDUUUEBAztiX9PtP2p6jfI%3D

Wow, this week we have a bumper episode with more resources than a GPT factory! Any time we get a guest making their second appearance, and therefore enter our Hall of Fame, then we officially dub them "Friend of the Show". And so this week, we've got Friend of the Show Leon Furze sharing his experiences and expertise.

Here are the links and posts related to our conversation with Leon around assessment.

Leon's blog - for all the updates and posts that he is working on https://leonfurze.com/blog

Leons's free e-Book on assessment can be found here: https://mailchi.mp/leonfurze/assessment (free ebook on assessment)

Leon Furze Linkedin profile is here if you want to follow his stream of thoughts, and to connect with him: Leon Furze - Furze Smith Consulting | LinkedIn

The Artificial Intelligence Assessment Scale (AIAS) paper we discussed can be found here: https://open-publishing.org/journals/index.php/jutlp/article/view/810

The online course around practical AI strategies:

https://practicalaistrategies.com/p/practical-ai-strategies

And, even better, he's given listeners a discount code that will save you 25% of the cost. Just use the magic word 'AIPODCAST'

The blog post we mentioned a couple of times during the episode:

https://leonfurze.com/2024/05/27/dont-use-genai-to-grade-student-work/

Leons new book and course can be also found on his main site here:

https://practicalaistrategies.com/

This week we set the episode timer for 15 minutes, and managed to get through just five papers before the buzzer went off! So we have plenty more papers to discuss in future episodes...

ENHANCING K-12 STUDENTS' PERFORMANCE IN CHEMISTRY THROUGH CHATGPT-POWERED BLENDED LEARNING IN THE EDUCATION 4.0 ERA

https://library.iated.org/view/ORTIZDEZARATE2024ENH

Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study

https://bera-journals.onlinelibrary.wiley.com/doi/pdfdirect/10.1111/bjet.13454

ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature

https://arxiv.org/abs/2403.16887

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

https://arxiv.org/abs/2403.07183

Large language models are able to downplay their cognitive abilities to fit the persona they simulate

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0298522

Assessment - Chris Goodall

In this episode of the AI Education Podcast, host Dan converses with Chris Goodall, the head of digital education at the Bourne Education Trust in England. They discuss the integration of AI into education, how it can be used to enhance teaching and learning processes, and the impact of personalized AI tools on students and educators.

The conversation covers practical applications of AI, the ongoing need for teacher and student adaptation to new technologies, as well as ethical considerations and future possibilities for AI in education.

Chris Goodalls Linkedin profile here: Chris Goodall | LinkedIn

Practical Advice for embedding IT in school: Embedding AI use in school

Research Update - 31st May 2024

Honestly folks, we've been trying to keep. We really have. But we have so much great content in the fortnightly (or is it bi-weekly?) interviews, that we've had to bite the bullet and switch to weekly podcasts, so that we can still fit in the Research Updates!

Going forwards you'll get a longer interview-style podcast once every two weeks, and a shorter 15-20 minute "Research Update" podcast every two weeks. Filling your Fridays with AI in Education podcast joy!

Here's the links to all the research papers discussed this week:

Remote Proctoring: Understanding the Debate

https://link.springer.com/referenceworkentry/10.1007/978-3-031-54144-5_150#DOI

Large language model-powered chatbots for internationalizing student support in higher education

https://arxiv.org/abs/2403.14702

ChatGPT in Veterinary Medicine: A Practical Guidance of Generative Artificial Intelligence in Clinics, Education, and Research

https://arxiv.org/abs/2403.14654

Investigation of the effectiveness of applying ChatGPT in Dialogic Teaching Using Electroencephalography

https://arxiv.org/abs/2403.16687

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

https://arxiv.org/abs/2403.18679

An MIT Exploration of Generative AI

https://mit-genai.pubpub.org/

MIT - Massachusetts Institute of Technology - have just published a series of really interesting papers about the impact of generative AI on a number of industries, and dive into the implications for society, education, human interaction and other areas. I actually think the whole set are interesting - and they're really easy to get - you can read them on the web, or get a PDF, an ebook, or even an audio book of every one!

We talked about the 3 education ones:

When Disruptive Innovations Drive Educational Transformation: Literacy, Pocket Calculator, Google Translate, ChatGPT

https://mit-genai.pubpub.org/pub/6chtnd56/release/3?readingCollection=0e231e9c

Generative AI and K-12 Education: An MIT Perspective

https://mit-genai.pubpub.org/pub/4k9msp17/release/1?readingCollection=0e231e9c

Generative AI and Creative Learning: Concerns, Opportunities, and Choices

https://mit-genai.pubpub.org/pub/gj6eod3e/release/2?readingCollection=0e231e9c

This week we continue our series on Assessment and AI. Ray talks with Jason Lodge from The University of Queensland, and who must have the longest business card in Australia, as he's Associate Professor of Educational Psychology in School of Education and Deputy Associate Dean in the Faculty of Humanities, Arts and Social Sciences!

The conversation talks about the challenges of assessment, and the options for rethinking assessment - and then we go deeper into Jason's views on the future of learning and assessment.

Jason's a great guest to share his experiences, as during 2023 he was on the TEQSA group of experts that came together to produce a report on assessment for Australian universities, Assessment reform for the age of artificial intelligence

https://www.teqsa.gov.au/guides-resources/resources/corporate-publications/assessment-reform-age-artificial-intelligence

Working on policy and guidance in an area where technology is developing so rapidly - and students are racing ahead of institutions, was interesting and Jason talks about the group dynamic. One of the interesting notes he talks about is the mindset: "The mantra we kept returning to is that we weren't trying to develop a map, but a compass. This is the direction we think we might need to head here."

AI and the Future of Assessment: Transforming Educational Practices

Episode Overview: In this episode of the AI Education Podcast, hosts Dan and Ray, alongside guests Adam Bridgman and Danny Liu, dive into the evolving landscape of academic assessment in the age of artificial intelligence. Recorded in the University of Sydney's own studios, this discussion explores the significant shifts in assessment strategies and the integration of AI in educational settings.

Guest Introductions:

• Professor Adam Bridgeman: Pro Vice Chancellor Educational Innovation at the University of Sydney - focused on enhancing teaching quality across the university. [University bio]
• Professor Danny Liu: Professor of Educational Technologies - dedicated to empowering educators to improve their teaching methods through innovative technologies. [University page - LinkedIn page]

Key Topics Discussed:

• The Persistence of Traditional Assessment Models: Despite the push to digital platforms during the COVID-19 pandemic, traditional assessment methods have largely remained unchanged, continuing the practice of replicating physical exam environments online.
• AI's Role in Rethinking Assessment: The guests discuss how AI challenges the conventional reasons for assessments, advocating for a paradigm shift towards assessments that truly measure student understanding and application of knowledge.
• Two-Lane Assessment Approach: Adam introduces a dual-lane strategy for assessment:
  • Lane One: Ensures the rigorous verification of student competencies necessary in professional fields.
  • Lane Two: Uses AI to foster skill development in using technology effectively, moving beyond traditional assessment forms to embrace innovative educational practices.
• Implementation Challenges and Solutions: The transition to new assessment models is recognised as a gradual process, needing careful planning and support for educators in rethinking their assessment strategies.
• Inclusivity and Access to Technology: Ensuring equitable access to AI tools for all students is highlighted as a critical aspect of the evolving educational landscape, emphasizing the need to support diverse student backgrounds and technological proficiencies.
• Future Outlook: The discussion concludes with reflections on the potential long-term impacts of AI on educational practices, the necessity of ongoing adaptation by educational institutions, and the importance of preparing students for a future where AI is seamlessly integrated into professional and everyday contexts.

Further Reading:

We recommend these three articles from the team, that give more detail on the topics discussed

• Where are we with generative AI as semester 1 starts? What to do about assessments if we can't out-design or out-run AI? Embracing the future of assessment at the University of Sydney