Calling all teachers! I would love to interview you about your experience using AI in the classroom - please visit https://www.theaiwhotaughtme.com/stories to sign up!

--

Have you ever sat with a child who is learning to read? It's fun to listen to them sound out the words and offer gentle pointers to help them along their way. But up until recently, most edtech software couldn't do that: they just guide kids without being able to tell what they are actually saying out loud.

But that is changing. In this episode, we cover the ground-breaking science behind Amira Learning. It started out of CMU's Project LISTEN and they have earned an impressive body of research (including a recent study in Louisiana) showing large learning gains in students who use an AI tutor that can listen and offer specific corrections. Future work out of Digital Promise will continue to study these effects.

But the latest LLMs have impressive speech capabilities out of the box, and it has enabled a new generation of reading apps that can listen to kids. Ello has surged in usage among home schoolers, while Project Read, Buddy Books and eSpark Reading Lab all offer great programs to enable phonics instruction that actually listens to kids.

In the future, perhaps we'll even have built in support within the major models, or the ability to bring-your-own-book and have an AI tutor read with you. In any case, this is one of the most interesting and widespread uses of serious AI in schools today

We dive into the recently announced learning modes from the big three LLM vendors: 

We also discuss the implications of a recent report on how people are using ChatGPT, including that more than 10% of conversations are classified as Tutoring or Teaching modes. Nearly half of ChatGPT usage comes from 18-25 year olds, which helps show why the university market is so important in this space.

And check out this TikTok from Jerard.b showing how ChatGPT is a little ... too ... helpful.

We look into the results from Alpha School, a private school in Texas that claims to use AI to help students gain 2x in only 2 hours a day.

When I dug in, I saw that their main "whitepaper" is reporting results based on using the wrong column from the NWEA MAP data! They haven't had any independent verification of their claims, which calls into question their validity.

I'm still quite intrigued by the school and I look forward to more robust reporting on their results from external researchers in the future.

Watch the accompanying YouTube video to see the charts and sources in more detail!

https://youtu.be/UPFCkbc3O3A

Sources:

https://www.astralcodexten.com/p/your-review-alpha-school

https://austinscholar.substack.com/p/austin-scholar-173-the-science-behind

https://2hourlearning.com/results/

https://www.youtube.com/watch?v=Qhov2Av7-5c

Writing feedback is a particularly focused use of AI in education. All the major AI education tools help improve writing, including MagicSchool, Brisk, Eduaide and more. But even with the immense power, there's a tension: how do you help students learn to be better writers without just doing it for them?

In this episode, we talk with David Vinca, the founder of the Sage AI Writing Tutor, which has gotten a lot of traction and positive feedback from teachers this year.  David and Luke discuss what makes a great writing tutor, why current LLMs don't always provide the best feedback, and what significant changes are coming for AI tools in schools in the upcoming school year.

You can learn more and try the Sage product at hellosage.io.

There has been a lot of drama and misinterpretation about a recent study preprint put out by MIT - Your Brain on ChatGPT. In this episode, we will look at that study in more detail, as well as a few others that have received less attention but I think do a better job of telling us how to use AI to teach writing.

We begin this episode by looking at a recent MIT pre-print, Your Brain on ChatGPT - preprint paper and the study site with FAQ and figures.

TIME Magazine covered this here, and then interviewed the study's main author, Dr. Nataliya Kosmyna - interview here. Fox News exaggerated the claims in their coverage of the story. It has been covered also by CNN and many other publications worldwide.

But the study doesn't actually make the claims that this breathless coverage suggests! Instead, we list three other studies that are more helpful at discerning how to use AI to teach writing:

Beware of Metacognitive Laziness.  https://arxiv.org/abs/2412.09315

In this experiment, some students used ChatGPT to write essays, while others did not. Those who used ChatGPT's help had better essays, but didn't have a significantly different level of learning or retention.

Modifying AI, Enhancing Essays. https://arxiv.org/pdf/2412.07200

A retrospective look at the CoAuthor dataset - a set of transcripts of students who used GPT-3 to help with their essays. This paper showed that students who engaged with feedback, rather than just accepting it blindly, had the most improvement.

Harnessing AI in Secondary Education to Enhance Writing Competence. https://arxiv.org/pdf/2412.12117v1

Not an experiment, but an overview and summary of other research.

"Research on feedback on writing shows that it is particularly valuable for students to receive help while they are engaged in the writing task. If students find themselves stuck or wonder how they have performed so far, feedback can be a great help. ... Unlike the human teacher, generative AI can provide instant feedback on a student’s writing (Jeon, 2023). This might include highlighting strengths, suggesting improvements, and pointing out areas where the student's voice shines through."

--

These three papers are available in the Stanford GenAI for Education Hub, which is an invaluable resource for finding quality research on AI in education.

Since the launch of ChatGPT in 2022, several AI education companies have sprung up to help teachers use these new tools in their classrooms. MagicSchool, founded by Adeel Khan, has been used by over 5 million teachers around the world. Meanwhile, Brisk Teaching has been growing fast and has over a million teachers using their extension as well.

These new platforms are the way that many teachers encounter and use AI in their classrooms. In this episode, we'll describe how they work, tell their story of how they began, and evaluate the evidence (or lack of evidence) on student learning.

Sources mentioned:

Tiers of Evidence

ESSA tiers of evidence (MagicSchool and Brisk both have earned the LOWEST tier, Tier 4, requiring just that there is a rationale behind their product but no evidence yet).

Common Sense privacy reviews for MagicSchool and Brisk are both 93%.

MagicSchool sources

Case studies from Aurora Public Schools(and the Aurora edtech page) and Innova. MagicSchool study from University of Jhang. Youtube short advertising rubric generator and Adeel Khan interview 

Brisk sources

Arman Jaffer interview on the Linkup Podcast and Brisk case studies.

Using LLMs can seem like a safe bet - but they have biases built in at every stage of the process. In this episode, we'll talk about how bias sneaks in and look at some studies that detail how LLMs are trained and how to watch out for and correct bias in an educational context.

Notes:

Follow up on the Tutor CoPilot study from the first week: https://lukeshepard.com/blog/preregistration-of-outcomes

Reserach about LLMs in classrooms: LLMs Are Biased Teachers

Meta's study on A Systematic Study of Bias Amplification

OpenAI discusses their own dataset: OpenAI: Is ChatGPT Biased?

I was influenced by Dr. Timnit Gebru- on AI Bias and the recently published The AI Con Book, which goes into more detail about how bias works in LLMs and the risks across industries.

* In Australia researchers found that AI helped boost scores on student writing - but only when students actually made edits and engaged with the suggestions. When they just copy/pasted the text, their learning was much worse.

https://lukeshepard.com/blog/chatgpt-in-schools

In this kickoff episode, host Luke Shepard and his AI co-host Emerald dive into how AI is entering classrooms. We will be looking at how AI is used in classrooms, starting with some recent research out of Harvard, Standford and Penn.

Studies

We look at four studies published in the last year:

Further Listening:

 Listen to this recent interview between Jane Rosenzweig and Michael Horn about how AI is affecting writing.

Sal Khan's TED talk: How AI Could Save and Not Destroy Education