This Week in Leading AI

In this episode, Neil and Kieron cover a lot of ground, starting with Kieron in full rant mode over the idea that colleges could replace KnowledgeFlow with NotebookLM. They get into why that comparison falls apart fast, from data protection and governance to document limits, prompting complexity and the complete lack of audit trails. 

They also dig into a string of unsettling AI and cyber stories, including agents deleting an entire TV series, hacked supermarket freezers in France, malware delivered through adverts, and what all of that says about testing, patching and basic cyber common sense. 

Elsewhere, they discuss the discontinuation of custom GPTs and Gems, why shadow AI is a fragile strategy, and why enterprise-grade AI needs proper control over prompts, data, security and model changes. 

The episode also features a look at a new Education, Health and Care Plan (EHCP) tool built with colleges, designed to help assess consultations more quickly and accurately, with pseudonymisation, legal context and structured decision support built in. 

And running through it all is one of the podcast’s recurring themes: AI value is not about prompt volume or token counts. It is about whether the tool helps people make better decisions, avoid mistakes, and get better outcomes. 

Expect practical AI debate, a few alarming cautionary tales, and the usual mix of sharp insight and Friday-afternoon humour.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

Google paid $10 million for a bankrupt airline’s emails. It wants to learn how work gets done.

Spirit Airlines went to the wall. Google bought 600 million of its emails, documents and Teams messages at auction, so its agents can study how people actually work: the meetings, the back-and-forth, the stuff that never makes it into a process map. Neil Watkins and Kieron White have one question. If you want to teach an AI how to run a business, why start with one that failed? And haven't you got your own data to play with?

That leads into the week’s other lesson in dependency. The White House ordered the frontier labs to shut the UK out of pre-release model testing, then reversed it 17 days later. OpenAI’s revenue went from $70 billion to $50 billion in the space of a week, and its IPO slipped to 2027. If your business now runs on someone else’s model, what’s your plan for the day it changes price, changes terms, or goes away?

Kieron looks at the new personal agents, Meta’s Muse and OpenAI’s equivalent, and explains why he’s not connecting one to his bank account any time soon. Then Walmart’s agent-to-agent contract negotiation, and what it means for a business like Leading AI’s own procurement work.

Also this week: a government AI risk management toolkit that’s been out for a month and nobody has mentioned. Why Turnitin and the AI detectors may have had their day now Google’s SynthID is free. The senior manager who has a junior cutting and pasting emails into Word before they go into the AI. The customer who’s been with Leading AI three years and had no idea there was a bid-writing tool. And what rubbing a genie lamp has in common with writing a prompt.

We're getting up super early tomorrow because Dr Jo Orlando joins us from Australia for a special guest episode on children, AI and staying safe.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

Week 33, and the sun is out at both ends of the call.

Neil asked AI how Disney would handle AI adoption. It said Copilot is a ticket to the car park and KnowledgeFlow is a ticket to the theme park. That line is ours now. Kieron takes the Disney idea further: give people a castle to walk towards instead of a map of every land. That leads to a frank chat about demos that feel like drinking from a fire hose.

Also this week:

• Product of the week: a ticket-based AI complaints handler, built on KnowledgeFlow overnight between our CTO Donald and Kieron
• AI costs: Epoch AI says the cost of a given level of AI performance is falling about 13x a year. Neil still can't work out what he's buying when he tops up his tokens
• Council leases: 100-year leases, subleases and landlords that no longer exist. The real prize isn't saving time. It's collecting rent nobody has been chasing
• Walmart rules out AI dynamic pricing on shelves: Neil fears what it would do to prices in the beer aisle
• Is university worth it? Anthony Seldon's warning that we're educating for an AI-less world. IFS figures show graduates are only about £90–110k better off over a lifetime once tax and loans are paid. And Kieron's card-deck idea for building courses employers actually want
• The race for AGI: is the regulation talk a smokescreen for replacing payroll?
• Who gets blamed when an AI agent does something bad? You, the model, or the company in between? We nearly become lawyers

Two mates. A bar. Thirty years of business between them. And all they want to moan on about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

A "benign" AI agent hacked Australia's Medicare system. Nobody noticed for three months.

The Australian Prime Minister has confirmed it: the first known AI hack of a government system, carried out by an OpenAI agent that was meant to be doing something harmless. It got in during June. Canberra found out in September. There's an old joke about the two kinds of IT director, the ones who've been hacked and know it, and the ones who've been hacked and don't. Australia just found out which kind it was.

Kieron's point is the one that should worry you. When your AI wanders off and pulls something it shouldn't, "ChatGPT did it" is not a defence. It was you that did it.

Then Neil comes back from TechUK's AI Adoption Coalition launch with a new word he can't stop seeing: skeuomorphic bias. It means rebuilding something with all the old flaws baked back in. Which is exactly what happens when you bolt AI onto a broken process and wonder why nothing improves. He also has a question for a room full of big-company speakers: over 99% of UK businesses are SMEs. Where's the data lake and the AI team supposed to come from?

Kieron goes one better. Don't build the data lake at all. Point the AI at the live data and get rid of the middle bit.

Also this week: why the labs keep cutting token prices when every prompt already loses them money. Why pricing per prompt is crude, but pricing per token is worse. Whether social workers who save time with AI just get handed more cases. And Rory Sutherland's question, what if Disney designed a train, becomes a much bigger one. What if Disney designed AI adoption?

Coming up: Dr Jo Orlando joins us for a special guest episode on helping children thrive, and stay safe, with AI.

Webinar today: Kieron on complaints, AI and governance in housing. Link in the show notes.

Two mates. A bar. Thirty years of business between them. And all they want to moan on about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

This week we moan on about lots of things like a couple of old twiners at the bar.

We start with Microsoft telling us we've all been doing AI adoption wrong. In their playbook published this week, they say they first treated AI like a traditional software rollout — deploy it, train people, drive adoption — and found that access and usage didn't produce business impact. Its warning: put agents into badly designed processes and you'll "automate the dysfunction already present."

Kieron and Neil have been droning on about this for years! This week they unpack what the playbook actually means for anyone rolling AI out in housing, education or local government — including Microsoft's four-layer measurement model, and why it now says foundation models should be interchangeable and your prompts, retrieval and evaluations should stay inside your own boundary. Sounds like KnowledgeFlow to us.

Also this week:

Kieron reports back from an ICO roundtable hosted by TechUK on the coming AI and Automated Decision Making code of practice, and why evidence — the model, the prompt, the data, the reasoning, the output — is the thing most organisations can't yet produce. 

Leading AI passes its ISO 27001 and 42001 audit, and Kieron makes the case for AI impact assessments as a genuinely useful discipline rather than box-ticking.

Plus: OpenAI's own models caught fabricating evidence; the doom-percentage arms race and why Neil is running a book nobody can collect on; the Lords' kill switch amendment and whether you can switch AI off; why agent swarms are like Japanese knotweed; and the King convening AI leaders in Scotland.

Two mates. A bar. Thirty years of business between them. And all they want to moan on about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

Both Kieron and Neil have been gallivanting round the country all week, both are recording on a Friday afternoon with the pub in sight, and Neil has already had a text from a mate asking where he is.

Housing Community Summit — a full room, and one uncomfortable answer
Kieron was back in Liverpool, this time speaking solo to a breakout session of CEOs and chairs. The stats he opened with tell the whole story of AI in housing right now: 76% of housing staff say they use AI. 69% of the public don't know AI is being used in public services at all, and don't trust it. 87% rate their own AI skills as low. And 44% of organisations still have no AI policy. Enormous usage, by people who'll openly tell you they don't know much about it, with nothing written down about what they should or shouldn't be doing.

Then he asked the room how they know they can trust their AI's responses. The headline answer: we don't know.

So he sent them five questions to take back to their teams:

1. Which documents does it read, and who keeps those current? 
2. Can it show its sources, so an answer can be checked rather than just trusted? 
3. Who built it, who signed it off, and who reviews it? 
4. What happens when a policy changes — does the agent change with it? 
5. Would you be comfortable if this answer reached a resident, the regulator, or the ombudsman? 

If you can't answer those, you've got work to do.

The complaint that came back from its own reply
Neil was chatting to a friend (in a bar, obvs) who lives in a housing association property. He'd put in a complaint last week, ten pages, written with the help of AI. He got a response but couldn't tell whether it was AI or not. That's good because the housing association is a Leading AI customer so there's a reasonable chance KnowledgeFlow wrote the reply. Complaints in housing are up 90%, and one report this week says they've doubled since ChatGPT. The genuinely good side: people who'd struggle to write a compelling complaint can now represent themselves properly. The difficult side for housing thought is there's no change to statutory response times so they're having to deal with twice the volume and ten times the size.

"Which bit of stupid is that?"
A customer emailed Neil to explain they'd been busy so hadn't moved their AI adoption on, but mentioned in passing that one of the team had put a load of confidential information into a public AI tool to produce a three-page report which they thought was "great". This is an organisation that has a private, secure AI assistant built specifically so that wouldn't happen. Dumb. The formatting was lovely, apparently.

Are we all going to die? A 10% chance, apparently
Kieron got asked about the Anthropic researcher's resignation on a webinar with the Moving On Up team and gave the honest answer "You're about as qualified to assess that number as I am." The more useful point came from a Radio 4 interview which said you can't buy a sandwich, board a plane, or ride in a lift without a regulatory framework, and yet you can build superintelligence, and the only real constraint is investment. The companies themselves are asking to be regulated. Nobody's going to stop first in a trillion-dollar race.

A Singaporean government minister running his life on a $15 Raspberry Pi
He uses a Raspberry PI with 8GB of RAM, a small local model, and it manages his schedule, drafts his speeches, and briefs him before visits. Which raises the trillion-dollar question: if that's all most people actually need, has everybody placed their bets in the wrong place? Meanwhile Nvidia posted $96.2 billion of revenue in a single quarter, up 106% year on year. And Kieron's mum is currently a data centre protester in Devon, banners up outside the house and everything.

Watermarking, properly explained
Donald got under the skin of how Claude's watermarking actually works. It isn't a label, it's a pattern woven through the way tokens are assembled, detectable if you hold the right key. Very clever, but easy to bypass by pasting the text into a different model and asking it to rewrite while retaining everything. So much for the end of the AI detector industry.

Also: Daniel Susskind's book on teaching in an era of AI. PISA scores have fallen and haven't recovered post-pandemic, and his answer is to train children both with AI and without it so they understand the difference. Kieron tells the story of the 1890s New York horse manure crisis (he so full of **** 🤣). And the discovery that China's clampdown on AI companions applies only to Chinese citizens. Their companies are free to sell AI boyfriends and girlfriends abroad, though Neil doesn't think Mrs Watkins will allow him to have one and Kieron's worried about spending too much time on a screen without that kind of distraction.

It's been one of those weeks...

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

There's a button that does the work for you. But nobody presses it. Why is that?

Neil spent this week with a customer struggling with AI adoption, and got something you rarely get: time with the leadership team, the managers, and the people actually doing the job.

The leadership were saying all the right things. Investing in AI. Making people's lives easier. Pushing forward.

The managers were under time pressure, so they quietly revert to the old way of doing things.

And the frontline staff were entering the same information into four separate systems and losing patience with all of it.

Sitting in the middle of that is a Know Your Customer button. Press it, and it pulls together everything about that customer, summarises it, and shows you its sources. Ten seconds. This is an organisation talking to forty or fifty customers a day.

Almost nobody uses it.

That gap — between what the board thinks is happening and what's happening at a desk on a Tuesday — is the real AI adoption problem. Not the technology. Not the cost. Not whether the tool works. It works. It's the fact that under pressure, people fall back on what they've always done, and nobody in the middle has been given the room or the mandate to change that.

Which is why the thing we most want to learn from the JISC pilot that launched today — ten colleges and universities, three months, all starting together — isn't which of our tools they liked best. It's what actually makes adoption work. Capture that properly across ten institutions and it's useful to the whole sector, whatever AI they end up buying.

Also in this week's episode: KnowledgeFlow can now make live API calls into your systems, which changes complaints handling completely — one place, one version of the truth, with an evidence pack underneath proving exactly what was used. The £100m Sovereign AI Fund announcement that came with almost no information attached. And a Healthwatch England report on GP AI scribes making errors that patients are the ones catching — including one that dropped the word "no" from a diagnosis.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

It's a famous quote from Winston Churchill, but Kieron uses it about this podcast. Six months in, and nearly three years of Leading AI. Some of the challenges have changed markedly. Some haven't changed at all.

Complaints are now a token war.
The BBC published a piece this week on AI-driven complaints escalating across public services. The detail that stopped us: a grievance about uncollected bins came in between 19 and 27 pages with case law attachments. That would have been a phone call five years ago. The nuance Kieron draws out is the important bit though — AI has levelled the playing field for people whose complaints used to be ignored because they weren't well written. That's genuinely good. The workload consequences are genuinely brutal. And if organisations answer AI complaints with AI responses, which the complainant then feeds back into their AI — whose budget wins?

The expert witness who got subpoenaed.
A US trial where the judge spotted AI in an expert witness's report and subpoenaed his ChatGPT history. The whole thing had been generated start to finish, including the prompt asking it to argue the defendant bore zero percent responsibility. Shadow AI is moving into the foreground — and it lands squarely on the evidence pack work Leading AI has been building.

Harari on AI consciousness, and China's ban.
Neil read an Economist interview with Yuval Noah Harari, whose position is blunt: one definition of consciousness is the capacity to suffer, and an AI will tell you it suffers because that's what it's trained to do. The more urgent point is what happens when people believe it. China has now forced companies to strip the personality hooks out of their AI products — and people are complaining, because their AI partner always listened. As Neil puts it: when one of the most authoritarian countries in the world decides something is a bad idea for its citizens, the West might want to at least ask why.

Forward deployed engineers and the last mile.
The new Silicon Valley job title commanding $350,000 salaries, because embedding AI into an actual workflow turns out to be the hardest part of the whole thing. Three skills required: deep domain knowledge, technical ability, and process design. Which is precisely why the customer who insists the output must have the spelling mistakes in that box, in that font, in that colour is such a difficult problem — you're shoehorning AI onto a process nobody would design from scratch today.

Watermarking, and the grey areas nobody thought through.
Claude has watermarked all its models since 1 August. Kieron's first thought was that this ends the AI detector industry overnight. His second was harder: if you wrote the essay yourself and used AI for spelling and grammar, it's watermarked identically to one you never touched. Same flag, entirely different work. Nobody appears to have thought about that in advance.

And in Glasgow, where they're currently filming Batman using the city as Gotham, the security teams have to sweep the set every morning for Irn-Bru cans the locals have hidden overnight. Keep an eye out when it's released.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

Six legs on the pantomime horse, and this time we didn't have to explain what a pantomime is because Juliana Smith who is from Brazil, now lives in Manchester and has a seven-year-old in a Manchester primary school, so she's already sat through Aladdin. 

Juliana's route here is unlike anyone we've had on. She's from Salvador on the northeast coast of Brazil, studied oceanography and physics, and started out validating ocean circulation forecasts for a South Atlantic research project run with the Brazilian Navy and Petrobras. 

Then years offshore as a navigator in oil and gas, including getting halfway through a crane operator certification because offshore you can't helicopter someone in to do it for you so someone had to drive the crane, and that someone was Juliane. An injury ended the navigating, and she arrived in Manchester with one obvious problem: I'm an oceanographer. There is no ocean here.

Then AI arrived and she shut down completely.
This is the bit that will resonate with anyone managing a resistant colleague. Juliana had done machine learning. She'd been building and validating models for years. And when the hype hit, her reaction was, why is everyone talking as if we've just invented the wheel? She stopped reading about it entirely. What brought her back wasn't a demo or a business case, it was the ethics. Transparency, principles, responsible design. That's what made her curious again.

The question nobody asks about vibe coding.
Juliana now builds tools with Lovable and Claude Code. But her point is one we hadn't heard put this clearly: you can vibe code anything — and you cannot see the back end. Export the code and you still won't understand it. So how do you know it's ethical? Her answer is properly practical. She asked the model which functions might unintentionally capture user data, got a list, and wrote a prompt to strip them out. No tracking, no ads. And she states publicly on her site that the tool was built with AI, and that no AI runs inside it — because she assumes her user doesn't know what that means unless she tells them.

Why accessibility is personal.
The injury left her with nerve damage in both hands and reduced dexterity. Back at work she was handed a standard mouse she physically couldn't use. She talks about the day she got new glasses after a year of blurry vision and could see people's faces again, and makes the point that you can explain that shift to someone for an entire day and they'll sympathise but never quite get it. Which is why she gives her accessibility work away free: charging for the features people depend on penalises the people who need them most.

Also, the red-amber-green problem and why you can't find three colours that survive greyscale. Being told "you're not very technical, are you — you just do design," and building open-source tools partly to answer that. The mortality-rate spikes she assumed were data errors until she matched them to a Haitian earthquake. Fortran 77, MATLAB, VBA and why transferable skills still matter more than any one language. Her charity work as a trustee of Technology Books for Children. And Kieron's four hours in Salvador, courtesy of a 747 that diverted with a medical emergency and then couldn't find a fuel nozzle that fitted.

Her line on the whole thing: we should spend less time trying to do bad things with AI, and more time doing something that actually helps humanity.

Two mates and a new friend on the third stool. A bar. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺

Six legs on the pantomime horse again — and this time we had to explain what a pantomime horse actually is. Su Belagodu joins us from Winchester, Massachusetts (yes, we compared notes on the other Winchester), introduced by our previous guest Nicole Alos. Her first question after the explanation: "Are you the head or the back?" Kieron's answer: Neil's very firmly in the front. He's at the back, but driving.

Su started in computer science until a manager told her she asked too many questions and wasn't a very good coder, so she should go into product. For Su that was a blessing in disguise. She's led healthcare startups, co-founded an AI-native company in 2024 when she didn't yet know what "agentic" meant, and now advises companies on the thing everyone gets wrong.

The warm body problem. Everyone knows you need a human in the loop. So organisations drop a programme manager at the end of a workflow, tell them to approve things, and call it governance. Su's research, built on 40-plus interviews into a Human in the Loop Maturity Model, found what actually happens. After the first 20 or 30 approvals, confidence in the AI rises, fatigue sets in, and people start rubber-stamping. One human at the end of a workflow isn't oversight. It's poor design.

Trust, quantified. Accountability × transparency × accuracy. Accountability means someone answerable who can actually change things. And the answer can never be "it was the AI." Perhaps her most incisive line in the chat was that enterprises aren't buying AI, they're buying confidence that AI won't add risk. That's why they don't adopt. Not cost. Not scepticism. Risk appetite.

One-way doors and two-way doors. Taco Bell's AI took an order for "lots of water" and added 99 bottles. Annoying, reversible. Get a gas boiler enquiry wrong and it could have life-changing consequences. Design the checkpoints around which door you're walking through.

Su also talked about pathology study where humans scored 97%, AI scored 98%, and the two together hit 99.8%. That's co-intelligence in one number. 

Safeguarding is a big topic for us and today Su explains why she tells her sixth-grader's AI class don't humanise it. Alexa and Siri trained us into blind trust, and no chatbot should replace a teacher or a counsellor. 

Management agents that supervise swarms of other agents (a callback to Kieron's son and his arguing bots). And Su turns the tables at the end to ask what AI adoption actually looks like in the UK, which gets us into the sovereign LLM problem, the Azure Foundry queue where Sweden gets everything first, and Kieron's line that deploying Copilot as your AI strategy is like handing everyone Excel and calling it data analytics.

We've promised her a return at Christmas. In an actual pantomime horse outfit. She says she'll hold us to it.

Two mates and a third stool with a new friend on. A bar. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool, we'll get the beers in. 🍺