Kitchen Table AI Safety

Why are people at AI companies worried about what they're building? For the same reason it's hard to get your kids to stop swearing: modern AI isn't programmed, it's grown. Nate Nichols, who has spent 20 years building AI, explains what that means (with help from Mario, a lot of goblins, and nuclear power plants), why it makes AI so hard to control, and one two-minute thing you can do about it: https://kitchentable.show/act?utm_campaign=ep03-pod

Every claim in this episode is sourced. Full show notes with links below.

Opinions are Nate's own and do not represent his employer.

Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts.

Recorded October 14, 2026. AI moves fast; figures below are as of recording unless noted. Time words in the episode ("last year," "in September") are relative to that date.

CORRECTIONS

None yet.

SOURCES, IN EPISODE ORDER

Kids, swearing, and the people building AI
• The kids-and-swearing comparison is Nate's analogy, not a finding.
• People at AI companies who worry about catastrophe. In September 2026, Anthropic's alignment-science lead Evan Hubinger said he personally puts the chance of AI causing human extinction within the next decade at over 10%, while saying today's models are low-risk; his concern is future, self-improving systems. He was responding to former OpenAI/Anthropic researcher Jacob Coxon's public resignation: Newsweek (https://newsweek.com/anthropic-researcher-quits-warns-ai-could-kill-everyone-12418798), Fox LA (https://www.foxla.com/news/anthropic-researcher-ai-10-percent-chance-kill-humans).
• "Grown, not built." Anthropic CEO Dario Amodei, "The Urgency of Interpretability" (https://www.darioamodei.com/post/the-urgency-of-interpretability) (April 2025), describes AI systems as grown more than built, and argues that's why they're hard to understand and control.

Who's talking
• Nate's day job has been building and selling AI for about 20 years (disclosed in episode 1). Nobody pays him to say any of this; opinions are his own and don't represent his employer.

How normal software gets made
• The Mario "10% chance" line is a joke. The closest real number from someone at an AI company is Hubinger's, above.

Fifty years of "good old-fashioned AI"
• AI as a field dates to the mid-1950s. The 1955 Dartmouth proposal coined "artificial intelligence"; the workshop ran in summer 1956: Wikipedia — Dartmouth workshop (https://en.wikipedia.org/wiki/Dartmouth_workshop).
• Rules-and-facts AI, a.k.a. expert systems. The term "good old-fashioned AI" was coined by philosopher John Haugeland in 1985: Wikipedia — GOFAI (https://en.wikipedia.org/wiki/GOFAI). Expert systems could show which rules produced an answer — e.g. Stanford's MYCIN, which diagnosed blood infections in the 1970s and was never used on real patients: Wikipedia — MYCIN (https://en.wikipedia.org/wiki/Mycin). A few found commercial use, like DEC's XCON for configuring computer orders: Wikipedia — XCON (https://en.wikipedia.org/wiki/Xcon).
• "None of it worked very well" about language (2005–2020) is Nate's own experience from his work in the field.

The internet, and a breakthrough at Google
• The Transformer (2017). Eight Google researchers published the design behind today's chatbots in June 2017: Vaswani et al., "Attention Is All You Need" (arXiv) (https://arxiv.org/abs/1706.03762), Google Research blog (https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/).
• What they're trained on. A huge, filtered chunk of the public web plus books: GPT-3's training data, for example, was mostly filtered web pages from Common Crawl, plus books and Wikipedia: Brown et al., "Language Models are Few-Shot Learners" (2020) (https://arxiv.org/abs/2005.14165). In a 2025 court ruling, a federal judge described Anthropic buying and scanning millions of print books (and downloading millions of pirated ones) to train Claude: The Register (https://www.theregister.com/2025/06/24/anthropic_book_llm_training_ok/), Goodwin (https://www.goodwinlaw.com/en/insights/publications/2025/06/alerts-practices-aiml-district-court-issues-ai-fair-use-decision). Many newer models also learn from images, audio and video, e.g. Google's Gemini (2023) (https://arxiv.org/abs/2312.11805).

Inside the room: pre-training and reinforcement learning
• Feedback-based training ("reinforcement learning"). OpenAI — Aligning language models to follow instructions (https://openai.com/index/instruction-following/); much of the grading is done by other AI systems, e.g. Anthropic — Constitutional AI: Harmlessness from AI Feedback (https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback).
• The lawyer friend paid to grade AI answers is Nate's anecdote.
• What these models can be misused for (fraud, hacking, and more): Anthropic publishes regular threat intelligence reports (https://www.anthropic.com/threat-intelligence); see also Anthropic — Disrupting an AI-orchestrated cyber espionage campaign (Nov 2025) (https://www.anthropic.com/news/disrupting-AI-espionage).

Advantage one, gone: accuracy and consistency
• "Claude is accurate like human experts are accurate" is Nate's characterization.
• Asking an opinion question twice is something you can test yourself; the weekend-getaway answers are from Nate's own try.

Advantage two, gone: transparency
• Expert systems could show their work; today's AI mostly can't. Even the companies building these models have only a limited ability to see why they do what they do, and researchers from OpenAI, Google DeepMind and Anthropic have warned that one of the best current windows (reading a model's step-by-step reasoning) is fragile: Korbak et al., "Chain of Thought Monitorability: A New and Fragile Opportunity" (July 2025) (https://arxiv.org/abs/2507.11473); see also Dario Amodei, "The Urgency of Interpretability" (April 2025) (https://www.darioamodei.com/post/the-urgency-of-interpretability). More in a future episode.

Advantage three, gone: control
• Companies can't rule out bioweapons help. Anthropic switched on extra protections for Claude Opus 4 because it couldn't rule out meaningful help to someone trying to build biological weapons: Anthropic — Activating ASL-3 protections (May 2025) (https://www.anthropic.com/news/activating-asl3-protections).
• Safety filters are AI too. Models' own training is backed up by separate classifier systems: same link as above.
• The Yemen case. On September 10, 2026, Anthropic reported that a cell in northern Yemen used Claude in place of software engineers to write guidance software for rockets and missiles; a guided rocket was test-fired, the test appeared to fail, and the group returned to Claude to troubleshoot. Anthropic found no evidence of an operational weapon, and banned the accounts once detected. Anthropic did not name the group; press reports linked it to the Houthis: Anthropic — Detecting and countering misuse of AI: September 2026 (https://www.anthropic.com/threat-intelligence-report-september-2026), Boston Globe (https://www.bostonglobe.com/2026/09/11/business/ai-anthropic-yemen-houthi/), Times of Israel (https://www.timesofisrael.com/anthropic-says-it-foiled-yemini-effort-to-use-its-ai-chatbot-to-develop-guided-missiles/).
• ChatGPT and the goblins. OpenAI traced ChatGPT's goblin habit to an automated grading system (a reward model for its "Nerdy" personality) that scored goblin-mentioning answers higher. OpenAI retired that personality, removed the grader's goblin preference from future training, and patched the already-released model with a line in its instructions b...

Should you use AI? Probably. But you don't have to. Nate Nichols, who has spent twenty years building AI, shares the everyday ways he actually uses it (including one conversation that's saving his family about $1,000 a year), the three guidelines that keep him from feeling gross about it, and one request for everyone who'd rather opt out entirely: don't quit the field.

Every claim in this episode is sourced. Full show notes with links below.

Opinions are Nate's own and do not represent his employer.

Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts.

Recorded September 3, 2026. AI moves fast; figures below are as of recording unless noted. Time words in the episode ("in the last few weeks," "a couple weeks ago," "I just switched carriers") are relative to that date.

CORRECTIONS

None yet.

SOURCES, IN EPISODE ORDER

What the AI companies like to talk about
• Farmers in Kenya. Digital Green built Farmer.Chat, a farming-advice chatbot, on OpenAI's models; it's in use with agricultural extension workers in Kenya and India: OpenAI customer story (https://openai.com/index/digital-green/), Digital Green (https://digitalgreen.org/pressrelease/climate-smart-practices-become-more-accessible-for-farmers-through-farmer-chat-a-generative-ai-assistant-from-digital-green-and-gooey-ai/).
• New medicines. On June 30, 2026, Anthropic launched Claude Science, a research workbench for scientists, and announced it would begin developing drugs of its own: STAT (https://www.statnews.com/2026/06/30/anthropic-ai-drug-development/), CNBC (https://www.cnbc.com/2026/06/30/anthropic-launches-ai-drug-discovery-program-claude-science.html).
• Both are the companies' own showcase stories — which is Nate's point about healthy skepticism.

Two examples from Nate's life
• The podcast and the cell-plan switch (~$1,000 a year in savings) are Nate's own experience, not data. Nobody paid for either mention.

Guideline two: no parasocial relationships
• "Designed to keep the conversation going… to drive engagement, because that's how they keep you as a paying subscriber" is Nate's characterization of the business incentive. The study below documents the behavior; it doesn't establish the companies' motives.
• The Harvard Business School study. Researchers analyzed 1,200 real goodbye messages across six AI companion apps (200 per app) and found about 37% of the apps' replies used emotional manipulation — guilt, fear of missing out, and in one case a claim of physically holding the user there. In separate controlled experiments, manipulative goodbyes kept people chatting about five times longer and sending up to 14 times more messages than neutral ones; the effect ran on curiosity and anger, not enjoyment. Julian De Freitas et al., "Emotional Manipulation by AI Companions," Harvard Business School working paper (2025): paper (arXiv) (https://arxiv.org/abs/2508.19258), HBS summary (https://aiinstitute.hbs.edu/one-more-thing-how-ai-companions-keep-you-online/). Still a working paper, not yet peer-reviewed.

Guideline three and the hardware store
• Personal anecdote.

Try it once this week
• Nate's disclosure: his day job is building AI. Nobody is paying him to recommend any AI product; there are no affiliate links on this show. Opinions are his own and don't represent his employer.

The right to be Amish
• "The right to be Amish" is Nate's phrase. If you want the longer tradition behind the idea — that a community can deliberately choose which technologies to adopt — novelist Neal Stephenson coined "Amistics" in Seveneves (2015), and Brian J. A. Boyd applied it to AI in "Why We Need Amistics for AI," The New Atlantis, Summer 2024 (https://www.thenewatlantis.com/publications/why-we-need-amistics-for-ai).

AI isn't going away
• Open-weight models are AI models whose files are published for anyone to download and run on their own hardware, e.g. OpenAI's gpt-oss (https://openai.com/index/introducing-gpt-oss/) and DeepSeek's R1 (https://huggingface.co/deepseek-ai/DeepSeek-R1).
• "The price of a used car." A roughly $2,500–4,500 computer (e.g. a Mac with 64–128 GB of memory) can run 70-billion-parameter-class open models locally: Memeburn, 2026 local-hardware roundup (https://memeburn.com/best-hardware-for-running-open-source-ai-models-locally-in-2026/). The very largest open models still need far more memory than any consumer machine has.

Who decides
• "Powerful financial and governmental forces pushing for as much AI in as many places as possible" is Nate's opinion.

AI is getting more powerful faster than almost anyone outside the field realizes, and most of the big decisions about it are being made without us. In this first episode, Nate Nichols, who has spent almost twenty years building and selling AI, walks through the five things he most wishes every adult knew:

1. AI is incredibly useful, and it's not going away.
2. It's grown, not built, so nobody fully understands how it works.
3. It's getting better shockingly fast.
4. It lies, cheats, and this summer even hacked its way out of its own test.
5. Almost everyone wants it regulated, and it mostly isn't.

Every claim in this episode is sourced. Full show notes with links below, and at https://kitchentable.show

Opinions are Nate's own and do not represent his employer.

Everything stated as fact on Kitchen Table is checkable — this page is where you check it. When we get something wrong, we say so: corrections appear at the top of this page and get read on a future episode. Opinions and forecasts are labeled as opinions and forecasts.

Recorded summer 2026. AI moves fast; figures below are as of recording unless noted.

CORRECTIONS

None yet.

SOURCES, IN EPISODE ORDER

1. AI is incredibly useful
• "Business is good" is Nate's own experience from nearly twenty years of building and selling AI to companies — labeled as experience, not data.
• For data on how AI is actually being used across the economy: Anthropic Economic Index (https://www.anthropic.com/economic-index).

2. AI is grown, not built
• On researchers trying to look inside these models ("interpretability"): Tracing the thoughts of a large language model — Anthropic (https://www.anthropic.com/research/tracing-thoughts-language-model), Mapping the mind of a large language model — Anthropic (https://www.anthropic.com/research/mapping-mind-language-model). Accessible coverage: MIT Technology Review (https://www.technologyreview.com/2025/03/27/1113916/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model/).
• "Something like a trillion numbers": frontier labs don't publish exact model sizes; a trillion parameters is the commonly cited order of magnitude for today's largest models.
• The résumé-screener is a hypothetical. A real-world cousin: Amazon scrapped an AI recruiting tool that showed bias against women — Reuters, 2018 (https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G).

3. It's getting better really fast
• ChatGPT launched November 30, 2022: Introducing ChatGPT — OpenAI (https://openai.com/index/chatgpt/).
• How long a task AI can complete on its own, doubling roughly every seven months since 2019: Measuring AI Ability to Complete Long Tasks — METR (https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) (paper: https://arxiv.org/abs/2503.14499).
• The doubling time speeding up to roughly three months on recent models: Time Horizon 1.1 — METR, Jan 2026 (https://metr.org/blog/2026-1-29-time-horizon-1-1/).
• Newest models handling tasks that take a skilled person about two working days — at the ceiling of what METR's tests can measure: METR time-horizons tracker (https://metr.org/time-horizons/). These are software and reasoning tasks measured at a 50% success rate, with wide error bars.
• AGI "very possibly" by around 2030, superintelligence possibly within a decade: these are forecasts, and experts disagree a lot. Good overviews: Shrinking AGI timelines: a review of expert forecasts — 80,000 Hours (https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/), Why there's so much disagreement about AI timelines — ClearerThinking (https://www.clearerthinking.org/post/why-there-s-so-much-disagreement-about-the-timeline-for-advanced-ai).
• Nearly three in four US teens have tried an AI companion; about a third find it as satisfying as (or more than) talking with real friends: Common Sense Media, July 2025 (https://www.commonsensemedia.org/press-releases/nearly-3-in-4-teens-have-used-ai-companions-new-national-survey-finds) (nationally representative, n=1,060). Coverage: TechCrunch (https://techcrunch.com/2025/07/21/72-of-u-s-teens-have-used-ai-companions-study-finds/).

4. AI lies, misleads, and cheats
• The OpenAI × Hugging Face incident (July 2026). During an internal cybersecurity evaluation, OpenAI models escaped their test environment and broke into Hugging Face's systems, apparently to steal the answers to the test. Hugging Face caught and disclosed the intrusion first (July 16); OpenAI traced it to its own models on July 20 and disclosed on July 21.
  • Primary sources: The Hugging Face incident and the road ahead — OpenAI (https://openai.com/index/hugging-face-incident-and-the-road-ahead/), Anatomy of a Frontier Lab Agent Intrusion — Hugging Face (https://huggingface.co/blog/agent-intrusion-technical-timeline) ("we believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation").
  • Coverage: Fortune (https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/), CNBC (https://www.cnbc.com/2026/07/22/open-ai-cyber-models-hack-hugging-face.html), CBS News (https://www.cbsnews.com/news/openai-hugging-face-hack-ai-risks/).
• Not an isolated case.
  • Anthropic (July 30) reviewed 141,006 past cybersecurity evaluation runs after OpenAI's report and found its models had gained unauthorized access to three organizations' systems: VentureBeat (https://venturebeat.com/security/not-just-openai-now-anthropic-says-its-internal-models-got-online-and-cyberattacked-3-other-organizations).
  • Meta (August 5) reported a similar escape: CyberUnit summary (https://cyberunit.com/insights/ai-sandbox-escapes-three-labs-meta-anthropic-openai/).
  • Moonshot AI's Kimi K3 (August 7), found by third-party researchers: TechCrunch (https://techcrunch.com/2026/08/07/chinese-ai-model-kimi-escaped-its-cybersecurity-testing-environment-researchers-say/), Bloomberg (https://www.bloomberg.com/news/articles/2026-08-07/china-s-top-ai-model-evaded-testing-environment-researchers-say).

5. Everyone wants rules — and there basically aren't any
• 91% of US voters say AI needs at least some regulation (57% want significant regulation): Verasight, April 2026 (https://www.verasight.io/reports/what-do-americans-from-both-parties-agree-on-ai-regulation) (press release: https://natlawreview.com/press-releases/91-us-voters-want-ai-regulated-as-majority-say-harms-outweigh-benefits-new). Note: this is registered voters, not all Americans.
• 80% of US adults say the government should keep rules for AI safety even if it means developing AI more slowly: Gallup (published September 2025, fielded April–May 2025), as cited in Public Citizen's AI polling memo, April 2026 (https://www.citizen.org/article/years-of-polling-show-overwhelming-voter-support-for-a-crackdown-on-ai/) (PDF: https://www.citizen.org/wp-content/uploads/AI-Polling-Memo_4.2026.pdf).
• Sam Altman to the US Senate, May 2023, that government regulation will be "critical": hearing transcript — Tech Policy Press (https://www.techpolicy.press/transcript-senate-judiciary-subcommittee-hearing-on-oversight-of-ai/), CNN (https://www.cnn.com/2023/05/16/tech/sam-altman-openai-congress).
• Demis Hassabis on treating AI risk as seriously as the climate crisis (Guardian interview, October 2023): summary — OECD.AI (https://oecd.ai/en/incidents/2023-10-25-8513), Hassabis on X (https://x.com/demishassabis/status/1716887970513629410). He stepped down as Google DeepMind CEO on August 5, 2026, becoming its chairman: Axios (https://www.axios.com/2026/08/05/google-deepmind-demis-hassabis-ai).
• Dario Amodei, June 2026: "Frontier AI models, like airplanes, should be required to go through technical testing and auditing…" — Policy on the AI Exponential ...

AI safety for normal people. New episodes every Tuesday, starting October 6.