Jim talks with recurring guest Forrest Landry about his arguments that continued AI development poses certain catastrophic risk to humanity. They discuss AI versus advanced planning systems (APS), the release of GPT-4, emergent intelligence from modest components, whether deep learning alone will produce AGI, Rice's theorem & the impossibility of predicting alignment, the likelihood that humans try to generalize AI, why the upside of AGI is an illusion, agency vs intelligence, instrumental convergence, implicit agency, deterministic chaos, theories of physics as theories of measurement, the relationship between human desire and AI tools, an analogy with human-animal relations, recognizing & avoiding multipolar traps, an environment increasingly hostile to humans, technology & toxicity, short-term vs long-term risks, why there's so much disagreement about AI risk, the substrate needs hypothesis, an inexorable long-term convergence process, why the only solution is avoiding the cycle, a boiling frog scenario, the displacement of humans, the necessity of understanding evolution, economic decoupling, non-transactional choices, the Forward Great Filter answer to the Fermi paradox, and much more.