Jim talks with Melanie Mitchell about her critique of applying standardized exams to LLMs and the debate over understanding in AI. They discuss ChatGPT and GPT-4's performance on standardized exams, questioning the underlying assumptions, OpenAI's lack of transparency, soon-to-be-released open-source LLMs, prompt engineering, making GPT its own skyhook to reduce hallucinations, the number of parameters in GPT-4, why LLMs should be probed differently than humans, how LLMs lie differently than humans, Stanford's holistic assessment for LLMs, a College Board for LLMs, why the term "understanding" is overstressed today, consciousness vs intelligence, the human drive for compression, working memory limitations as the secret to human intellectual abilities, episodic memory, embodied emotions, the idea that AIs don't care, calling for a new science of intelligence, the effects of differing evolutionary pressures, whether a model of physics could emerge from language learning, how little we understand these systems, and much more.

Episode Transcript

JRS Currents 036: Melanie Mitchell on Why AI is Hard

Complexity: A Guided Tour, by Melanie Mitchell

Artificial Intelligence: A Guide for Thinking Humans, by Melanie Mitchell

AI: A Guide for Thinking Humans (Substack)

"Did ChatGPT Really Pass Graduate-Level Exams?" (Part 1), by Melanie Mitchell

Currents 087: Shivanshu Purohit on Open-Source Generative AI

Holistic Evaluation of Language Models (HELM) - Stanford

"The Debate Over Understanding in AI's Large Language Models," by Melanie Mitchell and David Krakauer

Melanie Mitchell is Professor of Computer Science at Portland State University, and External Professor and Co-Chair of the Science Board at the Santa Fe Institute. Mitchell has also held faculty or professional positions at the University of Michigan, Los Alamos National Laboratory, and the OGI School of Science and Engineering. She is the author or editor of seven books and numerous scholarly papers in the fields of artificial intelligence, cognitive science, and complex systems, including her latest, Artificial Intelligence: A Guide for Thinking Humans.