Machine Learning Street Talk (MLST)

Support us! https://www.patreon.com/mlst

In this wide-ranging conversation, Tim Scarfe interviews Neel Nanda, a researcher at DeepMind working on mechanistic interpretability, which aims to understand the algorithms and representations learned by machine learning models. Neel discusses how models can represent their thoughts using motifs, circuits, and linear directional features which are often communicated via a "residual stream", an information highway models use to pass information between layers.

Neel argues that "superposition", the ability for models to represent more features than they have neurons, is one of the biggest open problems in interpretability. This is because superposition thwarts our ability to understand models by decomposing them into individual units of analysis. Despite this, Neel remains optimistic that ambitious interpretability is possible, citing examples like his work reverse engineering how models do modular addition. However, Neel notes we must start small, build rigorous foundations, and not assume our theoretical frameworks perfectly match reality.

The conversation turns to whether models can have goals or agency, with Neel arguing they likely can based on heuristics like models executing long term plans towards some objective. However, we currently lack techniques to build models with specific goals, meaning any goals would likely be learned or emergent. Neel highlights how induction heads, circuits models use to track long range dependencies, seem crucial for phenomena like in-context learning to emerge.

On the existential risks from AI, Neel believes we should avoid overly confident claims that models will or will not be dangerous, as we do not understand them enough to make confident theoretical assertions. However, models could pose risks through being misused, having undesirable emergent properties, or being imperfectly aligned. Neel argues we must pursue rigorous empirical work to better understand and ensure model safety, avoid "philosophizing" about definitions of intelligence, and focus on ensuring researchers have standards for what it means to decide a system is "safe" before deploying it. Overall, a thoughtful conversation on one of the most important issues of our time.

Support us! https://www.patreon.com/mlst

MLST Discord: https://discord.gg/aNPkGUQtc5

Twitter: https://twitter.com/MLStreetTalk

Neel Nanda: https://www.neelnanda.io/

TOC

[00:00:00] Introduction and Neel Nanda's Interests (walk and talk)

[00:03:15] Mechanistic Interpretability: Reverse Engineering Neural Networks

[00:13:23] Discord questions

[00:21:16] Main interview kick-off in studio

[00:49:26] Grokking and Sudden Generalization

[00:53:18] The Debate on Systematicity and Compositionality

[01:19:16] How do ML models represent their thoughts

[01:25:51] Do Large Language Models Learn World Models?

[01:53:36] Superposition and Interference in Language Models

[02:43:15] Transformers discussion

[02:49:49] Emergence and In-Context Learning

[03:20:02] Superintelligence/XRisk discussion

Transcript: https://docs.google.com/document/d/1FK1OepdJMrqpFK-_1Q3LQN6QLyLBvBwWW_5z8WrS1RI/edit?usp=sharing

Refs: https://docs.google.com/document/d/115dAroX0PzSduKr5F1V4CWggYcqIoSXYBhcxYktCnqY/edit?usp=sharing

Please check out Numerai - our sponsor using our link @

http://numer.ai/mlst

Numerai is a groundbreaking platform which is taking the data science world by storm. Tim has been using Numerai to build state-of-the-art models which predict the stock market, all while being a part of an inspiring community of data scientists from around the globe. They host the Numerai Data Science Tournament, where data scientists like us use their financial dataset to predict future stock market performance.

Support us! https://www.patreon.com/mlst

MLST Discord: https://discord.gg/aNPkGUQtc5

Twitter: https://twitter.com/MLStreetTalk

YT version: https://youtu.be/axJtywd9Tbo

In this fascinating interview, Dr. Tim Scarfe speaks with renowned philosopher Daniel Dennett about the potential dangers of AI and the concept of "Counterfeit People." Dennett raises concerns about AI being used to create artificial colleagues, and argues that preventing counterfeit AI individuals is crucial for societal trust and security.

They delve into Dennett's "Two Black Boxes" thought experiment, the Chinese Room Argument by John Searle, and discuss the implications of AI in terms of reversibility, reontologisation, and realism. Dr. Scarfe and Dennett also examine adversarial LLMs, mental trajectories, and the emergence of consciousness and semanticity in AI systems.

Throughout the conversation, they touch upon various philosophical perspectives, including Gilbert Ryle's Ghost in the Machine, Chomsky's work, and the importance of competition in academia. Dennett concludes by highlighting the need for legal and technological barriers to protect against the dangers of counterfeit AI creations.

Join Dr. Tim Scarfe and Daniel Dennett in this thought-provoking discussion about the future of AI and the potential challenges we face in preserving our civilization. Don't miss this insightful conversation!

TOC:

00:00:00 Intro

00:09:56 Main show kick off

00:12:04 Counterfeit People

00:16:03 Reversibility

00:20:55 Reontologisation

00:24:43 Realism

00:27:48 Adversarial LLMs are out to get us

00:32:34 Exploring mental trajectories and Chomsky

00:38:53 Gilbert Ryle and Ghost in machine and competition in academia

00:44:32 2 Black boxes thought experiment / intentional stance

01:00:11 Chinese room

01:04:49 Singularitarianism

01:07:22 Emergence of consciousness and semanticity

References:

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

https://arxiv.org/abs/2305.10601

The Problem With Counterfeit People (Daniel Dennett)

https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/

The knowledge argument

https://en.wikipedia.org/wiki/Knowledge_argument

The Intentional Stance

https://www.researchgate.net/publication/271180035_The_Intentional_Stance

Two Black Boxes: a Fable (Daniel Dennett)

https://www.researchgate.net/publication/28762339_Two_Black_Boxes_a_Fable

The Chinese Room Argument (John Searle)

https://plato.stanford.edu/entries/chinese-room/

https://web-archive.southampton.ac.uk/cogprints.org/7150/1/10.1.1.83.5248.pdf

From Bacteria to Bach and Back: The Evolution of Minds (Daniel Dennett)

https://www.amazon.co.uk/Bacteria-Bach-Back-Evolution-Minds/dp/014197804X

Consciousness Explained (Daniel Dennett)

https://www.amazon.co.uk/Consciousness-Explained-Penguin-Science-Dennett/dp/0140128670/

The Mind's I: Fantasies and Reflections on Self and Soul (Hofstadter, Douglas R; Dennett, Daniel C.)

https://www.abebooks.co.uk/servlet/BookDetailsPL?bi=31494476184

#DanielDennett #ArtificialIntelligence #CounterfeitPeople

Support us! https://www.patreon.com/mlst

Please check out Numerai - our sponsor @

https://numerai.com/mlst

Numerai is a groundbreaking platform which is taking the data science world by storm. Tim has been using Numerai to build state-of-the-art models which predict the stock market, all while being a part of an inspiring community of data scientists from around the globe. They host the Numerai Data Science Tournament, where data scientists like us use their financial dataset to predict future stock market performance.

Support us! https://www.patreon.com/mlst

MLST Discord: https://discord.gg/aNPkGUQtc5

Twitter: https://twitter.com/MLStreetTalk

Welcome to an exciting episode featuring an outstanding guest, Robert Miles! Renowned for his extraordinary contributions to understanding AI and its potential impacts on our lives, Robert is an artificial intelligence advocate, researcher, and YouTube sensation. He combines engaging discussions with entertaining content, captivating millions of viewers from around the world.

With a strong computer science background, Robert has been actively involved in AI safety projects, focusing on raising awareness about potential risks and benefits of advanced AI systems. His YouTube channel is celebrated for making AI safety discussions accessible to a diverse audience through breaking down complex topics into easy-to-understand nuggets of knowledge, and you might also recognise him from his appearances on Computerphile.

In this episode, join us as we dive deep into Robert's journey in the world of AI, exploring his insights on AI alignment, superintelligence, and the role of AI shaping our society and future. We'll discuss topics such as the limits of AI capabilities and physics, AI progress and timelines, human-machine hybrid intelligence, AI in conflict and cooperation with humans, and the convergence of AI communities.

Robert Miles:

@RobertMilesAI

https://twitter.com/robertskmiles

https://aisafety.info/

YT version: https://www.youtube.com/watch?v=kMLKbhY0ji0

Panel:

Dr. Tim Scarfe

Dr. Keith Duggar

Joint CTOs - https://xrai.glass/

Refs:

Are Emergent Abilities of Large Language Models a Mirage? (Rylan Schaeffer)

https://arxiv.org/abs/2304.15004

TOC:

Intro [00:00:00]

Numerai Sponsor Messsage [00:02:17]

AI Alignment [00:04:27]

Limits of AI Capabilities and Physics [00:18:00]

AI Progress and Timelines [00:23:52]

AI Arms Race and Innovation [00:31:11]

Human-Machine Hybrid Intelligence [00:38:30]

Understanding and Defining Intelligence [00:42:48]

AI in Conflict and Cooperation with Humans [00:50:13]

Interpretability and Mind Reading in AI [01:03:46]

Mechanistic Interpretability and Deconfusion Research [01:05:53]

Understanding the core concepts of AI [01:07:40]

Moon landing analogy and AI alignment [01:09:42]

Cognitive horizon and limits of human intelligence [01:11:42]

Funding and focus on AI alignment [01:16:18]

Regulating AI technology and potential risks [01:19:17]

Aligning AI with human values and its dynamic nature [01:27:04]

Cooperation and Allyship [01:29:33]

Orthogonality Thesis and Goal Preservation [01:33:15]

Anthropomorphic Language and Intelligent Agents [01:35:31]

Maintaining Variety and Open-ended Existence [01:36:27]

Emergent Abilities of Large Language Models [01:39:22]

Convergence vs Emergence [01:44:04]

Criticism of X-risk and Alignment Communities [01:49:40]

Fusion of AI communities and addressing biases [01:52:51]

AI systems integration into society and understanding them [01:53:29]

Changing opinions on AI topics and learning from past videos [01:54:23]

Utility functions and von Neumann-Morgenstern theorems [01:54:47]

AI Safety FAQ project [01:58:06]

Building a conversation agent using AI safety dataset [02:00:36]

Support us! https://www.patreon.com/mlst

In a historic and candid Senate hearing, OpenAI CEO Sam Altman, Professor Gary Marcus, and IBM's Christina Montgomery discussed the regulatory landscape of AI in the US. The discussion was particularly interesting due to its timing, as it followed the recent release of the EU's proposed AI Act, which could potentially ban American companies like OpenAI and Google from providing API access to generative AI models and impose massive fines for non-compliance.

The speakers openly addressed potential risks of AI technology and emphasized the need for precision regulation. This was a unique approach, as historically, US companies have tried their hardest to avoid regulation. The hearing not only showcased the willingness of industry leaders to engage in discussions on regulation but also demonstrated the need for a balanced approach to avoid stifling innovation.

The EU AI Act, scheduled to come into power in 2026, is still just a proposal, but it has already raised concerns about its impact on the American tech ecosystem and potential conflicts between US and EU laws. With extraterritorial jurisdiction and provisions targeting open-source developers and software distributors like GitHub, the Act could create more problems than it solves by encouraging unsafe AI practices and limiting access to advanced AI technologies.

One core issue with the Act is the designation of foundation models in the highest risk category, primarily due to their open-ended nature. A significant risk theme revolves around users creating harmful content and determining who should be held accountable – the users or the platforms. The Senate hearing served as an essential platform to discuss these pressing concerns and work towards a regulatory framework that promotes both safety and innovation in AI.

00:00 Show

01:35 Legals

03:44 Intro

10:33 Altman intro

14:16 Christina Montgomery

18:20 Gary Marcus

23:15 Jobs

26:01 Scorecards

28:08 Harmful content

29:47 Startups

31:35 What meets the definition of harmful?

32:08 Moratorium

36:11 Social Media

46:17 Gary's take on BingGPT and pivot into policy

48:05 Democratisation

Generative Deep Learning, 2nd Edition [David Foster]

https://www.oreilly.com/library/view/generative-deep-learning/9781098134174/

Support us! https://www.patreon.com/mlst

MLST Discord: https://discord.gg/aNPkGUQtc5

Twitter: https://twitter.com/MLStreetTalk

In this conversation, Tim Scarfe and David Foster, the author of 'Generative Deep Learning,' dive deep into the world of generative AI, discussing topics ranging from model families and auto regressive models to the democratization of AI technology and its potential impact on various industries. They explore the connection between language and true intelligence, as well as the limitations of GPT and other large language models. The discussion also covers the importance of task-independent world models, the concept of active inference, and the potential of combining these ideas with transformer and GPT-style models.

Ethics and regulation in AI development are also discussed, including the need for transparency in data used to train AI models and the responsibility of developers to ensure their creations are not destructive. The conversation touches on the challenges posed by AI-generated content on copyright laws and the diminishing role of effort and skill in copyright due to generative models.

The impact of AI on education and creativity is another key area of discussion, with Tim and David exploring the potential benefits and drawbacks of using AI in the classroom, the need for a balance between traditional learning methods and AI-assisted learning, and the importance of teaching students to use AI tools critically and responsibly.

Generative AI in music is also explored, with David and Tim discussing the potential for AI-generated music to change the way we create and consume art, as well as the challenges in training AI models to generate music that captures human emotions and experiences.

Throughout the conversation, Tim and David touch on the potential risks and consequences of AI becoming too powerful, the importance of maintaining control over the technology, and the possibility of government intervention and regulation. The discussion concludes with a thought experiment about AI predicting human actions and creating transient capabilities that could lead to doom.

TOC:

Introducing Generative Deep Learning [00:00:00]

Model Families in Generative Modeling [00:02:25]

Auto Regressive Models and Recurrence [00:06:26]

Language and True Intelligence [00:15:07]

Language, Reality, and World Models [00:19:10]

AI, Human Experience, and Understanding [00:23:09]

GPTs Limitations and World Modeling [00:27:52]

Task-Independent Modeling and Cybernetic Loop [00:33:55]

Collective Intelligence and Emergence [00:36:01]

Active Inference vs. Reinforcement Learning [00:38:02]

Combining Active Inference with Transformers [00:41:55]

Decentralized AI and Collective Intelligence [00:47:46]

Regulation and Ethics in AI Development [00:53:59]

AI-Generated Content and Copyright Laws [00:57:06]

Effort, Skill, and AI Models in Copyright [00:57:59]

AI Alignment and Scale of AI Models [00:59:51]

Democratization of AI: GPT-3 and GPT-4 [01:03:20]

Context Window Size and Vector Databases [01:10:31]

Attention Mechanisms and Hierarchies [01:15:04]

Benefits and Limitations of Language Models [01:16:04]

AI in Education: Risks and Benefits [01:19:41]

AI Tools and Critical Thinking in the Classroom [01:29:26]

Impact of Language Models on Assessment and Creativity [01:35:09]

Generative AI in Music and Creative Arts [01:47:55]

Challenges and Opportunities in Generative Music [01:52:11]

AI-Generated Music and Human Emotions [01:54:31]

Language Modeling vs. Music Modeling [02:01:58]

Democratization of AI and Industry Impact [02:07:38]

Recursive Self-Improving Superintelligence [02:12:48]

AI Technologies: Positive and Negative Impacts [02:14:44]

Runaway AGI and Control Over AI [02:20:35]

AI Dangers, Cybercrime, and Ethics [02:23:42]

https://www.perplexity.ai/

https://www.perplexity.ai/iphone

https://www.perplexity.ai/android

YT version: https://youtu.be/_vMOWw3uYvk

https://scholar.google.com/citations?user=GhrKC1gAAAAJ&hl=en

https://twitter.com/aravsrinivas?lang=en

The Importance of a Decluttered UI and User Experience [00:04:19]

Advertising in Search Engines and Potential Improvements [00:09:02]

Challenges and Opportunities in this new Search Modality [00:18:17]

Benefits of Perplexity and Personalized Learning [00:21:27]

Objective Truth and Personalized Wikipedia [00:26:34]

Opinions and Truth in Answer Engines [00:30:53]

Embracing the Digital Society with Language Models [00:37:30]

Impact on Jobs and Future of Learning [00:40:13]

Educating users on when perplexity works and doesn't work [00:43:13]

Improving user experience and the possibilities of voice-to-voice interaction [00:45:04]

The future of language models and auto-regressive models [00:49:51]

Performance of GPT-4 and potential improvements [00:52:31]

Building the ultimate research and knowledge assistant [00:55:33]

Revolutionizing note-taking and personal knowledge stores [00:58:16]

Patreon: https://www.patreon.com/mlst

In this exclusive interview, Dr. Tim Scarfe sits down with Minqi Jiang, a leading PhD student at University College London and Meta AI, as they delve into the fascinating world of deep reinforcement learning (RL) and its impact on technology, startups, and research. Discover how Minqi made the crucial decision to pursue a PhD in this exciting field, and learn from his valuable startup experiences and lessons.

Minqi shares his insights into balancing serendipity and planning in life and research, and explains the role of objectives and Goodhart's Law in decision-making. Get ready to explore the depths of robustness in RL, two-player zero-sum games, and the differences between RL and supervised learning.

As they discuss the role of environment in intelligence, emergence, and abstraction, prepare to be blown away by the possibilities of open-endedness and the intelligence explosion. Learn how language models generate their own training data, the limitations of RL, and the future of software 2.0 with interpretability concerns.

From robotics and open-ended learning applications to learning potential metrics and MDPs, this interview is a goldmine of information for anyone interested in AI, RL, and the cutting edge of technology. Don't miss out on this incredible opportunity to learn from a rising star in the AI world!

TOC

Tech & Startup Background [00:00:00]

Pursuing PhD in Deep RL [00:03:59]

Startup Lessons [00:11:33]

Serendipity vs Planning [00:12:30]

Objectives & Decision Making [00:19:19]

Minimax Regret & Uncertainty [00:22:57]

Robustness in RL & Zero-Sum Games [00:26:14]

RL vs Supervised Learning [00:34:04]

Exploration & Intelligence [00:41:27]

Environment, Emergence, Abstraction [00:46:31]

Open-endedness & Intelligence Explosion [00:54:28]

Language Models & Training Data [01:04:59]

RLHF & Language Models [01:16:37]

Creativity in Language Models [01:27:25]

Limitations of RL [01:40:58]

Software 2.0 & Interpretability [01:45:11]

Language Models & Code Reliability [01:48:23]

Robust Prioritized Level Replay [01:51:42]

Open-ended Learning [01:55:57]

Auto-curriculum & Deep RL [02:08:48]

Robotics & Open-ended Learning [02:31:05]

Learning Potential & MDPs [02:36:20]

Universal Function Space [02:42:02]

Goal-Directed Learning & Auto-Curricula [02:42:48]

Advice & Closing Thoughts [02:44:47]

References:

- Why Greatness Cannot Be Planned: The Myth of the Objective by Kenneth O. Stanley and Joel Lehman

https://www.springer.com/gp/book/9783319155234

- Rethinking Exploration: General Intelligence Requires Rethinking Exploration

https://arxiv.org/abs/2106.06860

- The Case for Strong Emergence (Sabine Hossenfelder)

https://arxiv.org/abs/2102.07740

- The Game of Life (Conway)

https://www.conwaylife.com/

- Toolformer: Teaching Language Models to Generate APIs (Meta AI)

https://arxiv.org/abs/2302.04761

- OpenAI's POET: Paired Open-Ended Trailblazer

https://arxiv.org/abs/1901.01753

- Schmidhuber's Artificial Curiosity

https://people.idsia.ch/~juergen/interest.html

- Gödel Machines

https://people.idsia.ch/~juergen/goedelmachine.html

- PowerPlay

https://arxiv.org/abs/1112.5309

- Robust Prioritized Level Replay: https://openreview.net/forum?id=NfZ6g2OmXEk

- Unsupervised Environment Design: https://arxiv.org/abs/2012.02096

- Excel: Evolving Curriculum Learning for Deep Reinforcement Learning

https://arxiv.org/abs/1901.05431

- Go-Explore: A New Approach for Hard-Exploration Problems

https://arxiv.org/abs/1901.10995

- Learning with AMIGo: Adversarially Motivated Intrinsic Goals

https://www.researchgate.net/publication/342377312_Learning_with_AMIGo_Adversarially_Motivated_Intrinsic_Goals

PRML

https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf

Sutton and Barto

https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

Patreon: https://www.patreon.com/mlst

Discord: https://discord.gg/ESrGqhf5CB

Twitter: https://twitter.com/MLStreetTalk

Chris Eliasmith is a renowned interdisciplinary researcher, author, and professor at the University of Waterloo, where he holds the prestigious Canada Research Chair in Theoretical Neuroscience. As the Founding Director of the Centre for Theoretical Neuroscience, Eliasmith leads the Computational Neuroscience Research Group in exploring the mysteries of the brain and its complex functions. His groundbreaking work, including the Neural Engineering Framework, Neural Engineering Objects software environment, and the Semantic Pointer Architecture, has led to the development of Spaun, the most advanced functional brain simulation to date. Among his numerous achievements, Eliasmith has received the 2015 NSERC "Polany-ee" Award and authored two influential books, "How to Build a Brain" and "Neural Engineering."

Chris' homepage:

http://arts.uwaterloo.ca/~celiasmi/

Interviewers: Dr. Tim Scarfe and Dr. Keith Duggar

TOC:

Intro to Chris [00:00:00]

Continuous Representation in Biologically Plausible Neural Networks [00:06:49]

Legendre Memory Unit and Spatial Semantic Pointer [00:14:36]

Large Contexts and Data in Language Models [00:20:30]

Spatial Semantic Pointers and Continuous Representations [00:24:38]

Auto Convolution [00:30:12]

Abstractions and the Continuity [00:36:33]

Compression, Sparsity, and Brain Representations [00:42:52]

Continual Learning and Real-World Interactions [00:48:05]

Robust Generalization in LLMs and Priors [00:56:11]

Chip design [01:00:41]

Chomsky + Computational Power of NNs and Recursion [01:04:02]

Spiking Neural Networks and Applications [01:13:07]

Limits of Empirical Learning [01:22:43]

Philosophy of Mind, Consciousness etc [01:25:35]

Future of human machine interaction [01:41:28]

Future research and advice to young researchers [01:45:06]

Refs:

http://compneuro.uwaterloo.ca/publications/dumont2023.html 

http://compneuro.uwaterloo.ca/publications/voelker2019lmu.html 

http://compneuro.uwaterloo.ca/publications/voelker2018.html

http://compneuro.uwaterloo.ca/publications/lu2019.html 

https://www.youtube.com/watch?v=I5h-xjddzlY