The Gradient: Perspectives on AI

In episode 88 of The Gradient Podcast, Daniel Bashir speaks to Professor Stevan Harnad.

Stevan Harnad is professor of psychology and cognitive science at Université du Québec à Montréal, adjunct professor of cognitive science at McGill University, and professor emeritus of cognitive science at the University of Southampton. His research is on category learning, categorical perception, symbol grounding, the evolution of language, and animal and human sentience (otherwise known as “consciousness”). He is also an advocate for open access and an activist for animal rights.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (05:20) Professor Harnad’s background: interests in cognitive psychobiology, editing Behavioral and Brain Sciences

* (07:40) John Searle submits the Chinese Room article

* (09:20) Early reactions to Searle and Prof. Harnad’s role

* (13:38) The core of Searle’s argument and the generator of the Symbol Grounding Problem, “strong AI”

* (19:00) Ways to ground symbols

* (20:26) The acquisition of categories

* (25:00) Pantomiming, non-linguistic category formation

* (27:45) Mathematics, abstraction, and grounding

* (36:20) Symbol manipulation and interpretation language

* (40:40) On the Whorf Hypothesis

* (48:39) Defining “grounding” and introducing the “T3” Turing Test

* (53:22) Turing’s concerns, AI and reverse-engineering cognition

* (59:25) Other Minds, T4 and zombies

* (1:05:48) Degrees of freedom in solutions to the Turing Test, the easy and hard problems of cognition

* (1:14:33) Over-interepretation of AI systems’ behavior, sentience concerns, T3 and evidence sentience

* (1:24:35) Prof. Harnad’s commentary on claims in The Vector Grounding Problem

* (1:28:05) RLHF and grounding, LLMs’ (ungrounded) capabilities, syntactic structure and propositions

* (1:35:30) Multimodal AI systems (image-text and robotic) and grounding, compositionality

* (1:42:50) Chomsky’s Universal Grammar, LLMs and T2

* (1:50:55) T3 and cognitive simulation

* (1:57:34) Outro

Links:

* Professor Harnad’s webpage and skywritings

* Papers:

* Category Induction and Representation

* Categorical Perception

* From Sensorimotor Categories to Grounded Symbols

* Minds, machines and Searle 2

* The Latent Structure of Dictionaries

In episode 87 of The Gradient Podcast, Daniel Bashir speaks to Professor Terry Winograd.

Professor Winograd is Professor Emeritus of Computer Science at Stanford University. His research focuses on human-computer interaction design and the design of technologies for development. He founded the Stanford Human-Computer Interaction Group, where he directed the teaching programs and HCI research. He is also a founding faculty member of the Stanford d.school and a founding member and past president of Computer Professionals for Social Responsibility.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (03:00) Professor Winograd’s background

* (05:10) At the MIT AI Lab

* (05:45) The atmosphere in the MIT AI Lab, Minsky/Chomsky debates

* (06:20) Blue-sky research, government funding for academic research

* (10:10) Isolation and collaboration between research groups

* (11:45) Phases in the development of ideas and how cross-disciplinary work fits in

* (12:26) SHRDLU and the MIT AI Lab’s intellectual roots

* (17:20) Early responses to SHRDLU: Minsky, Dreyfus, others

* (20:55) How Prof. Winograd’s thinking about AI’s abilities and limitations evolved

* (22:25) How this relates to current AI systems and discussions of intelligence

* (23:47) Repetitive debates in AI, semantics and grounding

* (27:00) The concept of investment, care, trust in human communication vs machine communication

* (28:53) Projecting human-ness onto AI systems and non-human things and what this means for society

* (31:30) Time after leaving MIT in 1973, time at Xerox PARC, how Winograd’s thinking evolved during this time

* (38:28) What Does It Mean to Understand Language? Speech acts, commitments, and the grounding of language

* (42:40) Reification of representations in science and ML

* (46:15) LLMs, their training processes, and their behavior

* (49:40) How do we coexist with systems that we don’t understand?

* (51:20) Progress narratives in AI and human agency

* (53:30) Transitioning to intelligence augmentation, founding the Stanford HCI group and d.school, advising Larry Page and Sergey Brin

* (1:01:25) Chatbots and how we consume information

* (1:06:52) Evolutions in journalism, progress in trust for modern AI systems

* (1:09:18) Shifts in the social contract, from institutions to personalities

* (1:12:05) AI and HCI in recent years

* (1:17:05) Philosophy of design and the d.school

* (1:21:20) Designing AI systems for people

* (1:25:10) Prof. Winograd’s perspective on watermarking for detecting GPT outputs

* (1:25:55) The politics of being a technologist

* (1:30:10) Echos of the past in AI regulation and competition and learning from history

* (1:32:34) Outro

Links:

* Professor Winograd’s Homepage

* Papers/topics discussed:

* SHRDLU

* Beyond Programming Languages

* What Does It Mean to Understand Language?

* The PageRank Citation Ranking

* Stanford Digital Libraries project

* Talk: My Politics as a Technologist

In episode 86 of The Gradient Podcast, Daniel Bashir speaks to Professor Gil Strang.

Professor Strang is one of the world’s foremost mathematics educators and a mathematician with contributions to finite element theory, the calculus of variations, wavelet analysis, and linear algebra. He has spent six decades teaching mathematics at MIT, where he was the MathWorks Professor of Mathematics. He was among the first MIT faculty members to publish a course on MIT’s OpenCourseware and has since championed both linear algebra education and open courseware.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:00) Professor Strang’s background and journey into teaching linear algebra

* (04:55) Undergrad interests

* (07:10) Writing textbooks

* (10:20) Prof. Strang’s interests in deep learning

* (11:00) How Professor Strang thought about teaching early on

* (16:20) MIT OpenCourseWare and education accessibility

* (19:50) Prof Strang’s applied/example-based approach to teaching linear algebra and closing the theory-practice gap

* (22:00) Examples!

* (27:20) Orthogonality

* (29:15) Singular values

* (34:40) Professor Strang’s favorite topics in linear algebra

* (37:55) Pedagogical approaches to deep learning, mathematical ingredients of deep learning’s complexity

* (42:04) Generalization and double descent in deep learning, powers and limitations

* (46:20) Did deep learning have to evolve as it did?

* (48:30) Teaching deep learning to younger students

* (50:50) How Prof. Strang’s approach to teaching linear algebra has evolved over time

* (53:00) The Four Fundamental Subspaces

* (56:15) Reflections on a career in teaching

* (59:49) Outro

Links:

* Professor Strang’s homepage

In episode 85 of The Gradient Podcast, Andrey Kurenkov speaks to Anant Agarwal

Anant Agarwal is the chief platform officer of 2U, and founder of edX. Anant taught the first edX course on circuits and electronics from MIT, which drew 155,000 students from 162 countries. He has served as the director of CSAIL, MIT's Computer Science and Artificial Intelligence Laboratory, and is a professor of electrical engineering and computer science at MIT. He is a successful serial entrepreneur, having co-founded several companies including Tilera Corporation, which created the Tile multicore processor, and Virtual Machine Works.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:30) History with research

* (05:56) Founding EdX

* (13:05) AI at EdX

* (18:40) Reaction to AI as a teacher

* (25:00) Student interest in AI

* (32:20) AI’s impact on academia

* (35:00) Future of AI in education

* (38:25) AI writing essays

* (43:38) Experiences playing with ChatGPT

In episode 84 of The Gradient Podcast, Daniel Bashir speaks to Professor Raphaël Millière.

Professor Millière is a Lecturer (Assistant Professor) in the Philosophy of Artificial Intelligence at Macquarie University in Sydney, Australia. Previously, he was the 2020 Robert A. Burt Presidential Scholar in Society and Neuroscience in Columbia University’s Center for Science and Society, and completed his DPhil in philosophy at the University of Oxford, where he focused on self-consciousness.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:20) Prof. Millière’s background

* (08:07) AI + philosophy questions and the human side / empiricism

* (18:38) Putting aside metaphysical issues

* (20:28) Prof. Millière’s work on self-consciousness, does consciousness constitutively involve self-consciousness?

* (32:05) Relationship to recent pronouncements about AI sentience

* (41:54) Chatbots’ self-presentation as having a “self”

* (51:05) Intro to grounding and related concepts

* (1:00:06) The different types of grounding

* (1:08:48) Lexical representations and things in the world, distributional hypothesis, concepts in LLMs

* (1:21:40) Representational content and overcoming the vector grounding problem

* (1:32:01) Causal-informational relations and teleology

* (1:43:45) Levels of grounding, extralinguistic aspects of meaning

* (1:52:12) Future problems and ongoing projects

* (2:04:05) Outro

Links:

* Professor Millière’s homepage and Twitter

* Research

* Are There Degrees of Self-Consciousness?

* The Varieties of Selflessness

* Selfless Memories

* The Vector Grounding Problem

In episode 83 of The Gradient Podcast, Daniel Bashir speaks to Peli Grietzer.

Peli is a scholar whose work borrows mathematical ideas from machine learning theory to think through “ambient” and ineffable phenomena like moods, vibes, cultural logics, and structures of feeling. He is working on a book titled Big Mood: A Transcendental-Computational Essay in Art and contributes to the experimental literature collective Gauss PDF. Peli has a PhD in mathematically informed literary theory from Harvard Comparative Literature in collaboration with the HUJI Einstein Institute of Mathematics.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:17) Peli’s background

* (10:40) Daniel takes 2 entire minutes to ask how Peli thinks about ~ Art ~

* (26:10) Idealism and art as revealing the nature of reality, extralinguistic experiences of truth through literature

* (52:05) The autoencoder as a way to understand Romantic theories of art

* (1:14:55) More on how Peli thinks about autoencoders

* (1:18:05) Connections to ambient meaning, stimmung/mood

* (1:37:18) Examples of poetry/literature as mathematical experience, aesthetic unity and totalizing worldviews

* (1:51:15) Moods clashing within a single work

* (2:10:14) Modernist writers

* (2:32:46) Outro

Links:

* Peli’s Twitter

* A Theory of Vibe

* Why poetry is a variety of mathematical experience

* Peli’s thesis

In episode 82 of The Gradient Podcast, Daniel Bashir speaks to Ryan Drapeau.

Ryan is a Staff Software Engineer at Stripe and technical lead for Stripe’s Payment Fraud organization, which uses machine learning to help prevent billions of dollars of credit card and payments fraud for business every year.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:15) Ryan’s background

* (05:28) Differences between adversarial problems (fraud, content moderation, etc.)

* (08:50) How fraud manifests for businesses

* (11:07) Types of fraud

* (15:49) Fraud as an industry

* (19:05) Information asymmetries between fraudsters and defenders

* (22:40) Fraud as an ML problem and Stripe Radar

* (25:45) Evolution of Stripe Radar

* (31:38) Architectural evolution

* (41:38) Why ResNets for Stripe Radar?

* (44:15) Future architectures for Stripe Radar and the explainability/performance tradeoff

* (48:58) War stories

* (52:55) Federated learning opportunities for Stripe Radar

* (55:50) Vectors for improvement in Stripe’s fraud detection systems

* (59:22) More ways of thinking about the fraud problem, multiclass models

* (1:03:30) Lessons Ryan has picked up from working on fraud

* (1:05:44) Outro

Links:

* How We Built It: Stripe Radar

* Stripe 2022 Update

In episode 81 of The Gradient Podcast, Daniel Bashir speaks to Shiv Rao.

Shiv Rao, MD is the co-founder and CEO of Abridge, a healthcare conversation company that uses cutting-edge NLP and generative AI to bring context and understanding to every medical conversation. Shiv previously served as an Executive Vice President at UPMC Enterprises, managing the provider-facing portfolio of technology investments and R&D. He is a practicing cardiologist in UPMC's Heart and Vascular Institute.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:34) Shiv’s medicine/technology/VC background

* (05:45) Difficulties for tech in healthcare and how this informs Shiv’s approach

* (10:52) “Productivity with a smile” and how AI can make medicine feel more human

* (12:35) Shiv’s experiences in medicine and how Abridge’s product helps doctors

* (16:10) How the role of a clinical team could evolve

* (19:30) Abridge’s partnerships and real-life use cases

* (23:00) Shiv’s perspectives on concerns about bias/trust/privacy

* (25:25) Clinical decision support vs “automating doctors”

* (29:07) Transparency and Abridge’s user experience

* (35:20) Algorithmic solutionism vs human-focused approaches to technology development

* (38:50) Ways AI might impact healthcare

* (41:10) Generative AI applications

* (45:00) Generative AI opportunities beyond documentation

* (49:25) Innovation and reducing friction, UX

* (50:56) Why people make wild predictions about AI

* (54:25) What it means to “automate away” a doctor, how we’re misusing the medical workforce

* (56:10) Shiv’s advice for people interested in AI + healthcare

* (1:00:04) Outro

Links:

* Abridge Homepage

In episode 80 of The Gradient Podcast, Daniel Bashir speaks to Professor Hugo Larochelle.

Professor Larochelle leads the Montreal Google DeepMind team and is adjunct professor at Université de Montréal and a Canada CIFAR Chair. His research focuses on the study and development of deep learning algorithms.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast: Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:38) Prof. Larochelle’s background, working in Bengio’s lab

* (04:53) Prof. Larochelle’s work and connectionism

* (08:20) 2004-2009, work with Bengio

* (08:40) Nonlocal Estimation of Manifold Structure, manifolds and deep learning

* (13:58) Manifold learning in vision and language

* (16:00) Relationship to Denoising Autoencoders and greedy layer-wise pretraining

* (21:00) From input copying to learning about local distribution structure

* (22:30) Zero-Data Learning of New Tasks

* (22:45) The phrase “extend machine learning towards AI” and terminology

* (26:55) Prescient hints of prompt engineering

* (29:10) Daniel goes on totally unnecessary tangent

* (30:00) Methods for training deep networks (strategies and robust interdependent codes)

* (33:45) Motivations for layer-wise pretraining

* (35:15) Robust Interdependent Codes and interactions between neurons in a single network layer

* (39:00) 2009-2011, postdoc in Geoff Hinton’s lab

* (40:00) Reflections on the AlexNet moment

* (41:45) Frustration with methods for evaluating unsupervised methods, NADE

* (44:45) How researchers thought about representation learning, toying with objectives instead of architectures

* (47:40) The Restricted Boltzmann Forest

* (50:45) Imposing structure for tractable learning of distributions

* (53:11) 2011-2016 at U Sherbooke (and Twitter)

* (53:45) How Prof. Larochelle approached research problems

* (56:00) How Domain Adversarial Networks came about

* (57:12) Can we still learn from Restricted Boltzmann Machines?

* (1:02:20) The ~ Infinite ~ Restricted Boltzmann Machine

* (1:06:55) The need for researchers doing different sorts of work

* (1:08:58) 2017-present, at MILA (and Google)

* (1:09:30) Modulating Early Visual Processing by Language, neuroscientific inspiration

* (1:13:22) Representation learning and generalization, what is a good representation (Meta-Dataset, Universal representation transformer layer, universal template, Head2Toe)

* (1:15:10) Meta-Dataset motivation

* (1:18:00) Shifting focus to the problem—good practices for “recycling deep learning”

* (1:19:15) Head2Toe intuitions

* (1:21:40) What are “universal representations” and manifold perspective on datasets, what is the right pretraining dataset

* (1:26:02) Prof. Larochelle’s takeaways from Fortuitous Forgetting in Connectionist Networks (led by Hattie Zhou)

* (1:32:15) Obligatory commentary on The Present Moment and current directions in ML

* (1:36:18) The creation and motivations of the TMLR journal

* (1:41:48) Prof. Larochelle’s takeaways about doing good science, building research groups, and nurturing a research environment

* (1:44:05) Prof. Larochelle’s advice for aspiring researchers today

* (1:47:41) Outro

Links:

* Professor Larochelle’s homepage and Twitter

* Transactions on Machine Learning Research

* Papers

* 2004-2009

* Nonlocal Estimation of Manifold Structure

* Classification using Discriminative Restricted Boltzmann Machines

* Zero-data learning of new tasks

* Exploring Strategies for Training Deep Neural Networks

* Deep Learning using Robust Interdependent Codes

* 2009-2011

* Stacked Denoising Autoencoders

* Tractable multivariate binary density estimation and the restricted Boltzmann forest

* The Neural Autoregressive Distribution Estimator

* Learning Attentional Policies for Tracking and Recognition in Video with Deep Networks

* 2011-2016

* Practical Bayesian Optimization of Machine Learning Algorithms

* Learning Algorithms for the Classification Restricted Boltzmann Machine

* A neural autoregressive topic model

* Domain-Adversarial Training of Neural Networks

* NADE

* An Infinite Restricted Boltzmann Machine

* 2017-present

* Modulating early visual processing by language

* Meta-Dataset

* A Universal Representation Transformer Layer for Few-Shot Image Classification

* Learning a universal template for few-shot dataset generalization

* Impact of aliasing on generalization in deep convolutional networks

* Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning

* Fortuitous Forgetting in Connectionist Networks

In episode 79 of The Gradient Podcast, Daniel Bashir speaks to Jeremie Harris.

Jeremie is co-founder of Gladstone AI, author of the book Quantum Physics Made Me Do It, and co-host of the Last Week in AI Podcast. Jeremy previously hosted the Towards Data Science podcast and worked on a number of other startups after leaving a PhD in physics.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:37) Jeremie’s physics background and transition to ML

* (05:19) The physicist-to-AI person pipeline, how Jeremie’s background impacts his approach to AI

* (08:20) A tangent on inflationism/deflationism about natural laws (I promise this applies to AI)

* (11:45) How ML implies a particular viewpoint on the above question

* (13:20) Jeremie’s first (recommendation systems) company, how startup founders can make mistakes even when they’ve read Paul Graham essays

* (17:30) Classic startup wisdom, different sorts of startups

* (19:35) OpenAI’s approach in shipping features for DALL-E 2 and generation vs. discrimination as an approach to product

* (24:55) Capabilities and risk

* (26:43) Commentary on fundamental limitations of alignment in LLMs

* (30:45) Intrinsic difficulties in alignment problems

* (41:15) Daniel tries to steel man / defend anti-longtermist arguments (nicely :) )

* (46:23) Anthropic’s paper on asking models to be less biased

* (47:20) Why Jeremie is excited about Anthropic’s Constitutional AI scheme

* (51:05) Jeremie’s thoughts on recent Eliezer discourse

* (56:50) Cheese / task vectors and steerability/controllability in LLMs

* (59:50) Difficulty of one-shot solutions in alignment work, better strategies

* (1:02:00) Lack of theoretical understanding of deep learning systems / alignment

* (1:04:50) Jeremie’s work and perspectives on AI policy

* (1:10:00) Incrementality in convincing policymakers

* (1:14:00) How recent developments impact policy efforts

* (1:16:20) Benefits and drawbacks of open source

* (1:19:30) Arguments in favor of (limited) open source

* (1:20:35) Quantum Physics (not Mechanics) Made Me Do It

* (1:24:10) Some theories of consciousness and corresponding physics

* (1:29:49) Outro

Links:

* Jeremie’s Twitter

* Quantum Physics Made Me Do It

* Gladstone AI