The Gradient: Perspectives on AI

In episode 98 of The Gradient Podcast, Daniel Bashir speaks to Laurence Liew.

Laurence is the Director for AI Innovation at AI Singapore. He is driving the adoption of AI by the Singapore ecosystem through the 100 Experiments, AI Apprenticeship Programmes and the Generational AI Talent Development initiative. He is the current Co-Chair of the Innovations and Commercialisation working group and Co-Chair of the "Broad Adoption of AI by SME" committee.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:25) Laurence’s background

* (07:00) AI Singapore and Singapore’s AI Strategy

* (08:27) Awareness and adoption of AI in Singapore

* (19:45) AI Apprenticeship Program stories

* (27:35) Developing generational AI talent within Singapore, literacy

* (32:25) Singapore’s place within the global AI ecosystem

* (38:30) How the generative AI boom has affected Singapore

* (43:50) Laurence’s vision for the future of Singapore’s tech ecosystem

* (49:41) Outro

Links:

* AI Singapore

In episode 97 of The Gradient Podcast, Daniel Bashir speaks to Professor Michael Levin and Adam Goldstein.

Professor Levin is a Distinguished Professor and Vannevar Bush Chair in the Biology Department at Tufts University. He also directs the Allen Discovery Center at Tufts. His group, the Levin Lab, focuses on understanding the biophysical mechanisms that implement decision-making during complex pattern regulation, and harnessing endogenous bioelectric dynamics toward rational control of growth and form. 

Adam Goldstein was a visiting scientist at the Levin Lab, where he worked on cancer research, and is the co-founder and Chairman of Astonishing Labs. Previously Adam founded Hipmunk, wrote tech books for O'Reilly, and was a Visiting Partner at Y Combinator.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:37) Intros

* (03:20) Prof. Levin intro

* (04:26) Adam intro

* (06:25) A perspective on intelligence

* (08:40) Diverse intelligence — in unconventional embodiments and unfamiliar spaces, substrate independence

* (12:23) Failure of the life-machine distinction, text-based systems, grounding, and embodiment

* (16:12) What it is to be a Self, fluidity and persistence

* (22:45) The combination problem in cognitive function, levels and representation

* (27:10) Goals for AI / cognitive science, Prof Levin’s perspective on building intelligent systems

* (31:25) Adam’s and Prof. Levin’s recent research—regenerative medicine and cancer

* (36:25) Examples of regeneration, Adam on the right approach to the regeneration problem as generation

* (45:25) Protein engineering vs. Adam and Prof. Levin’s program, implicit assumptions underlying biology

* (48:15) Regeneration example in liver disease

* (50:50) Perspectives on AI and its goals

Links:

* Levin Lab homepage

* Forms of life, forms of mind

* Adam’s homepage

* Research

* On Having No Head: Cognition throughout Biological Systems

* Technological Approach to Mind Everywhere

* Living Things Are Not (20th Century) Machines: Updating Mechanism Metaphors in Light of the Modern Science of Machine Behavior

* Life, death, and self: Fundamental questions of primitive cognition viewed through the lens of body plasticity and synthetic organisms

* Modular cognition

* Endless Forms

* Future Medicine: from molecular pathways to the collective intelligence of the body

* Technological Approach to Mind Everywhere: an experimentally-grounded framework for understanding diverse bodies and minds

* The Computational Boundary of a “Self”: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition

* Machine life

In episode 96 of The Gradient Podcast, Daniel Bashir speaks to Jonathan Frankle.

Jonathan is the Chief Scientist at MosaicML and (as of release). Jonathan completed his PhD at MIT, where he investigated the properties of sparse neural networks that allow them to train effectively through his lottery ticket hypothesis. He also spends a portion of his time working on technology policy, and currently works with the OECD to implement the AI principles he helped develop in 2019.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:35) Jonathan’s background and work

* (04:25) Origins of the Lottery Ticket Hypothesis

* (06:00) Jonathan’s empiricism and approach to science

* (08:25) More Karl Popper discourse + hot takes

* (09:45) Walkthrough of the Lottery Ticket Hypothesis

* (12:00) Issues with the Lottery Ticket Hypothesis as a statement

* (12:30) Jonathan’s advice for PhD students, on asking good questions

* (15:55) Strengths and Promise of the Lottery Ticket Hypothesis

* (18:55) More Lottery Ticket Hypothesis Papers

* (19:10) Comparing Rewinding and Fine-tuning

* (23:00) Care in making experimental choices

* (25:05) Linear Mode Connectivity and the Lottery Ticket Hypothesis

* (27:50) On what is being measured and how

* (28:50) “The outcome of optimization is determined to a linearly connected region”

* (31:15) On good metrics

* (32:54) On the Predictability of Pruning Across Scales — scaling laws for pruning

* (34:40) The paper’s takeaway

* (38:45) Pruning Neural Networks at Initialization — on a scientific disagreement

* (45:00) On making takedown papers useful

* (46:15) On what can be known early in training

* (49:15) Jonathan’s perspective on important research questions today

* (54:40) MosaicML

* (55:19) How Mosaic got started

* (56:17) Mosaic highlights

* (57:33) Customer stories

* (1:00:30) Jonathan’s work and perspectives on AI policy

* (1:05:45) The key question: what we want

* (1:07:35) Outro

Links:

* Jonathan’s homepage and Twitter

* Papers

* The Lottery Ticket Hypothesis and follow-up work

* Comparing Rewinding and Fine-tuning in Neural Network Pruning

* Linear Mode Connectivity and the LTH

* On the Predictability of Pruning Across Scales

* Pruning Neural Networks at Initialization: Why Are We Missing The Mark?

* Desirable Inefficiency

In episode 95 of The Gradient Podcast, Daniel Bashir speaks to Nao Tokui.

Nao Tokui is an artist/DJ and researcher based in Tokyo. While pursuing his Ph.D. at The University of Tokyo, he produced his first music album and singles using AI, including a 12-inch record with Nujabes, a legendary Japanese hip-hop producer. After completing his Ph.D. research, he founded Qosmo, AI Creativity and Music Lab, in 2009. Since then, he has been actively working at the intersection of AI technology and art. Nao and his team's works have been exhibited at renowned venues such as the New York MoMA and the Barbican Centre in London. Their performances have also been showcased at various music festivals, including MUTEK and Sonar. Additionally, he is leading the development of AI-based music instruments at his newly founded company, Neutone. In 2021, Nao received the Okawa Publishing Award for his Japanese book on art, creativity, and AI. The book is scheduled to be released in English as "Surfing human creativity with AI — A user's guide" in November 2023.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:15) Nao’s background and how he got into AI and music

* (05:10) Nao’s experiences as a DJ, collaboration with Nujabes

* (07:10) HCI and music

* (10:35) Leveraging the difference between AI systems and humans

* (12:40) Total control vs total chaos

* (13:45) Qosmo and the Neutone Project, misusable AI tools

* (17:25) On music and “creating something new”

* (21:00) Declarative and top-down vs. bottom-up creation, individual taste

* (23:50) How generative AI enables humans

* (26:25) On misusing technology and art

* (32:00) Dawn Patrol EP

* (36:00) A two-discriminator GAN for creating music in new genres

* (37:45) The AI DJ Project

* (38:20) The interactive vision of the project

* (42:10) How AI chooses music, breaking from constraints

* (43:15) Interpretability and how an AI system DJs differently

* (45:15) How the project altered Nao’s perspective on DJing, the role of humans

* (51:40) Nao’s book Creating with AI

* (55:15) Human-AI interaction as joint improvisation

* (58:10) Nao’s advice and takeaways for thinking about AI creatively

* (1:01:32) Outro

Links:

* Nao’s homepage and Twitter

* Other links:

* Neutone, AI audio plugin

* Real-time AI-generative DJ performance

* Qosmo

* Dawn Patrol EP

* Nao’s book: Surfing human creativity with AI — A user's guide

* Paper on Creative-GAN for deviating from existing music genres

In episode 94 of The Gradient Podcast, Daniel Bashir speaks to Divyansh Kaushik.

Divyansh is the Associate Director for Emerging Technologies and National Security at the Federation of American Scientists where his focus areas include, amongst other things, AI policy, STEM immigration, and US-China strategic competition. He holds a PhD from Carnegie Mellon University, where he focused on designing reliable AI systems that align with human values. In addition to his advocacy work on Capitol Hill, he also played a key role in establishing the Congressional Graduate Research and Development Caucus. He is a frequent contributor to leading publications, including Vox, National Defense Magazine, The Dispatch, Daily Caller, and Forbes.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:20) Divyansh intro/background

* (06:00) Zachary Lipton Appreciation Session ( + advice from Prof Lipton)

* (08:00) How Divyansh got involved in policy

* (11:30) What does policy work look like? Divyansh’s early experiences

* (15:42) AI policy issues, divides, party lines

* (19:15) Bringing AI talent into the US

* (26:45) US/China saber rattling, impact of Xi Jinping’s presidency

* (33:49) China’s AI regulations, CCP motivations, China’s disadvantages in AI and benefits of the US policy process

* (42:42) Trading off AI governance and stifling innovation

* (51:17) AI governance comments from Jeremy Howard / Connor Leahy / Andrew Maynard, regulating use vs basic technology, limits on scaling

* (1:01:30) Articulating and communicating the issues for AI governance

* (1:03:10) Existential risk concerns in AI governance, theories of change

* (1:10:15) How can AI researchers/practitioners better communicate with policymakers?

* (1:16:57) Outro

Links:

* Divyansh’s Twitter and FAS page

* Divyansh’s policy work:

* The impact of international scientists, engineers, and students on US research outputs and global competitiveness

* How Congress can shape AI governance without stifling innovation

* How Do OpenAI’s Efforts To Make GPT-4 “Safer” Stack Up Against The NIST AI Risk Management Framework?

* Six Policy Ideas for the National AI Strategy

* Other work mentioned/discussed:

* Jeremy Howard’s AI Safety and the Age of Dislightenment

* Proposals from Connor Leahy

* Andrew Maynard’s Regulating Frontier AI: To Open Source or Not?

In episode 93 of The Gradient Podcast, Daniel Bashir speaks to Professor Tal Linzen.

Professor Linzen is an Associate Professor of Linguistics and Data Science at New York University and a Research Scientist at Google. He directs the Computation and Psycholinguistics Lab, where he and his collaborators use behavioral experiments and computational methods to study how people learn and understand language. They also develop methods for evaluating, understanding, and improving computational systems for language processing.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:25) Prof. Linzen’s background

* (05:37) Back and forth between psycholinguistics and deep learning research, LM evaluation

* (08:40) How can deep learning successes/failures help us understand human language use, methodological concerns, comparing human representations to LM representations

* (14:22) Behavioral capacities and degrees of freedom in representations

* (16:40) How LMs are becoming less and less like humans

* (19:25) Assessing LSTMs’ ability to learn syntax-sensitive dependencies

* (22:48) Similarities between structure-sensitive dependencies, sophistication of syntactic representations

* (25:30) RNNs implicitly implement tensor-product representations—vector representations of symbolic structures

* (29:45) Representations required to solve certain tasks, difficulty of natural language

* (33:25) Accelerating progress towards human-like linguistic generalization

* (34:30) The pre-training agnostic identically distributed evaluation paradigm

* (39:50) Ways to mitigate differences in evaluation

* (44:20) Surprisal does not explain syntactic disambiguation difficulty

* (45:00) How to measure processing difficulty, predictability and processing difficulty

* (49:20) What other factors influence processing difficulty?

* (53:10) How to plant trees in language models

* (55:45) Architectural influences on generalizing knowledge of linguistic structure

* (58:20) “Cognitively relevant regimes” and speed of generalization

* (1:00:45) Acquisition of syntax and sampling simpler vs. more complex sentences

* (1:04:03) Curriculum learning for progressively more complicated syntax

* (1:05:35) Hypothesizing tree-structured representations

* (1:08:00) Reflecting on a prediction from the past

* (1:10:15) Goals and “the correct direction” in AI research

* (1:14:04) Outro

Links:

* Prof. Linzen’s Twitter and homepage

* Papers

* Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

* RNNS Implicitly Implement Tensor-Product Representations

* How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

* Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark

* How to Plant Trees in LMs: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

In episode 92 of The Gradient Podcast, Daniel Bashir speaks to Kevin K. Yang.

Kevin is a senior researcher at Microsoft Research (MSR) who works on problems at the intersection of machine learning and biology, with an emphasis on protein engineering. He completed his PhD at Caltech with Frances Arnold on applying machine learning to protein engineering. Before joining MSR, he was a machine learning scientist at Generate Biomedicines, where he used machine learning to optimize proteins.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:40) Kevin’s background

* (06:00) Protein engineering early in Kevin’s career

* (12:10) From research to real-world proteins: the process

* (17:40) Generative models + pretraining for proteins

* (22:47) Folding diffusion for protein structure generation

* (30:45) Protein evolutionary dynamics and generative models of protein sequences

* (40:03) Analogies and disanalogies between protein modeling and language models

* (41:45) In representation learning

* (45:50) Convolutions vs. transformers and inductive biases

* (49:25) Pretraining tasks for protein structure

* (51:45) More on representation learning for protein structure

* (54:06) Kevin’s thoughts on interpretability in deep learning for protein engineering

* (56:50) Multimodality in protein engineering and future directions

* (59:14) Outro

Links:

* Kevin’s Twitter and homepage

* Research

* Generative models + pre-training for proteins and chemistry

* Broad intro to techniques in the space

* Protein structure generation via folding diffusion

* Protein sequence design with deep generative models (review)

* Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins

* Protein generation with evolutionary diffusion: sequence is all you need

* ML for protein engineering

* ML-guided directed evolution for protein engineering (review)

* Learned protein embeddings for ML

* Adaptive machine learning for protein engineering (review)

* Multimodal deep learning for protein engineering

In episode 91 of The Gradient Podcast, Daniel Bashir speaks to Arjun Ramani and Zhengdong Wang.

Arjun is the global business and economics correspondent at The Economist.

Zhengdong is a research engineer at Google DeepMind.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (03:53) Arjun intro

* (06:04) Zhengdong intro

* (09:50) How Arjun and Zhengdong met in the woods

* (11:52) Overarching narratives about technological progress and AI

* (14:20) Setting up the claim: Arjun on what “transformative” means

* (15:52) What enables transformative economic growth?

* (21:19) From GPT-3 to ChatGPT; is there something special about AI?

* (24:15) Zhengdong on “real AI” and divisiveness

* (27:00) Arjun on the independence of bottlenecks to progress/growth

* (29:05) Zhengdong on bottleneck independence

* (32:45) More examples on bottlenecks and surplus wealth

* (37:06) Technical arguments—what are the hardest problems in AI?

* (38:00) Robotics

* (40:41) Challenges of deployment in high-stakes settings and data sources / synthetic data, self-driving

* (45:13) When synthetic data works

* (49:06) Harder tasks, process knowledge

* (51:45) Performance art as a critical bottleneck

* (53:45) Obligatory Taylor Swift Discourse

* (54:45) AI Taylor Swift???

* (54:50) The social arguments

* (55:20) Speed of technology diffusion — “diffusion lags” and dynamics of trust with AI

* (1:00:55) ChatGPT adoption, where major productivity gains come from

* (1:03:50) Timescales of transformation

* (1:10:22) Unpredictability in human affairs

* (1:14:07) The economic arguments

* (1:14:35) Key themes — diffusion lags, different sectors

* (1:21:15) More on bottlenecks, AI trust, premiums on human workers

* (1:22:30) Automated systems and human interaction

* (1:25:45) Campaign text reachouts

* (1:30:00) Counterarguments

* (1:30:18) Solving intelligence and solving science/innovation

* (1:34:07) Strengths and weaknesses of the broad applicability of Arjun and Zhengdong’s argument

* (1:35:34) The “proves too much” worry — how could any innovation have ever happened?

* (1:37:25) Examples of bringing down barriers to innovation/transformation

* (1:43:45) What to do with all of this information?

* (1:48:45) Outro

Links:

* Zhengdong’s homepage and Twitter

* Arjun’s homepage and Twitter

* Why transformative artificial intelligence is really, really hard to achieve

* Other resources and links mentioned:

* Allan-Feuer and Sanders: Transformative AGI by 2043 is <1% likely

* On AlphaStar Zero

* Hardmaru on AI as applied philosophy

* Robotics Transformer 2

* Davis Blalock on synthetic data

* Matt Clancy on automating invention and bottlenecks

* Michael Webb on 80,000 Hours Podcast

* Bob Gordon: The Rise and Fall of American Growth

* OpenAI economic impact paper

* David Autor: new work paper

* Baumol effect paper

* Pew research centre poll, public concern on AI

* Human premium Economist piece

* Callum Williams — London tube and AI/jobs

* Culture Series book 1, Iain Banks

In episode 90 of The Gradient Podcast, Daniel Bashir speaks to Miles Grimshaw.

Miles is General Partner at Benchmark. He was previously a General Partner at Thrive Capital, where he helped the firm raise its fourth and fifth funds, and sourced deals in Lattice, Mapbox, Benchling, and Airtable, among others.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:48) Miles’ background (note: Miles is now the second newest GP at Benchmark)

* (06:07) Miles’ investment philosophy and previous investments

* (12:25) Investing in the “decade of deep learning” and how Miles became interested in AI

* (18:53) Miles’ / Benchmark’s investment in Langchain

* (24:29) On AI advances and adoption

* (39:25) Hardware shortages, radically changing UX for LLMs

* (48:12) Opportunities for AI applications in new domains

* (50:15) Miles’ advice for potential founders in AI

* (1:00:00) Outro

Links:

* Miles’ Twitter

* Benchmark homepage

* LangChain homepage

In episode 89 of The Gradient Podcast, Daniel Bashir speaks to Shreya Shankar.

Shreya is a computer scientist pursuing her PhD in databases at UC Berkeley. Her research interest is in building end-to-end systems for people to develop production-grade machine learning applications. She was previously the first ML engineer at Viaduct, did research at Google Brain, and software engineering at Facebook. She graduated from Stanford with a B.S. and M.S. in computer science with concentrations in systems and artificial intelligence. At Stanford, helped run SHE++, an organization that helps empower underrepresented minorities in technology.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:22) Shreya’s background and journey into ML / MLOps

* (04:51) ML advances in 2013-2016

* (05:45) Shift in Stanford undergrad class ecosystems, accessibility of deep learning research

* (09:10) Why Shreya left her job as an ML engineer

* (13:30) How Shreya became interested in databases, data quality in ML

* (14:50) Daniel complains about things

* (16:00) What makes ML engineering uniquely difficult

* (16:50) Being a “historian of the craft” of ML engineering

* (22:25) Levels of abstraction, what ML engineers do/don’t have to think about

* (24:16) Observability for Production ML Pipelines

* (28:30) Metrics for real-time ML systems

* (31:20) Proposed solutions

* (34:00) Moving Fast with Broken Data

* (34:25) Existing data validation measures and where they fall short

* (36:31) Partition summarization for data validation

* (38:30) Small data and quantitative statistics for data cleaning

* (40:25) Streaming ML Evaluation

* (40:45) What makes a metric actionable

* (42:15) Differences in streaming ML vs. batch ML

* (45:45) Delayed and incomplete labels

* (49:23) Operationalizing Machine Learning

* (49:55) The difficult life of an ML engineer

* (53:00) Best practices, tools, pain points

* (55:56) Pitfalls in current MLOps tools

* (1:00:30) LLMOps / FMOps

* (1:07:10) Thoughts on ML Engineering, MLE through the lens of data engineering

* (1:10:42) Building products, user expectations for AI products

* (1:15:50) Outro

Links:

* Papers

* Towards Observability for Production Machine Learning Pipelines

* Rethinking Streaming ML Evaluation

* Operationalizing Machine Learning

* Moving Fast With Broken Data

* Blog posts

* The Modern ML Monitoring Mess

* Thoughts on ML Engineering After a Year of my PhD