In episode 116 of The Gradient Podcast, Daniel Bashir speaks to Kate Park.

Kate is the Director of Product at Scale AI. Prior to joining Scale, Kate worked on Tesla Autopilot as the AI team’s first and lead product manager building the industry’s first data engine. She has also published research on spoken natural language processing and a travel memoir.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:11) Kate’s background

* (03:22) Tesla and cameras vs. Lidar, importance of data

* (05:12) “Data is key”

* (07:35) Data vs. architectural improvements

* (09:36) Effort for data scaling

* (10:55) Transfer of capabilities in self-driving

* (13:44) Data flywheels and edge cases, deployment

* (15:48) Transition to Scale

* (18:52) Perspectives on shifting to transformers and data

* (21:00) Data engines for NLP vs. for vision

* (25:32) Model evaluation for LLMs in data engines

* (27:15) InstructGPT and data for RLHF

* (29:15) Benchmark tasks for assessing potential labelers

* (32:07) Biggest challenges for data engines

* (33:40) Expert AI trainers

* (36:22) Future work in data engines

* (38:25) Need for human labeling when bootstrapping new domains or tasks

* (41:05) Outro

Links:

* Scale Data Engine

* OpenAI case study

In episode 115 of The Gradient Podcast, Daniel Bashir speaks to Ben Wellington.

Ben is the Deputy Head of Feature Forecasting at Two Sigma, a financial sciences company. Ben has been at Two Sigma for more than 15 years, and currently leads efforts focused on natural language processing and feature forecasting. He is also the author of data science blog I Quant NY, which has influenced local government policy, including changes in NYC street infrastructure and the design of NYC subway vending machines. Ben is a Visiting Assistant Professor in the Urban and Community Planning program at the Pratt Institute in Brooklyn where he teaches statistics using urban open data. He holds a Ph.D. in Computer Science from New York University.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:30) Ben’s background

* (04:30) Why Ben was interested in NLP

* (05:48) Ben’s work on translational equivalence, dominant techniques

* (10:14) Scaling, large datasets at Two Sigma

* (12:50) Applying ML techniques to quantitative finance, features in financial ML systems

* (17:27) Baselines and time-dependence in constructing features, human knowledge

* (19:23) Black box models in finance

* (24:00) Two Sigma’s presence in the AI research community

* (26:55) Short- and long-term research initiatives at Two Sigma

* (30:42) How ML fits into Two Sigma’s investment strategy

* (34:05) Alpha and competition in investing

* (36:13) Temporality in data

* (40:38) Challenges for finance/AI and beating the market

* (44:36) Reproducibility

* (49:47) I Quant NY and storytelling with data

* (56:43) Descriptive statistics and stories

* (1:01:05) Benefits of simple methods

* (1:07:11) Outro

Links:

* Ben’s work on translational equivalence and scalable discriminative learning

* Two Sigma Insights

* Storytelling with data and I Quant NY

“There is this move from generality in a relative sense of ‘we are not as specialized as insects’ to generality in the sense of omnipotent, omniscient, godlike capabilities. And I think there's something very dangerous that happens there, which is you start thinking of the word ‘general’ in completely unhinged ways.”

In episode 114 of The Gradient Podcast, Daniel Bashir speaks to Venkatesh Rao.

Venkatesh is a writer and consultant. He has been writing the widely read Ribbonfarm blog since 2007, and more recently, the popular Ribbonfarm Studio Substack newsletter. He is the author of Tempo, a book on timing and decision-making, and is currently working on his second book, on the foundations of temporality. He has been an independent consultant since 2011, supporting senior executives in the technology industry. His work in recent years has focused on AI, semiconductor, sustainability, and protocol technology sectors. He holds a PhD in control theory (2003) from the University of Michigan. He is currently based in the Seattle area, and enjoys dabbling in robotics in his spare time. You can learn more about his work at venkateshrao.com

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:38) Origins of Ribbonfarm and Venkat’s academic background

* (04:23) Voice and recurring themes in Venkat’s work

* (11:45) Patch models and multi-agent systems: integrating philosophy of language, balancing realism with tractability

* (21:00) More on abstractions vs. tractability in Venkat’s work

* (29:07) Scaling of industrial value systems, characterizing AI as a discipline

* (39:25) Emergent science, intelligence and abstractions, presuppositions in science, generality and universality, cameras and engines

* (55:05) Psychometric terms

* (1:09:07) Inductive biases (yes I mentioned the No Free Lunch Theorem and then just talked about the definition of inductive bias and not the actual theorem 🤡)

* (1:18:13) LLM training and efficiency, comparing LLMs to humans

* (1:23:35) Experiential age, analogies for knowledge transfer

* (1:30:50) More clarification on the analogy

* (1:37:20) Massed Muddler Intelligence and protocols

* (1:38:40) Introducing protocols and the Summer of protocols

* (1:49:15) Evolution of protocols, hardness

* (1:54:20) LLMs, protocols, time, future visions, and progress

* (2:01:33) Protocols, drifting from value systems, friction, compiling explicit knowledge

* (2:14:23) Directions for ML people in protocols research

* (2:18:05) Outro

Links:

* Venkat’s Twitter and homepage

* Mediocre Computing

* Summer of Protocols and 2024 Call for Applications (apply!)

* Essays discussed

* Patch models and their applications to multivehicle command and control

* From Mediocre Computing

* Text is All You Need

* Magic, Mundanity, and Deep Protocolization

* A Camera, Not an Engine

* Massed Muddler Intelligence

* On protocols

* The Unreasonable Sufficiency of Protocols

* Protocols Don’t Build Pyramids

* Protocols in (Emergency) Time

* Atoms, Institutions, Blockchains

In episode 113 of The Gradient Podcast, Daniel Bashir speaks to Professor Sasha Rush.

Professor Rush is an Associate Professor at Cornell University and a Researcher at HuggingFace. His research aims to develop natural language processing systems that are safe, fast, and controllable. His group is interested primarily in tasks that involve text generation, and they study data-driven probabilistic methods that combine deep-learning based models with probabilistic controls. He is also interested in open-source NLP and deep learning, and develops projects to make deep learning systems safer, clearer, and easier to use.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:47) Professor Rush’s background

* (03:23) Professor Rush’s reflections on prior work—importance of learning and inference

* (04:58) How much engineering matters in deep learning, the Rush vs. Frankle Bet

* (07:12) On encouraging and incubating good research

* (10:50) Features of good research environments

* (12:36) 5% bets in Professor Rush’s research: State-Space Models (SSMs) as an alternative to Transformers

* (15:58) SSMs vs. Transformers

* (18:53) Probabilistic Context-Free Grammars—are (P)CFGs worth paying attention to?

* (20:53) Sequence-level knowledge distillation: approximating sequence-level distributions

* (25:08) Pruning and knowledge distillation — orthogonality of efficiency techniques

* (26:33) Broader thoughts on efficiency

* (28:31) Works on prompting

* (28:58) Prompting and In-Context Learning

* (30:05) Thoughts on mechanistic interpretability

* (31:25) Multitask prompted training enables zero-shot task generalization

* (33:48) How many data points is a prompt worth?

* (35:13) Directions for controllability in LLMs

* (39:11) Controllability and safety

* (41:23) Open-source work, deep learning libraries

* (42:08) A story about Professor Rush’s post-doc at FAIR

* (43:51) The impact of PyTorch

* (46:08) More thoughts on deep learning libraries

* (48:48) Levels of abstraction, PyTorch as an interface to motivate research

* (50:23) Empiricism and research commitments

* (53:32) Outro

Links:

* Research

* Early work / PhD

* Dual Decomposition and LP Relaxations

* Vine Pruning for Efficient Multi-Pass Dependency Parsing

* Improved Parsing and POS Tagging Using Inter-Sentence Dependency Constraints

* Research — interpretable and controllable natural language generation

* Compound Probabilistic Context-Free Grammars for Grammar Induction

* Multitask prompted training enables zero-shot task generalization

* Research — deep generative models

* A Neural Attention Model for Abstractive Sentence Summarization

* Learning Neural Templates for Text Generation

* How many data points is a prompt worth?

* Research — efficient algorithms and hardware for speech, translation, dialogue

* Sequence-Level Knowledge Distillation

* Open-source work

* NamedTensor

* Torch Struct

In episode 112 of The Gradient Podcast, Daniel Bashir speaks to Cameron Jones and Sean Trott.

Cameron is a PhD candidate in the Cognitive Science Department at the University of California, San Diego. His research compares how humans and large language models process language about world knowledge, situation models, and theory of mind.

Sean is an Assistant Teaching Professor in the Cognitive Science Department at the University of California, San Diego. His research interests include probing large language models, ambiguity in languages, how ambiguous words are represented, and pragmatic inference. He previously completed his PhD at UCSD.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:55) Cameron’s background

* (06:00) Sean’s background

* (08:15) Unexpected capabilities of language models and the need for embodiment to understand meaning

* (11:05) Interpreting results of Turing tests, separating what humans and LLMs do when behaving as though they “understand”

* (14:27) Internal mechanisms, interpretability, how we test theories

* (16:40) Languages are efficient, but for whom?

* (17:30) Initial motivations: lexical ambiguity

* (19:20) The balance of meanings across wordforms

* (22:35) Tension between speaker- and comprehender-oriented pressures in lexical ambiguity

* (25:05) Context and potential vs. realized ambiguity

* (27:15) LLM-ology

* (28:30) Studying LLMs as models of human cognition and as interesting objects of study in their own right

* (30:03) Example of explaining away effects

* (33:54) The internalist account of belief sensitivity—behavior and internal representations

* (37:43) LLMs and the False Belief Task

* (42:05) Hypothetical on observed behavior and inferences about internal representations

* (48:05) Distributional Semantics Still Can’t Account for Affordances

* (50:25) Tests of embodied theories and limitations of distributional cues

* (53:54) Multimodal models and object affordances

* (58:30) Language and grounding, other buzzwords

* (59:45) How could we know if LLMs understand language?

* (1:04:50) Reference: as a thing words do vs. ontological notion

* (1:11:38) The Role of Physical Inference in Pronoun Resolution

* (1:16:40) World models and world knowledge

* (1:19:45) EPITOME

* (1:20:20) The different tasks

* (1:26:43) Confounders / “attending” in LM performance on tasks

* (1:30:30) Another hypothetical, on theory of mind

* (1:32:26) How much information can language provide in service of mentalizing?

* (1:35:14) Convergent validity and coherence/validity of theory of mind

* (1:39:30) Interpretive questions about behavior w/r/t/ theory of mind

* (1:43:35) Does GPT-4 Pass the Turing Test?

* (1:44:00) History of the Turing Test

* (1:47:05) Interrogator strategies and the strength of the Turing Test

* (1:52:15) “Internal life” and personality

* (1:53:30) How should this research impact how we assess / think about LLM abilities?

* (1:58:56) Outro

Links:

* Cameron’s homepage and Twitter

* Sean’s homepage and Twitter

* Research — Language and NLP

* Languages are efficient, but for whom?

* Research — LLM-ology

* Do LLMs know what humans know?

* Distributional Semantics Still Can’t Account for Affordances

* In Cautious Defense of LLM-ology

* Should Psycholinguists use LLMs as “model organisms”?

* (Re)construing Meaning in NLP

* Research — language and grounding, theory of mind, reference [insert other buzzwords here]

* Do LLMs have a “theory of mind”?

* How could we know if LLMs understand language?

* Does GPT-4 Pass the Turing Test?

* Could LMs change language?

* The extended mind and why it matters for cognitive science research

* EPITOME

* The Role of Physical Inference in Pronoun Resolution

In episode 111 of The Gradient Podcast, Daniel Bashir speaks to Nicholas Thompson.

Nicholas is the CEO of The Atlantic. Previously, he served as editor-in-chief of Wired and editor of Newyorker.com. Nick also cofounded Atavist, which sold to Automattic in 2018. Publications under Nick’s leadership have won numerous National Magazine Awards and Pulitzer Prizes, and one WIRED story he edited was the basis for the movie Argo. Nick is also the co-founder of Speakeasy AI, a software platform designed to foster constructive online conversations about the world’s most pressing problems.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:12) Nick’s path into journalism

* (03:25) The Washington Monthly — a turning point

* (05:09) Perspectives from different positions in the journalism industry

* (08:16) What is great journalism?

* (09:42) Example from The Atlantic

* (11:00) Other examples/pieces of good journalism

* (12:20) Pieces on aging

* (12:56) Mortality and life-force associated with running — Nick’s piece in WIRED

* (15:30) On urgency

* (18:20) The job of an editor

* (22:23) AI in journalism — benefits and limitations

* (26:45) How AI can help writers, experimentation

* (28:40) Examples of AI in journalism and issues: CNET, Sports Illustrated, Nick’s thoughts on how AI should be used in journalism

* (32:20) Speakeasy AI and creating healthy conversation spaces

* (34:00) Details about Speakeasy

* (35:12) Business pivots and business model trouble

* (35:37) Remaining gaps in fixing conversational spaces

* (38:27) Lessons learned

* (40:00) Nick’s optimism about Speakeasy-like projects

* (43:14) Social simulacra, a “Troll WestWorld,” algorithmic adjustments in social media

* (46:11) Lessons and wisdom from journalism about engagement, more on engagement in social media

* (50:27) Successful and unsuccessful futures for AI in journalism

* (54:17) Previous warnings about synthetic media, Nick’s perspective on risks from synthetic media in journalism

* (57:00) Stop trying to build AGI

(59:13) Outro

Links:

* Nicholas’s Twitter and website

* Speakeasy AI

* Writing

* “To Run My Best Marathon at Age 44, I Had to Outrun My Past” in WIRED

* “The year AI actually changes the media business” in NiemanLab’s Predictions for Journalism 2023

In episode 110 of The Gradient Podcast, Daniel Bashir speaks to Professor Subbarao Kambhampati.

Professor Kambhampati is a professor of computer science at Arizona State University. He studies fundamental problems in planning and decision making, motivated by the challenges of human-aware AI systems. He is a fellow of the Association for the Advancement of Artificial Intelligence, American Association for the Advancement of Science, and Association for Computing machinery, and was an NSF Young Investigator. He was the president of the Association for the Advancement of Artificial Intelligence, trustee of the International Joint Conference on Artificial Intelligence, and a founding board member of Partnership on AI.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:11) Professor Kambhampati’s background

* (06:07) Explanation in AI

* (18:08) What people want from explanations—vocabulary and symbolic explanations

* (21:23) The realization of new concepts in explanation—analogy and grounding

* (30:36) Thinking and language

* (31:48) Conscious and subconscious mental activity

* (36:58) Tacit and explicit knowledge

* (42:09) The development of planning as a research area

* (46:12) RL and planning

* (47:47) What makes a planning problem hard?

* (51:23) Scalability in planning

* (54:48) LLMs do not perform reasoning

* (56:51) How to show LLMs aren’t reasoning

* (59:38) External verifiers and backprompting LLMs

* (1:07:51) LLMs as cognitive orthotics, language and representations

* (1:16:45) Finding out what kinds of representations an AI system uses

* (1:31:08) “Compiling” system 2 knowledge into system 1 knowledge in LLMs

* (1:39:53) The Generative AI Paradox, reasoning and retrieval

* (1:43:48) AI as an ersatz natural science

* (1:44:03) Why AI is straying away from its engineering roots, and what constitutes engineering

* (1:58:33) Outro

Links:

* Professor Kambhampati’s Twitter and homepage

* Research and Writing — Planning and Human-Aware AI Systems

* A Validation-structure-based theory of plan modification and reuse (1990)

* Challenges of Human-Aware AI Systems (2020)

* Polanyi vs. Planning (2021)

* LLMs and Planning

* Can LLMs Really Reason and Plan? (2023)

* On the Planning Abilities of LLMs (2023)

* Other

* Changing the nature of AI research

In episode 109 of The Gradient Podcast, Daniel Bashir speaks to Russ Maschmeyer.

Russ is the Product Lead for AI and Spatial Commerce at Shopify. At Shopify, he leads a team that looks at how AI can better empower entrepreneurs, with a particular interest in how image generation can help make the lives of business owners and merchants more productive. He previously led design for multiple services at Facebook and co-founded Primer, an AR-enabled interior design marketplace.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:50) Russ’s background and a hacked Kinect sensor

* (06:00) Instruments and emotion, embodiment and accessibility

* (08:45) Natural language as input and generative AI in creating emotive experiences

* (10:55) Work on search queries and recommendations at Facebook, designing for search

* (16:35) AI in the retail and entrepreneurial landscape

* (19:15) Shopify and AI for business owners

* (22:10) Vision and directions for AI in commerce

* (25:01) Personalized experiences for shopping

* (28:45) Challenges for creating personalized experiences

* (31:49) Intro to spatial commerce

* (34:48) AR/VR devices and spatial commerce

* (37:30) MR and AI for immersive product search

* (41:35) Implementation details

* (48:05) WonkaVision and difficulties for immersive web experiences

* (52:10) Future projects and directions for spatial commerce

* (55:10) Outro

Links:

* Russ’s Twitter and homepage

* With a Wave of the Hand, Improvising on Kinect in The New York Times

* Shopify Spatial Commerce Projects

* MR and AI for immersive product search

* A more immersive web with a simple optical illusion

* What if your room had a reset button?

In episode 108 of The Gradient Podcast, Daniel Bashir speaks to Professor Benjamin Breen.

Professor Breen is an associate professor of history at UC Santa Cruz specializing in the history of science, medicine, globalization, and the impacts of technological change. He is the author of multiple books including The Age of Intoxication: Origins of the Global Drug Trade and the more recent Tripping on Utopia: Margaret Mead, the Cold War, and the Troubled Birth of Psychedelic Science, which you can pre-order now.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:05) Professor Breen’s background

* (04:47) End of history narratives / millenarian thinking in AI/technology

* (09:53) Transformative technological change and societal change

* (16:45) AI and psychedelics

* (17:23) Techno-utopianism

* (26:08) Technologies as metaphors for humanity

* (32:34) McLuhanist thinking / brain as a computational machine, Prof. Breen’s skepticism

* (37:13) Issues with overblown narratives about technology

* (42:46) Narratives about transformation and their impacts on progress

* (45:23) The historical importance of today’s AI landscape

* (50:05) International aspects of the history of technology

* (53:13) Doomerism vs optimism, why doomerism is appealing

* (57:58) Automation, meta-skills, jobs — advice for early career

* (1:01:08) LLMs and (history) education

* (1:07:10) Outro

Links:

* Professor Breen’s Twitter and homepage

* Books

* Tripping on Utopia: Margaret Mead, the Cold War, and the Troubled Birth of Psychedelic Science

* The Age of Intoxication: Origins of the Global Drug Trade

* Writings

* Into the mystic

* ‘Alien Jesus’

* Simulating History with ChatGPT

In episode 107 of The Gradient Podcast, Daniel Bashir speaks to Professor Ted Gibson.

Ted is a Professor of Cognitive Science at MIT. He leads the TedLab, which investigates why languages look the way they do; the relationship between culture and cognition, including language; and how people learn, represent, and process language.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:13) Prof Gibson’s background

* (05:33) The computational linguistics community and NLP, engineering focus

* (10:48) Models of brains

* (12:03) Prof Gibson’s focus on behavioral work

* (12:53) How dependency distances impact language processing

* (14:03) Dependency distances and the origin of the problem

* (18:53) Dependency locality theory

* (21:38) The structures languages tend to use

* (24:58) Sentence parsing: structural integrations and memory costs

* (36:53) Reading strategies vs. ordinary language processing

* (40:23) Legalese

* (46:18) Cross-dependencies

* (50:11) Number as a cognitive technology

* (54:48) Experiments

* (1:03:53) Why counting is useful for Western societies

* (1:05:53) The Whorf hypothesis

* (1:13:05) Language as Communication

* (1:13:28) The noisy channel perspective on language processing

* (1:27:08) Fedorenko lab experiments—language for thought vs. communication and Chomsky’s claims

* (1:43:53) Thinking without language, inner voices, language processing vs. language as an aid for other mental processing

* (1:53:01) Dependency grammars and a critique of Chomsky’s grammar proposals, LLMs

* (2:08:48) LLM behavior and internal representations

* (2:12:53) Outro

Links:

* Ted’s lab page and Twitter

* Re-imagining our theories of language

* Research — linguistic complexity and dependency locality theory

* Linguistic complexity: locality of syntactic dependencies (1998)

* The Dependency Locality Theory: A Distance-Based Theory of Linguistic Complexity (2000)

* Consequences of the Serial Nature of Linguistic Input for Sentential Complexity (2005)

* Large-scale evidence of dependency length minimization in 37 languages (2015)

* Dependency locality as an explanatory principle for word order (2020)

* Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex (2022)

* A resource-rational model of human processing of recursive linguistic structure (2022)

* Research — language processing / communication and cross-linguistic universals

* Number as a cognitive technology: Evidence from Pirahã language and cognition (2008)

* The communicative function of ambiguity in language (2012)

* The rational integration of noisy evidence and prior semantic expectations in sentence interpretation (2013)

* Color naming across languages reflects color use (2017)

* How Efficiency Shapes Human Language (2019)