The Gradient: Perspectives on AI

In episode 108 of The Gradient Podcast, Daniel Bashir speaks to Professor Benjamin Breen.

Professor Breen is an associate professor of history at UC Santa Cruz specializing in the history of science, medicine, globalization, and the impacts of technological change. He is the author of multiple books including The Age of Intoxication: Origins of the Global Drug Trade and the more recent Tripping on Utopia: Margaret Mead, the Cold War, and the Troubled Birth of Psychedelic Science, which you can pre-order now.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:05) Professor Breen’s background

* (04:47) End of history narratives / millenarian thinking in AI/technology

* (09:53) Transformative technological change and societal change

* (16:45) AI and psychedelics

* (17:23) Techno-utopianism

* (26:08) Technologies as metaphors for humanity

* (32:34) McLuhanist thinking / brain as a computational machine, Prof. Breen’s skepticism

* (37:13) Issues with overblown narratives about technology

* (42:46) Narratives about transformation and their impacts on progress

* (45:23) The historical importance of today’s AI landscape

* (50:05) International aspects of the history of technology

* (53:13) Doomerism vs optimism, why doomerism is appealing

* (57:58) Automation, meta-skills, jobs — advice for early career

* (1:01:08) LLMs and (history) education

* (1:07:10) Outro

Links:

* Professor Breen’s Twitter and homepage

* Books

* Tripping on Utopia: Margaret Mead, the Cold War, and the Troubled Birth of Psychedelic Science

* The Age of Intoxication: Origins of the Global Drug Trade

* Writings

* Into the mystic

* ‘Alien Jesus’

* Simulating History with ChatGPT

In episode 107 of The Gradient Podcast, Daniel Bashir speaks to Professor Ted Gibson.

Ted is a Professor of Cognitive Science at MIT. He leads the TedLab, which investigates why languages look the way they do; the relationship between culture and cognition, including language; and how people learn, represent, and process language.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:13) Prof Gibson’s background

* (05:33) The computational linguistics community and NLP, engineering focus

* (10:48) Models of brains

* (12:03) Prof Gibson’s focus on behavioral work

* (12:53) How dependency distances impact language processing

* (14:03) Dependency distances and the origin of the problem

* (18:53) Dependency locality theory

* (21:38) The structures languages tend to use

* (24:58) Sentence parsing: structural integrations and memory costs

* (36:53) Reading strategies vs. ordinary language processing

* (40:23) Legalese

* (46:18) Cross-dependencies

* (50:11) Number as a cognitive technology

* (54:48) Experiments

* (1:03:53) Why counting is useful for Western societies

* (1:05:53) The Whorf hypothesis

* (1:13:05) Language as Communication

* (1:13:28) The noisy channel perspective on language processing

* (1:27:08) Fedorenko lab experiments—language for thought vs. communication and Chomsky’s claims

* (1:43:53) Thinking without language, inner voices, language processing vs. language as an aid for other mental processing

* (1:53:01) Dependency grammars and a critique of Chomsky’s grammar proposals, LLMs

* (2:08:48) LLM behavior and internal representations

* (2:12:53) Outro

Links:

* Ted’s lab page and Twitter

* Re-imagining our theories of language

* Research — linguistic complexity and dependency locality theory

* Linguistic complexity: locality of syntactic dependencies (1998)

* The Dependency Locality Theory: A Distance-Based Theory of Linguistic Complexity (2000)

* Consequences of the Serial Nature of Linguistic Input for Sentential Complexity (2005)

* Large-scale evidence of dependency length minimization in 37 languages (2015)

* Dependency locality as an explanatory principle for word order (2020)

* Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex (2022)

* A resource-rational model of human processing of recursive linguistic structure (2022)

* Research — language processing / communication and cross-linguistic universals

* Number as a cognitive technology: Evidence from Pirahã language and cognition (2008)

* The communicative function of ambiguity in language (2012)

* The rational integration of noisy evidence and prior semantic expectations in sentence interpretation (2013)

* Color naming across languages reflects color use (2017)

* How Efficiency Shapes Human Language (2019)

In episode 106 of The Gradient Podcast, Daniel Bashir speaks to Professor Harvey Lederman.

Professor Lederman is a professor of philosophy at UT Austin. He has broad interests in contemporary philosophy and in the history of philosophy: his areas of specialty include philosophical logic, the Ming dynasty philosopher Wang Yangming, epistemology, and philosophy of language. He has recently been working on incomplete preferences, on trying in the philosophy of language, and on Wang Yangming’s moral metaphysics.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:15) Harvey’s background

* (05:30) Higher-order metaphysics and propositional attitudes

* (06:25) Motivations

* (12:25) Setup: syntactic types and ontological categories

* (25:11) What makes higher-order languages meaningful and not vague?

* (25:57) Higher-order languages corresponding to the world

* (30:52) Extreme vagueness

* (35:32) Desirable features of languages and important questions in philosophy

* (36:42) Higher-order identity

* (40:32) Intuitions about mental content, language, context-sensitivity

* (50:42) Perspectivism

* (51:32) Co-referring names, identity statements

* (55:42) The paper’s approach, “know” as context-sensitive

* (57:24) Propositional attitude psychology and mentalese generalizations

* (59:57) The “good standing” of theorizing about propositional attitudes

* (1:02:22) Mentalese

* (1:03:32) “Does knowledge imply belief?” — when a question does not have good standing

* (1:06:17) Sense, Reference, and Substitution

* (1:07:07) Fregeans and the principle of Substitution

* (1:12:12) Follow-up work to this paper

* (1:13:39) Do Language Models Produce Reference Like Libraries or Like Librarians?

* (1:15:02) Bibliotechnism

* (1:19:08) Inscriptions and reference, what it takes for something to refer

* (1:22:37) Derivative and basic reference

* (1:24:47) Intuition: n-gram models and reference

* (1:28:22) Meaningfulness in sentences produced by n-gram models

* (1:30:40) Bibliotechnism and LLMs, disanalogies to n-grams

* (1:33:17) On other recent work (vector grounding, do LMs refer?, etc.)

* (1:40:12) Causal connections and reference, how bibliotechnism makes good on the meanings of sentences

* (1:45:46) RLHF, sensitivity to truth and meaningfulness

* (1:48:47) Intelligibility

* (1:50:52) When LLMs produce novel reference

* (1:53:37) Novel reference vs. find-replace

* (1:56:00) Directionality example

* (1:58:22) Human intentions and derivative reference

* (2:00:47) Between bibliotechnism and agency

* (2:05:32) Where do invented names / novel reference come from?

* (2:07:17) Further questions

* (2:10:04) Outro

Links:

* Harvey’s homepage and Twitter

* Papers discussed

* Higher-order metaphysics and propositional attitudes

* Perspectivism

* Sense, Reference, and Substitution

* Are Language Models More Like Libraries or Like Librarians? Bibliotechnism, the Novel Reference Problem, and the Attitudes of LLMs

In episode 105 of The Gradient Podcast, Daniel Bashir speaks to Eric Jang.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:25) Updates since Eric’s last interview

* (06:07) The problem space of humanoid robots

* (08:42) Motivations for the book “AI is Good for You”

* (12:20) Definitions of AGI

* (14:35) ~ AGI timelines ~

* (16:33) Do we have the ingredients for AGI?

* (18:58) Rediscovering old ideas in AI and robotics

* (22:13) Ingredients for AGI

* (22:13) Artificial Life

* (25:02) Selection at different levels of information—intelligence at different scales

* (32:34) AGI as a collective intelligence

* (34:53) Human in the loop learning

* (37:38) From getting correct answers to doing things correctly

* (40:20) Levels of abstraction for modeling decision-making — the neurobiological stack

* (44:22) Implementing loneliness and other details for AGI

* (47:31) Experience in AI systems

* (48:46) Asking for Generalization

* (49:25) Linguistic relativity

* (52:17) Language vs. complex thought and Fedorenko experiments

* (54:23) Efficiency in neural design

* (57:20) Generality in the human brain and evolutionary hypotheses

* (59:46) Embodiment and real-world robotics

* (1:00:10) Moravec’s Paradox and the importance of embodiment

* (1:05:33) How embodiment fits into the picture—in verification vs. in learning

* (1:10:45) Nonverbal information for training intelligent systems

* (1:11:55) AGI and humanity

* (1:12:20) The positive future with AGI

* (1:14:55) The negative future — technology as a lever

* (1:16:22) AI in the military

* (1:20:30) How AI might contribute to art

* (1:25:41) Eric’s own work and a positive future for AI

* (1:29:27) Outro

Links:

* Eric’s book

* Eric’s Twitter and homepage

In episode 104 of The Gradient Podcast, Daniel Bashir speaks to Nathan Benaich.

Nathan is Founder and General Partner at Air Street Capital, a VC firm focused on investing in AI-first technology and life sciences companies. Nathan runs a number of communities focused on AI including the Research and Applied AI Summit and leads Spinout.fyi to improve the creation of university spinouts. Nathan co-authors the State of AI Report.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:00) Updates in Nathan World — Air Street’s second fund, spinouts,

* (07:30) Events: Research and Applied AI Summit, State of AI Report launches

* (09:50) The State of AI: main messages, the increasing role of subject matter experts

* Research

* (14:13) Open and closed-source

* (17:55) Benchmarking and evaluation, small/large models and industry verticals

* (21:10) “Vibes” in LLM evaluation

* (24:00) Codegen models, personalized AI, curriculum learning

* (26:20) The exhaustion of human-generated data, lukewarm content, synthetic data

* (29:50) Opportunities for AI applications in the natural sciences

* (35:15) Reinforcement Learning from Human Feedback and alternatives

* (38:30) Industry

* (39:00) ChatGPT and productivity

* (42:37) General app wars, ChatGPT competitors

* (45:50) Compute—demand, supply, competition

* (50:55) Export controls and geopolitics

* (54:45) Startup funding and compute spend

* (59:15) Politics

* (59:40) Calls for regulation, regulatory divergence

* (1:04:40) AI safety

* (1:07:30) Nathan’s perspective on regulatory approaches

* (1:12:30) The UK’s early access to frontier models, standards setting, regulation difficulties

* (1:17:20) Jailbreaking, constitutional AI, robustness

* (1:20:50) Predictions!

* (1:25:00) Generative AI misuse in elections and politics (and, this prediction coming true in Bangladesh)

* (1:26:50) Progress on AI governance

* (1:30:30) European dynamism

* (1:35:08) Outro

Links:

* Nathan’s homepage and Twitter

* The 2023 State of AI Report

* Bringing Dynamism to European Defense

* A prediction coming true: How AI is disrupting Bangladesh’s election

* Air Street Capital is hiring a full-time Community Lead!

In episode 103 of The Gradient Podcast, Daniel Bashir speaks to Dr. Kathleen Fisher.

As the director of DARPA’s Information Innovation Office (I2O), Dr. Kathleen Fisher oversees a portfolio that includes most of the agency’s AI-related research and development efforts, including the recent AI Forward initiative. AI Forward explores new directions for AI research that will result in trustworthy systems for national security missions. This summer, roughly 200 participants from the commercial sector, academia, and the U.S. government attended workshops that generated ideas to inform DARPA’s next phase of AI exploratory projects. Dr. Fisher previously served as a program manager in I2O from 2011 to 2014. As a program manager, she conceptualized, created, and executed programs in high-assurance computing and machine learning, including Probabilistic Programming for Advancing Machine Learning (PPAML), making building ML applications easier. She was also a co-author of a recent paper about the threats posed by large language models.

Since 2018, DARPA has dedicated over $2 billion in R&D funding to AI research. The agency DARPA has been generating groundbreaking research and development for 65 years – leading to game-changing military capabilities and icons of modern society, such as initiating the research field that rendered self-driving cars and developing the technology that led to Apple’s Siri.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:30) Kathleen’s background

* (05:05) Intersections between programming languages and AI

* (07:15) Neuro-symbolic AI, trade-offs between flexibility and guarantees

* (09:45) History of DARPA and the Information Innovation Office (I2O)

* (13:55) DARPA’s perspective on research

* (17:10) Galvanizing a research community

* (20:06) DARPA’s recent investments in AI and AI Forward

* (26:35) Dual-use nature of generative AI, identifying and mitigating security risks, Kathleen’s perspective on short-term and long-term risk (note: the “Gradient podcast” Kathleen mentions is from Last Week in AI)

* (30:10) Concerns about deployment and interaction

* (32:20) Outcomes from AI Forward workshops and themes

* (36:10) Incentives in building and using AI technologies, friction

* (38:40) Interactions between DARPA and other government agencies

* (40:09) Future research directions

* (44:04) Ways to stay up to date on DARPA’s work

* (45:40) Outro

Links:

* DARPA I2O website

* Probabilistic Programming for Advancing Machine Learning (PPAML) (Archived)

* Assured Neuro Symbolic Learning and Reasoning (ANSR)

* AI Cyber Challenge

* AI Forward

* Identifying and Mitigating the Security Risks of Generative AI Paper

* FoundSci Solicitation

* FACT Solicitation

* Semantic Forensics (SemaFor)

* GARD Open Source Resources

* I2O Newsletter signup

In episode 102 of The Gradient Podcast, Daniel Bashir speaks to Peter Tse.

Professor Tse is a Professor of Cognitive Neuroscience and chair of the department of Psychological and Brain Sciences at Dartmouth College. His research focuses on using brain and behavioral data to constrain models of the neural bases of attention and consciousness, unconscious processing that precedes and constructs consciousness, mental causation, and human capacities for imagination and creativity. He is especially interested in the processing that goes into the construction of conscious experience between retinal activation at time 0 and seeing an event about a third of a second later.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:45) Prof. Tse’s background

* (03:25) Early experiences in physics/math and philosophy of physics

* (06:10) Choosing to study neuroscience

* (07:15) Prof Tse’s commitments about determinism

* (10:00) Quantum theory and determinism

* (13:45) Biases/preferences in choosing theories

* (20:41) Falsifiability and scientific questions, transition from physics to neuroscience

* (30:50) How neuroscience is unusual among the sciences

* (33:20) Neuroscience and subjectivity

* (34:30) Reductionism

* (37:30) Gestalt psychology

* (41:30) Introspection in neuroscience

* (45:30) The preconscious buffer and construction of conscious experience, color constancy

* (53:00) Perceptual and cognitive inference

* (55:00) AI systems and intrinsic meaning

* (57:15) Information vs. meaning

* (1:01:45) Consciousness and representation of bodily states

* (1:05:10) Our second-order free will

* (1:07:20) Jaegwon Kim’s exclusion argument

* (1:11:45) Why Kim thought his own argument was wrong

* (1:15:00) Resistance and counterarguments to Kim

* (1:19:45) Criterial causation

* (1:23:00) How neurons evaluate inputs criterially

* (1:24:00) Concept neurons in the hippocampus

* (1:31:57) Criterial causation and physicalism, mental causation

* (1:40:10) Daniel makes another attempt to push back 🤡

* (1:45:47) More on AI

* (1:47:05) Prof Tse’s perspective on modern AI systems, differences with human cognition

* (2:17:25) Consciousness, attention, spirituality

* (2:20:10) Prof Tse’s hopes for AI

* (2:23:30) Outro

Links:

* Professor Tse’s homepage

* Papers

* Vision/Perception

* Perceptual learning based on the learning of diagnostic features

* Complete mergeability and amodal completion

* Attention

* How Attention Can Alter Appearances

* How Top-down Attention Alters Bottom-up preconscious operations

* Consciousness

* Network structure and dynamics of the mental workspace

* On free will

* NDPR review of “Neural Basis of Free Will”

* Kripke’s Category Error

* Ontological Indeterminism undermines Kim’s Exclusion Argument

In episode 101 of The Gradient Podcast, Daniel Bashir speaks to Vera Liao.

Vera is a Principal Researcher at Microsoft Research (MSR) Montréal where she is part of the FATE (Fairness, Accountability, Transparency, and Ethics) group. She is trained in human-computer interaction research and works on human-AI interaction, currently focusing on explainable AI and responsible AI. She aims to bridge emerging AI technologies and human-centered design practices, and use both qualitative and quantitative methods to generate recommendations for technology design. Before joining MSR, Vera worked at IBM TJ Watson Research Center, and her work contributed to IBM products such as AI Explainability 360, Uncertainty Quantification 360, and Watson Assistant.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:41) Vera’s background

* (07:15) The sociotechnical gap

* (09:00) UX design and toolkits for AI explainability

* (10:50) HCI, explainability, etc. as “separate concerns” from core AI reseaarch

* (15:07) Interfaces for explanation and model capabilities

* (16:55) Vera’s earlier studies of online social communities

* (22:10) Technologies and user behavior

* (23:45) Explainability vs. interpretability, transparency

* (26:25) Questioning the AI: Informing Design Practices for Explainable AI User Experiences

* (42:00) Expanding Explainability: Towards Social Transparency in AI Systems

* (50:00) Connecting Algorithmic Research and Usage Contexts

* (59:40) Pitfalls in existing explainability methods

* (1:05:35) Ideal and real users, seamful systems and slow algorithms

* (1:11:08) AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

* (1:11:35) Vera’s earlier experiences with chatbots

* (1:13:00) Need to understand pitfalls and use-cases for LLMs

* (1:13:45) Perspectives informing this paper

* (1:20:30) Transparency informing goals for LLM use

* (1:22:45) Empiricism and explainability

* (1:27:20) LLM faithfulness

* (1:32:15) Future challenges for HCI and AI

* (1:36:28) Outro

Links:

* Vera’s homepage and Twitter

* Research

* Earlier work

* Understanding Experts’ and Novices’ Expertise Judgment of Twitter Users

* Beyond the Filter Bubble

* Expert Voices in Echo Chambers

* HCI / collaboration

* Exploring AI Values and Ethics through Participatory Design Fictions

* Ways of Knowing for AI: (Chat)bots as Interfaces for ML

* Human-AI Collaboration: Towards Socially-Guided Machine Learning

* Questioning the AI: Informing Design Practices for Explainable AI User Experiences

* Rethinking Model Evaluation as Narrowing the Socio-Technical Gap

* Human-Centered XAI: From Algorithms to User Experiences

* AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

* Fairness and explainability

* Questioning the AI: Informing Design Practices for Explainable AI User Experiences

* Expanding Explainability: Towards Social Transparency in AI Systems

* Connecting Algorithmic Research and Usage Contexts

In episode 100 of The Gradient Podcast, Daniel Bashir speaks to Professor Thomas Dietterich.

Professor Dietterich is Distinguished Professor Emeritus in the School of Electrical Engineering and Computer Science at Oregon State University. He is a pioneer in the field of machine learning, and has authored more than 225 refereed publications and two books. His current research topics include robust artificial intelligence, robust human-AI systems, and applications in sustainability. He is a former President of the Association for the Advancement of Artificial Intelligence, and the founding President of the International Machine Learning Society. Other major roles include Executive Editor of the journal Machine Learning, co-founder of the Journal for Machine Learning Research, and program chair of AAAI 1990 and NIPS 2000. He currently serves as one of the moderators for the cs.LG category on arXiv.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Episode 100 Note

* (02:03) Intro

* (04:23) Prof. Dietterich’s background

* (14:20) Kuhn and theory development in AI, how Prof Dietterich thinks about the philosophy of science and AI

* (20:10) Scales of understanding and sentience, grounding, observable evidence

* (23:58) Limits of statistical learning without causal reasoning, systematic understanding

* (25:48) A challenge for the ML community: testing for systematicity

* (26:13) Forming causal understandings of the world

* (28:18) Learning at the Knowledge Level

* (29:18) Background and definitions

* (32:18) Knowledge and goals, a note on LLMs

* (33:03) What it means to learn

* (41:05) LLMs as learning results of inference without learning first principles

* (43:25) System I/II thinking in humans and LLMs

* (47:23) “Routine Science”

* (47:38) Solving multiclass learning problems via error-correcting output codes

* (52:53) Error-correcting codes and redundancy

* (54:48) Why error-correcting codes work, contra intuition

* (59:18) Bias in ML

* (1:06:23) MAXQ for hierarchical RL

* (1:15:48) Computational sustainability

* (1:19:53) Project TAHMO’s moonshot

* (1:23:28) Anomaly detection for weather stations

* (1:25:33) Robustness

* (1:27:23) Motivating The Familiarity Hypothesis

* (1:27:23) Anomaly detection and self-models of competence

* (1:29:25) Measuring the health of freshwater streams

* (1:31:55) An open set problem in species detection

* (1:33:40) Issues in anomaly detection for deep learning

* (1:37:45) The Familiarity Hypothesis

* (1:40:15) Mathematical intuitions and the Familiarity Hypothesis

* (1:44:12) What’s Wrong with LLMs and What We Should Be Building Instead

* (1:46:20) Flaws in LLMs

* (1:47:25) The systems Prof Dietterich wants to develop

* (1:49:25) Hallucination/confabulation and LLMs vs knowledge bases

* (1:54:00) World knowledge and linguistic knowledge

* (1:55:07) End-to-end learning and knowledge bases

* (1:57:42) Components of an intelligent system and separability

* (1:59:06) Thinking through external memory

* (2:01:10) Outro

Links:

* Research — Fundamentals (Philosophy of AI)

* Learning at the Knowledge Level

* What Does it Mean for a Machine to Understand?

* Research – “Routine science”

* Ensemble methods in ML and error-correcting output codes

* Solving multiclass learning problems via error-correcting output codes

* An experimental comparison of bagging, boosting, and randomization

* ML Bias, Statistical Bias, and Statistical Variance of Decision Tree Algorithms

* The definitive treatment of these questions, by Gareth James

* Discovering/Exploiting structure in MDPs:

* MAXQ for hierarchical RL

* Exogenous State MDPs (paper with George Trimponias, slides)

* Research — Ecosystem Informatics and Computational Sustainability

* Project TAHMO

* Challenges for ML in Computational Sustainability

* Research — Robustness

* Steps towards robust AI (AAAI President’s Address)

* Benchmarking NN Robustness to Common Corruptions and Perturbations with Dan Hendrycks

* The familiarity hypothesis: Explaining the behavior of deep open set methods

* Recent commentary

* Toward High-Reliability AI

* What's Wrong with Large Language Models and What We Should Be Building Instead

In episode 99 of The Gradient Podcast, Daniel Bashir speaks to Professor Martin Wattenberg.

Professor Wattenberg is a professor at Harvard and part-time member of Google Research’s People + AI Research (PAIR) initiative, which he co-founded. His work, with long-time collaborator Fernanda Viégas, focuses on making AI technology broadly accessible and reflective of human values. At Google, Professor Wattenberg, his team, and Professor Viégas have created end-user visualizations for products such as Search, YouTube, and Google Analytics. Note: Professor Wattenberg is recruiting PhD students through Harvard SEAS—info here.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at [email protected]

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (03:30) Prof. Wattenberg’s background

* (04:40) Financial journalism at SmartMoney

* (05:35) Contact with the academic visualization world, IBM

* (07:30) Transition into visualizing ML

* (08:25) Skepticism of neural networks in the 1980s

* (09:45) Work at IBM

* (10:00) Multiple scales in information graphics, organization of information

* (13:55) How much information should a graphic display to whom?

* (17:00) Progressive disclosure of complexity in interface design

* (18:45) Visualization as a rhetorical process

* (20:45) Conversation Thumbnails for Large-Scale Discussions

* (21:35) Evolution of conversation interfaces—Slack, etc.

* (24:20) Path dependence — mutual influences between user behaviors and technology, takeaways for ML interface design

* (26:30) Baby Names and Social Data Analysis — patterns of interest in baby names

* (29:50) History Flow

* (30:05) Why investigate editing dynamics on Wikipedia?

* (32:06) Implications of editing patterns for design and governance

* (33:25) The value of visualizations in this work, issues with Wikipedia editing

* (34:45) Community moderation, bureaucracy

* (36:20) Consensus and guidelines

* (37:10) “Neutral” point of view as an organizing principle

* (38:30) Takeaways

* PAIR

* (39:15) Tools for model understanding and “understanding” ML systems

* (41:10) Intro to PAIR (at Google)

* (42:00) Unpacking the word “understanding” and use cases

* (43:00) Historical comparisons for AI development

* (44:55) The birth of TensorFlow.js

* (47:52) Democratization of ML

* (48:45) Visualizing translation — uncovering and telling a story behind the findings

* (52:10) Shared representations in LLMs and their facility at translation-like tasks

* (53:50) TCAV

* (55:30) Explainability and trust

* (59:10) Writing code with LMs and metaphors for using

* More recent research

* (1:01:05) The System Model and the User Model: Exploring AI Dashboard Design

* (1:10:05) OthelloGPT and world models, causality

* (1:14:10) Dashboards and interaction design—interfaces and core capabilities

* (1:18:07) Reactions to existing LLM interfaces

* (1:21:30) Visualizing and Measuring the Geometry of BERT

* (1:26:55) Note/Correction: The “Atlas of Meaning” Prof. Wattenberg mentions is called Context Atlas

* (1:28:20) Language model tasks and internal representations/geometry

* (1:29:30) LLMs as “next word predictors” — explaining systems to people

* (1:31:15) The Shape of Song

* (1:31:55) What does music look like?

* (1:35:00) Levels of abstraction, emergent complexity in music and language models

* (1:37:00) What Prof. Wattenberg hopes to see in ML and interaction design

* (1:41:18) Outro

Links:

* Professor Wattenberg’s homepage and Twitter

* Harvard SEAS application info — Professor Wattenberg is recruiting students!

* Research

* Earlier work

* A Fuzzy Commitment Scheme

* Stacked Graphs—Geometry & Aesthetics

* A Multi-Scale Model of Perceptual Organization in Information Graphics

* Conversation Thumbnails for Large-Scale Discussions

* Baby Names and Social Data Analysis

* History Flow (paper)

* At Harvard and Google / PAIR

* Tools for Model Understanding: Facets, SmoothGrad, Attacking discrimination with smarter ML

* TensorFlow.js

* Visualizing translation

* TCAV

* Other ML papers:

* The System Model and the User Model: Exploring AI Dashboard Design (recent speculative essay)

* Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

* Visualizing and Measuring the Geometry of BERT

* Artwork

* The Shape of Song