The Gradient: Perspectives on AI

In episode 29 of The Gradient Podcast, we chat with Rosanne Liu. Rosanne is a research scientist in Google Brain, and co-founder and executive director of ML Collective, a nonprofit organization for open collaboration and accessible mentorship. Before that she was a founding member of Uber AI. Outside of research, she supports underrepresented communities, and organizes symposiums, workshops, and a weekly reading group “Deep Learning: Classics and Trends” since 2018. She is currently thinking deeply how to democratize AI research even further, and improve the diversity and fairness of the field, while working on multiple fronts of machine learning research including understanding training dynamics, rethinking model capacity and scaling. 

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (01:30) How did you go into AI / research

* (6:45) AI research: the unreasonably narrow path and how not to be miserable

* (16:30) ML Collective Overview

* (21:45) Deep Learning: Classics and Trends Reading Group

* (26:25) More details about ML Collective

* (39:35) ICLR 2022 Diversity, Equity & Inclusion

* (48:00) Narrowness vs Variety in research

* (57:20) Favorite Papers 

* (58:50) Measuring the Intrinsic Dimension of Objective Landscapes 

* (01:01:40) Natural Adversarial Objects 

* (01:03:00) Interests outside of AI - Writing

* (01:08:05) Interests outside of AI - Narrating Travels with Charley

* (01:13:22) Outro

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 28 of The Gradient Podcast, Daniel Bashir speaks to Ben Green, postdoctoral scholar in the Michigan Society of Fellows and Assistant Professor at the Gerald R. Ford School of Public Policy. Ben’s work focuses on the social and political impacts of government algorithms.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Sections:

(00:00) Intro

(02:00) Getting Started

(06:15) Soul Searching

(11:55) Decentering Algorithms

(19:50) The Future of the City

(27:25) Ethical Lip Service

(32:30) Ethics Research and Industry Incentives

(36:30) Broadening our Vision of Tech Ethics

(47:35) What Types of Research are Valued?

(52:40) Outro

Episode Links:

* Ben’s Homepage

* Algorithmic Realism

* Special Issue of the Journal of Social Computing

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 27 of The Gradient Podcast, Andrey Kurenkov speaks to Max Braun, who leads the AI and robotics software engineering team at Everyday Robots, a moonshot to create robots that can learn to help people in their everyday lives. Previously, he worked on building frontier technology products as an entrepreneur and later at Google and X. Max enjoys exploring the intersection of art, technology, and philosophy as a writer and designer. 

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (01:00) Start in AI

* (5:45) Humanoid Research in Osaka

* (8:45) Joining Google X

* (12:15) Visual Search and Google Glass

* (15:58) Academia Industry Connection

* (18:45) Overview of Robotics Vision

* (26:00) Machine Learning for Robotics

* (32:00) Robot Platform

* (38:00) Development Process and History

* (43:35) QT-Opt

* (49:05) Imitation Learning

* (55:00) Simulation Platform

* (59:45) Sim2Real

* (1:07:00) SayCan

* (1:14:30) Current Objectives

* (1:17:00) Other Projects

* (1:21:40) Outro

Episode Links:

* Max Braun’s Website

* Everyday Robots

* Simulating Artificial Muscles for Controlling a Robotic Arm with Fluctuation

* Introducing the Everyday Robot Project

* Scalable Deep Reinforcement Learning from Robotic Manipulation (QT-Opt)

* Alphabet is putting its prototype robots to work cleaning up around Google’s offices

* Everyday robots are (slowly) leaving the lab

* Can Robots Follow Instructions for New Tasks?

* Efficiently Initializing Reinforcement Learning With Prior Policies

* Combining RL + IL at Scale

* Shortening the Sim to Real Gap

* Action-Image: Teaching Grasping in Sim

* SayCan

* I Made an AI Read Wittgenstein, Then Told It to Play Philosopher

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 26 of The Gradient Podcast, Daniel Bashir speaks to Yejin Choi, professor of Computer Science at the University of Washington, and senior research manager at the Allen Institute for Artificial Intelligence.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Sections:

(00:00) Intro

(01:42) Getting Started in the Winter

(09:17) Has NLP lost its way?

(12:57) The Mosaic Project, Commonsense Intelligence

(18:20) A Priori Intuitions and Common Sense in Machines

(21:35) Abductive Reasoning

(24:49) Benchmarking Common Sense

(33:00) DeLorean and COMET - Algorithms for Commonsense Reasoning

(43:30) Positive and Negative uses of Commonsense Models

(49:40) Moral Reasoning

(57:00) Descriptive Morality, Meta-Ethical Concerns

(1:04:30) Potential Misuse

(1:12:15) Future Work

(1:16:23) Outro

Episode Links:

* Yejin’s Homepage

* The Curious Case of Commonsense Intelligence in Daedalus

* Common Sense Comes Close to Computers in Quanta

* Can Computers Learn Common Sense? in The New Yorker

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 25 of The Gradient Podcast, Daniel Bashir speaks to David Chalmers, professor of philosophy and Philosophy and Neural Science at New York University, and co-director of NYU’s center for Mind, Brain, and Consciousness.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Sections:

(00:00) Intro(00:42) “Today’s neural networks may be slightly conscious”(03:55) Openness to Machine Consciousness(09:37) Integrated Information Theory(18:41) Epistemic Gaps, Verbal Reports(25:52) Vision Models and Consciousness(33:37) Reasoning about Consciousness(38:20) Illusionism(41:30) Best Approaches to the Hard Problem(44:21) Panpsychism(46:35) Outro

Episode Links:

* Chalmers’ Homepage

* Facing Up to the Hard Problem of Consciousness (1995)

* Reality+: Virtual Worlds and the Problems of Philosophy

* Amanda Askell on AI Consciousness

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 24 of The Gradient Podcast, Daniel Bashir talks to Greg Yang, senior researcher at Microsoft Research. Greg Yang’s Tensor Programs framework recently received attention for its role in the µTransfer paradigm for tuning the hyperparameters of large neural networks.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Sections:

(00:00) Intro(01:50) Start in AI / Research(05:55) Fear of Math in ML(08:00) Presentation of Research(17:35) Path to MSR(21:20) Origin of Tensor Programs(26:05) Refining TP’s Presentation(39:55) The Sea of Garbage (Initializations) and the Oasis(47:44) Scaling Up Further(55:53) On Theory and Practice in Deep Learning(01:05:28) Outro

Episode Links:

* Greg’s Homepage

* Greg’s Twitter

* µP GitHub

* Visual Intro to Gaussian Processes (Distill)

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In the 23rd interview of The Gradient Podcast, we talk to Nick Walton, the CEO and Co-Founder of Latitude, the goal of which is to make AI a tool of freedom and creativity for everyone, and which is currently developing AI Dungeon and Voyage.

Subscribe to The Gradient Podcast:  

* Apple Podcasts

* Spotify

* Pocket Casts

* RSS

Outline:(00:00) Intro(01:38) How did you go into AI / research(3:50) Origin of AI Dungeon(8:15) What is a Dungeon Master(12:!5) Brief history of AI Dungeon(17:30) AI in videogames, past and future(23:35) Early days of AI Dungeon(29:45) AI Dungeon as a Creative Tool(33:50) Technical Aspects of AI Dungeon(39:15) Voyage(48:27) Visuals in AI Dungeon(50:45) How to Control AI in Games(55:38) Future of AI in Games(57:50) Funny stories(59:45) Interests / Hobbies(01:01:45) Outro

Get full access to The Gradient at thegradientpub.substack.com/subscribe

How AI is Changing Chemical Discovery

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Preview:

While engineering, finance, and commerce have profited immensely from novel algorithms, they are not the only ones. Large-scale computation has been an integral part of the toolkit in the physical sciences for many decades - and some of the recent advances in AI have started to change how scientific discoveries are made.

There has been a lot of excitement about prominent achievements in the physical sciences, like using machine learning to render an image of a black hole or the contribution of AlphaFold towards protein folding. This article will cover some of the more prominent usages of AI in chemistry, the parent discipline of the aforementioned protein folding problem.

Get full access to The Gradient at thegradientpub.substack.com/subscribe

Reflections on Foundation Models

Preview:

Recently, we released our report on foundation models, launched the Stanford Center for Research on Foundation Models (CRFM) as part of the Stanford Institute for Human-Centered AI (HAI), and hosted a workshop to foster community-wide dialogue. Our work received an array of responses from a broad range of perspectives; some folks graciously shared their commentaries with us. We see open discourse as necessary for forging the right norms, best practices, and broader ecosystem around foundation models. In this blog post, we talk through why we believe these models are so important and clarify several points in relation to the community response. In addition, we support and encourage further community discussion of these complex issues; feel free to reach out at [email protected].

Get full access to The Gradient at thegradientpub.substack.com/subscribe

In episode 22 of The Gradient Podcast, we talk to Connor Leahy, an AI researcher focused on AI alignment and a co-founder of EleutherAI.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Connor is an AI researcher working on understanding large ML models and aligning them to human values, and a cofounder of EleutherAI, a decentralized grassroots collective of volunteer researchers, engineers, and developers focused on AI alignment, scaling, and open source AI research. The organization's flagship project is the GPT-Neo family of models designed to match those developed by OpenAI as GPT-3.

Sections:

(00:00:00) Intro(00:01:20) Start in AI(00:08:00) Being excited about GPT-2 (00:18:00) Discovering AI safety and alignment(00:21:10) Replicating GPT-2 (00:27:30) Deciding whether to relese GPT-2 weights(00:36:15) Life after GPT-2 (00:40:05) GPT-3 and Start of Eleuther AI(00:44:40) Early days of Eleuther AI(00:47:30) Creating the Pile, GPT-Neo, Hacker Culture(00:55:10) Growth of Eleuther AI, Cultivating Community(01:02:22) Why release a large language model(01:08:50) AI Risk and Alignment(01:21:30) Worrying (or not) about Superhuman AI(01:25:20) AI alignment and releasing powerful models(01:32:08) AI risk and research norms(01:37:10) Work on GPT-3 replication, GPT-NeoX(01:38:48) Joining Eleuther AI(01:43:28) Personal interests / hobbies(01:47:20) Outro

Links to things discussed:

* Replicating GPT2–1.5B , GPT2, Counting Consciousness and the Curious Hacker

* The Hacker Learns to Trust

* The Pile

* GPT-Neo

* GPT-J

* Why Release a Large Language Model?

* What A Long, Strange Trip It's Been: EleutherAI One Year Retrospective

* GPT-NeoX

Get full access to The Gradient at thegradientpub.substack.com/subscribe