The Inside View

Robert Miles has been making videos for Computerphile, then decided to create his own Youtube channel about AI Safety. Lately, he's been working on  a Discord Community that uses Stampy the chatbot to answer Youtube comments. We also spend some time discussing recent AI Progress and why Rob is not that optimistic about humanity's survival.

Transcript: https://theinsideview.ai/rob

Youtube: https://youtu.be/DyZye1GZtfk

Host: https://twitter.com/MichaelTrazzi

Rob: https://twitter.com/robertskmiles

OUTLINE

(00:00:00) Intro

(00:02:25) Youtube

(00:28:30) Stampy

(00:51:24) AI Progress

(01:07:43) Chatbots

(01:26:10) Avoiding Doom

(01:59:34) Formalising AI Alignment

(02:14:40) AI Timelines

(02:25:45) Regulations

(02:40:22) Rob’s new channel

Connor was the first guest of this podcast. In the last episode, we talked a lot about EleutherAI, a grassroot collective of researchers he co-founded, who open-sourced GPT-3 size models such as GPT-NeoX and GPT-J.  Since then, Connor co-founded Conjecture, a company aiming to make AGI safe through scalable AI Alignment research.

One of the goals of Conjecture is to reach a fundamental understanding of the internal mechanisms of current deep learning models using interpretability techniques.  In this episode, we go through the famous AI Alignment compass memes, discuss Connor’s inside views about AI progress, how he approaches AGI forecasting, his takes on Eliezer Yudkowsky’s secret strategy, common misconceptions and EleutherAI, and why you should consider working for his new company Conjecture.

youtube: https://youtu.be/Oz4G9zrlAGs

transcript: https://theinsideview.ai/connor2

twitter: https:/twitter.com/MichaelTrazzi

OUTLINE

(00:00) Highlights

(01:08) AGI Meme Review 

(13:36) Current AI Progress

(25:43) Defining AG

(34:36) AGI Timelines

(55:34) Death with Dignity

(01:23:00) EleutherAI

(01:46:09) Conjecture

(02:43:58) Twitter Q&A

Raphaël Millière is a Presidential Scholar in Society and Neuroscience at Columbia University. He has previously completed a PhD in philosophy in Oxford, is interested in the philosophy of mind, cognitive science, and artificial intelligence, and has recently been discussing at length the current progress in AI with popular Twitter threads on GPT-3, Dalle-2 and a thesis he called “scaling maximalism”. Raphaël is also co-organizing with Gary Marcus a workshop about compositionality in AI at the end of the month. 

Transcript: https://theinsideview.ai/raphael

Video: https://youtu.be/2EHWzK10kvw

Host: https://twitter.com/MichaelTrazzi

Raphaël : https://twitter.com/raphaelmilliere 

Workshop: https://compositionalintelligence.github.io 

Outline

(00:36) definitions of artificial general intelligence

(7:25) behavior correlates of intellience, chinese room

(19:11) natural language understanding, the octopus test, linguistics, semantics

(33:05) generating philosophy with GPT-3, college essays grades, bullshit

(42:45) Stochastic Chameleon, out of distribution generalization

(51:19) three levels of generalization, the Wozniak test

(59:38) AI progress spectrum, scaling maximalism

(01:15:06) Bitter Lesson

(01:23:08) what would convince him that scale is all we need

(01:27:04) unsupervised learning, lifelong learning

(01:35:33) goalpost moving

(01:43:30) what researchers "should" be doing, nuclear risk, climate change

(01:57:24) compositionality, structured representations

(02:05:57) conceptual blending, complex syntactic structure, variable binding

(02:11:51) Raphaël's experience with DALL-E

(02:19:02) the future of image generation

Blake Richards is an Assistant Professor in the Montreal Neurological Institute and the School of Computer Science at McGill University and a Core Faculty Member at MiLA. He thinks that AGI is not a coherent concept, which is why he ended up on a recent AGI political compass meme. When people asked on Twitter who was the edgiest people at MiLA, his name got actually more likes than Ethan, so  hopefully, this podcast will help re-establish the truth.

Transcript: https://theinsideview.ai/blake

Video: https://youtu.be/kWsHS7tXjSU

Outline:

(01:03) Highlights

(01:03) AGI good / AGI not now compass

(02:25) AGI is not a coherent concept

(05:30) you cannot build truly general AI

(14:30) no "intelligence" threshold for AI

(25:24) benchmarking intelligence

(28:34) recursive self-improvement

(34:47) scale is something you need

(37:20) the bitter lesson is only half-true

(41:32) human-like sensors for general agents

(44:06) the credit assignment problem

(49:50) testing for backpropagation in the brain

(54:42) burstprop (bursts of action potentials), reward prediction errors

(01:01:35) long-term credit-assignment in reinforcement learning

(01:10:48) what would change his mind on scaling and existential risk

Ethan is known on Twitter as the edgiest person at MILA. We discuss all the gossips around scaling large language models in what will be later known as the Edward Snowden moment of Deep Learning. On his free time, Ethan is a Master’s degree student at MILA in Montreal, and has published papers on out of distribution generalization and robustness generalization, accepted both as oral presentations and spotlight presentations at ICML and NeurIPS. Ethan has recently been thinking about scaling laws, both as an organizer and speaker for the 1st Neural Scaling Laws Workshop.

Transcript: https://theinsideview.github.io/ethan

Youtube: https://youtu.be/UPlv-lFWITI

Michaël: https://twitter.com/MichaelTrazzi

Ethan: https://twitter.com/ethancaballero

Outline

(00:00) highlights

(00:50) who is Ethan, scaling laws T-shirts

(02:30) scaling, upstream, downstream, alignment and AGI

(05:58) AI timelines, AlphaCode, Math scaling, PaLM

(07:56) Chinchilla scaling laws

(11:22) limits of scaling, Copilot, generative coding, code data

(15:50) Youtube scaling laws, constrative type thing

(20:55) AGI race, funding, supercomputers

(24:00) Scaling at Google

(25:10) gossips, private research, GPT-4

(27:40) why Ethan was did not update on PaLM, hardware bottleneck

(29:56) the fastest path, the best funding model for supercomputers

(31:14) EA, OpenAI, Anthropics, publishing research, GPT-4

(33:45) a zillion language model startups from ex-Googlers

(38:07) Ethan's journey in scaling, early days

(40:08) making progress on an academic budget, scaling laws research

(41:22) all alignment is inverse scaling problems

(45:16) predicting scaling laws, useful ai alignment research

(47:16) nitpicks aobut Ajeya Cotra's report, compute trends

(50:45) optimism, conclusion on alignment

Peter is the co-CEO of Rethink Priorities, a fast-growing non-profit doing research on how to improve the long-term future. On his free time, Peter makes money in prediction markets and is quickly becoming one of the top forecasters on Metaculus. We talk about the probability of London getting nuked, Rethink Priorities and why EA should fund projects that scale.

Check out the video and transcript here: https://theinsideview.github.io/peter

Emil is a resident at the Google Arts & Culture Lab were he explores the intersection between art and machine learning. He recently built his own Machine Learning server, or rig, which costed him €25000.

Emil's Story: https://www.emilwallner.com/p/ml-rig

Youtube: https://youtu.be/njbPpxhE6W0

00:00 Intro

00:23 Building your own rig

06:11 The Nvidia GPU rder hack

15:51 Inside Emil's rig

21:31 Motherboard

23:55 Cooling and datacenters

29:36 Deep Learning lessons from owning your hardware

36:20 Shared resources vs. personal GPUs

39:12 RAM, chassis and airflow

42:42 Amd, Apple, Arm and Nvidia

51:15 Tensorflow, TPUs, cloud minsdet, EleutherAI

Sonia is a graduate student applying ML to neuroscience at MILA. She was previously applying deep learning to neural data at Janelia, an NLP research engineer at a startup and graduated in computational neuroscience at Princeton University. 

Anonymous feedback: https://app.suggestionox.com/r/xOmqTW 

Twitter: https://twitter.com/MichaelTrazzi 

Sonia's December update: https://t.co/z0GRqDTnWm

Sonia's Twitter: https://twitter.com/soniajoseph_ 

Orthogonality Thesis: https://www.youtube.com/watch?v=hEUO6pjwFOo

Paperclip game: https://www.decisionproblem.com/paperclips/

Ngo & Yudkowsky on feedback loops: https://bit.ly/3ml0zFL 

Outline 

00:00 Intro

01:06 NFTs

03:38 Web 3

21:12 Digital Copy

29:09 ML x Neuroscience

43:44 Limits of the Orthogonality Thesis

01:01:25 Goal of perpetuating Information

01:08:14 Compressing information

01:10:52 Feedback loops are not safe

01:17:43 Another AI Safety aesthetic

01:23:46 Meaning of life

Phil Trammell is an Oxford PhD student in economics and research associate at the Global Priorities Institute. Phil is one of the smartest person I know, when considering the intersection of the long-term future and economic growth. Funnily enough, Phil was my roomate, a few years ago in Oxford, and last time I called him he casually said that he had written an extensive report on the econ of AI. A few weeks ago, I decided that I would read that report (which actually is a literature review), and that I would translate everything that I learn along the way to diagrams, so you too can learn what’s inside that paper. The video covers everything from MacroEconomics 101 to self-improving AI in about 30-ish diagrams.

Outline:

- 00:00 Podcast intro

- 01:19 Phil's intro

- 08:58 What's GDP

- 13:42 Decreasing growth

- 15:40 Permanent growth increase

- 19:02 Singularity of type I

- 22:58 Singularity of type II

- 23:24 Production function

- 24:10 The Economy as a two-tubes factory

- 25:09 Marginal Products of labor/capital

- 27:48 Labor/capital-augmenting technology

- 29:13 Technological progress since Ford

- 38:18 Factor payments

- 41:30 Elasticity of substitution

- 48:34 Production function with substitution

- 53:18 Perfect substitutability

- 54:00 Perfect complements

- 55:44 Exogenous growth

- 59:56 How to get long-run growth

- 01:05:40 Endogenous growth

- 01:10:40 The research feedback parameter

- 01:17:35 AI as an imperfect substitute for human labor

- 01:25:25 A simple model for perfect substitution

- 01:33:09 AI as a perfect substitute

- 01:36:07 Substitutability in robotics production

- 01:40:43 OpenAI automating coding

- 01:44:38 Growth impacts via impacts on savings

- 01:46:44 AI in task-based models of good productions

- 01:53:26 AI in technology production

- 02:03:55 Limits of the econ model

- 02:09:00 Conclusion

In this episode I discuss Brain Computer Interfaces with Slava Bobrov, a self-taught Machine Learning Engineer applying AI to neural biosignals to control robotic limbs. This episode will be of special interest to you if you're an engineer who wants to get started with brain computer interfaces, or just broadly interested in how this technology could enhance human intelligence. Fun fact: most of the questions I asked were sent by my Twitter followers, or come from a Discord I co-created on Brain Computer Interfaces. So if you want your questions to be on the next video or you're genuinely interested in this topic, you can find links for both my Twitter and our BCI discord in the description.

Outline: