Yann LeCun is perhaps the most prominent critic of the “LessWrong view” on AI safety, the only one of the three "godfathers of AI" to not acknowledge the risks of advanced AI. So, when he recently appeared on the Lex Fridman podcast, I listened with the intent to better understand his position. LeCun came across as articulate / thoughtful[1]. Though I don't agree with it all, I found a lot worth sharing.

Most of this post consists of quotes from the transcript, where I’ve bolded the most salient points. There are also a few notes from me as well as a short summary at the end.

Limitations of Autoregressive LLMs

Lex Fridman (00:01:52) You've said that autoregressive LLMs are not the way we’re going to make progress towards superhuman intelligence. These are the large language models like GPT-4, like Llama 2 and 3 soon and so on. [...]

---

Outline:

(00:42) Limitations of Autoregressive LLMs

(05:38) Grounding / Embodiment

(05:41) Richness of Interaction with the Real World

(10:32) Language / Video and Bandwidth

(15:09) Hierarchical Planning

(17:37) Skepticism of Autoregressive LLMs

(32:08) RL(HF)

(36:31) Bias / Open Source

(39:15) Business and Open Source

(41:13) Safety of (Current) LLMs

(43:23) LLaMAs

(46:29) GPUs vs the Human Brain

(49:04) What does AGI / AMI Look Like?

(51:29) AI Doomers

(57:25) What Does a World with AGI Look Like (Especially Re Safety)?

(01:03:08) Big Companies and AI

(01:05:18) Hope for the Future of Humanity

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Redwood (where Ryan works) recently released a series of blogposts proposing a framing and research agenda for reducing AI-risk that focuses on ensuring safety (and secondarily usefulness) under the conservative assumption that AIs are misaligned and actively scheming against human interests, under the name "AI Control".

This is in contrast to other work on AI risk which focuses on reducing the probability that AI systems pursue goals that are in conflict with human values in the first place (which might include having it not pursue goals in the relevant sense at all), usually called "AI Alignment". In other words, control aims to ensure that even if your models are actively misaligned, you'll be safe, because they are not capable of subverting your safety measures.

In this dialogue we dig into our disagreements on the degree to which this kind of work seems promising, and whether/how this reframing opens [...]

---

Outline:

(01:23) The Case for Control Work

(22:48) Problems with training against bad behavior, sample efficiency, and exploration-hacking

(53:50) Aside on trading with AI systems

(57:12) How easily can you change the goals of an AI system with training?

(01:03:05) Appendix: Other comments on Ryans shortform post

---

First published:

Source:

Narrated by TYPE III AUDIO.

Things were busy once again, partly from the Claude release but from many other sides as well. So even after cutting out both the AI coding agent Devin and the Gladstone Report along with previously covering OpenAI's board expansion and investigative report, this is still one of the longest weekly posts.

In addition to Claude and Devin, we got among other things Command-R, Inflection 2.5, OpenAI's humanoid robot partnership reporting back after only 13 days and Google DeepMind with an embodied cross-domain video game agent. You can definitely feel the acceleration.

The backlog expands. Once again, I say to myself, I will have to up my reporting thresholds and make some cuts. Wish me luck.

Table of Contents

---

Outline:

(00:52) Language Models Offer Mundane Utility

(06:17) Claude 3 Offers Mundane Utility

(11:57) Prompt Attention

(17:28) Clauding Along

(27:19) Language Models Don’t Offer Mundane Utility

(31:33) GPT-4 Real This Time

(32:22) Copyright Confrontation

(33:34) Fun with Image Generation

(37:49) They Took Our Jobs

(40:38) Get Involved

(43:53) Introducing

(51:55) Inflection 2.5

(55:04) Paul Christiano Joins NIST

(01:01:38) In Other AI News

(01:07:40) Quiet Speculations

(01:23:06) The Quest for Sane Regulations

(01:31:34) The Week in Audio

(01:31:44) Rhetorical Innovation

(01:41:23) A Failed Attempt at Adversarial Collaboration

(01:48:37) Spy Versus Spy

(01:52:38) Shouting Into the Void

(01:54:55) Open Model Weights are Unsafe and Nothing Can Fix This

(01:56:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:59:49) People Are Worried About AI Killing Everyone

(02:00:39) Other People Are Not As Worried About AI Killing Everyone

(02:07:39) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

For years I (Elizabeth) have been trying to write out my grant unified theory of [good/bad/high-variance/high-investment] [jobs/relationships/religions/social groups]. In this dialogue me (Elizabeth) and Ruby throw a bunch of component models back and forth and get all the way to better defining the question. 🎉

About a year ago someone published Common Knowledge About Leverage Research, which IIRC had some information that was concerning but not devastating. You showed me a draft of a reply you wrote to that post, that pointed out lots of similar things Lightcone/LessWrong did and how, yeah, they could look bad, but they could also be part of a fine trade-off. Before you could publish that, an ex-employee of Leverage published a much more damning account.

This feels to me like it encapsulates part of a larger system of trade-offs. Accomplishing big things sometimes requires weirdness, and sometimes sacrifice, but places telling you [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TikTok Might Get Banned Soon

This attempt is getting reasonably far rather quickly, passing the House with broad support.

Alec Stapp: TikTok bill to remove influence of CCP:

– passed unanimously out of committee

– GOP leadership says they’ll bring it to the floor for a vote next week

– Biden says he’ll sign the bill if passed

Can’t believe it's taken this long, but should be done soon.

It's been obvious for years that we shouldn’t let China control a black-box algorithm that influences >100 million American users.

JSM: Can this stand up to court scrutiny though?

Alec Stapp: Yes.

It then passed the house 352-65, despite opposition from Donald Trump.

Manifold is as of now around 72% that a bill will pass, similar to Metaculus. Consensus is that it is unlikely that ByteDance will divest. They [...]

---

Outline:

(00:03) TikTok Might Get Banned Soon

(02:48) Execution is Everything

(10:33) RTFB: Read The Bill

(19:57) How Popular is a TikTok Ban?

(20:15) Reciprocity is the Key to Every Relationship

(23:14) Call Your Congressman

(28:25) TikTok Data Sharing

(34:41) TikTok Promoting Chinese Interests

(45:41) Tyler Cowen Opposes the Bill

(48:23) Trump Opposes the Bill

(53:40) To Be Clear You Can Absolutely Go Too Far

(54:09) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

Crossposted by habryka after asking Eliezer whether I could post it under his account

i.

"Ignore all these elaborate, abstract, theoretical predictions," the Spokesperson for Ponzi Pyramid Incorporated said in a firm, reassuring tone. "Empirically, everyone who's invested in Bernie Bankman has received back 144% of what they invested two years later."

"That's not how 'empiricism' works," said the Epistemologist. "You're still making the assumption that --"

"You could only believe that something different would happen in the future, if you believed in elaborate theoretical analyses of Bernie Bankman's unobservable internal motives and internal finances," said the spokesperson for Ponzi Pyramid Incorporated. "If you are a virtuous skeptic who doesn't trust in overcomplicated arguments, you'll believe that future investments will also pay back 144%, just like in the past. That's the prediction you make if you predict based purely on empirical observations, instead of theories about a future nobody [...]

---

Outline:

(00:11) i.

(10:06) ii.

(31:23) iii.

(39:41) iiv.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Thanks to Ryan Greenblatt, Buck Shlegeris, Aryan Bhatt, and Akbir Khan for useful discussions and feedback on the draft of this post.

If AIs are potentially scheming and more knowledgeable than humans, and you want to answer a question, it may seem natural to not let AIs choose what answer they will argue for. For example, in AI debate, you make copies of an AI argue for the opposite answers to the question. Another example is consultancy (the usual baseline in debate papers), where you make AIs argue for a random answer to the question.

But letting potentially scheming AIs choose what side to argue for can work, as long as you train them to generate answers and justifications that are as convincing as possible. I call this approach open consultancy.

In this post, I argue that [...]

---

Outline:

(01:31) Open consultancy

(01:34) Definition of regular consultancy

(02:30) Definition of open consultancy

(03:31) Open consultancy is as robust against scheming as regular consultancy

(04:46) Open consultancy is sometimes better than debate

(05:22) Toy model 1: noisy convincingness

(06:25) Toy model 2: unknown non-expert weakness

(07:40) Toy model 3: experts finding CoT actually useful

(08:44) Combining open consultancy and debate

(10:12) Appendix

(10:15) Open consultancy is almost regular instruction-tuning RLHF

(11:17) Predictions

(12:59) Training the non-expert

(13:48) Weaknesses shared with debate

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The Good Judgement Project got some superforecasters to retrocast whether COVID started via zoonotic spillover or a lab leak. They in aggregate gave a 75% chance of zoonosis, but there was a range of views. GJP's executive summary is at the end of this linkpost.

Here is a link to the summary of the report on their substack, and here is a link to the full report (which is a total of 6 pages of content).

h/t John Halstead for drawing my attention to this.

Superforecasters assess that natural zoonosis is three times more likely to be the cause of the Covid-19 pandemic than either a biomedical research-related accident or some other process or mechanism. Asked to assign a probability to what caused the emergence of SARS-CoV-2 in human populations, more than 50 Superforecasters engaged in extensive online discussions starting on December 1, 2023.

---

First published:

Source:

Linkpost URL:
https://goodjudgment.substack.com/p/superforecasting-the-origins-of-the

Narrated by TYPE III AUDIO.

Previously: Some physics 101 students calculate that a certain pendulum will have a period of approximately 3.6 seconds. Instead, when they run the experiment, the stand holding the pendulum tips over and the whole thing falls on the floor.

The students, being diligent Bayesians, argue that this is strong evidence against Newtonian mechanics, and the professor's attempts to rationalize the results in hindsight are just that: rationalization in hindsight. What say the professor?

“Hold on now,” the professor answers, “‘Newtonian mechanics’ isn’t just some monolithic magical black box. When predicting a period of approximately 3.6 seconds, you used a wide variety of laws and assumptions and approximations, and then did some math to derive the actual prediction. That prediction was apparently incorrect. But at which specific point in the process did the failure occur?

For instance:

• Were there forces on the pendulum weight not included in the free [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

It is largely over.

The investigation into events has concluded, finding no wrongdoing anywhere.

The board has added four new board members, including Sam Altman. There will still be further additions.

Sam Altman now appears firmly back in control of OpenAI.

None of the new board members have been previously mentioned on this blog, or known to me at all.

They are mysteries with respect to AI. As far as I can tell, all three lack technical understanding of AI and have no known prior opinions or engagement on topics of AI, AGI and AI safety of any kind including existential risk.

Microsoft and investors indeed so far have came away without a seat. They also, however, lack known strong bonds to Altman, so this is not obviously a board fully under his control if there were to be another crisis. They now [...]

---

Outline:

(02:34) The New Board

(11:27) The Investigation Probably Was Not Real

(19:09) The New York Times Leak and Gwern's Analysis of It

(30:29) What Do We Now Think Happened?

(36:51) Altman's Statement

(39:26) Helen Toner and Tasha McCauley's Statement

(40:44) The Case Against Altman

(47:48) The Case For Altman and What We Will Learn Next

---

First published:

Source:

Narrated by TYPE III AUDIO.