Things were busy once again, partly from the Claude release but from many other sides as well. So even after cutting out both the AI coding agent Devin and the Gladstone Report along with previously covering OpenAI's board expansion and investigative report, this is still one of the longest weekly posts.

In addition to Claude and Devin, we got among other things Command-R, Inflection 2.5, OpenAI's humanoid robot partnership reporting back after only 13 days and Google DeepMind with an embodied cross-domain video game agent. You can definitely feel the acceleration.

The backlog expands. Once again, I say to myself, I will have to up my reporting thresholds and make some cuts. Wish me luck.

Table of Contents

---

Outline:

(00:52) Language Models Offer Mundane Utility

(06:17) Claude 3 Offers Mundane Utility

(11:57) Prompt Attention

(17:28) Clauding Along

(27:19) Language Models Don’t Offer Mundane Utility

(31:33) GPT-4 Real This Time

(32:22) Copyright Confrontation

(33:34) Fun with Image Generation

(37:49) They Took Our Jobs

(40:38) Get Involved

(43:53) Introducing

(51:55) Inflection 2.5

(55:04) Paul Christiano Joins NIST

(01:01:38) In Other AI News

(01:07:40) Quiet Speculations

(01:23:06) The Quest for Sane Regulations

(01:31:34) The Week in Audio

(01:31:44) Rhetorical Innovation

(01:41:23) A Failed Attempt at Adversarial Collaboration

(01:48:37) Spy Versus Spy

(01:52:38) Shouting Into the Void

(01:54:55) Open Model Weights are Unsafe and Nothing Can Fix This

(01:56:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:59:49) People Are Worried About AI Killing Everyone

(02:00:39) Other People Are Not As Worried About AI Killing Everyone

(02:07:39) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

For years I (Elizabeth) have been trying to write out my grant unified theory of [good/bad/high-variance/high-investment] [jobs/relationships/religions/social groups]. In this dialogue me (Elizabeth) and Ruby throw a bunch of component models back and forth and get all the way to better defining the question. 🎉

About a year ago someone published Common Knowledge About Leverage Research, which IIRC had some information that was concerning but not devastating. You showed me a draft of a reply you wrote to that post, that pointed out lots of similar things Lightcone/LessWrong did and how, yeah, they could look bad, but they could also be part of a fine trade-off. Before you could publish that, an ex-employee of Leverage published a much more damning account.

This feels to me like it encapsulates part of a larger system of trade-offs. Accomplishing big things sometimes requires weirdness, and sometimes sacrifice, but places telling you [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TikTok Might Get Banned Soon

This attempt is getting reasonably far rather quickly, passing the House with broad support.

Alec Stapp: TikTok bill to remove influence of CCP:

– passed unanimously out of committee

– GOP leadership says they’ll bring it to the floor for a vote next week

– Biden says he’ll sign the bill if passed

Can’t believe it's taken this long, but should be done soon.

It's been obvious for years that we shouldn’t let China control a black-box algorithm that influences >100 million American users.

JSM: Can this stand up to court scrutiny though?

Alec Stapp: Yes.

It then passed the house 352-65, despite opposition from Donald Trump.

Manifold is as of now around 72% that a bill will pass, similar to Metaculus. Consensus is that it is unlikely that ByteDance will divest. They [...]

---

Outline:

(00:03) TikTok Might Get Banned Soon

(02:48) Execution is Everything

(10:33) RTFB: Read The Bill

(19:57) How Popular is a TikTok Ban?

(20:15) Reciprocity is the Key to Every Relationship

(23:14) Call Your Congressman

(28:25) TikTok Data Sharing

(34:41) TikTok Promoting Chinese Interests

(45:41) Tyler Cowen Opposes the Bill

(48:23) Trump Opposes the Bill

(53:40) To Be Clear You Can Absolutely Go Too Far

(54:09) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

Crossposted by habryka after asking Eliezer whether I could post it under his account

i.

"Ignore all these elaborate, abstract, theoretical predictions," the Spokesperson for Ponzi Pyramid Incorporated said in a firm, reassuring tone. "Empirically, everyone who's invested in Bernie Bankman has received back 144% of what they invested two years later."

"That's not how 'empiricism' works," said the Epistemologist. "You're still making the assumption that --"

"You could only believe that something different would happen in the future, if you believed in elaborate theoretical analyses of Bernie Bankman's unobservable internal motives and internal finances," said the spokesperson for Ponzi Pyramid Incorporated. "If you are a virtuous skeptic who doesn't trust in overcomplicated arguments, you'll believe that future investments will also pay back 144%, just like in the past. That's the prediction you make if you predict based purely on empirical observations, instead of theories about a future nobody [...]

---

Outline:

(00:11) i.

(10:06) ii.

(31:23) iii.

(39:41) iiv.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Thanks to Ryan Greenblatt, Buck Shlegeris, Aryan Bhatt, and Akbir Khan for useful discussions and feedback on the draft of this post.

If AIs are potentially scheming and more knowledgeable than humans, and you want to answer a question, it may seem natural to not let AIs choose what answer they will argue for. For example, in AI debate, you make copies of an AI argue for the opposite answers to the question. Another example is consultancy (the usual baseline in debate papers), where you make AIs argue for a random answer to the question.

But letting potentially scheming AIs choose what side to argue for can work, as long as you train them to generate answers and justifications that are as convincing as possible. I call this approach open consultancy.

In this post, I argue that [...]

---

Outline:

(01:31) Open consultancy

(01:34) Definition of regular consultancy

(02:30) Definition of open consultancy

(03:31) Open consultancy is as robust against scheming as regular consultancy

(04:46) Open consultancy is sometimes better than debate

(05:22) Toy model 1: noisy convincingness

(06:25) Toy model 2: unknown non-expert weakness

(07:40) Toy model 3: experts finding CoT actually useful

(08:44) Combining open consultancy and debate

(10:12) Appendix

(10:15) Open consultancy is almost regular instruction-tuning RLHF

(11:17) Predictions

(12:59) Training the non-expert

(13:48) Weaknesses shared with debate

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The Good Judgement Project got some superforecasters to retrocast whether COVID started via zoonotic spillover or a lab leak. They in aggregate gave a 75% chance of zoonosis, but there was a range of views. GJP's executive summary is at the end of this linkpost.

Here is a link to the summary of the report on their substack, and here is a link to the full report (which is a total of 6 pages of content).

h/t John Halstead for drawing my attention to this.

Superforecasters assess that natural zoonosis is three times more likely to be the cause of the Covid-19 pandemic than either a biomedical research-related accident or some other process or mechanism. Asked to assign a probability to what caused the emergence of SARS-CoV-2 in human populations, more than 50 Superforecasters engaged in extensive online discussions starting on December 1, 2023.

---

First published:

Source:

Linkpost URL:
https://goodjudgment.substack.com/p/superforecasting-the-origins-of-the

Narrated by TYPE III AUDIO.

Previously: Some physics 101 students calculate that a certain pendulum will have a period of approximately 3.6 seconds. Instead, when they run the experiment, the stand holding the pendulum tips over and the whole thing falls on the floor.

The students, being diligent Bayesians, argue that this is strong evidence against Newtonian mechanics, and the professor's attempts to rationalize the results in hindsight are just that: rationalization in hindsight. What say the professor?

“Hold on now,” the professor answers, “‘Newtonian mechanics’ isn’t just some monolithic magical black box. When predicting a period of approximately 3.6 seconds, you used a wide variety of laws and assumptions and approximations, and then did some math to derive the actual prediction. That prediction was apparently incorrect. But at which specific point in the process did the failure occur?

For instance:

• Were there forces on the pendulum weight not included in the free [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

It is largely over.

The investigation into events has concluded, finding no wrongdoing anywhere.

The board has added four new board members, including Sam Altman. There will still be further additions.

Sam Altman now appears firmly back in control of OpenAI.

None of the new board members have been previously mentioned on this blog, or known to me at all.

They are mysteries with respect to AI. As far as I can tell, all three lack technical understanding of AI and have no known prior opinions or engagement on topics of AI, AGI and AI safety of any kind including existential risk.

Microsoft and investors indeed so far have came away without a seat. They also, however, lack known strong bonds to Altman, so this is not obviously a board fully under his control if there were to be another crisis. They now [...]

---

Outline:

(02:34) The New Board

(11:27) The Investigation Probably Was Not Real

(19:09) The New York Times Leak and Gwern's Analysis of It

(30:29) What Do We Now Think Happened?

(36:51) Altman's Statement

(39:26) Helen Toner and Tasha McCauley's Statement

(40:44) The Case Against Altman

(47:48) The Case For Altman and What We Will Learn Next

---

First published:

Source:

Narrated by TYPE III AUDIO.

Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock[1]

Today, the Forecasting Research Institute (FRI) released “Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration,” which discusses the results of an adversarial collaboration focused on forecasting risks from AI.

In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf

(This report is cross-posted to the EA Forum.)

We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The “concerned” participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the “skeptical” group (mainly “superforecasters”) predicted a 0.12% chance. Participants [...]

---

Outline:

(02:16) Extended Executive Summary

(02:47) Methods

(03:56) Results: What drives (and doesn’t drive) disagreement over AI risk

(04:35) Hypothesis #1 - Disagreements about AI risk persist due to lack of engagement among participants, low quality of participants, or because the skeptic and concerned groups did not understand each others arguments

(05:14) Hypothesis #2 - Disagreements about AI risk are explained by different short-term expectations (e.g. about AI capabilities, AI policy, or other factors that could be observed by 2030)

(07:56) Hypothesis #3 - Disagreements about AI risk are explained by different long-term expectations

(10:38) Hypothesis #4 - These groups have fundamental worldview disagreements that go beyond the discussion about AI

(11:34) Results: Forecasting methodology

(12:18) Broader scientific implications

(13:12) Directions for further research

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Linkpost URL:
https://forecastingresearch.org/s/AIcollaboration.pdf

Narrated by TYPE III AUDIO.

(Crossposted from twitter for easier linking.) (Intended for a broad audience—experts already know all this.)

When I talk about future “Artificial General Intelligence” (AGI), what am I talking about? Here's a handy diagram and FAQ:

“But GPT-4 can’t do any of those things.” Indeed it cannot—hence the word “future”!

“That will never happen because nobody wants AIs that are like that.” For one thing, imagine an AI where you can give it seed capital and ask it to go found a new company, and it does so, just as skillfully as Earth's most competent and experienced remote-only human CEO. And you can repeat this millions of times in parallel with millions of copies of this AI, and each copy costs $0.10/hour to run. You think nobody wants to have an AI that can do that? Really?? And also, just look around. Plenty of AI researchers and companies are trying [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.