I've been thinking about memetically fit Schelling points in the AGI policy space. I'll describe four such "Schelling policies", and use them as pedagogical examples.

Shut it all down

MIRI's new stated objective is the clearest example of a Schelling policy: "Shut it all down". MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies.

If you want to coordinate a large number of people coherently towards furthering a particular policy, "you get about five words" that you can make 'common knowledge' such that people can coordinate in a specific direction. The ease of communicating the policy makes a big difference in such conditions.

When you attempt to communicate an idea [...]

---

Outline:

(00:17) Shut it all down

(02:28) Effective Accelerationism (e/acc)

(04:58) We must beat China

(06:58) Only One Org

(09:36) Parting words

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Acknowledgements

The vision here was midwifed originally in the wild and gentle radiance that is Abram's company (though essentially none of the content is explicitly his).

The PIBBSS-spirit has been infused in this work from before it began (may it infuse us all), as have meetings with the Agent Foundations team at MIRI over the past ~2 years.

More recently, everyone who has been loving the High Actuation project into form (very often spontaneously and without being encumbered by self-consciousness of this fact):[1] individuals include Steve Petersen, Mateusz Baginski, Aditya Prasad, Harmony, TJ, Chris Lakin; the AISC 2024 team, Murray Buchanan, Matt Farr, Arpan Agrawal, Adam, Ryan, Quinn; various people from Topos, ALIFE, MAPLE, EA Bangalore. Published while at CEEALAR.

Disclaimers

• Very occasionally there are small remarks/questions from a remarkable human named Steve, since this and the next two posts are an edited transcript of me [...]

---

Outline:

(00:05) Acknowledgements

(01:01) Disclaimers

(01:40) Taking Intelligence Seriously

(03:04) Failing to Take Intelligence Seriously

(05:02) Opportunities from Moderate Intelligence at Scale

(09:14) Live Interfaces

(12:57) Live Theory (preface)

(14:33) Navigation

(17:02) Live Theory (teaser)

(17:06) Possibility

The original text contained 14 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The progress movement has grown a lot in the last few years. We now have progress journals, think tanks, and fellowships. The progress idea has spread and evolved into the “abundance agenda”, “techno-optimism”, “supply-side progressivism”, “American dynamism”. All of us want to see more scientific, technological, and economic progress for the good of humanity, and envision a bold, ambitious, flourishing future.

What we haven’t had so far is a regular gathering of the community.

Announcing Progress Conference 2024, a two-day event to connect people in the progress movement. Meet great people, share ideas in deep conversations, catalyze new projects, get energized and inspired.

Hosted by: the Roots of Progress Institute, together with the Foresight Institute, HumanProgress.org, the Institute for Humane Studies, the Institute for Progress, and Works in Progress magazine

When: October 18–19, 2024

Where: Berkeley, CA—at the Lighthaven campus, an inviting space perfect [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Throughout this post, we’re going to follow the Cognition -> Convergence -> Consequences methodology[1]. That means we’ll tackle tool-ness in three main stages, each building on the previous:

• Cognition: What does it mean, cognitively, to view or model something as a tool?
• Convergence: Insofar as different minds (e.g. different humans) tend to convergently model the same things as tools, what are the “real patterns” in the environment which give rise to that convergence?
• Consequences: Having characterized the real patterns convergently recognized as tool-ness, what other properties or consequences of tool-ness can we derive? What further predictions does our characterization make?

We’re not going to do any math in this post, though we will gesture at the spots where proofs or quantitative checks would ideally slot in.

Cognition: What does it mean, cognitively, to view or model something as a tool?

Let's start with a mental model of (the cognition [...]

---

Outline:

(01:02) Cognition: What does it mean, cognitively, to view or model something as a tool?

(03:27) Convergence: Insofar as different minds (e.g. different humans) tend to convergently model the same things as tools, what are the “real patterns” in the environment which give rise to that convergence?

(06:12) Consequences: From that characterization, what other properties or consequences of tool-ness can we derive?

(06:26) Modularity

(08:34) Loopiness

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.

I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I’d encourage them to spend time trying to name their fears, and see what they think then. And I’d encourage them to talk about these things with other people, or [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a linkpost for Parker Dimensional Analysis. Probably a little elementary for LessWrong, but I think it may still contain a few novel insights, particularly in the last section about Verison's error.

A couple years ago, there was an interesting clip on MSNBC.

A few weeks later, Matt Parker came out with a video analyzing why people tend to make mistakes like this. Now I'm normally a huge fan of Matt Parker. But in this case, I think he kinda dropped the ball.

He does have a very good insight. He realizes that people are treating the "million" as a unit, removing it from the numbers before performing the calculation, then putting it back on. This is indeed the proximate cause of the error. But Matt goes on to claim that the mistake is the treating of "million" as a unit; the implication being that, as a number [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives.

In this [...]

---

Outline:

(01:48) Formal verification for neural networks

(04:24) Heuristic explanations

(07:00) Surprise accounting

(09:06) Worked example: Boolean circuit

(09:39) Brute-force

(10:12) Final OR gate

(11:05) Pattern of NOT gates

(13:03) Pattern of AND and OR gates

(15:11) Discussion

(16:26) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

So this happened: DeepMind (with 48 authors, including a new member of the British nobility) decided to compete with me. Or rather, with some of my work from 10+ years ago.

Apparently, AlphaFold 3 can now predict how a given drug-like molecule will bind to its target protein. And it does so better than AutoDock Vina (the most cited molecular docking program, which I built at Scripps Research).

On top of this, it doesn’t even need a 3D structure of the target. It predicts it too!

But I’m a bit skeptical. I’ll try to explain why...

---

First published:

Source:

Narrated by TYPE III AUDIO.

I originally entered this to the ACX Book Review competition. Since it has not been selected as a finalist I'm now free to post it here. In truth it's a followup to my review of Morris's history of Israel's War of Independence.

In the wake of the October 7th attack on Israel and Israel's response, everyone seemed to agree that one side of the conflict was the epitome of evil, the reincarnation of the Nazis, with warfare in their blood and a pure unfiltered hatred of the enemy in their minds. The other side was a force for good, who just wanted peace and was doing the best they could in a difficult situation. The only problem is no one could agree which side was which.

This is unfair. While the loudest voices may paint the world in black and white, as soon as you ignore them, you begin [...]

---

Outline:

(02:22) Bias

(04:29) Style

(05:49) Synopsis

(05:53) From the Birth of Zionism till the War of Independence (1881 - 1949)

(06:00) Setting the Scene

(07:29) Pre State Zionism

(09:07) Relations with Arabs

(11:10) Partitioning Palestine

(12:26) The War Of Independence

(18:11) From the Armistice to the Six Day War (1949-1967)

(18:18) Peace?

(20:05) Border Infiltrations and Reprisals

(22:04) Suez Crisis

(23:59) The Road to War

(25:56) The 6 Day War

(28:43) From the War of Attrition to Peace with Egypt (1967 - 1982)

(28:50) The War of Attrition

(29:58) The Yom Kippur War

(32:22) Peace, Somehow

(35:23) Israel and Lebanon (1978-2001)

(35:30) The Invasion of Lebanon

(38:03) The Slow Retreat

(39:50) The Occupation (1967-2001)

(39:56) Israeli Governance

(41:50) Settlements

(43:09) The Rebirth of Palestinian Nationalism

(45:07) The First Intifada

(46:52) Oslo

(50:08) The second Intifada

(51:47) Morris's conclusions

(52:08) What Came Next?

(54:14) My Thoughts

(54:55) Israel

(58:22) Palestine

(59:29) Going Forward

(01:02:12) Should I Read This Book?

(01:03:07) Conclusion

The original text contained 28 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

As the title probably already indicates, this post contains community content rather than rationality content. Alternate, sillier version of this post here.

Motivation

I've been a co-organizer of the Bay Area Rationalist Summer Solstice for the past few years, and I've been thinking about how to make it a more meaningful and engaging experience, like what we have with Winter Solstice. The last few Summer Solstices, which I'd describe as mostly being big picnics, have been fun, but fairly low-effort, low-significance, and I think that's a missed opportunity.

Here's a few things that I'd like more of in Summer Solstice, non-exhaustive:

1. A sense of a temporary alternate world created around a shared purpose.
2. Time to connect with people and have deeper conversations.
3. Longer, more immersive collective experiences and thoughtfully designed rituals.
4. Thematic resonance with rationalist goals and community projects.
5. Ability to host the whole community, including children.

I [...]

---

Outline:

(00:18) Motivation

(02:41) Part 1. Smolstice

(04:12) Part 2. Swolestice

(05:38) Gathering interest

---

First published:

Source:

Narrated by TYPE III AUDIO.