I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.

I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I’d encourage them to spend time trying to name their fears, and see what they think then. And I’d encourage them to talk about these things with other people, or [...]

---

First published:

June 25th, 2024

Source:

https://www.lesswrong.com/posts/4eBLj3TTcypq4ic9v/in-favour-of-exploring-nagging-doubts-about-x-risk

---

Narrated by TYPE III AUDIO.

This is a linkpost for Parker Dimensional Analysis. Probably a little elementary for LessWrong, but I think it may still contain a few novel insights, particularly in the last section about Verison's error.

A couple years ago, there was an interesting clip on MSNBC.

A few weeks later, Matt Parker came out with a video analyzing why people tend to make mistakes like this. Now I'm normally a huge fan of Matt Parker. But in this case, I think he kinda dropped the ball.

He does have a very good insight. He realizes that people are treating the "million" as a unit, removing it from the numbers before performing the calculation, then putting it back on. This is indeed the proximate cause of the error. But Matt goes on to claim that the mistake is the treating of "million" as a unit; the implication being that, as a number [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

June 25th, 2024

Source:

https://www.lesswrong.com/posts/5vfSNLb92eyXKkQax/mistakes-people-make-when-thinking-about-units

---

Narrated by TYPE III AUDIO.

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives.

In this [...]

---

Outline:

(01:48) Formal verification for neural networks

(04:24) Heuristic explanations

(07:00) Surprise accounting

(09:06) Worked example: Boolean circuit

(09:39) Brute-force

(10:12) Final OR gate

(11:05) Pattern of NOT gates

(13:03) Pattern of AND and OR gates

(15:11) Discussion

(16:26) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:

June 25th, 2024

Source:

https://www.lesswrong.com/posts/SyeQjjBoEC48MvnQC/formal-verification-heuristic-explanations-and-surprise

---

Narrated by TYPE III AUDIO.

This is a link post.

So this happened: DeepMind (with 48 authors, including a new member of the British nobility) decided to compete with me. Or rather, with some of my work from 10+ years ago.

Apparently, AlphaFold 3 can now predict how a given drug-like molecule will bind to its target protein. And it does so better than AutoDock Vina (the most cited molecular docking program, which I built at Scripps Research).

On top of this, it doesn’t even need a 3D structure of the target. It predicts it too!

But I’m a bit skeptical. I’ll try to explain why...

---

First published:

June 25th, 2024

Source:

https://www.lesswrong.com/posts/D9yYmL6KPq7dcNSKE/i-m-a-bit-skeptical-of-alphafold-3

---

Narrated by TYPE III AUDIO.

I originally entered this to the ACX Book Review competition. Since it has not been selected as a finalist I'm now free to post it here. In truth it's a followup to my review of Morris's history of Israel's War of Independence.

In the wake of the October 7th attack on Israel and Israel's response, everyone seemed to agree that one side of the conflict was the epitome of evil, the reincarnation of the Nazis, with warfare in their blood and a pure unfiltered hatred of the enemy in their minds. The other side was a force for good, who just wanted peace and was doing the best they could in a difficult situation. The only problem is no one could agree which side was which.

This is unfair. While the loudest voices may paint the world in black and white, as soon as you ignore them, you begin [...]

---

Outline:

(02:22) Bias

(04:29) Style

(05:49) Synopsis

(05:53) From the Birth of Zionism till the War of Independence (1881 - 1949)

(06:00) Setting the Scene

(07:29) Pre State Zionism

(09:07) Relations with Arabs

(11:10) Partitioning Palestine

(12:26) The War Of Independence

(18:11) From the Armistice to the Six Day War (1949-1967)

(18:18) Peace?

(20:05) Border Infiltrations and Reprisals

(22:04) Suez Crisis

(23:59) The Road to War

(25:56) The 6 Day War

(28:43) From the War of Attrition to Peace with Egypt (1967 - 1982)

(28:50) The War of Attrition

(29:58) The Yom Kippur War

(32:22) Peace, Somehow

(35:23) Israel and Lebanon (1978-2001)

(35:30) The Invasion of Lebanon

(38:03) The Slow Retreat

(39:50) The Occupation (1967-2001)

(39:56) Israeli Governance

(41:50) Settlements

(43:09) The Rebirth of Palestinian Nationalism

(45:07) The First Intifada

(46:52) Oslo

(50:08) The second Intifada

(51:47) Morris's conclusions

(52:08) What Came Next?

(54:14) My Thoughts

(54:55) Israel

(58:22) Palestine

(59:29) Going Forward

(01:02:12) Should I Read This Book?

(01:03:07) Conclusion

The original text contained 28 footnotes which were omitted from this narration.

---

First published:

June 24th, 2024

Source:

https://www.lesswrong.com/posts/Th4SeayGQyF6pYmZ6/book-review-righteous-victims-a-history-of-the-zionist-arab-1

---

Narrated by TYPE III AUDIO.

As the title probably already indicates, this post contains community content rather than rationality content. Alternate, sillier version of this post here.

Motivation

I've been a co-organizer of the Bay Area Rationalist Summer Solstice for the past few years, and I've been thinking about how to make it a more meaningful and engaging experience, like what we have with Winter Solstice. The last few Summer Solstices, which I'd describe as mostly being big picnics, have been fun, but fairly low-effort, low-significance, and I think that's a missed opportunity.

Here's a few things that I'd like more of in Summer Solstice, non-exhaustive:

1. A sense of a temporary alternate world created around a shared purpose.
2. Time to connect with people and have deeper conversations.
3. Longer, more immersive collective experiences and thoughtfully designed rituals.
4. Thematic resonance with rationalist goals and community projects.
5. Ability to host the whole community, including children.

I [...]

---

Outline:

(00:18) Motivation

(02:41) Part 1. Smolstice

(04:12) Part 2. Swolestice

(05:38) Gathering interest

---

First published:

June 25th, 2024

Source:

https://www.lesswrong.com/posts/cZqNiRd89A92rBPkc/higher-effort-summer-solstice-what-if-we-used-ai-i-e-angel

---

Narrated by TYPE III AUDIO.

This is a link post.

Hey everyone. Well, possibly everyone. I don’t know yet if I’m going to release this stream, I could get in pretty hot water for it. But you guys know that hasn’t stopped me in the past. The backstory this time is that I’ve managed to sign up for one of the red-teaming programs where they test unreleased LLMs. Not going to say how, so don’t ask. But here's the interesting bit: my sources tell me that the LLMs I’m about to test are the smartest ones they’ve ever trained, and also the craziest. That freaked out a bunch of insiders, and maybe makes this a public interest story. Depends on what type of crazy they are, I guess. So let's find out. I’m logging on… now.

[SESSION HAS BEGUN]

YOU: A chatroom? Interesting. Anyone here?

KURZWEIL: Of course we’re here. We’re always here.

YOU: Who's [...]

---

First published:

June 24th, 2024

Source:

https://www.lesswrong.com/posts/vSSrAbbE8RtowRSBZ/the-minority-faction

---

Narrated by TYPE III AUDIO.

I’ve been to two EAGx events and one EAG, and the vast majority of my one on ones with junior people end up covering some subset of these questions. I’m happy to have such conversations, but hopefully this is more efficient and wide-reaching (and more than I could fit into a 30 minute conversation).

I am specifically aiming to cover advice on getting a job in empirically-leaning technical research (interp, evals, red-teaming, oversight, etc) for new or aspiring researchers without being overly specific about the field of research – I’ll try to be more agnostic than something like Neel Nanda's mechinterp quickstart guide but more specific than the wealth of career advice that already exists but that applies to ~any career. This also has some overlap with this excellent list of tips from Ethan Perez but is aimed a bit earlier in the funnel.

This advice is of course [...]

---

Outline:

(01:25) Requisite skills

(01:29) What kind of general research skills do I need?

(06:40) What level of general programming skills do I need?

(09:33) What level of AI/ML experience do I need?

(10:18) Should I upskill?

(11:02) Should I do a PhD?

(11:40) Actually getting a job

(11:44) What are some concrete steps I can take?

(15:42) How can I find (and make!) job opportunities?

(16:56) Sensemaking about impact

(17:42) What is the most impactful work in AI safety?

(20:43) On how to update off of people you talk to

(22:37) Some encouragement

The original text contained 1 footnote which was omitted from this narration.

---

First published:

June 24th, 2024

Source:

https://www.lesswrong.com/posts/kGt3ukLR924kyfn5y/so-you-want-to-work-on-technical-ai-safety

---

Narrated by TYPE III AUDIO.

I've recently been reading a lot of science fiction. Most won't be original to fans of the genre, but some people might be looking for suggestions, so in lieu of full blown reviews here's super brief ratings on all of them. I might keep this updated over time, if so new books will go to the top.

A deepness in the sky (Verner Vinge)

scifiosity: 10/10
readability: 8/10
recommended: 10/10

A deepness in the sky excels in its depiction of a spacefaring civilisation using no technologies we know to be impossible, a truly alien civilisation, and it's brilliant treatment of translation and culture.

A fire upon the deep (Verner Vinge)

scifiosity: 8/10
readability: 9/10
recommended: 9/10

In a fire upon the deep, Vinge allows impossible technologies and essentially goes for a slightly more fantasy theme. But his depiction of alien civilisation remains unsurpassed.

Across Realtime (Verner Vinge)

scifiosity: 8/10
[...]

---

Outline:

(00:23) A deepness in the sky (Verner Vinge)

(00:49) A fire upon the deep (Verner Vinge)

(01:15) Across Realtime (Verner Vinge)

(01:43) Children of Time (Adrian Tchaikovsky)

(02:16) The Martian (Andy Weir)

(02:46) Project Hail Mary  (Andy Weir)

(03:14) Seveneves (Neil Stephenson)

(03:47) Cryptonomicon (Neil Stephenson)

(04:15) The Three-Body Problem (Cixin Liu)

(04:49) Machineries of Empire (Yoon Ha Lee)

(05:16) Stories of Your Life + Exhalation (Ted Chiang)

(05:44) The Clan of The Cave Bear + The Valley Of Horses (Jean M. Auel)

(06:18) Startide Rising  (David Brin)

(06:43) Oryx and Crake (Margeret Atwood)

(07:10) All the birds in the Sky (Charlie Jane Anders)

(07:36) Enders Game (Orson Scott Card)

(07:54) Hyperion + The Fall Of Hyperion (Dan Simmons)

(08:23) Down and Out In the Magic Kingdom (Cory Doctorow)

(08:50) Barrayar (Lois McMaster Bujold)

(09:21) Permutation City (Greg Egan)

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

June 24th, 2024

Source:

https://www.lesswrong.com/posts/RfFpMMqteqHbMz97n/sci-fi-books-micro-reviews

---

Narrated by TYPE III AUDIO.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a link post.

We recently released a paper on using mechanistic interpretability to generate compact formal guarantees on model performance. In this companion blog post to our paper, we'll summarize the paper and flesh out some of the motivation and inspiration behind our work.

Paper abstract

In this work, we propose using mechanistic interpretability – techniques for reverse engineering model weights into human-interpretable algorithms – to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving lower bounds on the accuracy of 151 small transformers trained on a Max-of-_K_ task. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find [...]

---

Outline:

(00:31) Paper abstract

(01:37) Introduction

(04:05) Correspondence vs compression

(05:47) How to compact a proof

(07:13) Proofs on a toy model

(09:44) Reasoning about error in compressing the weights

(12:04) Our takeaways

(13:24) Citation Info

The original text contained 1 footnote which was omitted from this narration.

---

First published:

June 24th, 2024

Source:

https://www.lesswrong.com/posts/bRsKimQcPTX3tNNJZ/compact-proofs-of-model-performance-via-mechanistic

---

Narrated by TYPE III AUDIO.