The latest in the Feedback Loop Rationality series.

Periodically, people (including me) try to operationalize predictions, or bets, and... it doesn't seem to help much.

I think I recently "got good" at making "actually useful predictions." I currently feel on-the-cusp of unlocking a host of related skills further down the rationality tech tree. This post will attempt to spell out some of the nuances of how I currently go about it, and paint a picture of why I think it's worth investing in.

The takeaway that feels most important to me is: it's way better to be "fluent" at operationalizing predictions, compared to "capable at all."

Previously, "making predictions" was something I did separately from my planning process. It was a slow, clunky process.

Nowadays, reasonably often I can integrate predictions into the planning process itself, because it feels lightweight. I'm better at quickly feeling-around for "what [...]

---

Outline:

(02:00) Predicting Outcomes vs Comparing Plans

(04:14) Frictionless

(08:07) Feeling around for cruxiness

(08:41) Ideal Prerequisite:

(09:52) Frames 1: Costs and benefits

(10:32) Frame 2: Will this really help my deeper, longterm goals?

(11:12) Frame 3: Murphijitsu, and being dissatisfied with maybe

(12:38) The feeling of oh... thats cruxy

(13:46) Example

(13:49) The Fractal Strategy Workshop

(17:49) Tips and Tricks

(17:53) Practice via Videogames

(18:47) NOTES for Prediction-Followup

(19:33) Stuck on making a perfect prediction? Make an imperfect one

(19:40) (Relatedly: its okay to make fuzzy predictions that only make sense to you, but when doing so, try to make extremized ones)

(20:32) Make multiple predictions

(21:37) Takeaways

---

First published:

Source:

Narrated by TYPE III AUDIO.

I'd like to put forward another description of a basic issue that's been around for a while. I don't know if there's been significant progress on a solution, and would be happy to pointed to any such progress. I've opted to go for a relatively rough and quick post that doesn't dive too hard into the details, to avoid losing the thought at all. I may be up for exploring details further in comments or follow-ups.

The Question: How do you respect the wishes (or preferences) of a subject over whom you have a lot of control?

The core problem: any indicator/requirement/metric about respecting their wishes is one you can manipulate (even inadvertently).

For example, think about trying to respect the preferences of the child you're babysitting when you simply know from experience what they will [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a linkpost for https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin, posted in full here given its relevance to this community. Gerard has been one of the longest-standing malicious critics of the rationalist and EA communities and has done remarkable amounts of work to shape their public images behind the scenes.

Note: I am closer to this story than to many of my others. As always, I write aiming to provide a thorough and honest picture, but this should be read as the view of a close onlooker who has known about much within this story for years and has strong opinions about the matter, not a disinterested observer coming across something foreign and new. If you’re curious about the backstory, I encourage you to read my companion article after this one.

Introduction: Reliable Sources

Wikipedia administrator David Gerard cares a great deal about Reliable Sources. For the past half-decade, he has torn [...]

---

Outline:

(00:55) Introduction: Reliable Sources

(06:01) Gerard's Standards for Reliable Sources

(13:53) Who Is David Gerard?

(16:53) The Early Romantic Years

(28:00) Gerard's fling with LessWrong in the twilight of the old internet

(37:51) The bitter end

(45:26) The Vindictive Ex

(50:01) LessWrong

(01:04:17) Effective Altruism

(01:07:55) Scott Alexander

(01:16:22) Conclusion

(01:21:58) Companion article: A Young Mormon Discovers Online Rationality

The original text contained 24 footnotes which were omitted from this narration.

The original text contained 12 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's a new page collecting integrity incidents at the frontier AI labs.

Also a month ago I made a page on labs' policy advocacy.

If you have suggestions to improve these pages, or have ideas for other resources I should create, let me know.

Crossposted from AI Lab Watch. Subscribe on Substack.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Robin and I just had an interesting 2-hour AI doom debate. We picked up where the Hanson-Yudkowsky Foom Debate left off in 2008, revisiting key arguments in the light of recent AI advances.

My position is similar to Eliezer's: P(doom) on the order of 50%.

Robin's position remains shockingly different: P(doom) < 1%.

I think we managed to illuminate some of our cruxes of disagreement, though by no means all. Let us know your thoughts and feedback!

Topics

• AI timelines
• The "outside view" of economic growth trends
• Future economic doubling times
• The role of culture in human intelligence
• Lessons from human evolution and brain size
• Intelligence increase gradient near human level
• Bostrom's Vulnerable World hypothesis
• The optimization-power view
• Feasibility of AI alignment
• Will AI be "above the law" relative to humans

Where To Watch/Listen/Read

• YouTube video
• Podcast audio
• Transcript

About Doom Debates

My [...]

---

Outline:

(00:48) Topics

(01:19) Where To Watch/Listen/Read

(01:28) About Doom Debates

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Thanks to the many people I've chatted with this about over the past many months. And special thanks to Cunningham et al., Marks et al., Joseph Bloom, Trenton Bricken, Adrià Garriga-Alonso, and Johnny Lin, for crucial research artefacts and/or feedback.

Codebase: sparse_circuit_discovery

TL; DR: The residual stream in GPT-2-small, expanded with sparse autoencoders and systematically ablated, looks like the working memory of a forward pass. A few high-magnitude features causally propagate themselves through the model during inference, and these features are interpretable. We can see where in the forward pass, due to which transformer layer, those propagating features are written in and/or scrubbed out.

Introduction

What is GPT-2-small thinking about during an arbitrary forward pass?

I've been trying to isolate legible model circuits using sparse autoencoders. I was inspired by the following example, from the end of Cunningham et al. (2023):

---

Outline:

(00:54) Introduction

(01:46) Related Work

(02:15) Methodology

(03:52) Causal Graphs Key

(05:15) Results

(05:19) Parentheses Example

(06:05) Figure 1

(06:34) Figure 2

(07:12) Figure 3

(08:00) Figure 4

(09:08) Validation of Graph Interpretations

(09:18) Fig. 1 - Parentheticals

(10:19) Fig. 2 - C-words

(11:06) Fig. 3 - C-words

(11:33) Fig. 4 - Acronyms

(12:11) Other Validation

(12:59) Discussion

(13:02) The Residual Stream Is Working Memory...

(13:41) ...And It Is Easily Interpretable.

(14:19) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 3 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This time around, we cover the Hanson/Alexander debates on the value of medicine, and otherwise we mostly have good news.

Technology Advances

Regeneron administers a single shot in a genetically deaf child's ear, and they can hear after a few months, n=2 so far.

Great news: An mRNA vaccine in early human clinical trials reprograms the immune system to attack glioblastoma, the most aggressive and lethal brain tumor. It will now proceed to Phase I. In a saner world, people would be able to try this now.

More great news, we have a cancer vaccine trial in the UK.

And we’re testing personalized mRNA BioNTech canner vaccines too.

US paying Moderna $176 million to develop a pandemic vaccine against bird flu.

We also have this claim that Lorlatinib jumps cancer PFS rates from 8% to 60%.

The GLP-1 Revolution

Early [...]

---

Outline:

(00:12) Technology Advances

(01:04) The GLP-1 Revolution

(04:46) Claims About Hansoninan Medicine

(18:36) Pricing

(19:17) Epistemics

(19:39) DEA Worse Than FDA

(21:19) Study Harder

(23:05) FDA Delenda Est

(24:51) Bioethics

(27:28) Covid

(29:21) Demons

(32:48) Genetics

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

by Anne Marthe van der Bles, Sander van der Linden, Alexandra L. J. Freeman, and David J. Spiegelhalter. (2020)

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized communities, there are norms which encourage people to state numerical uncertainty when reporting a number. This is not often done when speaking to the public. The public might not understand what the uncertainty [...]

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Introduction

I happen to be in that happy stage in the research cycle where I ask for money so I can continue to work on things I think are important. Part of that means justifying what I want to work on to the satisfaction of the people who provide that money.

This presents a good opportunity to say what I plan to work on in a more layman-friendly way, for the benefit of LessWrong, potential collaborators, interested researchers, and funders who want to read the fun version of my project proposal

It also provides the opportunity for people who are very pessimistic about the chances I end up doing anything useful by pursuing this to have their say. So if you read this (or skim it), and have critiques (or just recommendations), I'd love to hear them! Publicly or privately.

So without further ado, in this post I will [...]

---

Outline:

(00:06) Introduction

(02:40) Reinforcement learning

(02:44) Agentic AIs vs Tool AIs

(04:54) Data walls

(07:05) Values

(07:49) Ok, but concretely what will you actually do?

(11:10) Call to action

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Editor's note: Somewhat after I posted this on my own blog, Max Chiswick cornered me at LessOnline / Manifest and gave me a whole new perspective on this topic. I now believe that there is a way to use poker to sharpen epistemics that works dramatically better than anything I had been considering. I hope to write it up—together with Max—when I have time. Anyway, I'm still happy to keep this post around as a record of my first thoughts on the matter, and because it's better than nothing in the time before Max and I get around to writing up our joint second thoughts.

As an epilogue to this story, Max and I are now running a beta test for a course on making AIs to play poker and other games. The course will a synthesis of our respective theories of pedagogy re [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.