This is a linkpost for https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin, posted in full here given its relevance to this community. Gerard has been one of the longest-standing malicious critics of the rationalist and EA communities and has done remarkable amounts of work to shape their public images behind the scenes.

Note: I am closer to this story than to many of my others. As always, I write aiming to provide a thorough and honest picture, but this should be read as the view of a close onlooker who has known about much within this story for years and has strong opinions about the matter, not a disinterested observer coming across something foreign and new. If you’re curious about the backstory, I encourage you to read my companion article after this one.

Introduction: Reliable Sources

Wikipedia administrator David Gerard cares a great deal about Reliable Sources. For the past half-decade, he has torn [...]

---

Outline:

(00:55) Introduction: Reliable Sources

(06:01) Gerard's Standards for Reliable Sources

(13:53) Who Is David Gerard?

(16:53) The Early Romantic Years

(28:00) Gerard's fling with LessWrong in the twilight of the old internet

(37:51) The bitter end

(45:26) The Vindictive Ex

(50:01) LessWrong

(01:04:17) Effective Altruism

(01:07:55) Scott Alexander

(01:16:22) Conclusion

(01:21:58) Companion article: A Young Mormon Discovers Online Rationality

The original text contained 24 footnotes which were omitted from this narration.

The original text contained 12 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's a new page collecting integrity incidents at the frontier AI labs.

Also a month ago I made a page on labs' policy advocacy.

If you have suggestions to improve these pages, or have ideas for other resources I should create, let me know.

Crossposted from AI Lab Watch. Subscribe on Substack.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Robin and I just had an interesting 2-hour AI doom debate. We picked up where the Hanson-Yudkowsky Foom Debate left off in 2008, revisiting key arguments in the light of recent AI advances.

My position is similar to Eliezer's: P(doom) on the order of 50%.

Robin's position remains shockingly different: P(doom) < 1%.

I think we managed to illuminate some of our cruxes of disagreement, though by no means all. Let us know your thoughts and feedback!

Topics

• AI timelines
• The "outside view" of economic growth trends
• Future economic doubling times
• The role of culture in human intelligence
• Lessons from human evolution and brain size
• Intelligence increase gradient near human level
• Bostrom's Vulnerable World hypothesis
• The optimization-power view
• Feasibility of AI alignment
• Will AI be "above the law" relative to humans

Where To Watch/Listen/Read

• YouTube video
• Podcast audio
• Transcript

About Doom Debates

My [...]

---

Outline:

(00:48) Topics

(01:19) Where To Watch/Listen/Read

(01:28) About Doom Debates

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Thanks to the many people I've chatted with this about over the past many months. And special thanks to Cunningham et al., Marks et al., Joseph Bloom, Trenton Bricken, Adrià Garriga-Alonso, and Johnny Lin, for crucial research artefacts and/or feedback.

Codebase: sparse_circuit_discovery

TL; DR: The residual stream in GPT-2-small, expanded with sparse autoencoders and systematically ablated, looks like the working memory of a forward pass. A few high-magnitude features causally propagate themselves through the model during inference, and these features are interpretable. We can see where in the forward pass, due to which transformer layer, those propagating features are written in and/or scrubbed out.

Introduction

What is GPT-2-small thinking about during an arbitrary forward pass?

I've been trying to isolate legible model circuits using sparse autoencoders. I was inspired by the following example, from the end of Cunningham et al. (2023):

---

Outline:

(00:54) Introduction

(01:46) Related Work

(02:15) Methodology

(03:52) Causal Graphs Key

(05:15) Results

(05:19) Parentheses Example

(06:05) Figure 1

(06:34) Figure 2

(07:12) Figure 3

(08:00) Figure 4

(09:08) Validation of Graph Interpretations

(09:18) Fig. 1 - Parentheticals

(10:19) Fig. 2 - C-words

(11:06) Fig. 3 - C-words

(11:33) Fig. 4 - Acronyms

(12:11) Other Validation

(12:59) Discussion

(13:02) The Residual Stream Is Working Memory...

(13:41) ...And It Is Easily Interpretable.

(14:19) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 3 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This time around, we cover the Hanson/Alexander debates on the value of medicine, and otherwise we mostly have good news.

Technology Advances

Regeneron administers a single shot in a genetically deaf child's ear, and they can hear after a few months, n=2 so far.

Great news: An mRNA vaccine in early human clinical trials reprograms the immune system to attack glioblastoma, the most aggressive and lethal brain tumor. It will now proceed to Phase I. In a saner world, people would be able to try this now.

More great news, we have a cancer vaccine trial in the UK.

And we’re testing personalized mRNA BioNTech canner vaccines too.

US paying Moderna $176 million to develop a pandemic vaccine against bird flu.

We also have this claim that Lorlatinib jumps cancer PFS rates from 8% to 60%.

The GLP-1 Revolution

Early [...]

---

Outline:

(00:12) Technology Advances

(01:04) The GLP-1 Revolution

(04:46) Claims About Hansoninan Medicine

(18:36) Pricing

(19:17) Epistemics

(19:39) DEA Worse Than FDA

(21:19) Study Harder

(23:05) FDA Delenda Est

(24:51) Bioethics

(27:28) Covid

(29:21) Demons

(32:48) Genetics

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

by Anne Marthe van der Bles, Sander van der Linden, Alexandra L. J. Freeman, and David J. Spiegelhalter. (2020)

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized communities, there are norms which encourage people to state numerical uncertainty when reporting a number. This is not often done when speaking to the public. The public might not understand what the uncertainty [...]

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Introduction

I happen to be in that happy stage in the research cycle where I ask for money so I can continue to work on things I think are important. Part of that means justifying what I want to work on to the satisfaction of the people who provide that money.

This presents a good opportunity to say what I plan to work on in a more layman-friendly way, for the benefit of LessWrong, potential collaborators, interested researchers, and funders who want to read the fun version of my project proposal

It also provides the opportunity for people who are very pessimistic about the chances I end up doing anything useful by pursuing this to have their say. So if you read this (or skim it), and have critiques (or just recommendations), I'd love to hear them! Publicly or privately.

So without further ado, in this post I will [...]

---

Outline:

(00:06) Introduction

(02:40) Reinforcement learning

(02:44) Agentic AIs vs Tool AIs

(04:54) Data walls

(07:05) Values

(07:49) Ok, but concretely what will you actually do?

(11:10) Call to action

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Editor's note: Somewhat after I posted this on my own blog, Max Chiswick cornered me at LessOnline / Manifest and gave me a whole new perspective on this topic. I now believe that there is a way to use poker to sharpen epistemics that works dramatically better than anything I had been considering. I hope to write it up—together with Max—when I have time. Anyway, I'm still happy to keep this post around as a record of my first thoughts on the matter, and because it's better than nothing in the time before Max and I get around to writing up our joint second thoughts.

As an epilogue to this story, Max and I are now running a beta test for a course on making AIs to play poker and other games. The course will a synthesis of our respective theories of pedagogy re [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: We build a comprehensive benchmark to measure situational awareness in LLMs. It consists of 16 tasks, which we group into 7 categories and 3 aspects of situational awareness (self-knowledge, situational inferences, and taking actions).

We test 19 LLMs and find that all perform above chance, including the pretrained GPT-4-base (which was not subject to RLHF finetuning). However, the benchmark is still far from saturated, with the top-scoring model (Claude-3.5-Sonnet) scoring 54%, compared to a random chance of 27.4% and an estimated upper baseline of 90.7%.

This post has excerpts from our paper, as well as some results on new models that are not in the paper.

Links: Twitter thread, Website (latest results + code), Paper

---

Outline:

(01:27) Abstract

(03:28) Introduction

(05:17) Motivation

(06:38) Benchmark

(08:45) Examples and prompts

(10:44) Results

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This summer, I’m supervising some research fellows through Cambridge's ERA AI Fellowship. The program started last week, and I’ve had conversations with about 6 fellows about their research projects & summer goals.

In this post, I’ll highlight a few pieces of advice I’ve found myself regularly giving to research fellows. This post reflects my own opinions and does not necessarily reflect the views of others at ERA.

Prioritize projects that have a clear target audience

Problem: One of the most common reasons why research products fail to add value is that they do not have a target audience. I think it can be easy to find a topic that is interesting/important, spend several months working on it, produce a 20-50 page paper, and then realize that you have no particular stakeholder(s) who find the work action-relevant.

Advice: Try to brainstorm what specific individuals you would want to [...]

---

Outline:

(00:32) Prioritize projects that have a clear target audience

(02:38) Be specific about your end products

(05:26) Take advantage of your network (and others’ networks)

(08:00) Miscellaneous

---

First published:

Source:

Narrated by TYPE III AUDIO.