I recently started working at Renaissance Philanthropy. It's a new organization, and most people I’ve met haven’t heard of it.1 So I thought I’d explain, in my own words and speaking for myself rather than my employers, what we (and I) are trying to do here.

Modern Medicis

The “Renaissance” in Renaissance Philanthropy is a reference to the Italian Renaissance, when wealthy patrons like the Medicis commissioned great artworks and inventions.

The idea is that “modern Medicis” — philanthropists — should be funding the great scientists and innovators of our day to tackle ambitious challenges.

RenPhil's role is to facilitate that process: when a philanthropist wants to pursue a goal, we help them turn that into a more concrete plan, incubate and/or administer new organizations to implement that plan, and recruit the best people in the world to work on that goal and make [...]

---

Outline:

(00:26) Modern Medicis

(06:15) Science Philanthropy Is Needed

(09:11) Startups Aren’t Everything

(12:53) Story Time: How Modern Medicis Fail

(16:55) The RenPhil Way

(21:32) Got A Dream? I’m Listening.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Each December, the LessWrong community reflects on the best blogposts of yesteryear, to decide which posts stood the tests of time.

In this post, I aim to:

• Explain some subtleties of what I'm hoping we get out of the Annual Review
• Make some explicit asks (even of busy people), which I think have reasonable cost/benefit ratios.
• Describe the new UI features. 

The most important changes are:

• Reviews are much more prominent than they used to be
• You can nominate posts from elsewhere on the internet using the Linkpost Importer. (Use this to nominate posts that contribute to the overall LessWrong or Alignment Forum conversation). It can import PDFs from arXiV, and blogposts from most urls.

For the Nomination Voting Phase, the main ask I'm making is: "spend ~30 minutes casting nomination votes, and write 2 short reviews explaining how you got value from the most important [...]

---

Outline:

(03:00) The Ask

(04:33) The concrete, minimal Civic Duty actions

(05:10) Whats new?

(06:48) Reviews are more prominent

(08:26) Linkpost Submission UI

(10:29) Nominations Page

(11:02) Improved Best of LessWrong browsing

(11:55) Sliding Scale of Voting Power

(12:37) Nuts and Bolts: How does the review work?

(12:42) Phase 1: Preliminary Voting

(15:39) Phase 2: Discussion

(16:50) Phase 3: Final Voting

The original text contained 4 footnotes which were omitted from this narration.

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There is little sign that the momentum of the situation is changing. Instead, things continue to slowly get worse, as nations in holes continue to keep digging. The longer we wait, the more expensive the ultimate price will be. We will soon find out what the new administration does, which could go any number of ways.

Table of Contents

---

Outline:

(00:29) Not Enough Dakka

(12:02) Embryo Selection

(15:44) Costs

(16:51) Proving that Dakka Works

(18:41) IVF

(22:18) Genetics

(22:43) Cultural Trends

(32:41) Denial

(33:49) Urbanization

(34:25) The Marriage Penalty

(35:24) The Biological Clock

(38:15) Technology Advances

(39:40) Big Families

(40:41) Au Pairs

(42:18) Childcare Regulations

(46:51) The Numbers

(47:18) The Housing Theory of Everything

(59:15) Causes

(01:07:39) The Iron Law of Wages

(01:10:37) South Korea

(01:15:36) Georgia (the Country)

(01:17:20) Japan

(01:18:38) China

(01:21:51) Italy

(01:22:04) Northwestern Spain

(01:23:59) Russia

(01:24:15) Taiwan

(01:26:34) The United Kingdom

(01:26:51) Ancient Greece

(01:27:24) Israel

(01:28:20) More Dakka

(01:33:21) Perception

(01:37:10) Your Own Quest

(01:42:21) Help Wanted

The original text contained 38 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The longtermist funding ecosystem needs certain functions to exist at a reasonable scale. I argue LTFF should continue to be funded because we're currently one of the only organizations comprehensively serving these functions. Specifically, we:

• Fund early-stage technical AI safety researchers working outside major labs
• Help talented people transition into existential risk work
• Provide independent funding voices to balance out AI company influence
• Move quickly to fund promising work in emerging areas, including "weirder" but potentially high-impact projects

Getting these functions right takes meaningful resources - well over $1M annually. This figure isn't arbitrary: $1M funds roughly 10 person-years of work, split between supporting career transitions and independent research. Given what we're trying to achieve - from maintaining independent AI safety voices to seeding new fields like x-risk focused information security - this is arguably a minimum viable scale.

While I'm excited to see some of these functions [...]

---

Outline:

(01:55) Core Argument

(04:11) Key Functions Currently (Almost) Unique to LTFF

(04:17) Technical AI Safety Funding

(04:40) Why arent other funders investing in GCR-focused technical AI Safety?

(06:27) Career Transitions and Early Researcher Funding

(07:44) Why arent other groups investing in improving career transitions in existential risk reduction?

(09:05) Going Forwards

(10:02) Providing (Some) Counterbalance to AI Companies on AI Safety

(11:39) Going Forwards

(12:21) Funding New Project Areas and Approaches

(13:18) Going Forwards

(14:01) Broader Funding Case

(14:05) Why Current Funding Levels Matter

(16:03) Going Forwards

(17:36) Conclusion

(18:41) Appendix: LTFFs Institutional Features

(18:57) Transparency and Communication

(19:41) Operational Features

---

First published:

Source:

Narrated by TYPE III AUDIO.

Audio note: this article contains 449 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Based off research performed in the MATS 5.1 extension program, under the mentorship of Alex Turner (TurnTrout). Research supported by a grant from the Long-Term Future Fund.

TLDR: I introduce a new framework for mechanistically eliciting latent behaviors in LLMs. In particular, I propose deep causal transcoding - modelling the effect of causally intervening on the residual stream of a deep (i.e. _gtrsim10_-layer) slice of a transformer, using a shallow MLP. I find that the weights of these MLPs are highly interpretable -- input directions serve as diverse and coherently generalizable steering vectors, while output directions induce predictable changes in model behavior via directional ablation.

Summary I consider deep causal transcoders (DCTs) with various activation functions [...]

---

Outline:

(05:40) Introduction

(07:16) Related work

(09:59) Theory

(18:28) Method

(22:04) Fitting a Linear MLP

(25:08) Fitting a Quadratic MLP

(30:12) Alternative formulation of tensor decomposition objective: causal importance minus similarity penalty

(34:03) Fitting an Exponential MLP

(36:35) On the role of _R_

(37:37) Relation to original MELBO objective

(38:54) Calibrating _R_

(41:44) Case Study: Learning Jailbreak Vectors

(41:49) Generalization of linear, quadratic and exponential DCTs

(51:49) Evidence for multiple harmless directions

(56:00) Many loosely-correlated DCT features elicit jailbreaks

(59:44) Averaging doesnt improve generalization when we add features to the residual stream

(01:01:05) Averaging does improve jailbreak scores when we ablate features

(01:03:15) Ablating (averaged) target-layer features also works

(01:04:49) Deeper models: constant depth horizon (_t-s_) suffices for learning jailbreaks

(01:09:32) Application: Jailbreaking Representation-Rerouted Mistral-7B

(01:17:15) Application: Eliciting Capabilities in Password-Locked Models

(01:18:42) Future Work

(01:19:01) Studying feature multiplicity

(01:20:03) Quantifying a broader range of behaviors

(01:20:24) Effect of pre-training hyper-parameters

(01:22:05) Acknowledgements

(01:22:16) Appendix

(01:22:19) Hessian auto diff details

The original text contained 26 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Preface

Several friends have asked me about what psychological effects I think could affect human judgement about x-risk.

This isn't a complete answer, but in 2018 I wrote a draft of "AI Research Considerations for Human Existential Safety" (ARCHES) that included an overview of cognitive biases I thought (and still think) will impair AI risk assessments. Many cognitive bias experiments had already failed to reproduce well in the psychology reproducibility crisis, so I thought it would be a good idea to point out some that did reproduce well, and that were obviously relevant to AI risk. Unfortunately, one prospective coauthor asked that this content be removed, because of the concern that academic AI researchers would be too unfamiliar with the relevant field: cognitive science. That coauthor ultimately decided not to appear on ARCHES at all, and I retrospect I probably should have added back this material.

Just to [...]

---

Outline:

(00:08) Preface

(01:42) Notes on definitions

(02:25) Risk Type 1b: Unrecognized prepotence

(06:38) Risk Type 1b: Unrecognized misalignment

---

First published:

Source:

Narrated by TYPE III AUDIO.

In the spirit of the season, you can book a call with me to help w/ your interp project (no large coding though)

Would you like someone to:

1. Review your paper or code?
2. Brainstorm ideas on next steps?
3. How to best communicate your results?
4. Discuss conceptual problems
5. Obvious Advice (e.g. being affected by SAD because it's winter, not exercising, not getting enough sleep)
6. [anything else that would be useful!]

When we're chatting, you can interrupt me to better focus on what you specifically want. We can set [5] minute timers to try to solve whatever problem you're having. You can steer the conversation or I can!

Who is this for? 

Anyone working on interp research. My niche is SAE's, so that's likely where I'm most helpful.

This is for both junior and senior researchers. If you're unsure, feel free to dm [...]

---

Outline:

(00:52) Who is this for?

(01:12) Im Useful, I Swear

(01:41) Why are You Doing this?

(02:03) How to Book

---

First published:

Source:

Narrated by TYPE III AUDIO.

Tom did the original thinking; Rose helped with later thinking, structure and writing.

Some plans for AI governance involve centralising western AGI development.[1] Would this actually be a good idea? We don’t think this question has been analysed in enough detail, given how important it is. In this post, we’re going to:

1. Explore the strategic implications of having one project instead of several
2. Discuss what we think the best path forwards is, given that strategic landscape

(If at this point you’re thinking ‘this is all irrelevant, because centralisation is inevitable’, we disagree! We suggest you read the appendix, and then consider if you want to read the rest of the post.)

On 2, we’re going to present:

• Our overall take: It's very unclear whether centralising would be good or bad.
• Our current best guess: Centralisation is probably net bad, because of risks from power concentration (but this is very uncertain).

---

Outline:

(01:39) What are the strategic implications of having one instead of several projects?

(02:36) Summary table

(02:53) Race dynamics

(03:23) Racing between western projects

(04:51) Racing between the US and China

(07:40) Why do race dynamics matter?

(08:50) Power concentration

(13:11) Why does power concentration matter?

(14:40) Infosecurity

(17:41) Why does infosecurity matter?

(19:13) What is the best path forwards, given that strategic landscape?

(20:05) Our overall take

(22:07) Our current best guess

(26:53) Overall conclusion

(29:22) Appendix: Why we don’t think centralisation is inevitable

The original text contained 10 footnotes which were omitted from this narration.

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

MIRI is a nonprofit research organization with a mission of addressing the most serious hazards posed by smarter-than-human artificial intelligence. In our general strategy update and communications strategy update earlier this year, we announced a new strategy that we’re executing on at MIRI, and several new teams we’re spinning up as a result. This post serves as a status update on where things stand at MIRI today.

We originally planned to run a formal fundraiser this year, our first one since 2019. However, we find ourselves with just over two years of reserves, which is more than we’ve typically had in our past fundraisers.

Instead, we’ve written this brief update post, discussing recent developments at MIRI, projects that are in the works, and our basic funding status. In short: We aren’t in urgent need of more funds, but we do have a fair amount of uncertainty about what the [...]

---

Outline:

(01:22) About MIRI

(02:55) Funding Status

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I work on Open Philanthropy's AI Governance and Policy team, but I’m writing this in my personal capacity – several senior employees at Open Phil have argued with me about this!

This is a brief-ish post addressed to people who are interested in making high-impact donations and are already concerned about potential risks from advanced AI. Ideally such a post would include a case that reducing those risks is an especially important (and sufficiently tractable and neglected) cause area, but I’m skipping that part for time and will just point you to this 80,000 Hours problem profile for now.

• Contrary to a semi-popular belief that donations in global catastrophic risks merely “funge” with major donors, there are several ways for individual donors, including those giving small amounts, to reduce global catastrophic risks from AI. These include donating to:
  • Work that would be less impactful if they were [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.