Sign up to save your podcasts
Or
Share “Thresholds for funding existential risk interventions” by NunoSempere
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Thresholds for funding existential risk interventions” by NunoSempere
This is some very old work I never published in the EA forum; publishing it now because I want to add it to my Estimating value series, mark the end of an era & reference it elsewhere. I don't think I could produce this work today since I've lost the youthful innocence to think it'd be worth the effort given the idosyncrasies of funding institutions. Thanks to SoGive for commissioning this piece a few years ago; you can see the original here.
1. Review of estimation strategies
The following are strategies we might consider using to determine a reasonable cost-effectiveness threshold for philanthropy to reduce existential risk.
1.1. Summary table
StrategyFeasible?Recommended?1. Randomized trialsImpossibleNo2. Shapley valuesEven more impossibleNo3. Natural experimentsUnlikelyNo4. Direct reduction in x-riskWith effortMaybe5. Relative value comparisonYesMaybe6. Estimate intermediate outputsYesYes7. Sanity checks or thresholdsYesYes8. Other funders as a benchmarkYesNot really9. Expert intuitionYesIn moderation10. Impact rubricYesNo1.2. Discussion of strategies
1.2.1. Ideal yet impracticable
Randomized trials are the gold standard in medicine and global health and development. However, they do have the problem of double-counting or triple-counting impact. Shapley values address that limitation.
However, randomized trials are impossible1 for the case of existential risk2; we can't boot up different possible worlds [...]
---
Outline:
(00:36) 1. Review of estimation strategies
(00:49) 1.1. Summary table
(01:06) 1.2. Discussion of strategies
(01:11) 1.2.1. Ideal yet impracticable
(03:10) 1.2.2. Methods at the sweet spot of feasibility, informativeness, ambitiousness
(06:19) 1.2.3. More common status quo methods
(07:51) 2. Review of past thresholds
(07:56) 2.1. List of past thresholds
(10:23) 2.2. Meaning of these thresholds
(13:24) 3. A few tentative thresholds
(13:42) 3.1. From first principles
(17:46) 3.2. Thresholds based on better specific interventions
(18:35) 3.2.1. Why not the LTFF
(20:28) 3.2.2. Comparison against robust AI safety technical research
(25:07) 3.2.3. Other interesting targets for comparison
(25:37) 3.3. Summary table and links to the models
(25:43) 3.3.1. Summary of meanings
(26:01) 3.3.2. Summary of values
(26:36) 3.4. A note on replicability of estimates
(28:16) Footnotes
The original text contained 11 footnotes which were omitted from this narration.
---
First published:
July 11th, 2026
Source:
https://forum.effectivealtruism.org/posts/aBAkov3iJNWDWu8qS/thresholds-for-funding-existential-risk-interventions
---
Narrated by TYPE III AUDIO.
View all episodes
By EA Forum TeamThis is some very old work I never published in the EA forum; publishing it now because I want to add it to my Estimating value series, mark the end of an era & reference it elsewhere. I don't think I could produce this work today since I've lost the youthful innocence to think it'd be worth the effort given the idosyncrasies of funding institutions. Thanks to SoGive for commissioning this piece a few years ago; you can see the original here.
1. Review of estimation strategies
The following are strategies we might consider using to determine a reasonable cost-effectiveness threshold for philanthropy to reduce existential risk.
1.1. Summary table
StrategyFeasible?Recommended?1. Randomized trialsImpossibleNo2. Shapley valuesEven more impossibleNo3. Natural experimentsUnlikelyNo4. Direct reduction in x-riskWith effortMaybe5. Relative value comparisonYesMaybe6. Estimate intermediate outputsYesYes7. Sanity checks or thresholdsYesYes8. Other funders as a benchmarkYesNot really9. Expert intuitionYesIn moderation10. Impact rubricYesNo1.2. Discussion of strategies
1.2.1. Ideal yet impracticable
Randomized trials are the gold standard in medicine and global health and development. However, they do have the problem of double-counting or triple-counting impact. Shapley values address that limitation.
However, randomized trials are impossible1 for the case of existential risk2; we can't boot up different possible worlds [...]
---
Outline:
(00:36) 1. Review of estimation strategies
(00:49) 1.1. Summary table
(01:06) 1.2. Discussion of strategies
(01:11) 1.2.1. Ideal yet impracticable
(03:10) 1.2.2. Methods at the sweet spot of feasibility, informativeness, ambitiousness
(06:19) 1.2.3. More common status quo methods
(07:51) 2. Review of past thresholds
(07:56) 2.1. List of past thresholds
(10:23) 2.2. Meaning of these thresholds
(13:24) 3. A few tentative thresholds
(13:42) 3.1. From first principles
(17:46) 3.2. Thresholds based on better specific interventions
(18:35) 3.2.1. Why not the LTFF
(20:28) 3.2.2. Comparison against robust AI safety technical research
(25:07) 3.2.3. Other interesting targets for comparison
(25:37) 3.3. Summary table and links to the models
(25:43) 3.3.1. Summary of meanings
(26:01) 3.3.2. Summary of values
(26:36) 3.4. A note on replicability of estimates
(28:16) Footnotes
The original text contained 11 footnotes which were omitted from this narration.
---
First published:
July 11th, 2026
Source:
https://forum.effectivealtruism.org/posts/aBAkov3iJNWDWu8qS/thresholds-for-funding-existential-risk-interventions
---
Narrated by TYPE III AUDIO.