Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shapes of Mind and Pluralism in Alignment, published by Adam Shimi on August 13, 2022 on The AI Alignment Forum.

This work was done while at Conjecture.

This post has been written for the first Refine blog post day, at the end of a week of readings, discussions, and exercises about epistemology for doing good conceptual research.

I have recently presented my model behind the Refine incubator that I'm running. Yet in the two weeks since this post was published, multiple discussions helped me make legible an aspect of my intuitions that I didn't discuss in this post: the notion of different "shapes of mind".

There are two points to this intuition:

Different people will have different "shapes of mind" — ways of revealing hidden bits of evidence from the world;

And alignment is the kind of hard problem where the bits of evidence are dispersed, such that there's no one-trick that is enough.

I've given my current best model of the different forms of pluralism and when to use them in another recent post. What I want to explore here is the first point: this notion of shape of mind. For that, let's recall the geometric model of bits of evidence I introduced in Levels of Pluralism.

We have a high-dimensional space with objects in it. The space is the problem and the objects are bits of evidence.

Because we suck at high-dimensional geometry, we use frames/perspectives that reduce the dimensionality and highlight some aspects of the space. These are operationalizations.

There are clusters of bits of evidence in the space (whether they are rich or poor). These clusters are veins of evidence.

Here the shapes of mind are favored operationalizations — that is, the favored low-dimensional compression of the high-dimensional space where the bits of evidence lie. More precisely, a shape of mind is a cluster of "close" such transforms.

What makes someone have a given shape of mind?

(Education) One of the most obvious I've observed on how people tackle problems come from their background. For an alignment example, John tackle problems like a statistical physicist whereas Paul tackles problem like a theoretical computer scientists, leading to very different perspectives: True Names vs Building-Breaker.

(Knowledge) What you know influences your shape of mind, since you can see and link more things. But when I'm talking about knowing here, I mean the kind of deep knowledge that framing exercises are supposed to provide.

(Past Life) This one feels easily missed by most people; after all, why should what you did in your past (especially personal life) should influence your scientific research? Because it clearly does. A notable example is how easy it becomes to see hidden assumptions about how the world is supposed to work when you don't come from a background where they make any sense.

One thing this handle makes clear is the difference between my model for different programs as Refine, SERI MATS, and PIBBSS respectively aim at:

Refine is looking for different shapes of mind that the mind currently at work in conceptual alignment, and aims at empowering them to contribute productively.

MATS (according to my current model) is mainly looking for shapes of mind closed to archetypal ones (the mentors), and focuses on making them fit for alignment research by helping them approach this initial example (while still maintaining enough diversity for productive disagreement).

PIBBSS is looking for new shapes of minds, but ones that are visibly relevant and useful from existing object-level shapes of mind. That is, PIBBSS starts with current conceptual alignment researchers and the shapes of mind that they feel they might get something out of in their own research.

I'm excited to finally be in a field with all three.

Thus we can see framing exercises as a way of shaping your mind to see the hidden bit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How I think about alignment, published by Linda Linsefors on August 13, 2022 on The AI Alignment Forum.

This was written as part of the first Refine blog post day. Thanks for comments by Chin Ze Shen, Tamsin Leake, Paul Bricman, Adam Shimi.

Magic agentic fluid/force

Somewhere in my brain there is some sort of physical encoding of my values. This encoding could be spread out over the entire brain, it could be implicit somehow. I’m not making any claim of how values are implemented in a brain, just that the information is somehow in there.

Somewhere in the future a super intelligent AI is going to do some action.

If we solve alignment, then there will be some causal link between the values in my head (or some human head) and the action of that AI. In some way, whatever the AI does, it should do it because that is what we want.

This is not purely about information. Technically I have some non-zero causal influence over everything in my future lightcone, but most of this influence is too small to matter. More relevant, but still not the thing we want is deceptive AI. In this case the AI’s action is guided by our values in a non-neglectable way, but not in the way we want.

I have a placeholder concept which I call magic agentic fluid or magic agentic force. (I’m using “magic” in the traditional rationalist way of tagging that I don’t yet have a good model of how this works.)

MAF is a high-level abstraction. I expect it to not be there when you zoom in too much. Same as how solid objects just exist at a macro scale, and if you zoom in too much there are just atoms. There is no essence, only shared properties. I think the same way about agents and agency.

MAF is also in some way like energy. In physics energy is a clearly defined concept, but you can’t have just energy. There is no energy without a medium. Same as how you can’t have speed without there being anything that is moving. This is not a perfect analogy.

But I think that this concept does point to something real and/or useful, and that it would be valuable to try to get a better grasp on what it is, and what it is made of.

Let’s say I want to eat an apple, and later I am eating an apple. There is some causal chain that we can follow from me wanting the apple to me eating the apple. There is a chain of events propagating through spacetime, and it carries with it my will and it enacts it. This chain is the MAF.

I would very much like to understand the full chain of how my values are causing my actions. If you have any relevant information, please let me know.

Trial and error as an incomplete example of MAF

I want an apple -> I try to get an apple and if it doesn't work, I try something else until I have an apple -> I have an apple

This is an example of MAF, because wanting an apple caused me to have an apple through some causal chain. But also notice that there are steps missing. How was “I want an apple” transformed into “I try to get an apple and if it doesn't work, I try something else until I have an apple”? Why not the alternative plan “I cry until my parents figure out what I want and provide me with an apple”. Also, the second step involves generating actual things to try. When humans execute trial and error, we don’t just do random things, we do something smarter. This is not just about efficiency. If I try to get an apple by trying random actions sequences, with no learning other than “that exact sequence did not work”, then I’ll never get an apple, because I’ll die first.

I could generate a more complete example, involving neural nets or something, and that would be useful. But I’ll stop here for now.

Mapping the territory along the path

When we have solved Alignment, there will be a causal chain, carrying MAF from somewhere inside my brain, all the way to the actions of the AI. The MAF has to survive passa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Internationalism is a key value in EA, published by freedomandutility on August 12, 2022 on The Effective Altruism Forum.

With increasing coverage of EA, I’m seeing many critiques on Twitter along the line of “EA billionaires should just pay more taxes” and “philanthropy bypasses democratic processes”.

A key value which EAs share and most people do not hold (or at least do not adhere to in practice) is internationalism and the idea that people matter equally, regardless of nationality.

I think it’s very important to emphasise this when communicating about EA.

Internationalism explains why paying taxes isn’t better than funding EA charities and projects (because governments typically spend <1% of tax revenue on improving things for people in other countries, while EA spending values everyone equally).

Internationalism also justifies philanthropists using their influence to promote policies which improve things for people around the world (because the policies of powerful governments can have large impacts on foreigners, but foreigners are excluded from the democratic processes which determine the policies).

(Also, side note, but I think when EAs observe that people don’t care as much about people who are far away from them, the reason is more about nationality and nationalism than physical distance)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Large-scale International Educational Migration: A Shallow Investigation, published by Johannes Haushofer on August 12, 2022 on The Effective Altruism Forum.

Jasmin Baier, Johannes Haushofer, Hannah Lea Shaw

with input from Toman Barsbai and Merve Demirel

1. Introduction

Two thirds of the world’s population live on less than $10 per day (Our World in Data), and 60% of the global variability in income is explained by where one is born (Milanovich, 2015). Perhaps as a result, many people want to permanently migrate to another country: in low-income countries, the share ranges from 20–59%; among Africans under 25, it is 33% (Gallup World Poll, own calculations; Figure 1).

At the same time, many high-income countries experience significant gaps in their labor force, both in terms of skills and raw numbers, resulting from low birth rates and aging populations. The OECD estimates that this workforce gap will amount to 450 million people by 2050.

These figures suggest that increased international migration would have benefits both for migrants themselves, and for destination countries. Indeed, the economic benefits of removing international migration barriers amount to large fractions of world GDP — one or two orders of magnitude larger than those of trade barriers (Clemens, 2011).

Recognizing this, many high-income countries are actively working to reduce migration barriers. This is especially true for educational migration: many countries are working towards internationalizing their higher education systems, simplifying student visa and work requirements, and reducing tuition costs (e.g., in the case of Germany, keeping university completely free, even for third-country nationals). For the migrants, international educational migration provides access to the labor market in the destination country, and additionally combines the returns to migration with those to education.

The goal of this investigation is therefore to explore international educational migration as a possible mechanism to increase migration opportunities and alleviate poverty.

2. Potential Impact

2.1 Direct Economic Impact on Migrants

Cross-country estimates of the returns to migration suggest potential impacts on income around 1,500% (Figure 2). Studies that permit causal claims have shown that international migration generates income increases of several hundreds of percent. For instance, labor migrants from Tonga to New Zealand increase their income by 263% one year after migration (McKenzie, Stillman, and Gibson 2010). Similarly, Mobarak et al. (2020) estimate returns of around 200% in terms of income from low-skilled migration between Bangladesh and Malaysia.

Because these studies estimate individual economic gains from labor migration, often low-skill, we view these estimates as lower bounds for the impact of educational migration, which combines the benefits of migration with those of education. Specifically, education itself has large returns, on the order of 10% per year (Peet, Fink, and Fawzi 2015). Importantly, these returns are unlikely to decay over time. (This stands in sharp contrast to e.g. cash transfer programs, whose effects have been shown to dissipate over time; e.g. Blattman, Fiala, and Martinez 2020).

Unfortunately, we are not aware of causal evidence on the returns to international educational migration. However, to provide a concrete example for the likely returns, we briefly summarize estimates for the impact of the educational migration program Malengo, with which we are associated. Malengo enables Ugandan secondary school graduates to apply for and complete an English-taught Bachelor’s program in Germany.

The cost of the program per student net of administration costs is EUR 11,971, which includes a living expense stipend and semester fees for the first year; application expense...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind alignment team opinions on AGI ruin arguments, published by Vika on August 12, 2022 on LessWrong.

We had some discussions of the AGI ruin arguments within the DeepMind alignment team to clarify for ourselves which of these arguments we are most concerned about and what the implications are for our work. This post summarizes the opinions of a subset of the alignment team on these arguments. Disclaimer: these are our own opinions that do not represent the views of DeepMind as a whole or its broader community of safety researchers.

This doc shows opinions and comments from 8 people on the alignment team (without attribution). For each section of the list, we show a table summarizing agreement / disagreement with the arguments in that section (the tables can be found in this sheet). Each row is sorted from Agree to Disagree, so a column does not correspond to a specific person. We also provide detailed comments and clarifications on each argument from the team members.

For each argument, we include a shorthand description in a few words for ease of reference, and a summary in 1-2 sentences (usually copied from the bolded parts of the original arguments). We apologize for some inevitable misrepresentation of the original arguments in these summaries. Note that some respondents looked at the original arguments while others looked at the summaries when providing their opinions (though everyone has read the original list at some point before providing opinions).

A general problem when evaluating the arguments was that people often agreed with the argument as stated, but disagreed about the severity of its implications for AGI risk. A lot of these ended up as "mostly agree / unclear / mostly disagree" ratings. It would have been better to gather two separate scores (agreement with the statement and agreement with implications for risk).

Summary of agreements, disagreements and implications

Most agreement:

Section A ("strategic challenges"): #1 (human level is nothing special), #2 (unaligned superintelligence could easily take over), #8 (capabilities generalize out of desired scope)

Section B1 (distributional leap): #14 (some problems only occur in dangerous domains)

Section B2 (outer/inner alignment): #16 (inner misalignment), #18 (no ground truth), #23 (corrigibility is anti-natural)

Section B3 (interpretability): #28 (large option space)

Section B4 (miscellaneous): #36 (human flaws make containment difficult)

Most disagreement:

#6 (pivotal act is necessary). We think it's necessary to end the acute risk period, but don't agree with the "pivotal act" framing that assumes that the risk period is ended through a discrete unilateral action by a small number of actors.

#24 (sovereign vs corrigibility). We think this kind of equivocation isn't actually happening much in the alignment community. Our work focuses on building corrigible systems (rather than sovereigns), and we expect that the difficulties of this approach could be surmountable, especially if we can figure out how to avoid building arbitrarily consequentialist systems.

#39 (can't train people in security mindset). Most of us don't think it's necessary to generate all the arguments yourself in order to make progress on the problems.

#42 (there's no plan). The kind of plan we imagine Eliezer to be thinking of does not seem necessary for a world to survive.

Most controversial among the team:

Section A ("strategic challenges"): #4 (can't cooperate to avoid AGI), #5 (narrow AI is insufficient), #7 (no weak pivotal acts), #9 (pivotal act is a dangerous regime)

Section B1 (distributional leap): #13 and 15 (problems above intelligence threshold and correlated capability gains)

Section B2 (outer/inner alignment): #17 (inner properties), #21 (capabilities go further), #22 (simple alignment core)

Section B3 (interpreta...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Common-sense cases where "hypothetical future people" matter, published by levin on August 12, 2022 on The Effective Altruism Forum.

Motivation

A few weeks ago, I was speaking with a student (call him Josh) who was skeptical of the importance of existential risk and the validity of longtermism. He said something like, "What matters to me is kids going hungry today, not hypothetical future people." I found this to be a moving, sincere objection, so I thought about where it seemed to go wrong and offered Josh a version of Case 1 below, and he seemed pretty convinced.

Josh's skepticism echoes the dismissal of "merely possible people" expressed by critics who hold presentist person-affecting views — that is, they believe that "an act can only be bad [or good] if it is bad [or good] for someone," where "someone" is a person who exists at the time of the act. The current non-existence of future people is a common objection to taking their well-being into moral consideration, and it would be good for longtermists to have cases ready to go that illustrate the weaknesses of this view.

I developed a couple more cases in Twitter threads and figured I'd combine them into a linkable forum post.

Case 1: The Reformer

You work in a department of education. You spend a full year working on a report on a new kindergarten curriculum that makes kids happier and learn better. It takes a few years for this to circulate and get approved, and a few more for teachers to learn it.

By the time it's being taught, 6 years have passed since your work. I think your work, 6 years ago, was morally significant because of the happier, better-educated students now. But these kindergarteners are (mostly) 5 years old. They didn't even exist at the time of your work.

You remember a conversation you had, while working on the curriculum, with your friend who thinks that "hypothetical future people can't have interests" (and who is familiar with the turnaround times of education reform). The friend shook her head. "I don't know why you're working on this kindergarten curriculum for future people," she said. "You could be helping real people who are alive today. Why not switch to working on a second-grade curriculum?"

Indeed, if only already-existing people matter, you'd be in the weird position where your work would've been morally valuable if you'd written a 2nd grade curriculum but your kindergarten curriculum is morally worthless. Why should the birth year of beneficiaries affect this evaluation?

Case 2: The Climate Resiliency Project

After finishing architecture school, you choose to work at a firm that designs climate resiliency projects. The government of Bangladesh has contracted that firm to design sea walls, on the condition that the work be expedited. You could have worked at a commercial firm for more pay and shorter hours, but you choose to work at the climate-focused firm.

The team works for a year on the sea walls project. The Bangladeshi government builds it over the next 20 years. In 2042, a typhoon strikes, and the walls save thousands of lives.

Now, you consider how your choice to work at the climate resiliency firm compared to its alternatives. You think your work on the sea walls accounted for, say, 1% of the impact, saving dozens of lives. But maybe you could have donated a big share of your larger salary to Against Malaria and saved dozens of lives that way instead.

If "nonexistent future people" don't matter, we are again in the absurd position of asking, "Well, how many of the lives saved were over the age of 20?" After all, those under 20 didn't exist yet, so you should not have taken their non-existent interests into consideration.

As the decades progress, the sea walls save more lives, as the effects of climate change get worse. But the "future people don't matter" view holds that ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anti-squatted AI x-risk domains index, published by plex on August 12, 2022 on The Effective Altruism Forum.

I heard about a recent case of a relevant domain being squatted, and figured that the people who buy up domains then sell them at a markup might start to notice the growth of interest in AI x-risk and claim some domains which would otherwise host good projects. I've bought the following, and will make them available for use by anyone in the x-risk community.

aisafety.global - Conference?aisafety.careers - 80k said they were happy for others to join the spacealignment.careersalignment.academy - Training program?alignment.courses - List of all training programs (using Stampy answer as backend)?aisafety.ventures - Entrepreneurs org?aisafety.fund - AIS-specific funding org?aisafety.foundationaisafety.institutexrisk.fundxrisk.foundationagisafety.info - Useful if we fall back to AGI Safetyagisafety.xyzaisafety.coach - More orgs in the direction of AI Safety Support?aisafety.devaisafety.network - Peer-to-peer researchers something?aisafety.quest - Project Eurler for AIS?agisafety.questaisafety.worldalignment.fyialignment.plusagisafety.wiki - alt. Stampy's Wiki branding?alignment.wiki

Others:

x-risks.com - Drew Spartz

This cost ~$200 of personal funds for the first year, and closer to $500/y going forward as the onboarding discounts wear off. I estimate saving one or two wanted domains from being squatted and the hassle / cost of acquiring it or moving to a worse domain would make this worthwhile for several years, so this seems probably positive EV.

The main downside risk would be from people who want these domains not knowing they're available, so I have forwarded all of these domains to my home page. Hopefully people who want them will check there to track me down and request them. This is not perfect, but seems better than waiting for them to be sniped. Edit: Most (but not all! despite apparently identical settings!) of the redirects are not currently functioning. Will look into and fix.

If you would like to list your own available relevant domains in the comments that would be welcome, I'll add them to this post and the LW cross-post.

I will point them towards the servers of anyone who claims to have a half decent plan for the domain, and hand over the domain fully if they build something which seems useful.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Oversight Misses 100% of Thoughts The AI Does Not Think, published by johnswentworth on August 12, 2022 on The AI Alignment Forum.

Problem: an overseer won’t see the AI which kills us all thinking about how to kill humans, not because the AI conceals that thought, but because the AI doesn’t think about how to kill humans in the first place. The AI just kills humans as a side effect of whatever else it’s doing.

Analogy: the Hawaii Chaff Flower didn’t go extinct because humans strategized to kill it. It went extinct because humans were building stuff nearby, and weren’t thinking about how to keep the flower alive. They probably weren’t thinking about the flower much at all.

More generally: how and why do humans drive species to extinction? In some cases the species is hunted to extinction, either because it's a threat or because it's economically profitable to hunt. But I would guess that in 99+% of cases, the humans drive a species to extinction because the humans are doing something that changes the species' environment a lot, without specifically trying to keep the species alive. DDT, deforestation, introduction of new predators/competitors/parasites, construction. that’s the sort of thing which I expect drives most extinction.

Assuming this metaphor carries over to AI (similar to the second species argument), what kind of extinction risk will AI pose?

Well, the extinction risk will not come from AI actively trying to kill the humans. The AI will just be doing some big thing which happens to involve changing the environment a lot (like making replicators, or dumping waste heat from computronium, or deciding that an oxygen-rich environment is just really inconvenient what with all the rusting and tarnishing and fires, or even just designing a fusion power generator), and then humans die as a side-effect. Collateral damage happens by default when something changes the environment in big ways.

What does this mean for oversight? Well, it means that there wouldn't necessarily be any point at which the AI is actually thinking about killing humans or whatever. It just doesn't think much about the humans at all, and then the humans get wrecked by side effects. In order for an overseer to raise an alarm, the overseer would have to figure out itself that the AI's plans will kill the humans, i.e. the overseer would have to itself predict the consequences of a presumably-very-complicated plan.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Refining the sharp left turn threat model, published by Victoria Krakovna on August 12, 2022 on The AI Alignment Forum.

This is our current distillation of the sharp left turn threat model and an attempt to make it more concrete. We will discuss our understanding of the claims made in this threat model, and propose some mechanisms for how a sharp left turn could happen. This is a work in progress, and we welcome feedback and corrections.

What are the main claims of the “sharp left turn” threat model?

Claim 1. Capabilities will generalize far (i.e., to many domains)

There is an AI system that:

Performs well: it can accomplish impressive feats, or achieve high scores on valuable metrics.

Generalizes, i.e., performs well in new domains, which were not optimized for during training, with no domain-specific tuning.

Generalization is a key component of this threat model because we're not going to directly train an AI system for the task of disempowering humanity, so for the system to be good at this task, the capabilities it develops during training need to be more broadly applicable.

Some optional sub-claims can be made that increase the risk level of the threat model:

Claim 1a [Optional]: Capabilities (in different "domains") will all generalize at the same time

Claim 1b [Optional]: Capabilities will generalize far in a discrete phase transition (rather than continuously)

Claim 2. Alignment techniques that worked previously will fail during this transition

Qualitatively different alignment techniques are needed. The ways the techniques work apply to earlier versions of the AI technology, but not to the new version because the new version gets its capability through something new, or jumps to a qualitatively higher capability level (even if through “scaling” the same mechanisms).

Claim 3: Humans can’t intervene to prevent or align this transition

Path 1: humans don't notice because it's too fast (or they aren’t paying attention)

Path 2: humans notice but are unable to make alignment progress in time

Some combination of these paths, as long as the end result is insufficiently correct alignment

Arguments for the claims in this threat model

Claim 1: There is a "core" of general intelligence - a most effective way of updating beliefs and selecting actions (Ruin #22). Speculation about what the core is: consequentialism / EU maximization / "doing things for reasons".

Claim 1a: Capability gains from intelligence are highly correlated (Ruin #15)

Claim 2: There is no simple core for alignment (Ruin #22), Corrigibility is anti-natural (Ruin #23)

Claims 1 & 2: arguments in Will capabilities generalize more? (+ Ruin #21)

Claim 3: We can't coordinate to avoid AGI (Ruin #4)

Mechanisms for capabilities generalizing far (Claim 1)

Capabilities will generalize far if learning / reasoning / cognitive work is done outside of the optimization process, similarly to how human cultural evolution happens outside genetic evolution and proceeds faster. Here are some mechanisms for cognitive work getting done outside the optimization process:

Mechanisms proposed by Paul Christiano: performing search internally, natural selection occurring internally, reasoning about how to think better, deploying subagents

The AI develops a way to outpace SGD as the way to learn / improve, similar to how humans outpace evolution by cultural transmission + within-lifetime learning. E.g., it writes Google docs to itself, or searches over prompts (or whatever is the prompt analog at that point).

The optimization process finds a general meta-learning algorithm, which replaces the previous solution of domain-specific heuristics and greatly increases the rate of learning

Sequential reasoning: sharp increase in effective sequential memory/cognitive time between the "aligned" regime and the sharp left turn regime. Ways in ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some concerns about policy work funding and the Long Term Future Fund, published by weeatquince on August 12, 2022 on The Effective Altruism Forum.

Situation

As far as I can tell The Long-Term Future Fund (LTFF) wants to fund work that will influence public policy. They say they are looking to fund “policy analysis, advocacy ...” work and post in policy Facebook groups asking for applications.

However, as far as I can tell, in practice, the LTFF appears to never (or only very rarely) fund such projects that apply for funding. Especially new projects.My view that such funding is rare is based on the following pieces of evidence:

Very few policy grants have been made. Looking at the payout reports of the LTFF, they funded a grant for policy influencing work in 2019 (to me, for a project that was not a new project). Upon asking them they say they have funded at least one more policy grant that has not been written up.

Very many policy people have applied for LTFF grants. Without actively looking for it I know of: someone in an established think tank looking for funds for EA work, 3 groups wanting to start EA think tank type projects, a group wanting to do mass campaigning work. All these groups looked competent and were rejected. I am sure many others apply too. I know one of these has gone on to get funding elsewhere (FTX).

Comments from staff at leading EA orgs. In January last year, a senior staff member at a leading EA institution mentioned, to my surprise, that EA funders tend not to fund any new longtermist policy projects (except perhaps with very senior trusted people like OpenPhil funding CSET). Recently I spoke to someone at CEA about this and asked if it matched their views too and they said they do think there is a problem here. Note this was about EA funders in general, not specifically the LTFF.

Comments from EA policy folk looking for funding. There seems to be (at least there was in 2021) a general view from EAs working in the policy space that it has been very hard to find funding for policy work. Note this is about EA funders in general, not the LTFF.

Odd lines of questioning. When I applied, the line of questioning was very odd. I faced an hour of: Why should we do any policy stuff? Isn't all policy work a waste of time? Didn’t [random unconnected policy thing] not work? Etc. Of course it can be useful to check applicants have a good understanding of what they are doing, but it made me question whether they wanted to fund policy work at all.

Odd feedback. Multiple applicants to the LTFF have reported receiving feedback along the lines of: We see high downside risks but cannot clarify what those risks are. Or: We want to fund you but an anonymous person vetoed you and we cannot say who or why. Or: Start-up policy projects are too risky. Or: We worry you might be successful and hard to shut down if we decided we don’t like you in future. This makes me worry that the fund managers do not think through the risks and reasons for funding or not funding policy work in as much depth as I would like, and that they maybe do not fund any new/start-up policy projects.

Acknowledgment that they do apply a higher bar for policy work. Staff at the LTFF have told me that they apply a higher bar for policy work than for other grants.

Of course, this is all circumstantial, and not necessarily a criticism of the LTFF. The fund managers might argue they never get any policy projects worth funding and that all the promising projects I happened to hear of were actually net negative and it was good not to fund them. It is also possible that things have improved in the last year (the notes that make up this post have been sitting in an email chain for a long while now).

Relevance and recommendation

That said I thought it was worth me writing this up publicly as the possibilit...