Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety and Neighboring Communities: A Quick-Start Guide, as of Summer 2022, published by Sam Bowman on September 1, 2022 on LessWrong.

Getting into AI safety involves working with a mix of communities, subcultures, goals, and ideologies that you may not have encountered in the context of mainstream AI technical research. This document attempts to briefly map these out for newcomers.

This is inevitably going to be biased by what sides of these communities I (Sam) have encountered, and it will quickly become dated. I expect it will still be a useful resource for some people anyhow, at least in the short term.

AI Safety/AI Alignment/AGI Safety/AI Existential Safety/AI X-Risk

The research project of ensuring that future AI progress doesn’t yield civilization-endingly catastrophic results.

Good intros:

Carlsmith Report

What misalignment looks like as capabilities scale

Vox piece

Why are people concerned about this?

My rough summary:

It’s plausible that future AI systems could be much faster or more effective than us at real-world reasoning and planning.

Probably not plain generative models, but possibly models derived from generative models in cheap ways

Once you have a system with superhuman reasoning and planning abilities, it’s easy to make it dangerous by accident.

Most simple objective functions or goals become dangerous in the limit, usually because of secondary or instrumental subgoals that emerge along the way.

Pursuing typical goals arbitrarily well requires a system to prevent itself from being turned off, by deception or force if needed.

Pursuing typical goals arbitrarily well requires acquiring any power or resources that could increase the chances of success, by deception or force if needed.

Toy example: Computing pi to an arbitrarily high precision eventually requires that you spend all the sun’s energy output on computing.

Knowledge and values are likely to be orthogonal: A model could know human values and norms well, but not have any reason to act on them. For agents built around generative models, this is the default outcome.

Sufficiently powerful AI systems could look benign in pre-deployment training/research environments, because they would be capable of understanding that they’re not yet in a position to accomplish their goals.

Simple attempts to work around this (like the more abstract goal ‘do what your operators want’) don’t tend to have straightforward robust implementations.

If such a system were single-mindedly pursuing a dangerous goal, we probably wouldn’t be able to stop it.

Superhuman reasoning and planning would give models with a sufficiently good understanding of the world many ways to effectively gain power with nothing more than an internet connection. (ex: Cyberattacks on banks.)

Consensus within the field is that these risks could become concrete within ~4–25 years, and have a >10% chance of being leading to a global catastrophe (i.e., extinction or something comparably bad). If true, it’s bad news.

Given the above, we either need to stop all development toward AGI worldwide (plausibly undesirable or impossible), or else do three possible-but-very-difficult things:

(i) build robust techniques to align AGI systems with the values and goals of their operators,

(ii) ensure that those techniques are understood and used by any group that could plausibly build AGI, and

(iii) ensure that we're able to govern the operators of AGI systems in a way that makes their actions broadly positive for humanity as a whole.

Does this have anything to do with sentience or consciousness?

No.

Influential people and institutions:

Present core community as I see it: Paul Christiano, Jacob Steinhardt, Ajeya Cotra, Jared Kaplan, Jan Leike, Beth Barnes, Geoffrey Irving, Buck Shlegeris, David Krueger, Chris Olah, Evan Hubinger, Richard Ngo, Rohin Shah; ARC, R...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stop Discouraging Microwave Formula Preparation, published by jefftk on September 2, 2022 on LessWrong.

In the early days of microwaves, a small number of babies were hurt by being fed milk that had been overheated (case reports:

Sando

1984,

Hibbard

1988). Public health authorities now universally recommend against all forms of microwaving milk for babies:

FDA:

Never use a microwave oven for heating infant formulas. Microwaving may cause the bottle to remain cool while hot spots develop in the formula. Overheated formula can cause serious burns to your baby.

CDC:

If you do decide to warm the bottle, never use a microwave. Microwaves heat milk and food unevenly, resulting in "hot spots" that can burn your baby's mouth and throat.

NHS:

Never warm up formula in a microwave, as it may heat the feed unevenly and burn your baby's mouth.

Australia's

Department of Health and Aged Care:

Do not use a microwave to heat up formula as this can make the milk too hot and can burn baby's mouth.

The baby formula label:

Warning: do not use microwave to prepare or warm formula. Serious burns may occur.

This is all nuts, and they should stop. The microwave is great way to heat the water for making formula because it's fast and efficient. Instead of telling people never to do this we should be giving the small amount of guidance necessary to do it safely. Ideally the guidelines for preparing infant formula using the microwave would look something like:

Heat the water, taking care not to overheat. A 6oz bottle in a typical microwave requires approximately 15 seconds to warm from room temperature, or 30 seconds from the fridge. Add the powder, cap the bottle, and shake thoroughly. Test a drop on the back of your hand: it should not feel hot or even very warm.

As long as you're heating the water before adding formula you can't get hot spots, because you still have to shake after adding the powder. And testing on your hand ensures it's not too hot.

Microwaving mixed formula or breast milk is probably also fine, but there is at least a little more reason to worry there: if you get it too hot you'll likely lose some of the nutritional value. Exactly how hot it has to get before this is a problem isn't something we know, though what studies there are indicate gentle heating is fine (Fay 1991, Sigman-Grant

1992, Ovesen 1996, Sierra 2001).

I wish public health authorities would cite their sources and give background. It needn't be in the primary public-facing documentation, but each claim ("never use a microwave") should have a link to their reasoning. I think in this case if they needed to explain their reasoning it would be clear that it doesn't hold up, and they should change their recommendations.

(I've written about this before, but only in the context of breast milk. I think the case is strongest for heating water for formula because there's no question of nutritional degradation.)

Comment via: facebook

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Young EAs should choose projects more carefully, published by sabrinac on September 2, 2022 on The Effective Altruism Forum.

Thanks to Emma Williamson for her helpful feedback.

Epistemic status: I wrote this up pretty quickly so take what I say with a grain of salt. A lot of this post is based on an intuition I’ve developed after talking to lots of young EAs. I also think this post is most relevant for people working within community-building and meta-EA stuff. I wish someone would’ve given me this advice a year ago, so I hope it’s useful for other people.

The problem

Lots of young EAs are uncertain about their career paths and enjoy working within the EA community, so they often default to working on hopping between independent projects that have the EA stamp of approval. (e.g. doing ops for Atlas, independent community-building projects, going to the Bay to learn about AI safety, running one-off retreats, etc.)

While hopping between independent projects can allow you to quickly test your fit for something, I think many young EAs should do this less and/or think more carefully about which projects they’re working on.

There are a bunch of reasons why I think this:

Independent, unrelated projects often lack good mentorship and learning structures. You don’t get to work at an established org where you get lots of feedback from your boss/mentor. You also usually don’t build up expertise in a field.

It encourages young people to stay within the EA bubble, instead of leaving, acquiring diverse skill-sets, and getting feedback from the real world. This also exacerbates talent bottlenecks further down the funnel once the community lacks expertise in specific fields/career paths.

Young people often equate “working on an EA project” with “doing work that substantially improves the world/reduces x-risk.”

While projects might seem valuable in the short term, I think it damages young people’s longer-term impact. (i.e. by delaying you from developing deep expertise and career capital.)

Instead of grappling with the complexity of the problems you want to solve, hopping between projects gives you a shallow understanding of several different things.

I think my claim mostly applies to a certain subset of young people in EA, particularly community-builders and people with a lack of direction/concrete career plans.

There’s another related trend where I see community-builders incentivized to put out fires and solve the small-scale problems immediately in front of them.

They get the reputation as someone who can “get shit done” but in practice, they’re usually solving ops bottlenecks at the cost of building harder-to-acquire skills.

I’m concerned that conscientious young women with high executive functioning disproportionately get trapped here.

Caveats

This post doesn’t apply to the majority of young EAs.

Some independent projects are useful for figuring out if you enjoy specific kinds of work, allowing you to quickly eliminate career options.

I’m not arguing that young people should spend less time working on projects, but rather that they should choose projects more carefully, prioritizing ones with (a) good mentorship/learning opportunities and (b) a narrow focus within the fields they’re interested in.

I don’t think this trend is indicative across the entire community. I think when people want to work within a specific cause area—e.g. AI safety, biosecurity, animal welfare, etc.—they have a lot more direction and this phenomenon happens less.

This problem overlaps with community-builders spending too much time community-building, and not engaging enough with the problems they want to solve.

Suggestions

Don’t work on a project just because it has the EA stamp of approval—make sure you have a clear theory of change for why you’re working on something.

(Gently and kindly) tell your frie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Replacement for PONR concept, published by Daniel Kokotajlo on September 2, 2022 on The AI Alignment Forum.

Writing this in my personal capacity, as a follow up to my previous post on this topic.

I am now contemplating replacing the "Point of No Return" concept with the following three concepts to use instead:

Overpower Ability: The first moment where there exists a group of APS-AIs which, if they collectively decided to eliminate 50%+ of the human population within a month, could do so.

APS-AI stands for Advanced, Planning, Strategically Aware AI. Maybe there is some "cheap" way to eliminate 50% of humanity, e.g. by hacking a nuclear silo and launching a missile, that the AIs would never actually do because it wouldn't benefit them. For now I'm happy to say "whatever, still counts" but I'm open to being convinced the definition should be revised.

Powerbase Ability: The first moment where there exists a group of APS-AIs which could, if they collectively decided, end up effectively in control of a cutting-edge APS-AI lab within a month.

I'm imagining two main ways this could happen: 1. They use persuasion, charisma, deceptive alignment, etc. to convince the AI project that built them that they are aligned & trustworthy. 2. They fail at this, but escape (via hacking?) to a more gullible human institution which then becomes a cutting-edge APS-AI project thanks to their help. Yes, "effectively in control" is not a sharp line & may be somewhat controversial, but maybe this is good enough for now.

Crunch time: The period where it's relatively more important to optimize for direct effects rather than P2B. (P2B stands for Plan to P2B Better and basically means “learning and growing.”)

My favorite way to understand what crunch time means is via this meme from the CLR common room:

Transcript for those who can’t read my handwriting:

[Small brain]: Pull a lever in a direction that seems good

[Glowing brain]: Pull the biggest lever hard in a direction that seems good

[Shining brain]: Do research to find bigger levers & better decide which direction to pull, then do the above

[Sparkle brain]: Do movement building to get more researchers + influence + more movement building, then do the above

[Kant brain]: Do all three but shift portfolio towards lever-pulling as crunch time draws near.

[Dr. Manhattan brain]: Do research to figure out when crunch time is. Build models with Tristan and Hjalmar.

[von Neumann brain]: Crunch time was in 2015

In other words, Crunch Time is when you should be spending down your capital and playing your cards instead of learning and growing. Obviously you still shouldn't pass up great opportunities to learn and grow, it's a matter of relative emphasis. Note that most projects have both direct and meta/P2B/learning-and-growing effects; crunch time is when you should primarily be choosing what to do on the basis of direct effects.

How these concepts relate:

Ultimately what we are interested in is Crunch Time. The goal of AI timelines forecasting is to inform decision-makers, and decision-makers very often face choices that can be summarized as “Option A is safe and increases our influence & knowledge, whereas Option B looks like it might actually do something to solve the problem but only on certain assumptions.”

But in order to estimate when crunch time is, it helps to first think about what sort of capabilities AIs might have, and when. I propose “Overpower Ability” and “Powerbase Ability” as two notable milestones to forecast. Probably most people will agree that crunch time is before overpower ability. Insofar as you think things like “taking over the world is really hard” and “takeoff will be slow” you might think that crunch time happens around powerbase ability, or even later. Insofar as you think things like “Civilization is inadequate” and “takeoff...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI coordination needs clear wins, published by Evan Hubinger on September 1, 2022 on The AI Alignment Forum.

Thanks to Kate Woolverton and Richard Ngo for useful conversations, comments, and feedback.

EA and AI safety have invested a lot of resources into building our ability to get coordination and cooperation between big AI labs. So far, however, despite that investment, it doesn’t seem to me like we’ve had that many big coordination “wins” yet. I don’t mean to say that to imply that our efforts have failed, however—the obvious other hypothesis is just that we don’t really have that much to coordinate on right now, other than the very nebulous goal of improving our general coordination/cooperation capabilities.

In my opinion, however, I think that our lack of clear wins is actually a pretty big problem—and not just because I think there are useful things that we can plausibly coordinate on right now, but also because I expect our lack of clear wins now to limit our ability to get the sort of cooperation we need in the future.

In the theory of political capital, it is a fairly well-established fact that “Everybody Loves a Winner.” That is: the more you succeed at leveraging your influence to get things done, the more influence you get in return. This phenomenon is most thoroughly studied in the context of the ability of U.S. presidents’ to get their agendas through Congress—contrary to a naive model that might predict that legislative success uses up a president’s influence, what is actually found is the opposite: legislative success engenders future legislative success, greater presidential approval, and long-term gains for the president’s party.

I think many people who think about the mechanics of leveraging influence don’t really understand this phenomenon and conceptualize their influence as a finite resource to be saved up over time so it can all be spent down when it matters most. But I think that is just not how it works: if people see you successfully leveraging influence to change things, you become seen as a person who has influence, has the ability to change things, can get things done, etc. in a way that gives you more influence in the future, not less.

Of course, you do have to actually succeed to make this work—if you try to spend your influence to make something happen and fail, you get the opposite effect. This suggests the obvious strategy, however, of starting with small but nevertheless clear coordination wins and working our way up towards larger ones—which is exactly the strategy that I think we should be pursuing.[1]

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 21 criticisms of EA I'm thinking about, published by Peter Wildeford on September 1, 2022 on The Effective Altruism Forum.

Note — I’m writing this in a personal capacity, and am not representing the views of either of my employers (Rethink Priorities and EA Funds). This post was also not reviewed in advance by anyone.

I really like all the engagement with criticism of EA as a part of the criticism and red-teaming contest and I hope our movement becomes much stronger as a result. Here's some criticism I've recently found myself thinking about.

Similar to Abraham's style I'm going to write these out in bullet points because while I'd love to have spent more time to make this a longer post, you can tell by the fact that I'm rushing it out on the last day that this plan didn't come together. So consider these more of starting points for conversation than decisive critiques.

Also apologies if someone else has already articulated these critiques - I'm happy to be pointed to resources and would be happy to edit my post to include them. I'd be very excited also for others to expand upon any of these points. Also, if there's particular interest, I might be able to personally expand on a particular point at a later date.

Furthermore, apologies if any of these criticisms are wrongheaded or lack nuance - some nuance can't be included given the format, but I'd be happy to remove or alter criticism I no longer stand by or link to discussion threads where more nuance is discussed.

Lastly I concede that these aren't necessarily the most important criticisms of EA. These are criticisms I am listing with the intention of producing novelty[1] and articulating some things I've been thinking about rather than producing a prioritized list of the most important criticisms. In many cases, I think the correct response to a criticism I list here could be to ignore it and keep going.

With those caveats in mind, here it goes:

1.) I think criticizing EA is kinda hard and slippery. I agree that EA is a tower of assumptions but I think it is pretty hard to attack this tower as it can instead operate as a motte and bailey. I think a lot of my criticisms are of the form of attacks on specific projects / tactics rather than EA as a philosophy/strategy/approach. And usually these criticisms are met with "Yeah but that doesn't undermine EA [as a philosophy/strategy/approach] and we can just change X". But then X doesn't actually change. So I think the criticism is still important, assuming EA-with-X is indeed worse than EA-without-X or EA-with-different-X. It's still important to criticize things as they are actually are implemented. Though of course this isn't necessarily EA's fault as driving change in institutions - especially more diffuse leaderless institutions - is very hard.

2.) I think criticism of EA may be more discouraging than it is intended to be and we don't think about this enough. I really like this contest and the participatory spirit and associated scout mindset. EA is a unique and valuable approach and being willing to take criticism is a big part of that. (Though the changing may be harder, as mentioned.) But I think filling up the EA Forum with tons of criticism may not be a good experience for newcomers and may be particularly poorly timed with Will's big "What We Owe the Future" media launch bringing EA into the public eye for the first time. I think as self-aggrandizing as it might appear, I should probably make it clear to everyone that I actually think the EA community is pretty great and dramatically much better than every other movement I've been a part of.

I think it's human nature to see people focus on the criticisms and think that there are serious problems and get demotivated, so perhaps we ought to balance that out with more positives[^though of course I don't think a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My take on What We Owe the Future, published by elifland on September 1, 2022 on The Effective Altruism Forum.

Cross-posted from Foxy Scout

Overview

What We Owe The Future (WWOTF) by Will MacAskill has recently been released with much fanfare. While I strongly agree that future people matter morally and we should act based on this, I think the book isn’t clear enough about MacAskill’s views on longtermist priorities, and to the extent it is it presents a mistaken view of the most promising longtermist interventions.

I argue that MacAskill:

Underestimates risk of misaligned AI takeover. more

Overestimates risk from stagnation. more

Isn’t clear enough about longtermist priorities. more

I highlight and expand on these disagreements in part to contribute to the debate on these topics, but also make a practical recommendation.

While I like some aspects of the book, I think The Precipice is a substantially better introduction for potential longtermist direct workers, e.g. as a book given away to talented university students. For instance, I’m worried people will feel bait-and-switched if they get into EA via WWOTF then do an 80,000 Hours call or hang out around their EA university group and realize most people think AI risk is the biggest longtermist priority, many thinking this by a large margin. more

What I disagree with[1]

Underestimating risk of misaligned AI takeover

Overall probability of takeover

In endnote 2.22 (p. 274), MacAskill writes [emphasis mine]:

I put that possibility [of misaligned AI takeover] at around 3 percent this century. I think most of the risk we face comes from scenarios where there is a hot or cold war between great powers.

I think a 3% chance of misaligned AI takeover this century is too low, with 90% confidence.[2] Most of the risk coming from scenarios with hot or cold great power wars may be technically true if one thinks a war between US and China is >50% likely soon which might be reasonable with a loose definition of cold war. That being said, I strongly think MacAskill’s claim about great power war gives the wrong impression of the most probable AI takeover threat models.

My credence on misaligned AI takeover is 40% this century, of which not much depends on a great power war scenario.

Below I’ll explain why my best-guess credence is 40%: the biggest input is a report on power-seeking AI, but I’ll also list some other inputs then aggregate the inputs.

Power-seeking AI report

The best analysis estimating the chance of existential risk (x-risk) from misaligned AI takeover that I’m aware of is Is Power-Seeking AI an Existential Risk? by Joseph Carlsmith.[3]

Carlsmith decomposes a possible existential catastrophe from AI into 6 steps, each conditional on the previous ones:

Timelines: By 2070, it will be possible and financially feasible to build APS-AI: systems with advanced capabilities (outperform humans at tasks important for gaining power), agentic planning (make plans then acts on them), and strategic awareness (its plans are based on models of the world good enough to overpower humans).

Incentives: There will be strong incentives to build and deploy APS-AI.

Alignment difficulty: It will be much harder to build APS-AI systems that don’t seek power in unintended ways, than ones that would seek power but are superficially attractive to deploy.

High-impact failures: Some deployed APS-AI systems will seek power in unintended and high-impact ways, collectively causing >$1 trillion in damage.

Disempowerment: Some of the power-seeking will in aggregate permanently disempower all of humanity.

Catastrophe: The disempowerment will constitute an existential catastrophe.

I’ll first discuss my component probabilities for a catastrophe by 2100 rather than 2070[4], then discuss the implications of Carlsmith’s own assessment as well as reviewers of hi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 9/1/22: Meet the New Booster, published by Zvi on September 1, 2022 on LessWrong.

We will soon have a new, updated booster shot only eight or so months behind schedule. Given only 30% of America is boosted even once, not many people will accept this kind and generous (if rather late) offer. It won’t make much difference even to those who take advantage of it at this point, but better to update than not update, the usual various objections aside. Hopefully this makes the next update easier.

Other things were mostly as one would expect.

This week I also wrote a long analysis of the student loan forgiveness situation.

Executive Summary

Booster update available soon.

Student loan forgiveness bigger than most realize.

Monkeypox situation looking better.

Let’s run the numbers.

The Numbers

Predictions

North Carolina reported almost 500 deaths, which is obviously a backlog, so I adjusted that number to the still-rather-higher-than-reality 150.

Prediction from last week: 542k cases (-5%) and 3,030 deaths (-5%)

Results: 523k cases (-9%) and 3,183 deaths (-1%).

Prediction for next week (HOLIDAY): 420k deaths (-20%) and 2,550 deaths (-20%).

Labor Day is going to mess with reporting a bunch, so noting that this does not represent a predicted real decline in either cases or deaths.

Deaths

Cases

Moderna Sues Pfizer

Moderna is charging Pfizer with patent infringement.

Vaccine maker Moderna announced Friday that it’s suing rival drugmakers Pfizer and BioNtech for patent infringement. The lawsuit alleges the two companies used certain key features of technology Moderna developed to make their COVID-19 vaccine. It argues that Pfizer and BioNtech’s vaccine infringes patents Moderna filed between 2010 and 2016 for its messenger RNA or mRNA technology.

In October 2020, Moderna pledged not to enforce its COVID-19 related patents while the pandemic was ongoing, according to a statement from the company. In March this year, it said it will stick to its commitment not to enforce its COVID-19 related patents in low and middle-income countries, but expects rival companies like Pfizer to respect its intellectual property.

Moderna is not seeking to remove the Pfizer and BioNTech vaccine from the market, but is seeking monetary damages.

On the one hand, this is horrible, it was a pandemic, you use what tools you have. Moderna pledged not to enforce the patent during the pandemic.

On the other hand, fair. Moderna developed key real innovations that were foundational for the Pfizer vaccine. Pfizer should pay a license fee for that, which is all Moderna is asking for. It would not be reasonable to expect Moderna to waive this forever.

My solution would be that they agree on a reasonable fee, ideally going forward only, and that the United States happily cover the costs involved by letting Pfizer raise the price to match. Moderna deserves to be paid, and Pfizer shouldn’t have to pay, and we got orders of magnitude more benefit than all of this costs. A little bonus payment seems quite reasonable to me.

Meet the New Booster

Bivariant boosters (aka ‘updated boosters’) have been FDA approved. That only took (checks date) about 8 months longer than necessary. Weekend Reading has the write-up from a very straightforward meeting.

Should you get the new booster shot? Certainly you should get the new rather than the old, if you are getting boosted. Getting boosted is a better deal than it was prior to this authorization, once the new doses have shipped.

I do not intend to get an additional booster at this time. I consider it a reasonable decision and small mistake either way, with the note that if you are going to get boosted within the next year or so, now seems like the right time to do so.

There are standard calls for what the public ‘deserves to know’ and I would direct everyone to how we deal with ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA is too reliant on personal connections, published by sawyer on September 1, 2022 on The Effective Altruism Forum.

Epistemic status: Personal experience and anecdotes have led me to this bundle of weakly-held views, and I’m wondering what other people think. My experience is heavily biased towards the existential risk subculture of EA, so it’s possible that my criticism applies much less (or not at all) outside of that subculture.

I think the EA community is too reliant on personal connections in a number of ways. I think that many individuals and organizations would benefit (by their own lights) from taking steps to be less reliant on personal connections, and that such steps would also increase overall impact by supporting better ideas, increasing community health, and improving the community’s reputation among outsiders.

Evidence that EA is very reliant on personal connections

Many large donors (and donation advisors) do not take general applications. This includes Open Philanthropy (“In general, we expect to identify most giving opportunities via proactive searching and networking”), Longview, REG, CERR, CLR, and the new Longtermism Fund. The 80,000 Hours job board does not take applications, saying, “Due to resource constraints, we are currently unable to process unsolicited requests to list vacancies on this board.”

Compared to other communities, I think EA organizations and grantees are more likely to get funding from one of a small group of funders. Historical EA funding data is dominated by Open Philanthropy, Givewell, and (starting in 2022) FTX. Effective Altruism Data suggests that Founders Pledge might also be in this small dominant group. I don’t know how to compare this to other communities or research fields so I can’t really back up my claim with data, but I’d be interested if anyone has thoughts on this.

Compared to other communities, EA grantees and grantors share a higher number of personal and professional connections. I can’t figure out how to operationalize this and don’t have any sources, but it seems obvious to me. How many physicists live in a group house with their NSF grant evaluator? How many physicists have a part-time job making grants to their colleagues? Both of these things are common among employees of organizations in the EA community.

As a counterexample to my claim about the lack of applications, it’s important to note that some large EA-affiliated funders do take applications. For example, EA Funds, SFF, ACE, and FTX (though only once, and they “do not currently have any plans to resume accepting applications and do not know if or when we will do so”). Also, Givewell takes applications “for organizations [to] apply for a recommendation without receiving an invitation from GiveWell.”

Finally, while 80k’s job board doesn’t take applications, it does include lots of jobs at organizations that don’t normally consider themselves part of the EA community, which I see as evidence against EA overreliance on personal connections.

Why is overreliance on personal connections a problem?

The more reliant we are on personal connections to achieve our altruistic goals, the more we entangle the EA community with the EA project. Having an EA community is good[citation needed], but the community is not identical to the project. Communities enforce certain norms and styles (any community that does not do so loses out on many of the benefits of being a community), and so necessarily exclude people who don’t want to follow those norms or who dislike that style. Furthermore, identifying the community with the project risks making people feel like once they leave the community, they might as well stop pursuing the project as well. Put another way, “I don’t feel like I belong in this community, so I guess I’m not meant to do [job/project/skill set tha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Enantiodromia, published by ChristianKl on August 31, 2022 on LessWrong.

Oxford Languages defines enantiodromia as “the tendency of things to change into their opposites, especially as a supposed governing principle of natural cycles and of psychological development”. While Plato’s works are so influential that the project of Western philosophy is sometimes described as footnotes to Plato, his discussion of enantiodromia is not part of our standard Western worldview. To the extent that the principle of enantiodromia exists, it matters individually, it matters for choosing the right public policy and it matters for AI safety.

Enantiodromia in Jewish college admissions

One practical example of enantiodromia is a public policy with the goal of creating racial equality. In the 20th century, elite universities found that they admitted a higher proportion of Jewish people than people of other backgrounds. They were admitting more Jewish people because Jewish people scored on average higher than non-Jewish people. To cut down the number of Jewish admissions they thought up a clever way where they added subjective character judgments as an admission criterion which allowed them to cut down the percentage of Jewish admission.

In a world where the performance of a lawyer or doctor correlates with their SAT score if the SAT score is the only thing that matters for admissions, whether or not a lawyer or doctor is Jewish is not going to correlate strongly with their SAT scores. Once you have a policy, where Jewish people need a higher SAT score to be accepted, the Jewish lawyers and doctors will on average have a higher SAT score and thus higher competency.

This competency difference in turn reinforces stereotypes. In a world, where the admission criteria result in Jewish lawyers being on average more competent than non-Jewish lawyers the stereotype of Jewish lawyers being more competent is justified. A smart person who seeks a competent lawyer has an easier time determining whether the last name of the lawyer sounds Jewish than determining the SAT score of the lawyer.

In the end, the policy to create equality between Jewish people and non-Jewish created stereotypes about how Jewish people are unequal. This is enantiodromia.

Given that the dynamic of enantiodromia does not exist in the public consciousness it’s very hard to talk about enantiodromia plays out with modern social justice policies, so it’s left as an exercise for the reader to think through the details.

Examples in society

Celibacy as a means to reduce sexually driven misbehavior provided fertile ground for an environment of sexual abuse scandals.

Over in Buddism which has its own celibacy expectations for ordained monks we see an abundance of sex scandals as well even when they are of a different nature than the scandals of the catholic church.

Research driven by the desire for pandemic preparedness led to Chinese researchers doing gain-of-function experiments on Coronaviruses under biosafety level II (not suitable for protecting researchers against airborne viruses) in Wuhan. Shortly afterward we see an airborne Coronavirus emerging from Wuhan which a genome that looks at first glace inconsistent with expectations from evolutionary theory which causes a global pandemic.

In my native country of Germany, our population cares very deeply about climate change. While you might expect that this leads to lower CO2 output as other European countries like Great Britain or France where environmentalism is weaker, the opposite is the case. German environmentalism managed to shut down nuclear power plants which in turn led to higher CO2 emissions. While unsuccessful German environmentalists even fought the newly created Tesla plant near Berlin.

Enantiodromia clashes with enlightenment values. In the protest work ethic, hard wo...